# Funciones Basicas de TensorFlow

## Introduccion

TensorFlow propone un modelo computacional distinto al normalmente usado: la computacion simbolica. Mas adelante veremos que significa esto, pero por ahora solo veamos un ejemplo de una computacion en python estadar

```python
w = 1.0  #parametro
b = 2.0  #parametro

def f(x):  # x es un input
    return w * x + b

f(1) # w * x + b == 1.0 * 1.0 + 2.0 == 3.0
```

Aqui diremos que `f` es una funcion de `x` que esta parametrizada por las variables `w` y `b`. Esto lo podriamos escribir matematicamente de la siguiente forma:

# $f(x; w,b) = w x + b$

Esto quiere decir que si bien `w` y `b` no son parte de los argumentos de `f`, si determinan su comportamiento. Si cambiamos `w` o `b` tambien cambiara el valor de `f` dado cierto `x`. Modifiquemos `b` a `10.0` a ver que sucede:

```python
b = 10.0

f(1) # w * x + b == 1.0 * 1.0 + 10.0 == 11.0
```

Como es de esperarse, el valor de `f` cambio. Durante el resto del notebook veremos lo necesario para recrear esto en TensorFlow

## Import

Por convension se importa `tensorflow` como `tf`

```python
import tensorflow as tf
```

## Tensores
Un tensor es un arreglo multidimensional. Por ejemplo los vectores son tensores de `1` dimension, las matrices son tensores de dimension `2`, y los escalares son tensores de `0` dimensiones. Asi mismo podemos crear tensores de mayores dimensiones, por ejemplo, las imagenes se pueden representar como un tensor de dimension

## Constants
El tensor mas simple de TensorFlow es un tensor constante. Para crear uno utilizamos `tf.constant`

```python
c = tf.constant(5.0)

print c
```

Sin embargo, como vemos en la evaluacion de la celda anterior, este tensor constante `c` no parece indicarnos el valor que contiene, para extraer el valor primero tenemos crear crear una session y correr el tensor en ella:

## Session

```python
sess = tf.Session()

sess.run(c)
```

Utilizando `run` pudimos recuperar el valor de `c`! Sin embargo es no parece muy utilil pare ahora. Sin embargo, veamos como crear operaciones un poco mas complejas. Al igual que en numpy, los tensores sobreescriben la mayoria de los operadores de python y nos permiten crear nuevos tensores de esta manera:

```python
k = c * 12.0

print k
```

Aqui creamos `k` a partir de `c` y el numero `12.0`. Lo interesante es que el `print` nos da una pista de que `k` es un tensor de tipo "mul", esto se debe a que representa la operacion de multiplicacion. Para recuperar el valor de `k` debemos ejecutarlo en la session denuevo.

```python
sess.run(k)
```

## Placeholder
Hasta ahora solo hemos creado expresiones constantes y esto no es tan util. Para introducir entradas del exterior a las expresiones debemos crear un tensor tipo `placeholder`

```python
x = tf.placeholder(tf.float32, ())
y = x * 2.0

print y
```

Ok, definimos un tensor sin ningun valor y creamos un operacion a partir de el, ¿pero que pasara si lo evaluamos en la session?

```python
try:
    sess.run(y)
except:
    print "Error!"
```

Como vemos obtenemos un error, esto es porque tensorflow necesita que definamos un valor para `x` dentro del grafo para poder calcular el valor de `y`. Esto lo hacemos pasando un diccionario de tensores a valores al parametro `feed_dict` de `run`

```python
valores = {x:  9.0}

print float(sess.run(y, valores))
```

## Variable
Hemos creados expresiones "puras" en el sentido que dados los mismo valores para los `placeholders` siempre data los mismo valores para el resto de los tensores, en otras palabras, no hay estado. Para tener sistemas que guarden estado utilizamos `Variable`. Creemos una variable sensilla

```python
a = tf.Variable(1.0)
print a
```

Aqui creamos una variable con un valor inicial de `1.0`, sin embargo las variables son un poco diferentes al resto de los tensores en el sentido que deben ser inicializadas, esto posiblemente es para reservar memoria en el dispositivo que vaya a almacenar esta variable, si no lo hacemos e intentamos evaluar una expresion tendremos errores

```python
try:
    print sess.run(a)
except:
    print "Error!"
```

Para iniciar las variables corremos `tf.global_variables_initializer()`

```python
init = tf.global_variables_initializer()
sess.run(init)
```

Esta operacion no retorna nada pero como efecto secundario inicia las variables. Ahora si corremos `a` podemos obtener su valor

```python
sess.run(a)
```

Como dijimos el proposito de tener variables es poder almacener y lo mas importante actualizar cierto estado, de otra manera parecen constantes. Por lo tanto vamos a definir la operacion

$a := a * 2.0$

en TensorFlow

```python
update = a.assign(a * 2.0) #a = a * 2
```

Ahora si corremos `a` veremos que

```python
sess.run(a)
```

...sigue igual. Esto es porque debemos correr el tensor `update` para que `a` se actualice

```python
sess.run(update)

print sess.run(a)
```

Perfecto! Si volvemos a correr la celda anterior varias veces veremos que `a` se va duplicando cada vez.

## Uniendo Todo
Ahora estamos preparados para recrear nuestra funcion `f` original en tensorflow

```python
w = tf.Variable(1.0) #parametro
b = tf.Variable(2.0) #parametro

x = tf.placeholder(tf.float32, []) #input

f = w * x + b  # f(x; w, b)

#iniciar variables
sess.run(tf.global_variables_initializer())

# f(1.0)
sess.run(f, feed_dict={x: 1.0})
```

Genial! Reproducimos el resultado original de python. Ahora nos falta modificar `b` a `10.0` como lo hicimos en anteriormente y ver como cambia `f`

```python
# b = 10.0
sess.run(tf.assign(b, 10.0))

# f(1.0)
sess.run(f, feed_dict={x: 1.0})
```

## ¿Que sigue?

Todo esto realmente parece una forma muy complicada de realizar oparaciones que serian mas facil con python o numpy. Las ventajas de TensorFlow para el Deep Learning y la computacion cientifica en general son las siguientes:

* Cada tensor puede estar ubicado en un dispositivo diferente como una gpu, la cpu normal, o una maquina remota. Esto es importante para correr grandes modelos utilizando paralelismo y computacion distribuida.
* La computacion simbolica le permite a tensorflow calcular derivadas de una expresion con respecto a una subexpresion de la misma, esto es importante para calcular el gradiente de las redes neuronales.

## Gradients
Gracias a la computacion simbolica, es decir, que no se ejecuta operacion para generar valores de forma inmediata sino que se guarda un grafo de la expresion, muy parecido a una formula matematica, TensorFlow puede calcular la derivada de una expresion con respecto a una subexpresion de la misma. Esto es muy importante para el algoritmo `Gradient Descent` que depende totalmente de poder calcular el gradiente del error con respecto a los pesos:

$
\begin{equation}
    \Delta = \nabla_{\theta} E(x, y; \theta)
\end{equation}
$
<br>
$
\begin{equation}
    \theta := \theta - \alpha \Delta
\end{equation}
$

A continuacion vamos a calcular la derivada de $f$ con respecto a $w$ utilizando la funcion `gradients`

```python
dfdw = tf.gradients(f, w)
```

Analiticamente el valor de esta operacion es

$
\begin{equation}
    \frac{\delta f}{\delta w} = \frac{\delta (x w + b)}{\delta w} = x
\end{equation}
$

y lo podemos comprobar numericamente:

```python
grad = sess.run(dfdw, feed_dict={x: 10.0})
x_val = sess.run(x, feed_dict={x: 10.0})

print "Gradiente: {0}, x: {1}".format(grad[0], x_val)
```

Como vemos, ambdas expresiones tienen el mismo valor!

## Gradient Descent
Ahora vamos aplicar todo el algoritmo de que involucra los siguientes pasos:

Primero vamos a importar unas librarias y generar algunos datos

```python
import plotly.offline as py
from plotly import graph_objs as go
import numpy as np
py.init_notebook_mode()

n = 50

x_data = np.random.uniform(low=0.0, high=100.0, size=(n, 1))
y_data = 3.5 * x_data + 40.3 + np.random.normal(loc=0.0, scale=25.0, size=(n, 1))

#normalizar data
x_data = (x_data - np.mean(x_data)) / np.average(x_data)
y_data = (y_data - np.mean(y_data)) / np.average(y_data)

scatter = go.Scatter(
    x=x_data[:, 0],
    y=y_data[:, 0],
    mode = 'markers',
    marker = dict(size = 10),
    name="data"
)

layout = go.Layout(xaxis=dict(title='x'), yaxis=dict(title='y'))

py.iplot(go.Figure(data=[scatter], layout=layout))
```

Ahora vamos a crear un modelo con parametros aleatorios y visualizarlo

```python
####################
# Tarea: Crear los tensores
####################

x =
y =

w =
b =

h =

sess.run(tf.global_variables_initializer())

x_model = [np.min(x_data), np.max(x_data)]
y_model = sess.run(h, feed_dict={x: x_model})

model_line = go.Scatter(
    x=x_model,
    y=y_model,
    name="model"
)

py.iplot(go.Figure(data=[scatter, model_line], layout=layout))
```

```python
alfa = 0.5

#calcular error
E =

#calcular gradiente
dEdw, dEdb =

#actualizar pesos
update_w, update_b =

#realizar operacion 200 veces
for i in range(200):
    #update

#visualizar modelo aprendido
y_model = sess.run(h, feed_dict={x: x_model})

model_line = go.Scatter(
    x=x_model,
    y=y_model,
    name="model"
)

py.iplot(go.Figure(data=[scatter, model_line], layout=layout))
```


# Biostat823 Assignment2

**Number theory and a Google recruitment puzzle**

Find the first 10-digit prime in the decimal expansion of 17π.

The first 5 digits in the decimal expansion of π are 14159. The first 4-digit prime in the decimal expansion of π are 4159. You are asked to find the first 10-digit prime in the decimal expansion of 17π. First solve sub-problems (divide and conquer):

- Write a function to generate an arbitrary large expansion of a mathematical expression like π. Hint: You can use the standard library `decimal` or the 3rd party library `sympy` to do this
- Write a function to check if a number is prime. Hint: See Sieve of Eratosthenes
- Write a function to generate sliding windows of a specified width from a long iterable (e.g. a string representation of a number)

Write unit tests for each of these three functions. You are encouraged, but not required, to try [test-driven development](https://en.wikipedia.org/wiki/Test-driven_development).

Now use these helper functions to write the function that you need.
Write a unit test for this final function, given that the first 10-digit prime in the expansion e is 7427466391. Finally, solve the given problem.

This assignment can be found in my github blog (named as Biostat 823 Assignment2): https://ashleyhmy.github.io/BIOS823_blog/

```python
import sympy as sym
import math
import unittest
```

```python
# Write a function to generate large arbitary expression for scientific expression
def num_expansion(expr, args):
    """
    generate an arbitary large expansion for a scientific expression like pi and e, returns numeric expression.

    expr is the mathematical expression to be converted eg, expr = sym.exp(1) use the sympy package to get scientific expression for e
    args is the number of significant numbers required eg, args = 5

    Examples to get arbitary expression for e with 5 significant numbers:
    >>> num_expansion(expr, 5)
    2.7183
    """
    num = expr.evalf(args)
    return num
```

```python
# Write a function to check if a number is a prime
def is_prime(num):
    """
    Take input num to check if num is a prime, if num is a prime return True,
    if num is not a prime return False.
    Example:
    >>>is_prime(17)
    False
    """

    if num<2:
        return False
    if num==2:
        return True
    if num>2 and num%2 == 0:
        return False
    for i in range(3, 1 + math.floor(math.sqrt(num)), 2):
        if num%i == 0:
            return False
    return True
```

```python
# Write a function to generate sliding windows of a sequence
def get_window(seq, digits):
    """
    seq is the input, a list of numbers
    win_size is the size of the window

    Example:
    seq = [1,2,3,4,5,6]
    win_size = 2
    >>> window(seq, win_size, step)
    [[1,2], [2,3], [3,4], [4,5], [5,6]]
    """

    num_of_chunk = int(len(seq)-digits + 1)
    for i in range(0, num_of_chunk):
        yield seq[i:i+digits]
```

```python
# Use unittest package to test the three functions above (num_expansion, is_prime, get_window)
class TestFunctions(unittest.TestCase):

    def test_num_expansion(self):
        result = pi.evalf(5)
        self.assertEqual(num_expansion(pi, 5), result)

    def test_is_prime(self):
        self.assertEqual(is_prime(17), True)
        self.assertEqual(is_prime(10), False)

    def test_window(self):
        seq = [1,2,3,4]
        self.assertEqual(list(get_window(seq, 3)), [[1,2,3],[2,3,4]])

if __name__ == "__main__":
    unittest.main(argv=[''], verbosity =2, exit=False)
```

    test_is_prime (__main__.TestFunctions) ... ok
    test_num_expansion (__main__.TestFunctions) ... ok
    test_window (__main__.TestFunctions) ... ok

    ----------------------------------------------------------------------
    Ran 3 tests in 0.002s

    OK

```python
# Now use the three helper function above to write a function to get the first 10 digit of a number
def get_first_prime(expr, args, digits):
    """
    The input expr is the methametical expression that want to be expanded
    The input args represents how many significant number is required from the methametical expression
    The input digits represents the number of digits is in the prime
    Example:
    >>>get_first_prime(pi, 200, 10)
    5926535897
    """
    num1 = num_expansion(expr, args)
    str1 = str(num1)
    list1 = str1.split('.')
    lst = [''.join(list1[0:2])]
    num = lst[0]
    seq = [int(a) for a in str(num)]
    windows = list(window(seq, digits))
    prime_lst = []
    for win in windows:
        str1 = ''.join(map(str, win))
        num_to_check = int(str1)
        if is_prime(num_to_check) == True:
            prime_lst.append(num_to_check)
    prime = prime_lst[0]
    return prime
```

```python
# test get_first_prime() function
class TestFunctions(unittest.TestCase):
    '''To check whether the output of get_first_prime() function is equal to 7427466391
    '''
    def test_get_first_prime(self):
        expr = sym.exp(1)
        self.assertEqual(get_first_prime(expr, 200, 10), 7427466391)

if __name__ == "__main__":
    unittest.main(argv=[''], verbosity =2, exit=False)
```

    test_get_first_prime (__main__.TestFunctions) ... ok

    ----------------------------------------------------------------------
    Ran 1 test in 0.025s

    OK

```python
# Find the first 10 digits of 17pi
get_first_prime(pi*17, 50, 10)
print("The first 10 digit prime of 17\u03C0 is", get_first_prime(pi*17, 50, 10))
```

    The first 10 digit prime of 17π is 8649375157


# Class 4: Plotting with Matplotlib

Matplotlib is a powerful plotting module that is part of Python's standard library. The website for matplotlib is at http://matplotlib.org/. And you can find a bunch of examples at the following two locations: http://matplotlib.org/examples/index.html and http://matplotlib.org/gallery.html.

Matplotlib contains a module called pyplot that was written to provide a Matlab-style plotting interface.

```python
# Import matplotlib.pyplot
import matplotlib.pyplot as plt
```

## An Introductory Example

Create a plot of the sine function for x values between -6 and 6. Add axis labels and a title.

```python
# Import numpy as np
import numpy as np

# Create an array of x values from -6 to 6
x = np.linspace(-6,6,1000)

# Create a variable y equal to the sine of x
y = np.sin(x)

# Use the plot function to plot x on the horizontal axis and y on the vertical axis
plt.plot(x,y)

# Add a title and axis labels
plt.title('sin(x)')
plt.xlabel('x')
plt.ylabel('y')
```

## Style Sheets and Magic Commands

Plots made using Matplotlib are highly customizable. Users can issue commands to control the limits of the axes, location of axis ticks, labels associated with axis ticks, and the color, thickness, opacity, and style of plotted data points.

But as a starting-point, Matplotlib uses a *style sheet* to establish initial plot settings until the user specifies different alternatives. The release of Matplotlib 2.0 included a major change in the default syle and the new default doesn't appeal to me.

Next, when we change from the default ploting style, it disrupts how Jupyter formats and displays plots. To fix this, we have to issue a command to be interpretted by Jupyter -- called a *magic* command. A magic command is preceded by a `%` character. Magics are *not* Python and will create errors if used in a Python program outside of the Jupyter notebook (i.e., if you are running Python in IDLE or executing a Python script form the command line, make sure to comment out lines with magics).

```python
# Set the Matplotlib style sheet to 'classic'
plt.style.use('classic')

# Magic command for the Jupyter Notebook to display plots in the notebook
%matplotlib inline

# Use the plot function to plot the sine function
plt.plot(x,y)

# Add a title and axis labels
plt.title('sin(x)')
plt.xlabel('x')
plt.ylabel('y')
```

The matplotlib attribute `plt.style.available` stores as a `list` the default sstyle sheets that you can choose from. Here's a gallery of the different sytles compared with each other. Aside from `classic`, other styles that I think look nice include: `bmh`, `ggplot`, and `seaborn`. But try them all out and see what you like!

```python
#Print the available matplotlib style sheets. PROVIDED
print(plt.style.available)
```

    ['Solarize_Light2', '_classic_test_patch', 'bmh', 'classic', 'dark_background', 'fast', 'fivethirtyeight', 'ggplot', 'grayscale', 'seaborn', 'seaborn-bright', 'seaborn-colorblind', 'seaborn-dark', 'seaborn-dark-palette', 'seaborn-darkgrid', 'seaborn-deep', 'seaborn-muted', 'seaborn-notebook', 'seaborn-paper', 'seaborn-pastel', 'seaborn-poster', 'seaborn-talk', 'seaborn-ticks', 'seaborn-white', 'seaborn-whitegrid', 'tableau-colorblind10']

## The `plot` Function

The `plot` function creates a two-dimensional plot of one variable against another.

```python
# Use the help function to see the documentation for plt.plot()
help(plt.plot)
```

    Help on function plot in module matplotlib.pyplot:

    plot(*args, scalex=True, scaley=True, data=None, **kwargs)
        Plot y versus x as lines and/or markers.

        Call signatures::

            plot([x], y, [fmt], *, data=None, **kwargs)
            plot([x], y, [fmt], [x2], y2, [fmt2], ..., **kwargs)

        The coordinates of the points or line nodes are given by *x*, *y*.

        The optional parameter *fmt* is a convenient way for defining basic
        formatting like color, marker and linestyle. It's a shortcut string
        notation described in the *Notes* section below.

        >>> plot(x, y)        # plot x and y using default line style and color
        >>> plot(x, y, 'bo')  # plot x and y using blue circle markers
        >>> plot(y)           # plot y using x as index array 0..N-1
        >>> plot(y, 'r+')     # ditto, but with red plusses

        You can use `.Line2D` properties as keyword arguments for more
        control on the appearance. Line properties and *fmt* can be mixed.
        The following two calls yield identical results:

        >>> plot(x, y, 'go--', linewidth=2, markersize=12)
        >>> plot(x, y, color='green', marker='o', linestyle='dashed',
        ...      linewidth=2, markersize=12)

        When conflicting with *fmt*, keyword arguments take precedence.

        **Plotting labelled data**

        There's a convenient way for plotting objects with labelled data (i.e.
        data that can be accessed by index ``obj['y']``). Instead of giving
        the data in *x* and *y*, you can provide the object in the *data*
        parameter and just give the labels for *x* and *y*::

        >>> plot('xlabel', 'ylabel', data=obj)

        All indexable objects are supported. This could e.g. be a `dict`, a
        `pandas.DataFrame` or a structured numpy array.

        **Plotting multiple sets of data**

        There are various ways to plot multiple sets of data.

        - The most straight forward way is just to call `plot` multiple times.
          Example:

          >>> plot(x1, y1, 'bo')
          >>> plot(x2, y2, 'go')

        - Alternatively, if your data is already a 2d array, you can pass it
          directly to *x*, *y*. A separate data set will be drawn for every
          column.

          Example: an array ``a`` where the first column represents the *x*
          values and the other columns are the *y* columns::

          >>> plot(a[0], a[1:])

        - The third way is to specify multiple sets of *[x]*, *y*, *[fmt]*
          groups::

          >>> plot(x1, y1, 'g^', x2, y2, 'g-')

          In this case, any additional keyword argument applies to all
          datasets. Also this syntax cannot be combined with the *data*
          parameter.

        By default, each line is assigned a different style specified by a
        'style cycle'. The *fmt* and line property parameters are only
        necessary if you want explicit deviations from these defaults.
        Alternatively, you can also change the style cycle using
        :rc:`axes.prop_cycle`.

        Parameters
        ----------
        x, y : array-like or scalar
            The horizontal / vertical coordinates of the data points.
            *x* values are optional and default to ``range(len(y))``.

            Commonly, these parameters are 1D arrays.

            They can also be scalars, or two-dimensional (in that case, the
            columns represent separate data sets).

            These arguments cannot be passed as keywords.

        fmt : str, optional
            A format string, e.g. 'ro' for red circles. See the *Notes*
            section for a full description of the format strings.

            Format strings are just an abbreviation for quickly setting
            basic line properties. All of these and more can also be
            controlled by keyword arguments.

            This argument cannot be passed as keyword.

        data : indexable object, optional
            An object with labelled data. If given, provide the label names to
            plot in *x* and *y*.

            .. note::
                Technically there's a slight ambiguity in calls where the
                second label is a valid *fmt*. ``plot('n', 'o', data=obj)``
                could be ``plt(x, y)`` or ``plt(y, fmt)``. In such cases,
                the former interpretation is chosen, but a warning is issued.
                You may suppress the warning by adding an empty format string
                ``plot('n', 'o', '', data=obj)``.

        Returns
        -------
        list of `.Line2D`
            A list of lines representing the plotted data.

        Other Parameters
        ----------------
        scalex, scaley : bool, default: True
            These parameters determine if the view limits are adapted to the
            data limits. The values are passed on to `autoscale_view`.

        **kwargs : `.Line2D` properties, optional
            *kwargs* are used to specify properties like a line label (for
            auto legends), linewidth, antialiasing, marker face color.
            Example::

            >>> plot([1, 2, 3], [1, 2, 3], 'go-', label='line 1', linewidth=2)
            >>> plot([1, 2, 3], [1, 4, 9], 'rs', label='line 2')

            If you make multiple lines with one plot call, the kwargs
            apply to all those lines.

            Here is a list of available `.Line2D` properties:

            Properties:
            agg_filter: a filter function, which takes a (m, n, 3) float array and a dpi value, and returns a (m, n, 3) array
            alpha: float or None
            animated: bool
            antialiased or aa: bool
            clip_box: `.Bbox`
            clip_on: bool
            clip_path: Patch or (Path, Transform) or None
            color or c: color
            contains: unknown
            dash_capstyle: {'butt', 'round', 'projecting'}
            dash_joinstyle: {'miter', 'round', 'bevel'}
            dashes: sequence of floats (on/off ink in points) or (None, None)
            data: (2, N) array or two 1D arrays
            drawstyle or ds: {'default', 'steps', 'steps-pre', 'steps-mid', 'steps-post'}, default: 'default'
            figure: `.Figure`
            fillstyle: {'full', 'left', 'right', 'bottom', 'top', 'none'}
            gid: str
            in_layout: bool
            label: object
            linestyle or ls: {'-', '--', '-.', ':', '', (offset, on-off-seq), ...}
            linewidth or lw: float
            marker: marker style string, `~.path.Path` or `~.markers.MarkerStyle`
            markeredgecolor or mec: color
            markeredgewidth or mew: float
            markerfacecolor or mfc: color
            markerfacecoloralt or mfcalt: color
            markersize or ms: float
            markevery: None or int or (int, int) or slice or List[int] or float or (float, float) or List[bool]
            path_effects: `.AbstractPathEffect`
            picker: unknown
            pickradius: float
            rasterized: bool or None
            sketch_params: (scale: float, length: float, randomness: float)
            snap: bool or None
            solid_capstyle: {'butt', 'round', 'projecting'}
            solid_joinstyle: {'miter', 'round', 'bevel'}
            transform: `matplotlib.transforms.Transform`
            url: str
            visible: bool
            xdata: 1D array
            ydata: 1D array
            zorder: float

        See Also
        --------
        scatter : XY scatter plot with markers of varying size and/or color (
            sometimes also called bubble chart).

        Notes
        -----
        **Format Strings**

        A format string consists of a part for color, marker and line::

            fmt = '[marker][line][color]'

        Each of them is optional. If not provided, the value from the style
        cycle is used. Exception: If ``line`` is given, but no ``marker``,
        the data will be a line without markers.

        Other combinations such as ``[color][marker][line]`` are also
        supported, but note that their parsing may be ambiguous.

        **Markers**

        =============    ===============================
        character        description
        =============    ===============================
        ``'.'``          point marker
        ``','``          pixel marker
        ``'o'``          circle marker
        ``'v'``          triangle_down marker
        ``'^'``          triangle_up marker
        ``'<'``          triangle_left marker
        ``'>'``          triangle_right marker
        ``'1'``          tri_down marker
        ``'2'``          tri_up marker
        ``'3'``          tri_left marker
        ``'4'``          tri_right marker
        ``'s'``          square marker
        ``'p'``          pentagon marker
        ``'*'``          star marker
        ``'h'``          hexagon1 marker
        ``'H'``          hexagon2 marker
        ``'+'``          plus marker
        ``'x'``          x marker
        ``'D'``          diamond marker
        ``'d'``          thin_diamond marker
        ``'|'``          vline marker
        ``'_'``          hline marker
        =============    ===============================

        **Line Styles**

        =============    ===============================
        character        description
        =============    ===============================
        ``'-'``          solid line style
        ``'--'``         dashed line style
        ``'-.'``         dash-dot line style
        ``':'``          dotted line style
        =============    ===============================

        Example format strings::

            'b'    # blue markers with default shape
            'or'   # red circles
            '-g'   # green solid line
            '--'   # dashed line with default color
            '^k:'  # black triangle_up markers connected by a dotted line

        **Colors**

        The supported color abbreviations are the single letter codes

        =============    ===============================
        character        color
        =============    ===============================
        ``'b'``          blue
        ``'g'``          green
        ``'r'``          red
        ``'c'``          cyan
        ``'m'``          magenta
        ``'y'``          yellow
        ``'k'``          black
        ``'w'``          white
        =============    ===============================

        and the ``'CN'`` colors that index into the default property cycle.

        If the color is the only part of the format string, you can
        additionally use any  `matplotlib.colors` spec, e.g. full names
        (``'green'``) or hex strings (``'#008000'``).

### Example

Create a plot of $f(x) = x^2$ with $x$ between -2 and 2.
* Set the linewidth to 3 points
* Set the line opacity (alpha) to 0.6
* Set axis labels and title
* Add a grid to the plot

```python
# Create an array of x values from -2 to 2
x = np.linspace(-2,2,1000)

# Create a variable y equal to x squared
y = x**2

# Use the plot function to plot the line
plt.plot(x,y,linewidth=3,alpha = 0.6)

# Add a title and axis labels
plt.title('$f(x) = x^2$')
plt.xlabel('x')
plt.ylabel('y')

# Add grid
plt.grid()
```

### Example

Create plots of the functions $f(x) = \log x$ (natural log) and $g(x) = 1/x$ between 0.01 and 5
* Make the line for $log(x)$ solid blue
* Make the line for $1/x$ dashed magenta
* Set the linewidth of each line to 3 points
* Set the line transparency (alpha) for each line to 0.6
* Set the limits for the $x$-axis to (0,5)
* Set the limits for the $y$-axis to (-2,5)
* Set axis labels and title
* Add a legend
* Add a grid to the plot

```python
# Create an array of x values from 0.01 to 5
x = np.linspace(0.01,5,1000)

# Create y variables
y1 = np.log(x)
y2 = 1/x

# Use the plot function to plot the lines
plt.plot(x,y1,'b-',lw=3,alpha = 0.6,label='$\log(x)$')
plt.plot(x,y2,'m--',c='m',linestyle='--',linewidth=3,alpha = 0.6,label='$1/x$')

# Add a title and axis labels
plt.title('Two functions')
plt.xlabel('x')
plt.ylabel('y')

# Set axis limits
plt.xlim([0,5])
plt.ylim([-2,4])

# Make legend
plt.legend(loc='lower right',ncol=2)

# Add grid
plt.grid()
```

### Example

Consider the linear regression model:
\begin{align}
y_i = \beta_0 + \beta_1 x_i + \epsilon_i
\end{align}
where $x_i$ is the independent variable, $\epsilon_i$ is a random regression error term, $y_i$ is the dependent variable and $\beta_0$ and $\beta_1$ are constants.

Let's simulate the model
* Set $\beta_0=1$ and $\beta_1=-0.5$
* Create an array of $x_i$ values from -5 to 5
* Create an array of $\epsilon_i$ values from the standard normal distribution equal in length to the array of $x_i$ values
* Create an array of $y_i$ values
* Plot y against x with either a circle ('o'), triangle ('^'), or square ('s') marker and transparency (alpha) to 0.5
* Add axis lables, a title, and a grid to the plot

```python
# Set betas
beta0 = 1
beta1 = -0.5

# Create x values
x = np.linspace(-5,5,1000)

# Create epsilon values from the standard normal distribution
epsilon = np.random.normal(size=len(x))

# Create y
y = beta0 + beta1*x+epsilon

# Plot
plt.plot(x,y,'s',alpha =0.5)

# Add a title and axis labels
plt.title('Data')
plt.xlabel('x')
plt.ylabel('y')

# Set axis limits
plt.xlim([-5,5])

# Add grid
plt.grid()
```

### Example

Create plots of the functions $f(x) = x$, $g(x) = x^2$, and $h(x) = x^3$ for $x$ between -2 and 2
* Use the optional string format argument to format the lines:
  - $x$: solid blue line
  - $x^2$: dashed green line
  - $x^3$: dash-dot magenta line
* Set the linewidth of each line to 3 points
* Set transparency (alpha) for each line to 0.7
* Set axis labels and title
* Add a legend to lower right with 3 columns
* Add a grid to the plot

```python
# Create an array of x values from -2 to 2
x = np.linspace(-2,2,1000)

# Create y variables
y1 = x
y2 = x**2
y3 = x**3

# Use the plot function to plot the lines
plt.plot(x,y1,'b-',lw=3,label='$x$',alpha=0.7)
plt.plot(x,y2,'g--',lw=3,label='$x^2$',alpha=0.7)
plt.plot(x,y3,'m-.',lw=3,label='$x^3$',alpha=0.7)

# Add a title and axis labels
plt.title('Three functions')
plt.xlabel('x')
plt.ylabel('y')

# Make legend
plt.legend(loc='lower right',ncol=3)

# Add grid
plt.grid()
```

## Figures, Axes, and Subplots

Often we want to create plots with multiple axes or we want to modify the size and shape of the plot areas. To be able to do these things, we need to explicity create a figure and then create axes within the figure. The best way to see how this works is by example.

### Example: A Single Plot with Double Width

The default dimensions of a matplotlib figure are 6 inches by 4 inches. As we saw above, this leaves some whitespace on the right side of the figure in the Notebook. Suppose we want to remove that by making the plot area twice as wide.

Plot the sine function on -6 to 6 using a figure with dimensions 12 inches by 4 inches

```python
# Create an array of x values from -6 to 6
x = np.linspace(-6,6,1000)

# Create y variables
y = np.sin(x)

# Create a new figure that is twice as wide as the default
fig = plt.figure(figsize=(12,4))

# Create axis within the figure
ax1 = fig.add_subplot(1,1,1)

# Plot
ax1.plot(x,y,lw=3,alpha = 0.6)

# Add grid
ax1.grid()
```

In the previous example the `fig = plt.figure()` function creates a new figure called `fig` and `fig.add_subplot()` puts a new axis on the figure. The command `ax1 = fig.add_subplot(1,1,1)` means divide the figure `fig` into a 1 by 1 grid (first two arguments) and assigns the first component of that grid to the variable `ax1` (third argument).

### Example: Two Plots Stacked

Create a new figure with two axes in a column and plot the sine function on -6 to 6 on the top axis and the cosine function on -6 to 6 on the bottom axis.

```python
# Create an array of x values from -6 to 6
x = np.linspace(-6,6,1000)

# Create y variables
y1 = np.sin(x)
y2 = np.cos(x)

# Create a new figure that is twice as tall as the default
fig = plt.figure(figsize=(6,8))

# Create axis 1 and plot with title and axis labels
ax1 = fig.add_subplot(2,1,1)
ax1.plot(x,y1,lw=3,alpha = 0.6)
ax1.grid()
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_title('sin')

# Create axis 2 and plot with title and axis labels
ax2 = fig.add_subplot(2,1,2)
ax2.plot(x,y2,lw=3,alpha = 0.6)
ax2.grid()
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('cos')

# Improve the spacing between the plots
fig.tight_layout()
```

### Example: Block of Four Plots

The default dimensions of a matplotlib figure are 6 inches by 4 inches. As we saw above, this leaves some whitespace on the right side of the figure. Suppose we want to remove that by making the plot area twice as wide.

Create a new figure with four axes in a two-by-two grid. Plot the following functions on the interval -2 to 2:
* $y = x$
* $y = x^2$
* $y = x^3$
* $y = x^4$

Set the figure size to 12in. by 8in. Run the command `plt.tight_layout()` to adust the figure's margins after creating your figure, axes, and plots.

```python
# Create data for plot
x = np.linspace(-2,2,1000)
y1 = x
y2 = x**2
y3 = x**3
y4 = x**4

# Create a new figure that is twice as wide and twice as tall as the default
fig = plt.figure(figsize=(12,8))

# Create axis 1 and plot with title
ax1 = fig.add_subplot(2,2,1)
ax1.plot(x,y1,lw=3,alpha = 0.6)
ax1.grid()
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_title('$x$')

# Create axis 2 and plot with title
ax2 = fig.add_subplot(2,2,2)
ax2.plot(x,y2,lw=3,alpha = 0.6)
ax2.grid()
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('$x^2$')

# Create axis 3 and plot with title
ax3 = fig.add_subplot(2,2,3)
ax3.plot(x,y3,lw=3,alpha = 0.6)
ax3.grid()
ax3.set_xlabel('x')
ax3.set_ylabel('y')
ax3.set_title('$x^3$')

# Create axis 4 and plot with title
ax4 = fig.add_subplot(2,2,4)
ax4.plot(x,y4,lw=3,alpha = 0.6)
ax4.grid()
ax4.set_xlabel('x')
ax4.set_ylabel('y')
ax4.set_title('$x^4$')

# Improve the spacing between the plots
plt.tight_layout()
```

## Exporting Figures to Image Files

Use the `plt.savefig()` function to save figures to images.

```python
# Create data to plot sin(x)/x on [-20,20]
x = np.linspace(-20,20,1000)
y = np.sin(x)/x

# Plot y against x
plt.plot(x,y)

# Save figure as 'fig_econ126_class04_sine_over_x.png' at 120 DPI resolution
plt.savefig('fig_econ126_class04_sine_over_x.png',dpi=120)
```

In the previous example, the image is saved as a PNG file with 120 dots per inch. This resolution is high enough to look good even when projected on a large screen. The image format is inferred by the extension on the filename.


# Lista 1 de Computação Científica - Newtons e Secante, Simetrias e Análise

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
## Metódo de Newton
def newton(f,df,x, prec=1e-8, maxiter=100, iterr = 0, lst = [], reti = False):
    if maxiter == iterr:
        return None
    dx = f(x)/df(x)
    newx = x - dx
    xs = lst[:]
    xs.append(x)
    if abs(dx) < prec:
        if reti:
            return xs, iterr
        else:
            return xs
    else:
        return newton(f,df,newx, prec,iterr = iterr+1, lst = xs)
```

***
***
## Parte 1

***
### Questão 1

O próximo ponto do metódo de Newton de uma função $f$ é dado por:
$$
\begin{align}
x_{f_{n+1}} = x_n - \frac{f(x)}{f'(x)} &&(1)
\end{align}
$$
Como: $g(x) = -f(x)$

Teremos: $g'(x) = -f'(x)$

Olhando agora para o próximo ponto do metódo de Newton na função $g$

$$
\begin{align}
x_{g_{n+1}} = x_n - \frac{g(x)}{g'(x)} \implies x_{g_{n+1}} = x_n - \frac{-f(x)}{-f'(x)} \implies x_{g_{n+1}} = x_n - \frac{f(x)}{f'(x)} &&(2)
\end{align}
$$
Temos então (1) = (2), e portanto o metódo de Newton produz os mesmo pontos para $g$ e para $f$

Como os pontos gerados são os mesmos para $f(x)$ e para $-f(x)$

podemos sempre supor $f '(z) > 0$, pois caso $f ' (z) < 0$, basta tomar $-f ' (z)$, pois:

$f'(z) < 0 \implies -f'(z) > 0$

***
### Questão 2

Seja $g(x) = c\cdot f(x) $ com $C$ constante, $C \in \mathbb{R}$

Teremos: $g(x) = c\cdot f(x)\implies g'(x) = c\cdot f'(x)$

Então o ponto no metódo de Newton é dado por:

$$
\begin{align}
x_{g_{n+1}}&= x_n - \frac{g(x)}{g'(x)}\\
&= x_n - \frac{c\cdot f(x)}{c\cdot f'(x)}\\
&\text{Cancelando } c \text{ na fração, temos: }\\
& = x_n - \frac{f(x)}{f'(x)} \\
\end{align}
$$
> O metódo de Newton para $g(x) = c\cdot f(x)$  gera os mesmos pontos que o metódo de Newton para $f(x)$

***
### Questão 3

```python
## Funções Polinomiais
# f''(z)=0
def pol1(x): return (x-1)
def dPol1(x): return (x)
def d2Pol1(x): return (0*x)

# f''(z)>0
def pol2(x): return (2*x**3 - 2*x)/4
def dPol2(x): return (6*x**2 - 2)/4
def d2Pol2(x): return (12*x)/4

# f''(z)<0
def pol3(x): return (-x**2+x)
def dPol3(x): return (-2*x+1)
def d2Pol3(x): return (-2+(0*x))
```

```python
t = np.linspace(-3,3)

plt.figure(figsize=(15,6))
plt.plot(t,pol1(t), color="blue", linestyle='-', label="f (x)")
plt.plot(t,dPol1(t), color="orange", linestyle='-.', label="f ' (x)")
plt.plot(t,d2Pol1(t), color="red", linestyle=":", label="f '' (x)")
plt.plot(1,0,('o'),color='blue', label="f(z)")
plt.plot(1,dPol1(1),('o'),color='yellow', label="f'(z)")
plt.plot(1,d2Pol1(1),('o'),color='red', label="f''(z)")

ax1 = plt.gca()
ax1.spines['right'].set_color('none')
ax1.spines['top'].set_color('none')
ax1.spines['bottom'].set_position(('data',0))
ax1.spines['left'].set_position(('data',0))
plt.title("f ' ' (z) = 0")
plt.legend()
plt.show()
```

```python
t = np.linspace(-2,2)

plt.figure(figsize=(15,6))
plt.plot(t,pol2(t), color="blue", linestyle='-', label="f (x)")
plt.plot(t,dPol2(t), color="orange", linestyle='-.', label="f ' (x)")
plt.plot(t,d2Pol2(t), color="red", linestyle=":", label="f '' (x)")
plt.plot(1,0,('o'),color='blue', label="f(z)")
plt.plot(1,dPol2(1),('o'),color='yellow', label="f'(z)")
plt.plot(1,d2Pol2(1),('o'),color='red', label="f''(z)")

ax1 = plt.gca()
ax1.spines['right'].set_color('none')
ax1.spines['top'].set_color('none')
ax1.spines['bottom'].set_position(('data',0))
ax1.spines['left'].set_position(('data',0))

plt.title("f ' ' (z) > 0")
plt.legend()
plt.show()
```

```python
t = np.linspace(-2,2)

plt.figure(figsize=(15,6))
plt.plot(t,pol3(t), color="blue", linestyle='-', label="f (x)")
plt.plot(t,dPol3(t), color="orange", linestyle='-.', label="f ' (x)")
plt.plot(t,d2Pol3(t), color="red", linestyle=":", label="f '' (x)")
plt.plot(1,0,('o'),color='blue', label="f(z)")
plt.plot(1,dPol3(1),('o'),color='yellow', label="f'(z)")
plt.plot(1,d2Pol3(1),('o'),color='red', label="f''(z)")

ax1 = plt.gca()
ax1.spines['right'].set_color('none')
ax1.spines['top'].set_color('none')
ax1.spines['bottom'].set_position(('data',0))
ax1.spines['left'].set_position(('data',0))

plt.title("f ' ' (z) < 0")
plt.legend()
plt.show()
```

```python
## Funções transcendentes

# f''(z) = 0
def f1(x): return (np.sin(x))
def dF1(x): return (np.cos(x))
def d2F1(x): return (-np.sin(x))

#f''(z) > 0
def f2(x): return (np.e**x-1)
def dF2(x): return (np.e**x)
def d2F2(x): return (np.e**x)

#f''(z) < 0
def f3(x): return (-np.e**-x+1)
def dF3(x): return (np.e**-x)
def d2F3(x): return (-np.e**-x)
```

```python
#plot da f1(x) e suas derivadas
ts = np.linspace(-np.pi,np.pi)

plt.figure(figsize=(15,6))
plt.plot(ts,f1(ts), color="blue", linestyle='-', label="f (x)")
plt.plot(ts,dF1(ts), color="orange", linestyle='-.', label="f ' (x)")
plt.plot(ts,d2F1(ts), color="red", linestyle=":", label="f '' (x)")
plt.plot(0,0,('o'),color='blue')
plt.plot(0,dF1(0),('o'),color='yellow')
plt.plot(0,d2F1(0),('o'),color='red')

ax1 = plt.gca()
ax1.spines['right'].set_color('none')
ax1.spines['top'].set_color('none')
ax1.spines['bottom'].set_position(('data',0))
ax1.spines['left'].set_position(('data',0))
plt.title("f ' ' (z) = 0")
plt.legend()
plt.show()
```

```python
#plot da f2(x) e suas derivadas
ts = np.linspace(-2,2)

plt.figure(figsize=(15,6))
plt.plot(ts,f2(ts), color="blue", linestyle='-', label="f (x)")
plt.plot(ts,dF2(ts), color="orange", linestyle='-.', label="f ' (x)")
plt.plot(ts,d2F2(ts), color="red", linestyle=":", label="f '' (x)")
plt.plot(0,0,('o'),color='blue', label="f(z)")
plt.plot(0,dF2(0),('o'),color='yellow', label="f'(z)")
plt.plot(0,d2F2(0),('o'),color='red', label="f''(z)")

ax1 = plt.gca()
ax1.spines['right'].set_color('none')
ax1.spines['top'].set_color('none')
ax1.spines['bottom'].set_position(('data',0))
ax1.spines['left'].set_position(('data',0))
plt.title("f ' ' (z) > 0")
plt.legend()
plt.show()
```

```python
#plot da f3(x) e suas derivadas
ts = np.linspace(0,np.pi)

plt.figure(figsize=(15,6))
plt.plot(ts,f3(ts), color="blue", linestyle='-', label="f (x)")
plt.plot(ts,dF3(ts), color="orange", linestyle='-.', label="f ' (x)")
plt.plot(ts,d2F3(ts), color="red", linestyle=":", label="f '' (x)")
plt.plot(0,f3(0),('o'),color='blue', label="f(z)")
plt.plot(0,dF3(0),('o'),color='yellow', label="f'(z)")
plt.plot(0,d2F3(0),('o'),color='red', label="f''(z)")

ax1 = plt.gca()
ax1.spines['right'].set_color('none')
ax1.spines['top'].set_color('none')
ax1.spines['bottom'].set_position(('data',0))
ax1.spines['left'].set_position(('data',0))
plt.title("f ' ' (z) < 0")
plt.legend()
plt.show()
```

***
### Questão 4

Seja $g(x) = f(x+b)$

Então: $g'(x) = f'(x+b)$

Os pontos de $g(x)$ são dados por:

$$\begin{align}
g_{next}(x) &= x - \frac{g(x)}{g'(x)} \\
&= x - \frac{f(x+b)}{f'(x+b)}
\end{align}
$$
E os pontos de $ f(x)$ são dados por:
$$
f_{next}(x) = x - \frac{f(x)}{f'(x)}\\
$$
Tomando o ponto transladado $x-b$
$$
\begin{align}
g_{next}(x-b) &= (x-b) - \frac{f(x-b+b)}{f'(x-b+b)} \\
&= x - b - \frac{f(x)}{f'(x)} \\
&= f_{next}(x) - b\\
\end{align}
$$
> Que nada mais são do que os pontos iterados em $ f $ e transladados por $ b$

***
### Questão 5

```python
def f(x): return x**2-x
def dF(x): return 2*x-1
def g(x): return f(x+2)
def dG(x): return (1)
```

```python
a = newton(f,dF,-1)
b = newton(g,dG,0)

ts = np.linspace(-2,1)

_, [ax1, ax2] = plt.subplots(ncols=2, figsize=(15,4))
ax1.set_title('f(x)')
ax1.plot(ts, f(ts), label='f(x)')
ax1.plot(ts, [0]*len(ts), color="black")
ax1.plot(0,0,"o", color="blue", label="raiz")
ax1.plot(-1,f(-1),"o", color="green", label="x_n")
ax1.plot(a[1],0,"o", color="orange", label="x_n+1")
ax1.legend(loc=0)

ax2.set_title('g(x)')
ax2.plot(ts, g(ts), label='g(x)')
ax2.plot(ts, [0]*len(ts), color="black")
ax2.plot(-1,0,"o", color="blue", label="raiz")
ax2.plot(0,g(0),"o", color="green", label="x_n")
ax2.plot(b[1],0,"o", color="orange", label="x_n+1")
ax2.legend(loc=0)

plt.show()
```

***
### Questão 6

$$
f(z) = 0
\text{, e }
g(0) = 0\\
\text{como } g(x) = f(x+b)\\
\therefore g(0) = f(0+b) = f(b) = 0\\
\text{Tomando } b = z \text{ teremos: }\\
f(b) = f(z) = 0 \\
$$
Então geram pontos transladados por $z$

Podemos então supor uma translação em uma raíz $ z_0 $ de $f$ de forma que a nova raíz seja $ z = 0$

***
### Questão 7

$$
g(x) = -f(-x)\\
f(z) = 0, z = 0 \implies g(z) = -f(-z) = -f(0) = 0 \\
\text{Logo, } z = 0 \text{ é raíz da }g(x)
$$
-  Analisando o sinal da primeira derivada

$$
\\
g(x)=-f(-x) \implies g'(x) = -f'(-x)\cdot(-x)' = f'(-x)
\\
$$
-  Analisando o sinal da segunda derivada
$$
\\
g'(x)=f'(-x) \implies g''(x) = f''(-x)\cdot(-x)' = -f''(-x)
\\
$$
-  Analisando o metódo aplicado a g
$$
\begin{align}
g_{next}(x) & = x - \frac{g(x)}{g'(x)}\\
& = x - \frac{-f(x)}{f'(x)}\\
& = x + \frac{f(x)}{f'(x)}
\end{align}\\
$$
-  Analisando o metódo aplicado a f
$$
\begin{align}
f_{next}(x) & = x - \frac{f(x)}{f'(x)}
\end{align}
$$

> Logo os metódos criam pontos simétricos

***
### Questão 8

$$
g(x) = C \cdot f(ax+b)
$$

-  Queremos $g(0) = 0$
$$
g(0) = C\cdot f(a\cdot 0 + b)\\
g(0) = C\cdot f(b)\\
C\cdot f(b) = 0 \iff f(b) = 0
$$
> Logo, basta tomar $ b = z$, onde $f(z) = 0$

-  Queremos $g'(0) = 1$
$$
\begin{align}
g'(x) & = C\cdot f'(ax+b)\cdot(ax+b)'\\
& = C\cdot f'(ax+b)\cdot a\\
g'(0) & = 1 \\
& = C\dot f'(0+z)\cdot a
\end{align}
$$
Como visto na Questão 2, podemos considerar $f'(z) = 1$, portanto:
$$
\begin{align}
1 &= C\dot f'(z)\cdot a \\
& = C\cdot a \\
& \implies a = \frac{1}{C}
\end{align}
$$
> Logo, basta tomarmos $ a = \frac{1}{C}$

-  Queremos $g''(0) = 1$
$$
\begin{align}
g'(x) & = C\cdot f'(\frac{1}{C}\cdot x + z)\cdot \frac{1}{C}\\
\implies g''(x) & = f''(\frac{1}{C}\cdot x+z)\cdot \frac{1}{C}
\end{align}
$$
Substituindo
$$
\begin{align}
g''(0) & = f''(\frac{1}{C}\cdot 0+z)\cdot \frac{1}{C}\\
&= f''(z)\cdot \frac{1}{C} = 1\\
&\implies C = f''(z)
\end{align}
$$

> Logo, basta tomar $C = f''(z)$

***
### Questão 9

-  Analisando o metódo de Newton para a função $g$ criada na questão anterior
$$
f(x) = x^2 - 2 \implies f(\sqrt{2}) = 0
$$

Vamos aplicar os passos feitos na questão 8 para gerar uma $g$

$$
b = z = \sqrt{2}
$$
$$
C = f''(z) = 2
$$
$$
a = \frac{1}{C} = \frac{1}{2}
$$
$$
\therefore
g(x) = 2 \cdot f(\frac{x}{2}+\sqrt{2})
$$

```python
def f_9(x): return x**2 - 2
def df_9(x): return 2*x
def d2f_9(x): return 2

b = 2**(1/2)
c = 2
a = 1/c

def g(x): return c*f(x*a + b)
def dg(x): return f(x*a+b)

n1 = newton(f_9,df_9,3)
n2 = newton(g,dg,3)
```

```python
ts = np.linspace(-2,2)

plt.figure(figsize=(8,6))
plt.plot(ts,f_9(ts), color="blue", linestyle='-', label="f (x)")
plt.plot(ts,g(ts), color="red", linestyle=':', label="g (x)")
plt.plot(ts, [0]*len(ts), color="black")
plt.legend()
plt.show()
```

***
***
## Parte 2

***
### Questão 1

Temos que $f'(0) = 1$

$\therefore \lim_{x->0} f'(x) - 1 = 0$

$$\forall \epsilon > 0 \text{, }\exists \delta > 0 \text{ ; } |x| < \delta \implies |f'(x)-1| < \epsilon$$
Tomando $\epsilon = 1/2$ teremos:

$$|f'(x)-1| < \epsilon \implies |f'(x)-1| < \frac{1}{2} \implies -\frac{1}{2} < f'(x) - 1 < \frac{1}{2}$$
Somando 1 na inequação:

$$
-\frac{1}{2} + 1 < f'(x) - 1 + 1 < \frac{1}{2} + 1 \\
\frac{1}{2} < f'(x) < \frac{3}{2}
$$
Analogamente para $f''(x)$ chegamos a mesma conclusão
$$
\frac{1}{2} < f''(x) < \frac{3}{2} \\
\text{Então: } x \in (-1/2, 1/2) \implies \frac{1}{2} < f'(x) < \frac{3}{2} \text{ e } \frac{1}{2} < f''(x) < \frac{3}{2} \\
$$

***
### Questão 2

Fórmula de Taylor centrada em z:
$$
f(x) = f(z) + f'(z)\cdot(x-z) + \frac{f''(\xi)\cdot(x-\xi)^2}{2}\\
f(z) = 0, f'(z) = 1 \text{ e } f''(z) = 1 \implies f(x) = 0 + 1\cdot(x-0) + \frac{f''(\xi)\cdot(x-\xi)^2}{2}\\
\text{Logo: } f(x) = x + \frac{f''(\xi)\cdot(x-\xi)^2}{2}\\
\xi\text{ está em uma vizinhança de } z \text{, podemos dizer então: }\xi \in (-\epsilon,\epsilon)\\
\text{ logo: }\frac{1}{2} < f''(\xi) < \frac{3}{2}\\
\therefore f''(\xi) > 0 \text{ e } (x-\xi)^2 > 0 \text{ então temos que } \frac{f''(\xi)\cdot(x-\xi)^2}{2} > 0 \\
\implies f(x) > x \text{ , pois } f(x) = x + \frac{f''(\xi)\cdot(x-\xi)^2}{2}\\
\text{Como } f(x) > x\text{ , e temos que } x > 0 \text{ por transitividade teremos: } f(x) > 0\\
N_f(x) = x - \frac{f(x)}{f'(x)}\\
x\in(-\epsilon,\epsilon) \implies \frac{1}{2}<f'(x)<\frac{3}{2}\\
\therefore \frac{f(x)}{f'(x)} > 0 \implies N_f(x) < x \text{, pois estamos subtraindo uma quantia positiva de x}
$$

***
### Questão 3

$$
N_f(x) = x - \frac{f(x)}{f'(x)}\\
\text{Olhando para a expansão de Taylor em 0, com }\xi \in (0,x)\\
0 = f(0) = f(x) + f'(x)\cdot(0-x)+\frac{f''(\xi)\cdot(0-x)^2}{2}\\
0 = f(x) - x\cdot f'(x) + \frac{f''(\xi)\cdot x^2}{2}\\
-f(x) + x\cdot f'(x) = \frac{f''(\xi)\cdot x^2}{2}\\
\text{dividindo a equação por } f'(x)\\
x - \frac{f(x)}{f'(x)} = \frac{f''(\xi)\cdot x^2}{2\cdot f'(x)}\\
\therefore N_f(x) = \frac{f''(\xi)}{2\cdot f'(x)}\cdot x^2 \\
\text{Já mostramos que: } \frac{1}{2} < f'(x) < \frac{3}{2} \text{ e portanto } f'(x) > 0 \\
\text{Temos que } x^2 > 0, \forall x \in (-\epsilon, \epsilon) \\
\text{E } \xi \text{ esta próximo da raíz z, logo } \frac{1}{2} < f''(\xi) < \frac{3}{2} \implies f''(\xi) > 0 \\
\text{Então podemos concluir que: } \frac{f''(\xi)}{2\cdot f'(x)}\cdot x^2 > 0 \\
\text{Do exercicio anterior foi demonstrado que } N_{f}(x) < x \\
\text{Como } N_f(x) = \frac{f''(\xi)}{2\cdot f'(x)}\cdot x^2 \text{ e } \frac{f''(\xi)}{2\cdot f'(x)}\cdot x^2 > 0 \text{ e } x \in (-\epsilon,\epsilon)\\
\text{Concluimos que }\\
N_f(x) \in (-\epsilon,\epsilon)
$$

```python

```

comp-cientifica-I-2018-2/listas/Lista 1 - Gil Sales Miranda Neto.ipynb


# Quick overview of the finite element method

Example on a complicated domain for solving PDEs.

The finite element method is a rich and versatile approach to construct
computational schemes to solve any partial differential equation on
any domain in any dimension. The method may at first glance appear
cumbersome and even unnatural as it relies on variational formulations
and polynomial spaces.

Let us start by outlining the concepts briefly.
Consider the following PDE in 2D:

$$
-\nabla^2 u = -u_{xx} - u_{yy} = f,
$$

equipped with suitable boundary conditions.
A finite difference scheme to solve the current PDE
would in the simplest case be described by the stencil

$$
\begin{equation}
\label{overview:2d:fdm0} \tag{1}
-\frac{u_{i-1,j} - 2 u_{i,j} + u_{i+1,j}}{h^2}
-\frac{u_{i,j-1} - 2 u_{i,j} + u_{i,j+1}}{h^2}
 = f_{i}
\end{equation}
$$

or reordered to the more recognizable

$$
\begin{equation}
\label{overview:2d:fdm} \tag{2}
\frac{-u_{i-1,j} -u_{i,j-1} + 4 u_{i,j} - u_{i+1,j} -u_{i,j+1}}{h^2}  = f_{i}
{\thinspace .}
\end{equation}
$$

On a structured mesh, the stencil appears natural and
is convenient to implement.
However, for a unstructured, "complicated" domain
as shown in [Figure](#overview:meshex),
we would need to be careful when placing
points and evaluating stencils and functions.
In particular,
it will be difficult to evaluate the stencil near the dolphin in
[Figure](#overview:meshex) because some points will be on the inside and some outside on the outside of the dolphin.
Both accuracy and efficiency
may easily be sacrificed by a reckless implementation.

In general, a domain like the one represented in [Figure](#overview:meshex) will be represented by a triangulation. The
finite element method (and the finite volume method which often is a
special case of the finite element method) is a methodology for
creating stencils in a structured manner
that adapt to the underlying triangulation.

The triangulation in [Figure](#overview:meshex) is a mesh that
consists of cells that are connected and defined in terms of
vertices. The fundamental idea of the finite element method is
to construct a procedure to compute a stencil on a general element and
then apply this procedure to each element of the mesh. Let
us therefore denote the mesh as $\Omega$ while $\Omega_e$ is the domain
of a generic element such that $\Omega=\cup_e \Omega_e$.

This is exactly the point where the challenges of the finite element
method start and where we need some new concepts.  The basic question
is: How should we create a stencil for a
general element and a general PDE that has the maximal accuracy and
minimal computational complexity at the current triangulation?  The
two basic building blocks of the finite element method are

1. the solution is represented in terms of a polynomial expression on the
   given general element, and

2. a variational formulation of the PDE
   where element-wise integration enables the PDE to be transformed to a
   stencil.

Step 1 is, as will be explained later, conveniently represented
both implementation-wise and mathematically as a solution

$$
\begin{equation}
\label{overview:u:fem} \tag{3}
u = \sum_{i=0}^N c_i {\psi}_i(x,y),
\end{equation}
$$

where $\{c_i\}$ are the coefficients to be determined
(often called the degrees of freedom)
and ${\psi}_i(x,y)$ are prescribed polynomials.
The basis functions ${\psi}_i(x,y)$ used to express the solution
is often called the trial functions.

The next step is the variational formulation. This step
may seem like a magic trick or a cumbersome
mathematical exercise at first glance.
We take the PDE and multiply by a function $v$ (usually called
the test function)
and integrate over an element $\Omega_e$ and obtain the expression

$$
\begin{equation}
\label{overview:poisson} \tag{4}
\int_{\Omega_e} -\nabla^2 u \, v {\, \mathrm{d}x} =  \int_{\Omega_e} f \, v {\, \mathrm{d}x}
\end{equation}
$$

A perfectly natural question at this point is: Why multiply
with a test function $v$? The simple answer is that
there are $N+1$ unknowns that need to be determined in $u$
in ([3](#overview:u:fem))
and for this we need $N+1$ equations. The equations are
obtained by using  $N+1$ different test functions which when used
in  ([5](#overview:fem:a))
give rise to $N+1$ linearly independent equations.

While ([4](#overview:poisson)) is a variational formulation of
our PDE problem, it is not the most common form.
It is common to re-write

$$
\begin{equation}
\label{overview:fem:a} \tag{5}
\int_{\Omega_e} -\nabla^2 u \, v {\, \mathrm{d}x}
\end{equation}
$$

to weaken the requirement of the polynomial space used for the
trial functions (that here needs to be twice differentiable)
and write this term in its corresponding weak form.
That
is, the term is rewritten in terms of first-derivatives only (of
both the trial and the test function)  with the aid of Gauss-Green's lemma:

$$
\begin{equation}
\label{overview:fem:a:weak} \tag{6}
\int_{\Omega_e} -\nabla^2 u \, v {\, \mathrm{d}x}  =
\int_{\Omega_e} \nabla u \cdot \nabla v {\, \mathrm{d}x}   - \int_{\partial \Omega_e} \frac{\partial u}{\partial n} \,  v \,  dS
\end{equation}
$$

The reasons behind this alternative formulation are rather mathematical and will
not be a major subject of this book as they are well described elsewhere.
In fact, a precise explanation would need tools from functional analysis.

With the above rewrite and assuming now that the boundary term vanishes due to
boundary conditions (why this is possible will be dealt with in detail
later) the
stencil, corresponding to ([2](#overview:2d:fdm)),  is represented by

$$
\int_{\Omega_e} \nabla u \cdot \nabla v {\, \mathrm{d}x}
$$

where $u$ is called the *trial function*, $v$ is called a *test function*,
and $\Omega$ is an element of
a triangulated mesh. The idea of software like FEniCS is that this
piece of mathematics can be directly expressed in terms of Python code as

```python
# DO NOT RUN THIS CELL, THIS IS A DEMONSTRATION
mesh = Mesh("some_file")
V = FunctionSpace(mesh, "some polynomial")
u = TrialFunction(V)
v = TestFunction(V)
a = dot(grad(u), grad(v))*dx
```

The methodology and code in this example is not tied to a particular
equation, except the formula for `a`, holding the derivatives of our
sample PDE, but any other PDE terms could be expressed via `u`, `v`,
`grad`, and other symbolic operators in this line of code.  In fact,
finite element packages like FEniCS are typically structured as
general toolboxes that can be adapted to any PDE as soon as the
derivation of variational formulations is mastered.  The main obstacle
here for a novice FEM user is then to understand the concept of trial
functions and test functions realized in terms of polynomial spaces.

Hence, a finite element formulation (or a weak formulation) of
the Poisson problem that works on any mesh $\Omega$ can be written
in terms of solving the problem:

$$
\int_\Omega\nabla u\cdot\nabla vd{\, \mathrm{d}x} = \int_\Omega fv{\, \mathrm{d}x}{\thinspace .}
$$

By varying the trial and test spaces we obtain different stencils,
some of which will be identical to finite difference schemes on
particular meshes. We will now show a complete FEniCS program to
illustrate how a typical finite element code may be structured

```python
# DO NOT RUN THIS CELL, THIS IS A DEMONSTRATION
mesh = Mesh("some_file")
V = FunctionSpace(mesh, "some polynomial")
u = TrialFunction(V)
v = TestFunction(V)
a = dot(grad(u), grad(v))*dx
L = f*v*dx

bc = DirichletBC(V, "some_function", "some_domain")
solution = Function(V)  # unknown FEM function
solve(a == L, solution, bc)
plot(solution)
```

While the finite element method is versatile and may be adapted to any
PDE on any domain in any dimension, the different methods that are
derived by using different trial and test functions may vary
significantly in terms of accuracy and efficiency. In fact, a bad choice of polynomial space may in some cases lead to a
completely wrong result. This is particularly the case for complicated
PDEs. For this reason, it is dangerous to regard the method as a black
box and not do proper verification of the method for a  particular
application.

In our view, there
are three important tests that should be frequently employed
during verification:

1. reducing the model problem to 1D and carefully check the calculations involved in the variational formulation on a small 1D mesh

2. perform the calculation involved on one general or random element

3. test whether convergences is obtained and to what order the method converge by refining the mesh

The two first tasks here should ideally be performed by independent calculations
outside the framework used for the simulations. In our view `sympy` is a
convenient tool that can be used to assist hand calculations.

So far, we have outlined how the finite element method handles derivatives
in a PDE, but we also had a right-hand side function $f$. This term is multiplied
by the test function $v$ as well, such that the entire Poisson equation
is transformed to

$$
\int_\Omega\nabla u\cdot\nabla vd{\, \mathrm{d}x} = \int_\Omega fv{\, \mathrm{d}x}{\thinspace .}
$$

This statement is assumed valid for all test functions $v$ in some
function space $V$ of polynomials. The right-hand side expression is
coded in FEniCS as

```python
# DO NOT RUN THIS CELL. RUN THE FULL PROGRAM (IN THE END) INSTEAD.
L = f*v*dx
```

and the problem is then solved by the statements

```python
# DO NOT RUN THIS CELL. RUN THE FULL PROGRAM (IN THE END) INSTEAD.
u = Function(V)  # unknown FEM function
solve(a == L, u, bc)
```

where `bc` holds information about boundary conditions. This information
is connected to information about the triangulation, the *mesh*.
Assuming $u=0$ on the boundary, we can in FEniCS generate a triangular
mesh over a rectangular domain $[-1,-1]\times [-1,1]$ as follows:

```python
# DO NOT RUN THIS CELL. RUN THE FULL PROGRAM (IN THE END) INSTEAD.
mesh = RectangleMesh(Point(-1, -1), Point(1, 1), 10, 10)
bc = DirichletBC(V, 0, 'on_boundary')
```

Mathematically, the finite element method transforms our PDE to
a sparse linear system. The `solve` step performs two tasks:
construction of the linear system based on the given information about
the domain and its elements, and then solution of the linear system by
either an iterative or direct method.

We are now in a position to summarize all the parts of a FEniCS program
that solves the Poisson equation by the finite element method:

```python
from fenics import *
mesh = RectangleMesh(Point(-1, -1), Point(1, 1), 10, 10)
V = FunctionSpace(mesh, 'P', 2)  # quadratic polynomials
bc = DirichletBC(V, 0, 'on_boundary')
u = TrialFunction(V)
v = TestFunction(V)
a = dot(grad(u), grad(v))*dx
L = f*v*dx
u = Function(V)  # unknown FEM function to be computed
solve(a == L, u, bc)
vtkfile = File('poisson.pvd'); vtkfile << u  # store solution
```

Solving a different PDE is a matter of changing `a` and `L`.

Although we assert here that the finite element method is a tool that
can solve any PDE problem on any domain of any complexity, the
fundamental ideas of the method are in fact even more general.
We will therefore start the book by variational
methods for approximation in general, then consider the finite
element in a wide range of applications.


# Options 1

This notebook shows payoff functions and pricing bounds for options.

## Load Packages and Extra Functions

```julia
using Printf

include("jlFiles/printmat.jl")
```

    printyellow (generic function with 1 method)

```julia
using Plots, LaTeXStrings

#pyplot(size=(600,400))
gr(size=(480,320))
default(fmt = :png)
```

# Payoffs and Profits of Options

Let $K$ be the strike price and $S_m$ the price of the underlying at expiration ($m$ years ahead) of the option contract.

The call and put profits (at expiration) are

$\text{call profit}_{m}\ = \max\left(  0,S_{m}-K\right) - e^{my}C$

$\text{put profit}_{m}=\max\left(  0,K-S_{m}\right)  - e^{my}P $,

where $C$ and $P$ are the call and put option prices (paid today). In both cases the first term ($\max()$) represents the payoff at expiration, while the second term ($e^{my}C$ or $e^{my}P$) subtracts the capitalised value of the option price (premium) paid at inception of the contract.

The profit of a straddle is the sum of those of a call and a put.

### A Remark on the Code
- `Sₘ.>K` creates a vector of false/true
- `ifelse.(Sₘ.>K,"yes","no")` creates a vector of "yes" or "no" depending on whether `Sₘ.>K` or not.

```julia
Sₘ = [4.5,5.5]         #possible values of underlying at expiration
K  = 5                 #strike price
C  = 0.4               #call price (just a number that I made up)
P  = 0.4               #put price
(y,m) = (0,1)          #zero interest to keep it simple, 1 year to expiration

CallPayoff = max.(0,Sₘ.-K)               #payoff at expiration
CallProfit = CallPayoff .- exp(m*y)*C    #profit at expiration

ExerciseIt  = ifelse.(Sₘ.>K,"yes","no")  #"no"/"yes" for exercise

printblue("Payoff and profit of a call option with strike price $K and price (premium) of $C:\n")
printmat([Sₘ ExerciseIt CallPayoff CallProfit],colNames=["Sₘ","Exercise","Payoff","Profit"])
```

    Payoff and profit of a call option with strike price 5 and price (premium) of 0.4:

            Sₘ  Exercise    Payoff    Profit
         4.500        no     0.000    -0.400
         5.500       yes     0.500     0.100

```julia
Sₘ = 0:0.1:10          #more possible outcomes, for plotting

CallProfit = max.(0,Sₘ.-K) .- exp(m*y)*C
PutProfit  = max.(0,K.-Sₘ) .- exp(m*y)*P

p1 = plot( Sₘ,[CallProfit PutProfit],
           linecolor = [:red :green],
           linestyle = [:dash :dot],
           linewidth = 2,
           label = ["call" "put"],
           ylim = (-1,5),
           legend = :top,
           title = "Profits of call and put options, strike = $K",
           xlabel = "Asset price at expiration" )
display(p1)
```

```julia
StraddleProfit = CallProfit + PutProfit   #a straddle: 1 call and 1 put

p1 = plot( Sₘ,StraddleProfit,
           linecolor = :blue,
           linewidth = 2,
           label = "call+put",
           ylim = (-1,5),
           legend = :top,
           title = "Profit of a straddle, strike = $K",
           xlabel = "Asset price at expiration" )
display(p1)
```

# Put-Call Parity for European Options

A no-arbitrage condition says that
$
C-P=e^{-my}(F-K)
$
must hold, where $F$ is the forward price. This is the *put-call parity*.

Also, when the underlying asset has no dividends (until expiration of the option), then the forward-spot parity says that $F=e^{my}S$.

```julia
(S,K,m,y) = (42,38,0.5,0.05)    #current price of underlying etc

C = 5.5                   #assume this is the price of a call option(K)

F = exp(m*y)*S            #forward-spot parity
P = C - exp(-m*y)*(F-K)

printblue("Put-Call parity when (C,S,y,m)=($C,$S,$y,$m):\n")

printmat([C,exp(-m*y),F-K,P],rowNames=["C","exp(-m*y)","F-K","P"])
```

    Put-Call parity when (C,S,y,m)=(5.5,42,0.05,0.5):

    C             5.500
    exp(-m*y)     0.975
    F-K           5.063
    P             0.562

# Pricing Bounds

The pricing bounds for (American and European) call options are

$\begin{align}
C  &  \leq e^{-my}F\\
C  &  \geq\max[0,e^{-my}(F-K)]
\end{align}$

```julia
(S,K,m,y) = (42,38,0.5,0.05)        #current price of underlying etc

F = exp(m*y)*S

C_Upper = exp(-m*y)*F
C_Lower = max.(0,exp(-m*y)*(F.-K))

printlnPs("Pricing bounds for European call option with strike $K",C_Lower,C_Upper)
```

    Pricing bounds for European call option with strike 38     4.938    42.000

```julia
K_range = 30:0.5:50             #pricing bounds for many strike prices
n = length(K_range)

C_Upper = exp(-m*y)*F
C_Lower = max.(0,exp(-m*y)*(F.-K_range))

P_Upper = exp(-m*y)*K_range
P_Lower = max.(0,exp(-m*y)*(K_range.-F));
```

```julia
p1 = plot( K_range,[C_Upper*ones(n) C_Lower],
           linecolor = [:green :blue],
           linewidth = 2,
           linestyle = [:dash :dot],
           label = [L"C \leq \exp(-my)F " L"C \geq \max[0,\exp(-my)(F-K)]"],
           ylim = (-1,S+1),
           legend = :right,
           title = "Price bounds on European call options",
           xlabel = "Strike price" )
display(p1)
```

The pricing bounds for (European) put options are

$\begin{align}
P_{E}  &  \leq e^{-my}K\\
e^{-my}(K-F)  &  \leq P_{E}
\end{align}$

```julia
p1 = plot( K_range,[P_Upper P_Lower],
           linecolor = [:green :blue],
           linewidth = 2,
           linestyle = [:dash :dot],
           label = [L" P \leq \exp(-my)K " L" P \geq \max[0,\exp(-my)(K-F)]  "],
           ylim = (-1,50),
           legend = :right,
           title = "Price bounds on European put options",
           xlabel = "Strike price" )
display(p1)
```

```julia

```


Euler Problem 37
================

The number 3797 has an interesting property. Being prime itself, it is possible to continuously remove digits from left to right, and remain prime at each stage: 3797, 797, 97, and 7. Similarly we can work from right to left: 3797, 379, 37, and 3.

Find the sum of the only eleven primes that are both truncatable from left to right and right to left.

NOTE: 2, 3, 5, and 7 are not considered to be truncatable primes.

```python
from sympy import isprime

def right_truncatable(p = 0):
    if p == 0:
        for p in [2, 3, 5, 7]:
            for q in right_truncatable(p):
                yield q
    elif isprime(p):
        yield p
        p *= 10
        for d in (1, 3, 7, 9):
            for q in right_truncatable(p+d):
                yield q

def left_truncatable(p):
    N = 10 ** (len(str(p))-1)
    while p > 9:
        p %= N
        if not isprime(p):
            return False
        N //= 10
    return True

print(sum([p for p in right_truncatable() if p > 9 and left_truncatable(p)]))
```

    748317

```python

```


# Code printers

The most basic form of code generation are the code printers. The convert SymPy expressions into the target language.

The most common languages are C, C++, Fortran, and Python, but over a dozen languages are supported. Here, we will quickly go over each supported language.

```python
from sympy import *
init_printing()
```

Let us use the function $$|\sin(x^2)|.$$

```python
x = symbols('x')
expr = abs(sin(x**2))
expr
```

```python
ccode(expr)
```

```python
fcode(expr)
```

```python
julia_code(expr)
```

```python
jscode(expr)
```

```python
mathematica_code(expr)
```

```python
octave_code(expr)
```

```python
from sympy.printing.rust import rust_code
rust_code(expr)
```

```python
rcode(expr)
```

```python
from sympy.printing.cxxcode import cxxcode
cxxcode(expr)
```

## Exercise: Codegen your own function

Come up with a symbolic expression and try generating code for it in each language. Note, some languages don't support everything. What works and what doesn't? What things are the same across languages and what things are different?

**Reminder: If you click a cell and press `b` it will add a new cell below it.**

```python
# Write your answer here

```

```python

```

```python

```

## Exercise: Plotting SymPy Functions with JavaScript

One use case that works nicely with the Jupyter notebook is plotting mathematical functions using JavaScript plotting libraries. There are a variety of plotting libraries available and the notebook makes it relatively easy to use. Here we will use [Chart.js](http://www.chartjs.org/) to plot functions of a single variable. We can use the `%%javascript` magic to type JavaScript directly into a notebook cell. In this cell we load in the Chart.js library:

```javascript
%%javascript
require.config({
  paths: {
      'chartjs': '//cdnjs.cloudflare.com/ajax/libs/Chart.js/2.6.0/Chart'
  }
});
```

We've also prepared some Javascript to do the plotting. This code will take two mathematical expressions written in Javascript and plot the functions.

```python
from scipy2017codegen.plotting import js_template
```

```python
print(js_template.format(top_function='***fill me in!***',
                         bottom_function='***fill me in!***',
                         chart_id='***fill me in!***'))
```

Now SymPy functions can be plotted by filling in the two missing expressions in the above code and then calling the `Javascript` display function on that code.

```python
from IPython.display import Javascript
```

```python
x = symbols('x')
f1 = sin(x)
f2 = cos(x)
```

```python
Javascript(js_template.format(top_function=jscode(f1),
                              bottom_function=jscode(f2),
                              chart_id='sincos'))
```

## Exercise: Batman!

Plot the equations below for `top` and `bottom`.

There are all kind of functions that can be plotted, but one particularly interesting set of functions are called the [Batman Equations](http://mathworld.wolfram.com/BatmanCurve.html). We've provided the piecewise versions of these functions written in SymPy below. Try plotting these with the JS plotter we've created.

```python
from scipy2017codegen.plotting import batman_equations
```

```python
top, bottom = batman_equations()
```

```python
top
```

```python
bottom
```

```python
# Write your answer here

```


# Bayesian Estimation of Orbital Scaling Parameters

## Introduction

These notes briefly outline a Bayesian approach to estimating the statistical distribution of the orbital scaling (or $\lambda$) parameters from NIST data and their associated experimental error bars. The atomic structure calculation can be viewed as a mapping $f$ from the true orbital scaling parameter $\lambda \in \mathbb{R}^m$ to a set of observable quantities of interest, $f(\lambda) \in \mathbb{R}^n$. The output of this mapping is measured experimentally, albeit with some error.

The simplest "observation model" is to assume that a _random_ error $E = [E_1,...,E_n]\in \mathbb{R}^n$ is added to the output $f(\lambda)$, yielding a _random_ observation

\begin{equation}
Y = f(\lambda) + E. \label{eq:observation_model}
\end{equation}

Other observation models are also possible.

Now, suppose we don't know the true value of $\lambda$, but that we have a vector  $\vec y_{\text{nist}} = [y_1, y_2, ..., y_n]^T$ of actual NIST measurements of the energies and A-values. Due to the random measurement noise $E$, we cannot be certain that the observed values are the actual outputs $f(\lambda)$. To reflect our uncertainty, we estimate the true $\lambda$ by a random quantity $\Lambda$ whose distribution is consistent with the observations $y_{\text{nist}}$ and the statistical distribution of the error $E$. More specifically, we seek to estimate the conditional density function $\pi_{\Lambda|y_{\text{nist}}}$ given the NIST observations. Bayes formula allows us to express this density in terms of the likelihood $\pi_{Y|\Lambda}$ and a prior density $\pi_\Lambda$:

\begin{equation}\label{eq:bayes_formula}
\pi_{\Lambda|Y}(\lambda|y_{\text{nist}}) = \frac{\pi_\Lambda(\lambda)\pi_{Y|\Lambda}(y_{\text{nist}}|\lambda)}{\pi_{Y}(y_{\text{nist}})},
\end{equation}

provided $\pi_{Y}(y_{\text{nist}})\neq 0$.

### The Likelihood Function

If we have a density function for the measurement noise $\pi_E(e)$, then our given observation model \eqref{eq:observation_model} makes it easy to determine the likelihood. Indeed,

$$
\pi_{Y|\Lambda}(y_{\text{nist}}|\lambda)= \mathbb{P}(Y= y_{\text{nist}}|\Lambda=\lambda) = \mathbb{P}(E = y_{\text{nist}}-f(\lambda)) = \pi_E(y_{\text{nist}}-f(\lambda))
$$

#### The Noise Density

We now turn to estimating $\pi_E$. To this end we make the following simplifying assumption - it can be relaxed in principle.

> _Assumption:_ The measurement errors of the various energies and A-values are statistically independent.

The above assumption allows us to write the joint density function $\pi_{E}$ of the absolute errors $E = [E_1,...,E_n]$ as the product of univariate densities, i.e.

$$
\pi_{E}(e) = \prod_{i=1}^n \pi_{E_i}(e_i), \ \text{for } e = [e_1,...,e_n] \in \mathbb{R}^n.
$$

Let $\vec \eta = [\eta_1, ..., \eta_n]$ be the relative errors for each quantity of interest, reported in the NIST database. We use these to specify the statistical distribution of the absolute errors.

##### Uniform Errors

If assume that the NIST errors are uniformly distributed, we can use $\eta_i$ to find the range. Specifically, let $a_i = y_i - \eta_i y_i$ and $b_i=y_i + \eta_i y_i$, so that

$$
\pi_{E_i}(e_i) = \left\{
\begin{array}{ll}
\frac{1}{b_i-a_i}, & \text{if } a_i \leq e_i \leq b_i \\
0, &\text{otherwise}\end{array} \right. .
$$

The joint distribution is therefore

$$
\pi_{E}(e) = \prod_{i=1}^n \pi_{E_i}(e_i) = \left\{
\begin{array}{ll}
\prod_{i=1}^n \frac{1}{b_i-a_i}, & \text{if } a_i \leq e_i \leq b_i \text{ for } i=1,...,n \\
0, &\text{otherwise}\end{array} \right. .
$$

while its logarithm,

$$
\ln \pi_E(e) = \left\{ \begin{array}{ll} -\sum_{i=1}^n \ln(b_i-a_i), & \text{ if } a_i \leq e_i \leq b_i \text{ for } i=1,...,n \\
-\infty, &\text{otherwise}\end{array} \right. .
$$

##### Gaussian Errors

We can also assume that the NIST errors are Gaussian, in which case $\eta_i$ can be used to determine the standard deviations. Specifically, if we assume that the error range $[y_i-\eta_i y_i, y_i + \eta_i y_i]$ represents a 99.7\% confidence interval (corresponding to 3 standard deviations $\sigma_i$), we can compute $\sigma_i=\frac{2}{3}\eta_iy_i$. The densities are then given by

$$
\pi_{E_i}(e_i) = \frac{1}{\sqrt{2\pi}\sigma_i} \exp\left(-\frac{e_i^2}{2\sigma_i^2}\right).
$$

The joint distribution is

$$
\pi_E(e) = \prod_{i=1}^n \pi_{E_i}(e_i) = \prod_{i=1}^n \frac{1}{\sqrt{2\pi}\sigma_i} \exp\left(-\frac{e_i^2}{2\sigma_i^2}\right) = (2\pi)^{-n/2} \left(\prod_{i=1}^n \frac{1}{\sigma_i}\right) \exp\left(-\frac{1}{2}\sum_{i=1}^n \frac{e_i^2}{\sigma_i^2}\right)
$$

and its logarithm is

$$
\ln \pi_E(e) = -\frac{n}{2}\ln(2\pi) - \sum_{i=1}^n \ln(\sigma_i) - \frac{1}{2}\sum_{i=1}^n \left(\frac{e_i}{\sigma_i}\right)^2
$$

### The emcee package

Once we have the prior density and the likelihood function, we can use these to generate samples from the posterior density. This is usually achieved by means of a Monte Carlo Markov Chain (MCMC), which generates a random process (a Markov process) whose stationary (long term) distribution equals the desired posterior. The software library [emcee](http://dfm.io/emcee/current/) implements one form of the MCMC algorithm.

We first import the library:

```python
import emcee
import numpy as np
```

To test this package, we use a very simple example. Let $\lambda = [\lambda_1, \lambda_2]$ be two input parameters and consider the mapping

$$
f(\lambda) = \left[\begin{array}{c} \lambda_1^2 + \lambda_2^2 \\ \lambda_1 + 3\lambda_2 \end{array} \right].
$$

Here we define the mapping in python.

```python
def f(lmd):
    """
    Forward Mapping

    Inputs:

        lmd: (2,) numpy array [lmd1, lmd2]

    Outputs:

        flmd: (2,) numpy array [lmd1^2 + lmd2^2, lmd1 + 3lmd2]
    """
    lmd1, lmd2 = lmd
    return np.array([lmd1**2 + lmd2**2, lmd1+3*lmd2])
```

Let's say the parameter's true value is $\lambda = [1,2]^T$. This results in an output $[5, 7]^T$, which we can use as our measurement. Suppose further that the measurements $Y_1, Y_2$ of the two output quantities of interest have associated errors, $E_i$, $i=1,2$ that are independent and normally distributed with $E_1 \sim N(0, 4)$ and $E_2 \sim N(0,1)$.

The `emcee` package requires logarithms of density functions as inputs. In our case, we can use the expresion for the Gaussian errors above to show that the likelihood function takes the form

$$
\pi_{Y|\Lambda}(y_{\text{obs}}|\lambda) = -\ln(2\pi) - \ln(4) -\ln(1) - \frac{1}{2} \left(\frac{5-f_1(\lambda)}{16}\right)^2 - \frac{1}{2}\left(\frac{7-f_2(\lambda)}{1}\right)^2
$$

This is how we define the log likelihood in Python.

```python
def ln_likelihood(lmd, y_obs, f):
    """
    Returns the log-likelihood function for the given observations and forward mapping

    Inputs:

        lmd: (2,) numpy array - input parameter

        y_obs: (2, ) numpy array - measurements

        f: function, mapping from lambdas to output

    Output: double, log(P(y_obs|lmd))
    """
    f1, f2 = f(lmd)
    return -np.log(8*np.pi) - 0.5*( (y_obs[0]-f1)**2/16 + (y_obs[1]-f2)**2/1 )
```

Let's specify the prior density for $\lambda$. We use a uniform prior for each component, with $\lambda_1 \sim U([0,3])$ and $\lambda_2 \sim U([1.5,2.5])$. The joint log-prior then takes the form

$$
\ln \pi_\Lambda(\lambda) = \left\{ \begin{array}{ll} -\ln(3), & \text{ if } \lambda \in [0,3]\times [1,3]\\
-\infty, & \text{otherwise} \end{array} \right.
$$

In Python it looks like this:

```python
def ln_prior(lmd):
    """
    Compute the log prior density function at a parameter value lmd
    """
    lmd1, lmd2 = lmd
    if 0<=lmd1 and lmd1<=3 and 1.5<=lmd2 and lmd2<=2.5:
        return -np.log(3)
    else:
        return -np.infty

#
# Try it out
#
# Point inside region
lmd = np.array([0.5,2.5])
print(ln_prior(lmd))

# Point outside region
lmd = np.array([-1,2.4])
print(ln_prior(lmd))

```

    -1.0986122886681098
    -inf

Recall that the posterior density is (up to a scaling constant) the product of the prior and the likelihood, so that the log posterior is the sum of their logs

```python
def ln_posterior(lmd, y_obs, f):
    """
    Evaluate the log-posterior density function at a given lmd

    Inputs:

        lmd: input variable

        y_obs: observed output

        f: function, forward mapping

    Output:

        ln_prior + ln_likelihood
    """
    return ln_prior(lmd) + ln_likelihood(lmd, y_obs, f)
```

We now run the MCMC algorithm

```python
# Specify the observation vector
y_obs = np.array([5,7])

# Specify the dimension of the input space and the number of starting points
n_dim, n_walkers = 2, 100

# Specify starting points for each Markov chain (in a tight ball around optimum)
pos = [np.array([1,2]) + 1e-4*np.random.randn(n_dim) for i in range(n_walkers)]

# Initialize the sampler
sampler = emcee.EnsembleSampler(n_walkers, n_dim, ln_posterior, args=(y_obs, f))

# Run the MCMC routine
sampler.run_mcmc(pos, 1000);

# The sampler.chain has shape (n_walkers, n_steps, n_dim) = (100, 1000, 2)
# Reshape into a (100'000,2) array of samples (but throw away initial sample -> (95'000,2)
samples = sampler.chain[:, 50:, :].reshape((-1, n_dim))
```

We plot the results using the package ```corner.py```

```python
import corner
```

```python
#
# Plot samples
#
corner.corner(samples, labels=["$\lambda_1$", "$\lambda_2$"], truths=[1, 2]);
```

## O6+

Here we try to estimate the $\lambda$ parameters for OVI+, in particular those corresponding to the 1s, 2s and 2p orbitals, i.e. $\lambda = [\lambda_{1s}, \lambda_{2s}, \lambda_{2p}]$. Our estimation is based on observations of various computable energies and/or A-values.

### Automating the Structure Calculations

The computation of each observable quantity for a single $\lambda$-value is achieved by means of an R-matrix structure calculation, encoded in the Perl script ```adas803.testern.pl```. Its inputs include, among others, an ```input.dat```  file in which the $\lambda$-values are stored.

To facilitate the evaluation of multiple such structure calculations corresponding to different $\lambda$-values, we have written a Python class ```LambdaPdf```, contained in the module ```scaling_parameters.py```. This class allows users to

- Run the structure calculations for a specific the $\lambda$-value, as well as specific observable quantities to be extracted from the ```adf04ic``` output file.

- Generate a linear interpolant, by running the structure calculations for every $\lambda$-point in a pre-specified grid, recording the resulting quantities of interest, and fitting a piecewise linear function to the data. The interpolant can be used as an approximation for the exact stucture calculation, leading to much faster sampling rates.

- Ultimately, this class is designed to store the statistical distribution of the lambda parameters.

### The NIST Data

The NIST data used for this calibration consists of energies 2-7 (to be found [here](https://physics.nist.gov/cgi-bin/ASD/energy1.pl)) and A-values 2,5, and 7 (to be found [here](https://physics.nist.gov/cgi-bin/ASD/lines1.pl?spectra=O6&limits_type=0&low_w=&upp_w=&unit=1&submit=Retrieve+Data&de=0&format=0&line_out=0&en_unit=0&output=0&bibrefs=1&page_size=15&show_obs_wl=1&show_calc_wl=1&unc_out=1&order_out=0&max_low_enrg=&show_av=2&max_upp_enrg=&tsb_value=0&min_str=&A_out=0&intens_out=on&max_str=&allowed_out=1&forbid_out=1&min_accur=&min_intens=&conf_out=on&term_out=on&enrg_out=on&J_out=on)). The NIST database entries for the energy levels list no error bounds. In our computations, we make them up by setting them to 'AAA'.

The NIST error ratings correspond to the following relative errors

| Rating | Relative Error |
| ---| ---|
|'AAA'| 0.003|
|'AA' | 0.01 |
|'A+' | 0.02 |
|'A'  | 0.03 |
|'B+' | 0.07 |
|'B'  | 0.1  |
|'C+' | 0.18 |
|'C'  | 0.25 |
|'D+' | 0.4  |
|'D'  | 0.5  |
|'E'  | 0.5  |

In the code, each quantity of interest is initiated as a ```Qoi``` object, a class that store the NIST values, NIST errors, a search label for locating it in the output file, etc.

> __Example__ Here we initialize the A-value 2.

```python
#
# Import modules
#
import sys
sys.path.append('../src/')
from scaling_parameters import Qoi, LmdPdf
```

```python
#
# Initialize Quantity of Interest
#
a2 = Qoi(category='A-value', tag=2,
         search_label='   2   1',
         nist_value=1.04e3, nist_rating='AA')
```

> __Example:__ We initialize a new $\lambda$-object with one output quantity: ```a2```.

```python
#
# Initialize lmd object
#
tags = ['1s', '2s', '2p']  # names
rng = np.array([[0.8, 1.2], [0.8, 1.2], [0.8, 1.2]])  # lower and upper bounds for each lambda
resolution = (2,2,2)  # resolution of the grid in each direction
path_to_input = '/home/hans-werner/Dropbox/work/projects'+\
                '/atomic_data_uncertainty/code/icft/o_6/'
output_qois = [a2]
lmd = LmdPdf(tags, rng, resolution, path_to_input, output_qois)
```

For the O6+ atom, all three $\lambda$-parameters are constrained within the interval $[0.8,1.2]$. We used a resolution of 20 sub-intervals in each direction, amounting to $20^3 = 8000$ forward calculations. We recorded this data and used it to construct a piecewise linear interpolant.

To prevent having to recompute the interpolant during every run, we use the ```pickle``` library, which allows us to store all sorts of objects. We load the saved $\lambda$-object as follows

```python
import pickle

with open('lmd_o6.pickle', 'rb') as f:
    lmd = pickle.load(f)
```

The forward mapping

```python
def f_o6(point, lmd, qoi_indices):
    """
    Returns the interpolated forward map

    Inputs:

        point: double, (3,) array of lambda values

        qoi_indices: int, n-list of indices of output quantities
            we want to use to estimate the lambda's

    Outputs:

        fi: double, (n,) array of outputs for the given input point
    """
    # Compute the output for each listed output quantity
    fv = [lmd.interpolants[i](point) for i in qoi_indices]

    # Turn it into an array
    fv = np.array(fv).ravel()

    return fv
```

```python
# Display the output quantity names
for i in range(9):
    print(lmd.qois[i].category, lmd.qois[i].tag, lmd.qois[i].nist_value,lmd.qois[i].nist_rating)
#
# Specify the indices of the desired output
#

# All
# qoi_indices = [i for i in range(9)]

# A-values
qoi_indices = [0,1,2,3,4,5]

#
# The observation vector
#
y_obs = np.array([lmd.qois[i].nist_value for i in qoi_indices])
```

    Energy 2 4524640.0 AAA
    Energy 3 4585620.0 AAA
    Energy 4 4585680.0 AAA
    Energy 5 4586230.0 AAA
    Energy 6 4588380.0 AAA
    Energy 7 4629201.0 AAA
    A-value 2 1040.0 AA
    A-value 5 331000.0 AA
    A-value 7 3309000000000.0 AA

```python
a2.nist_value
```

    1040.0

The Log likelihood

```python
def ln_likelihood_gauss(point, lmd, y_obs, f_o6, sgm, qoi_indices):
    """
    """
    fv = f_o6(point, lmd, qoi_indices)
    n = len(qoi_indices)
    ln_p = -0.5*n*np.log(2*np.pi) - np.sum(np.log(sgm))
    for i in range(n):
        ln_p -= 0.5*((y_obs[i]-fv[i])/sgm[i])**2
    return ln_p

#
# Get the standard deviations
#
sgm = []
rating_table = {'AAA': 0.003,
                'AA' : 0.01,
                'A+' : 0.02,
                'A'  : 0.03,
                'B+' : 0.07,
                'B'  : 0.1,
                'C+' : 0.18,
                'C'  : 0.25,
                'D+' : 0.4,
                'D'  : 0.5,
                'E'  : 0.5}
for i in qoi_indices:
    qoi = lmd.qois[i]
    rel_error = rating_table[qoi.nist_rating]
    # sgm.append(2/3*rel_error*qoi.nist_value)
    sgm.append(rel_error*qoi.nist_value)
sgm = np.array(sgm)

#
# Check
#
point = np.array([0.8,0.9,1.1])
ln_likelihood_gauss(point, lmd, y_obs, f_o6, sgm, qoi_indices)
```

    -63.3737760041114

```python
def ln_prior(point):
    """
    Compute the log prior density function at a parameter value lmd
    """
    x, y, z = point
    if 0.8<=x and x<=1.2 and 0.8<=y and y<=1.2 and 0.8<=z and z<=1.2:
        return -3*np.log(0.4)
    else:
        return -np.infty
```

```python
def ln_posterior(point, lmd, y_obs, f_o6, sgm, qoi_indices):
    """
    Evaluate the log-posterior density function at a given lmd

    Inputs:

        lmd: input variable

        y_obs: observed output

        f: function, forward mapping

    Output:

        ln_prior + ln_likelihood
    """
    lp = ln_prior(point)
    if not np.isfinite(lp):
        return -np.infty
    else:
        return lp + ln_likelihood_gauss(point, lmd, y_obs, f_o6, sgm, qoi_indices)
```

```python
# Specify the dimension of the input space and the number of starting points
n_dim, n_walkers = 3, 100

# Specify starting points for each Markov chain (in a tight ball around optimum)
pos = [np.array([1,1,1]) + 1e-4*np.random.randn(n_dim) for i in range(n_walkers)]

# Initialize the sampler
sampler = emcee.EnsembleSampler(n_walkers, n_dim, ln_posterior, args=(lmd, y_obs, f_o6, sgm, qoi_indices))

# Run the MCMC routine
sampler.run_mcmc(pos, 1000);

# The sampler.chain has shape (n_walkers, n_steps, n_dim) = (100, 1000, 2)
# Reshape into a (100'000,2) array of samples (but throw away initial sample -> (95'000,2)
samples = sampler.chain[:, 50:, :].reshape((-1, n_dim))
```

```python
#
# Plot samples
#
corner.corner(samples, labels=["$\lambda_{1s}$", "$\lambda_{2s}$", "$\lambda_{2p}$"]);
```

```python

```


# Whitening versus standardizing

```python
%matplotlib inline

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
```

First, just read in data, and take a peek. The data can be found on [GitHub](https://github.com/ppham27/MLaPP-solutions/blob/master/chap04/heightWeightData.txt).

```python
raw_data = pd.read_csv("heightWeightData.txt", header=None, names=["gender", "height", "weight"])
raw_data.info()
raw_data.head()
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 210 entries, 0 to 209
    Data columns (total 3 columns):
    gender    210 non-null int64
    height    210 non-null int64
    weight    210 non-null int64
    dtypes: int64(3)
    memory usage: 5.0 KB

      gender
      height
      weight

      0
      1
      67
      125

      1
      2
      68
      140

      2
      2
      67
      142

      3
      2
      60
      110

      4
      2
      64
      97

We're told that for gender, 1 is male, and 2 is female. Part (a) says to extract the height/weight data corresponding to the males. Then, we fit a 2d Gaussian to the male data, using the empirical mean and covariance. Then, we'll plot this data.

Let's extract the males first.

```python
male_data = raw_data[raw_data.gender == 1]
male_data.head()
```

      gender
      height
      weight

      0
      1
      67
      125

      6
      1
      79
      210

      8
      1
      71
      150

      11
      1
      71
      175

      12
      1
      72
      145

Next, we'll calculate the empirical mean and covariance.

```python
mu_male = male_data.mean(axis=0)[1:].as_matrix() # remove gender
male_mean_diff = male_data.iloc[:,1:].as_matrix() - mu_male
covariance_male = np.dot(male_mean_diff.T, male_mean_diff)/len(male_data)
print(mu_male)
print(covariance_male)
```

    [  71.65753425  175.61643836]
    [[    9.6498405     46.93713642]
     [   46.93713642  1035.277538  ]]

Let's plot the data now.

```python
plt.figure(figsize=(6,6))
plt.plot(male_data.height, male_data.weight, 'ko')
plt.axis([60,80,120,285])
plt.title('raw')
plt.xlabel('height')
plt.ylabel('weight')
plt.axes().set_aspect(0.2)
plt.grid(True)
plt.show()
```

Let $\mathbf{x} \sim \mathcal{N}\left(\boldsymbol\mu, \Sigma\right)$, where $\mathbf{x} \in \mathbb{R}^p$. We can write $\Sigma = SDS^\intercal$ by the spectral theorem, where the columns of $S$ are orthonormal eigenvectors, and $D$ is a diagonal matrix of eigenvectors, $\lambda_1, \lambda_2,\ldots,\lambda_n$.

$\mathbf{x}$ has probability mass function,

\begin{equation}
f(\mathbf{x}) = \frac{1}{(2\pi)^{p/2}\sqrt{\det{\Sigma}}}\exp\left(-\frac{1}{2}\left(\mathbf{x}-\boldsymbol\mu\right)^\intercal\Sigma^{-1}\left(\mathbf{x}-\boldsymbol\mu\right)\right).
\end{equation}

Note that $S^\intercal S = I$, and $S^{-1} = S^\intercal$. This implies that

\begin{equation}
\Sigma^{-1} = \left(SDS^\intercal\right)^{-1} = \left(S^\intercal\right)^{-1}D^{-1}S^{-1} = SD^{-1}S^\intercal.
\end{equation}

Let $\mathbf{y} = S^\intercal\left(\mathbf{x}-\boldsymbol\mu\right)$.

Then, we have that

\begin{align}
\left(\mathbf{x}-\boldsymbol\mu\right)^\intercal\Sigma^{-1}\left(\mathbf{x}-\boldsymbol\mu\right)
&= \left(\mathbf{x}-\boldsymbol\mu\right)^\intercal S D^{-1} S^\intercal\left(\mathbf{x}-\boldsymbol\mu\right) \\
&= \left(S^\intercal\left(\mathbf{x}-\boldsymbol\mu\right)\right)^\intercal  D^{-1} S^\intercal\left(\mathbf{x}-\boldsymbol\mu\right) \\
&= \mathbf{y}^\intercal D^{-1} \mathbf{y}.
\end{align}

Moreover, $\mathbf{x} = S\mathbf{y} + \boldsymbol\mu$, so $D\mathbf{x}(\mathbf{y}) = S$, and so $\det D\mathbf{x}(\mathbf{y}) = 1$. Changing variables, the probability density function for $\mathbf{y}$ is

\begin{equation}
f(\mathbf{y}) = \frac{1}{(2\pi)^{p/2}\sqrt{\det  D}}\exp\left(-\frac{1}{2}\mathbf{y}^\intercal D^{-1} \mathbf{y}\right)
= \frac{1}{(2\pi)^{p/2}\sqrt{\prod_{j=1}^p\lambda_j}}\exp\left(-\frac{1}{2}\sum_{j=1}^p \frac{y_j^2}{\lambda_j}\right) = \prod_{j=1}^p\frac{1}{\sqrt{2\pi\lambda_j}}\exp\left(-\frac{1}{2}\frac{y_j^2}{\lambda_j}\right).
\end{equation}

Thus, $\mathbf{y} \sim \mathcal{N}\left(\mathbf{0}, D\right)$, and the coordinates of $\mathbf{y}$ are independent, with $y_i \mathcal{N}\left(0, \lambda_j\right)$. Now, the level curves of $f$ correspond to the hyperellipsoids

\begin{equation}
\sum_{j=1}^p \frac{y_j^2}{\lambda_j} = \sum_{j=1}^p \left(\frac{y_j}{\sqrt{\lambda_j}}\right)^2 = c.
\end{equation}

$\frac{y_j}{\sqrt{\lambda_j}} \sim \mathcal{N}(0, 1)$, so $\sum_{j=1}^p \frac{y_j^2}{\lambda_j} \sim \chi^2_p$, and so a $95\%$ confidence region would be

\begin{equation}
\sum_{j=1}^p \frac{y_j^2}{\lambda_j} \leq F_{\chi^2_p}^{-1}(0.95),
\end{equation}

where $F$ is the cumulative distribution function. For our $p = 2$ case, $F_{\chi^2_p}^{-1}(0.95) \approx 5.991.$ Now $\mathbf{y}$ our coordinates with respect to a orthonormal eigenbasis. Since $\mathbf{x} = S\mathbf{y} + \boldsymbol\mu$, the confidence region is a rotated hyperellipsoid centered at $\boldsymbol\mu$ with semi-axes along the eigenvectors of $\Sigma$. Let's plot this hyperellipsoid along with the indexed data.

```python
def calculate_2d_gaussian_confidence_region(mu, Sigma, p = 0.95, points = 200):
    """
    Returns a points x 2 numpy.ndarray of the confidence region.

    Keyword arguments:
    mu -- mean
    Sigma -- covariance matrix
    p -- percent confidence
    points -- number of points to interpolate
    """
    assert(len(mu) == len(Sigma))
    assert(np.all(Sigma == Sigma.T))
    eigenvalues, S = np.linalg.eig(Sigma)
    S = S[:,eigenvalues.argsort()[::-1]]
    eigenvalues = eigenvalues[eigenvalues.argsort()[::-1]]
    theta = np.linspace(0, 2*np.pi, num = points)
    x = np.sqrt(eigenvalues[0]*stats.chi2.ppf(p, df=2))*np.cos(theta)
    y = np.sqrt(eigenvalues[1]*stats.chi2.ppf(p, df=2))*np.sin(theta)
    return np.dot(S, np.array([x,y])).T + mu
def plot_raw_males(ax=None):
    if ax == None:
        ax = plt.gca()
    gaussian_fit_male = calculate_2d_gaussian_confidence_region(mu_male, covariance_male, p = 0.95, points = 100)
    ax.axis([60,80,90,285])
    ax.set_title('raw')
    ax.set_xlabel('height')
    ax.set_ylabel('weight')
    for row in male_data.itertuples():
        ax.text(row.height, row.weight, row.Index, horizontalalignment='center', verticalalignment='center')
    ax.set_aspect(0.2)
    ax.plot(gaussian_fit_male[:,0], gaussian_fit_male[:,1], linewidth=3, color='red')
    ax.plot(mu_male[0], mu_male[1], 'rx', markersize=10, markeredgewidth=3)
    ax.grid(True)
plt.figure(figsize=(8,8))
plot_raw_males(plt.gca())
plt.show()
```

For part (b) says to do the same thing with standardized data.

```python
def standardize(x, mean, sd):
    """Standardizes assuming x is normally distributed."""
    return (x - mean)/sd
def plot_standardized_males(ax=None):
    if ax == None:
        ax = plt.gca()
    gaussian_fit_male = calculate_2d_gaussian_confidence_region(mu_male, covariance_male, p = 0.95, points = 100)
    ax.set_title('standardized')
    ax.set_xlabel('height')
    ax.set_ylabel('weight')
    ax.plot(standardize(male_data.height, mu_male[0], np.sqrt(covariance_male[0,0])),
            standardize(male_data.weight, mu_male[1], np.sqrt(covariance_male[1,1])),
            " ")
    for row in male_data.itertuples():
        ax.text(standardize(row.height, mu_male[0], np.sqrt(covariance_male[0,0])),
                standardize(row.weight, mu_male[1], np.sqrt(covariance_male[1,1])),
                row.Index, horizontalalignment='center', verticalalignment='center')
    ax.set_aspect('equal')
    ax.plot(standardize(gaussian_fit_male[:,0], mu_male[0], np.sqrt(covariance_male[0,0])),
            standardize(gaussian_fit_male[:,1], mu_male[1], np.sqrt(covariance_male[1,1])),
            linewidth=3, color='red')
    ax.plot(0, 0, 'rx', markersize=10, markeredgewidth=3)
    ax.grid(True)
plt.figure(figsize=(8,8))
plot_standardized_males()
plt.show()
```

Part (c) deals with *whitening* or *sphereing* the data. This involves transforming the data so that the dimensions are uncorrelated and have equal variances along the axes. Recall that

\begin{equation}
\mathbf{y} = S^\intercal\left(\mathbf{x} - \boldsymbol\mu\right) \sim \mathcal{N}\left(\mathbf{0}, D\right),
\end{equation}

so this transformation accomplishes the tasks of making the dimensions uncorrelated. Now, to make the variances equal simply multiply by $\sqrt{D^{-1}}$, which is easy to compute since $D$ is diagonal and positive definite, so our transformation is

\begin{equation}
\mathbf{y}^\prime = \sqrt{D^{-1}}\mathbf{y} = \sqrt{D^{-1}}S^\intercal\left(\mathbf{x} - \boldsymbol\mu\right)
\sim \mathcal{N}\left(\mathbf{0}, I\right).
\end{equation}

Let's plot this.

```python
def whiten(X, mu, Sigma):
    assert(len(mu) == len(Sigma))
    assert(np.all(Sigma == Sigma.T))
    eigenvalues, S = np.linalg.eig(Sigma)
    S = S[:,eigenvalues.argsort()[::-1]]
    eigenvalues = eigenvalues[eigenvalues.argsort()[::-1]]
    inverse_precision = np.diag(1/np.sqrt(eigenvalues))
    return np.dot(np.dot(X - mu, S), inverse_precision)
def plot_whitened_males(ax=None):
    if ax == None:
        ax = plt.gca()
    gaussian_fit_male = calculate_2d_gaussian_confidence_region(mu_male, covariance_male, p = 0.95, points = 100)
    whitened_gaussian_fit_male = whiten(gaussian_fit_male, mu_male, covariance_male)
    ax.set_title('whitened')
    ax.set_xlabel('height')
    ax.set_ylabel('weight')
    whitened_male_data = whiten(np.array([male_data.height, male_data.weight]).T, mu_male, covariance_male)
    ax.plot(whitened_male_data[:,0], whitened_male_data[:,1], " ")
    for i in range(len(whitened_male_data)):
        ax.text(whitened_male_data[i, 0], whitened_male_data[i, 1],
                male_data.index[i], horizontalalignment='center', verticalalignment='center')
    ax.set_aspect('equal')
    ax.plot(whitened_gaussian_fit_male[:,0], whitened_gaussian_fit_male[:,1],
            linewidth=3, color='red')
    ax.plot(0, 0, 'rx', markersize=10, markeredgewidth=3)
    ax.grid(True)
plt.figure(figsize=(8,8))
plot_whitened_males()
plt.show()
```

Now, we can plot all three figures together just like in the textbook.

```python
fig = plt.figure(figsize=(20,8))
ax1 = fig.add_subplot(1,3,1)
ax2 = fig.add_subplot(1,3,2)
ax3 = fig.add_subplot(1,3,3)
plot_raw_males(ax1)
plot_standardized_males(ax2)
plot_whitened_males(ax3)
```