# Session 2 - Training a Network w/ Tensorflow

Assignment: Teach a Deep Neural Network to Paint

Parag K. Mital

Creative Applications of Deep Learning w/ Tensorflow

Kadenze Academy

#CADL

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

# Learning Goals

* Learn how to create a Neural Network
* Learn to use a neural network to paint an image
* Apply creative thinking to the inputs, outputs, and definition of a network

# Outline

- [Assignment Synopsis](#assignment-synopsis)
- [Part One - Fully Connected Network](#part-one---fully-connected-network)
    - [Instructions](#instructions)
    - [Code](#code)
    - [Variable Scopes](#variable-scopes)
- [Part Two - Image Painting Network](#part-two---image-painting-network)
    - [Instructions](#instructions-1)
    - [Preparing the Data](#preparing-the-data)
    - [Cost Function](#cost-function)
    - [Explore](#explore)
    - [A Note on Crossvalidation](#a-note-on-crossvalidation)
- [Part Three - Learning More than One Image](#part-three---learning-more-than-one-image)
    - [Instructions](#instructions-2)
    - [Code](#code-1)
- [Part Four - Open Exploration \(Extra Credit\)](#part-four---open-exploration-extra-credit)
- [Assignment Submission](#assignment-submission)

This next section will just make sure you have the right version of python and the libraries that we'll be using.  Don't change the code here but make sure you "run" it (use "shift+enter")!

```python
# First check the Python version
import sys
if sys.version_info < (3,4):
    print('You are running an older version of Python!\n\n' \
          'You should consider updating to Python 3.4.0 or ' \
          'higher as the libraries built for this course ' \
          'have only been tested in Python 3.4 and higher.\n')
    print('Try installing the Python 3.5 version of anaconda '
          'and then restart `jupyter notebook`:\n' \
          'https://www.continuum.io/downloads\n\n')

# Now get necessary libraries
try:
    import os
    import numpy as np
    import matplotlib.pyplot as plt
    from skimage.transform import resize
    from skimage import data
    from scipy.misc import imresize
except ImportError:
    print('You are missing some packages! ' \
          'We will try installing them before continuing!')
    !pip install "numpy>=1.11.0" "matplotlib>=1.5.1" "scikit-image>=0.11.3" "scikit-learn>=0.17" "scipy>=0.17.0"
    import os
    import numpy as np
    import matplotlib.pyplot as plt
    from skimage.transform import resize
    from skimage import data
    from scipy.misc import imresize
    print('Done!')

# Import Tensorflow
try:
    import tensorflow as tf
except ImportError:
    print("You do not have tensorflow installed!")
    print("Follow the instructions on the following link")
    print("to install tensorflow before continuing:")
    print("")
    print("https://github.com/pkmital/CADL#installation-preliminaries")

# This cell includes the provided libraries from the zip file
# and a library for displaying images from ipython, which
# we will use to display the gif
try:
    from libs import utils, gif
    import IPython.display as ipyd
except ImportError:
    print("Make sure you have started notebook in the same directory" +
          " as the provided zip file which includes the 'libs' folder" +
          " and the file 'utils.py' inside of it.  You will NOT be able"
          " to complete this assignment unless you restart jupyter"
          " notebook inside the directory created by extracting"
          " the zip file or cloning the github repo.")

# We'll tell matplotlib to inline any drawn figures like so:
%matplotlib inline
plt.style.use('ggplot')
```

```python
# Bit of formatting because I don't like the default inline code style:
from IPython.core.display import HTML
HTML("""<style> .rendered_html code {
    padding: 2px 4px;
    color: #c7254e;
    background-color: #f9f2f4;
    border-radius: 4px;
} </style>""")
```

# Assignment Synopsis

In this assignment, we're going to create our first neural network capable of taking any two continuous values as inputs.  Those two values will go through a series of multiplications, additions, and nonlinearities, coming out of the network as 3 outputs.  Remember from the last homework, we used convolution to filter an image so that the representations in the image were accentuated.  We're not going to be using convolution w/ Neural Networks until the next session, but we're effectively doing the same thing here: using multiplications to accentuate the representations in our data, in order to minimize whatever our cost function is.  To find out what those multiplications need to be, we're going to use Gradient Descent and Backpropagation, which will take our cost, and find the appropriate updates to all the parameters in our network to best optimize the cost.  In the next session, we'll explore much bigger networks and convolution.  This "toy" network is really to help us get up and running with neural networks, and aid our exploration of the different components that make up a neural network.  You will be expected to explore manipulations of the neural networks in this notebook as much as possible to help aid your understanding of how they effect the final result.

We're going to build our first neural network to understand what color "to paint" given a location in an image, or the row, col of the image.  So in goes a row/col, and out goes a R/G/B.  In the next lesson, we'll learn what this network is really doing is performing regression.  For now, we'll focus on the creative applications of such a network to help us get a better understanding of the different components that make up the neural network.  You'll be asked to explore many of the different components of a neural network, including changing the inputs/outputs (i.e. the dataset), the number of layers, their activation functions, the cost functions, learning rate, and batch size.  You'll also explore a modification to this same network which takes a 3rd input: an index for an image.  This will let us try to learn multiple images at once, though with limited success.

We'll now dive right into creating deep neural networks, and I'm going to show you the math along the way.  Don't worry if a lot of it doesn't make sense, and it really takes a bit of practice before it starts to come together.

# Part One - Fully Connected Network

## Instructions
Create the operations necessary for connecting an input to a network, defined by a `tf.Placeholder`, to a series of fully connected, or linear, layers, using the formula:

$$\textbf{H} = \phi(\textbf{X}\textbf{W} + \textbf{b})$$

where $\textbf{H}$ is an output layer representing the "hidden" activations of a network, $\phi$ represents some nonlinearity, $\textbf{X}$ represents an input to that layer, $\textbf{W}$ is that layer's weight matrix, and $\textbf{b}$ is that layer's bias.

If you're thinking, what is going on?  Where did all that math come from?  Don't be afraid of it.  Once you learn how to "speak" the symbolic representation of the equation, it starts to get easier.  And once we put it into practice with some code, it should start to feel like there is some association with what is written in the equation, and what we've written in code.  Practice trying to say the equation in a meaningful way: "The output of a hidden layer is equal to some input multiplied by another matrix, adding some bias, and applying a non-linearity".  Or perhaps: "The hidden layer is equal to a nonlinearity applied to an input multiplied by a matrix and adding some bias".  Explore your own interpretations of the equation, or ways of describing it, and it starts to become much, much easier to apply the equation.

The first thing that happens in this equation is the input matrix $\textbf{X}$ is multiplied by another matrix, $\textbf{W}$.  This is the most complicated part of the equation.  It's performing matrix multiplication, as we've seen from last session, and is effectively scaling and rotating our input.  The bias $\textbf{b}$ allows for a global shift in the resulting values.  Finally, the nonlinearity of $\phi$ allows the input space to be nonlinearly warped, allowing it to express a lot more interesting distributions of data.  Have a look below at some common nonlinearities.  If you're unfamiliar with looking at graphs like this, it is common to read the horizontal axis as X, as the input, and the vertical axis as Y, as the output.

```python
xs = np.linspace(-6, 6, 100)
test = np.maximum(xs,0)

plt.plot(xs, np.maximum(xs, 0), label='relu')
plt.plot(xs, 1 / (1 + np.exp(-xs)), label='sigmoid')
plt.plot(xs, np.tanh(xs), label='tanh')
plt.xlabel('Input')
plt.xlim([-6, 6])
plt.ylabel('Output')
plt.ylim([-1.5, 1.5])
plt.title('Common Activation Functions/Nonlinearities')
plt.legend(loc='lower right')

test
```

Remember, having series of linear followed by nonlinear operations is what makes neural networks expressive.  By stacking a lot of "linear" + "nonlinear" operations in a series, we can create a deep neural network!  Have a look at the output ranges of the above nonlinearity when considering which nonlinearity seems most appropriate.  For instance, the `relu` is always above 0, but does not saturate at any value above 0, meaning it can be anything above 0.  That's unlike the `sigmoid` which does saturate at both 0 and 1, meaning its values for a single output neuron will always be between 0 and 1.  Similarly, the `tanh` saturates at -1 and 1.

Choosing between these is often a matter of trial and error.  Though you can make some insights depending on your normalization scheme.  For instance, if your output is expected to be in the range of 0 to 1, you may not want to use a `tanh` function, which ranges from -1 to 1, but likely would want to use a `sigmoid`.  Keep the ranges of these activation functions in mind when designing your network, especially the final output layer of your network.

## Code

In this section, we're going to work out how to represent a fully connected neural network with code.  First, create a 2D `tf.placeholder` called $\textbf{X}$ with `None` for the batch size and 2 features.  Make its `dtype` `tf.float32`.  Recall that we use the dimension of `None` for the batch size dimension to say that this dimension can be any number.  Here is the docstring for the `tf.placeholder` function, have a look at what args it takes:

Help on function placeholder in module `tensorflow.python.ops.array_ops`:

```python
placeholder(dtype, shape=None, name=None)
```

    Inserts a placeholder for a tensor that will be always fed.

    **Important**: This tensor will produce an error if evaluated. Its value must
    be fed using the `feed_dict` optional argument to `Session.run()`,
    `Tensor.eval()`, or `Operation.run()`.

    For example:

```python
x = tf.placeholder(tf.float32, shape=(1024, 1024))
y = tf.matmul(x, x)

with tf.Session() as sess:
  print(sess.run(y))  # ERROR: will fail because x was not fed.

  rand_array = np.random.rand(1024, 1024)
  print(sess.run(y, feed_dict={x: rand_array}))  # Will succeed.
```

    Args:
      dtype: The type of elements in the tensor to be fed.
      shape: The shape of the tensor to be fed (optional). If the shape is not
        specified, you can feed a tensor of any shape.
      name: A name for the operation (optional).

    Returns:
      A `Tensor` that may be used as a handle for feeding a value, but not
      evaluated directly.

TODO! COMPLETE THIS SECTION!

```python
# Create a placeholder with None x 2 dimensions of dtype tf.float32, and name it "X":
X = tf.placeholder(tf.float32, shape=(None, 2), name='X')
```

Now multiply the tensor using a new variable, $\textbf{W}$, which has 2 rows and 20 columns, so that when it is left mutiplied by $\textbf{X}$, the output of the multiplication is None x 20, giving you 20 output neurons.  Recall that the `tf.matmul` function takes two arguments, the left hand ($\textbf{X}$) and right hand side ($\textbf{W}$) of a matrix multiplication.

To create $\textbf{W}$, you will use `tf.get_variable` to create a matrix which is `2 x 20` in dimension.  Look up the docstrings of functions `tf.get_variable` and `tf.random_normal_initializer` to get familiar with these functions.  There are many options we will ignore for now.  Just be sure to set the `name`, `shape` (this is the one that has to be [2, 20]), `dtype` (i.e. tf.float32), and `initializer` (the `tf.random_normal_intializer` you should create) when creating your $\textbf{W}$ variable with `tf.get_variable(...)`.

For the random normal initializer, often the mean is set to 0, and the standard deviation is set based on the number of neurons.  But that really depends on the input and outputs of your network, how you've "normalized" your dataset, what your nonlinearity/activation function is, and what your expected range of inputs/outputs are.  Don't worry about the values for the initializer for now, as this part will take a bit more experimentation to understand better!

This part is to encourage you to learn how to look up the documentation on Tensorflow, ideally using `tf.get_variable?` in the notebook.  If you are really stuck, just scroll down a bit and I've shown you how to use it.

TODO! COMPLETE THIS SECTION!

```python
W = tf.get_variable(
    name='W',
    shape=[2, 20],
    dtype=tf.float32,
    initializer=tf.random_normal_initializer(mean=0.0, stddev=0.1))

h = tf.matmul(X, W)   # Missing variable b here
```

```python
# Test, document for tensofflow get_variable
tf.nn.bias_add?
```

And add to this result another new variable, $\textbf{b}$, which has [20] dimensions.  These values will be added to every output neuron after the multiplication above.  Instead of the `tf.random_normal_initializer` that you used for creating $\textbf{W}$, now use the `tf.constant_initializer`.  Often for bias, you'll set the constant bias initialization to 0 or 1.

TODO! COMPLETE THIS SECTION!

```python
b = tf.get_variable(
    name='b',
    shape=[20],
    dtype=tf.float32,
    initializer=tf.random_normal_initializer(mean=0.0, stddev=0.1))

h = tf.nn.bias_add(h, b)
```

So far we have done:
$$\textbf{X}\textbf{W} + \textbf{b}$$

Finally, apply a nonlinear activation to this output, such as  `tf.nn.relu`, to complete the equation:

$$\textbf{H} = \phi(\textbf{X}\textbf{W} + \textbf{b})$$

TODO! COMPLETE THIS SECTION!

```python
h = tf.nn.relu(h)
```

Now that we've done all of this work, let's stick it inside a function.  I've already done this for you and placed it inside the `utils` module under the function name `linear`.  We've already imported the `utils` module so we can call it like so, `utils.linear(...)`.  The docstring is copied below, and the code itself.  Note that this function is slightly different to the one in the lecture.  It does not require you to specify `n_input`, and the input `scope` is called `name`.  It also has a few more extras in there including automatically converting a 4-d input tensor to a 2-d tensor so that you can fully connect the layer with a matrix multiply (don't worry about what this means if it doesn't make sense!).

```python
utils.linear??
```

```python
def linear(x, n_output, name=None, activation=None, reuse=None):
    """Fully connected layer

    Parameters
    ----------
    x : tf.Tensor
        Input tensor to connect
    n_output : int
        Number of output neurons
    name : None, optional
        Scope to apply

    Returns
    -------
    op : tf.Tensor
        Output of fully connected layer.
    """
    if len(x.get_shape()) != 2:
        x = flatten(x, reuse=reuse)

    n_input = x.get_shape().as_list()[1]

    with tf.variable_scope(name or "fc", reuse=reuse):
        W = tf.get_variable(
            name='W',
            shape=[n_input, n_output],
            dtype=tf.float32,
            initializer=tf.contrib.layers.xavier_initializer())

        b = tf.get_variable(
            name='b',
            shape=[n_output],
            dtype=tf.float32,
            initializer=tf.constant_initializer(0.0))

        h = tf.nn.bias_add(
            name='h',
            value=tf.matmul(x, W),
            bias=b)

        if activation:
            h = activation(h)

        return h, W
```

## Variable Scopes

Note that since we are using `variable_scope` and explicitly telling the scope which name we would like, if there is *already* a variable created with the same name, then Tensorflow will raise an exception!  If this happens, you should consider one of three possible solutions:

1. If this happens while you are interactively editing a graph, you may need to reset the current graph:
```python
    tf.reset_default_graph()
```
You should really only have to use this if you are in an interactive console!  If you are creating Python scripts to run via command line, you should really be using solution 3 listed below, and be explicit with your graph contexts!
2. If this happens and you were not expecting any name conflicts, then perhaps you had a typo and created another layer with the same name!  That's a good reason to keep useful names for everything in your graph!
3. More likely, you should be using context managers when creating your graphs and running sessions.  This works like so:

    ```python
    g = tf.Graph()
    with tf.Session(graph=g) as sess:
        Y_pred, W = linear(X, 2, 3, activation=tf.nn.relu)
    ```

    or:

    ```python
    g = tf.Graph()
    with tf.Session(graph=g) as sess, g.as_default():
        Y_pred, W = linear(X, 2, 3, activation=tf.nn.relu)
    ```

You can now write the same process as the above steps by simply calling:

```python
h, W = utils.linear(
    x=X, n_output=20, name='linear', activation=tf.nn.relu)
```

# Part Two - Image Painting Network

## Instructions

Follow along the steps below, first setting up input and output data of the network, $\textbf{X}$ and $\textbf{Y}$. Then work through building the neural network which will try to compress the information in $\textbf{X}$ through a series of linear and non-linear functions so that whatever it is given as input, it minimized the error of its prediction, $\hat{\textbf{Y}}$, and the true output $\textbf{Y}$ through its training process.  You'll also create an animated GIF of the training which you'll need to submit for the homework!

Through this, we'll explore our first creative application: painting an image.  This network is just meant to demonstrate how easily networks can be scaled to more complicated tasks without much modification.  It is also meant to get you thinking about neural networks as building blocks that can be reconfigured, replaced, reorganized, and get you thinking about how the inputs and outputs can be anything you can imagine.

## Preparing the Data

We'll follow an example that Andrej Karpathy has done in his online demonstration of "image inpainting".  What we're going to do is teach the network to go from the location on an image frame to a particular color.  So given any position in an image, the network will need to learn what color to paint.  Let's first get an image that we'll try to teach a neural network to paint.

TODO! COMPLETE THIS SECTION!

```python
# First load an image
#img = utils.get_celeb_imgs()[0]
img = plt.imread('000026.jpg')

# Be careful with the size of your image.
# Try a fairly small image to begin with,
# then come back here and try larger sizes.
img = imresize(img, (100, 100))
plt.figure(figsize=(5, 5))
plt.imshow(img)

# Make sure you save this image as "reference.png"
# and include it in your zipped submission file
# so we can tell what image you are trying to paint!
plt.imsave(fname='reference.png', arr=img)
```

In the lecture, I showed how to aggregate the pixel locations and their colors using a loop over every pixel position.  I put that code into a function `split_image` below.  Feel free to experiment with other features for `xs` or `ys`.

```python
def split_image(img):
    # We'll first collect all the positions in the image in our list, xs
    xs = []

    # And the corresponding colors for each of these positions
    ys = []

    # Now loop over the image
    for row_i in range(img.shape[0]):
        for col_i in range(img.shape[1]):
            # And store the inputs
            xs.append([row_i, col_i])
            # And outputs that the network needs to learn to predict
            ys.append(img[row_i, col_i])

    # we'll convert our lists to arrays
    xs = np.array(xs)
    ys = np.array(ys)
    return xs, ys
```

Let's use this function to create the inputs (xs) and outputs (ys) to our network as the pixel locations (xs) and their colors (ys):

```python
xs, ys = split_image(img)

# and print the shapes
xs.shape, ys.shape
```

    ((10000, 2), (10000, 3))

Also remember, we should normalize our input values!

TODO! COMPLETE THIS SECTION!

```python
# Normalize the input (xs) using its mean and standard deviation
xs = (xs - np.mean(xs)) / np.std(xs)

# Just to make sure you have normalized it correctly:
print(np.min(xs), np.max(xs))
assert(np.min(xs) > -3.0 and np.max(xs) < 3.0)
```

    -1.71481604244 1.71481604244

Similarly for the output:

```python
print(np.min(ys), np.max(ys))
```

    2 255

We'll normalize the output using a simpler normalization method, since we know the values range from 0-255:

```python
ys = ys / 255.0
print(np.min(ys), np.max(ys))
```

    0.0078431372549 1.0

Scaling the image values like this has the advantage that it is still interpretable as an image, unlike if we have negative values.

What we're going to do is use regression to predict the value of a pixel given its (row, col) position.  So the input to our network is `X = (row, col)` value.  And the output of the network is `Y = (r, g, b)`.

We can get our original image back by reshaping the colors back into the original image shape.  This works because the `ys` are still in order:

```python
print(ys.shape)
plt.imshow(ys.reshape(img.shape))
```

But when we give inputs of (row, col) to our network, it won't know what order they are, because we will randomize them.  So it will have to *learn* what color value should be output for any given (row, col).

Create 2 placeholders of `dtype` `tf.float32`: one for the input of the network, a `None x 2` dimension placeholder called $\textbf{X}$, and another for the true output of the network, a `None x 3` dimension placeholder called $\textbf{Y}$.

TODO! COMPLETE THIS SECTION!

```python
# Let's reset the graph:
tf.reset_default_graph()

# Create a placeholder of None x 2 dimensions and dtype tf.float32
# This will be the input to the network which takes the row/col
X = tf.placeholder(tf.float32, shape=(None, 2), name='X')

# Create the placeholder, Y, with 3 output dimensions instead of 2.
# This will be the output of the network, the R, G, B values.
Y = tf.placeholder(tf.float32, shape=(None,3), name='Y')
```

Now create a deep neural network that takes your network input $\textbf{X}$ of 2 neurons, multiplies it by a linear and non-linear transformation which makes its shape [None, 20], meaning it will have 20 output neurons.  Then repeat the same process again to give you 20 neurons again, and then again and again until you've done 6 layers of 20 neurons.  Then finally one last layer which will output 3 neurons, your predicted output, which I've been denoting mathematically as $\hat{\textbf{Y}}$, for a total of 6 hidden layers, or 8 layers total including the input and output layers.  Mathematically, we'll be creating a deep neural network that looks just like the previous fully connected layer we've created, but with a few more connections. So recall the first layer's connection is:

\begin{align}
\textbf{H}_1=\phi(\textbf{X}\textbf{W}_1 + \textbf{b}_1) \\
\end{align}

So the next layer will take that output, and connect it up again:

\begin{align}
\textbf{H}_2=\phi(\textbf{H}_1\textbf{W}_2 + \textbf{b}_2) \\
\end{align}

And same for every other layer:

\begin{align}
\textbf{H}_3=\phi(\textbf{H}_2\textbf{W}_3 + \textbf{b}_3) \\
\textbf{H}_4=\phi(\textbf{H}_3\textbf{W}_4 + \textbf{b}_4) \\
\textbf{H}_5=\phi(\textbf{H}_4\textbf{W}_5 + \textbf{b}_5) \\
\textbf{H}_6=\phi(\textbf{H}_5\textbf{W}_6 + \textbf{b}_6) \\
\end{align}

Including the very last layer, which will be the prediction of the network:

\begin{align}
\hat{\textbf{Y}}=\phi(\textbf{H}_6\textbf{W}_7 + \textbf{b}_7)
\end{align}

Remember if you run into issues with variable scopes/names, that you cannot recreate a variable with the same name! Revisit the section on Variable Scopes if you get stuck with name issues.

TODO! COMPLETE THIS SECTION!

```python
tf.reset_default_graph()
```

```python
# We'll create 6 hidden layers.  Let's create a variable
# to say how many neurons we want for each of the layers
# (try 20 to begin with, then explore other values)
n_neurons = 20

# Create the first linear + nonlinear layer which will
# take the 2 input neurons and fully connects it to 20 neurons.
# Use the `utils.linear` function to do this just like before,
# but also remember to give names for each layer, such as
# "1", "2", ... "5", or "layer1", "layer2", ... "layer6".
h1, W1 = utils.linear(
         x=X, n_output=n_neurons, name='layer1', activation=tf.nn.relu)

# Create another one:
h2, W2 = utils.linear(
          x=h1, n_output=n_neurons, name='layer2', activation=tf.nn.relu)

# and four more (or replace all of this with a loop if you can!):
h3, W3 = utils.linear(
          x=h2, n_output=n_neurons, name='layer3', activation=tf.nn.relu)

h4, W4 = utils.linear(
          x=h3, n_output=n_neurons, name='layer4', activation=tf.nn.relu)

h5, W5 = utils.linear(
          x=h4, n_output=n_neurons, name='layer5', activation=tf.nn.relu)

h6, W6 = utils.linear(
          x=h5, n_output=n_neurons, name='layer6', activation=tf.nn.relu)

# Now, make one last layer to make sure your network has 3 outputs:
Y_pred, W7 = utils.linear(h6, 3, activation=None, name='pred')
```

```python
assert(X.get_shape().as_list() == [None, 2])
assert(Y_pred.get_shape().as_list() == [None, 3])
assert(Y.get_shape().as_list() == [None, 3])
```

## Cost Function

Now we're going to work on creating a `cost` function.  The cost should represent how much `error` there is in the network, and provide the optimizer this value to help it train the network's parameters using gradient descent and backpropagation.

Let's say our error is `E`, then the cost will be:

$$cost(\textbf{Y}, \hat{\textbf{Y}}) = \frac{1}{\text{B}} \displaystyle\sum\limits_{b=0}^{\text{B}} \textbf{E}_b
$$

where the error is measured as, e.g.:

$$\textbf{E} = \displaystyle\sum\limits_{c=0}^{\text{C}} (\textbf{Y}_{c} - \hat{\textbf{Y}}_{c})^2$$

Don't worry if this scares you.  This is mathematically expressing the same concept as: "the cost of an actual $\textbf{Y}$, and a predicted $\hat{\textbf{Y}}$ is equal to the mean across batches, of which there are $\text{B}$ total batches, of the sum of distances across $\text{C}$ color channels of every predicted output and true output".  Basically, we're trying to see on average, or at least within a single minibatches average, how wrong was our prediction?  We create a measure of error for every output feature by squaring the predicted output and the actual output it should have, i.e. the actual color value it should have output for a given input pixel position.  By squaring it, we penalize large distances, but not so much small distances.

Consider how the square function (i.e., $f(x) = x^2$) changes for a given error.  If our color values range between 0-255, then a typical amount of error would be between $0$ and $128^2$.  For example if my prediction was (120, 50, 167), and the color should have been (0, 100, 120), then the error for the Red channel is (120 - 0) or 120.  And the Green channel is (50 - 100) or -50, and for the Blue channel, (167 - 120) = 47.  When I square this result, I get: (120)^2, (-50)^2, and (47)^2.  I then add all of these and that is my error, $\textbf{E}$, for this one observation.  But I will have a few observations per minibatch.  So I add all the error in my batch together, then divide by the number of observations in the batch, essentially finding the mean error of my batch.

Let's try to see what the square in our measure of error is doing graphically.

```python
error = np.linspace(0.0, 128.0**2, 100)
loss = error**2.0
plt.plot(error, loss)
plt.xlabel('error')
plt.ylabel('loss')
```

This is known as the $l_2$ (pronounced el-two) loss.  It doesn't penalize small errors as much as it does large errors.  This is easier to see when we compare it with another common loss, the $l_1$ (el-one) loss.  It is linear in error, by taking the absolute value of the error.  We'll compare the $l_1$ loss with normalized values from $0$ to $1$.  So instead of having $0$ to $255$ for our RGB values, we'd have $0$ to $1$, simply by dividing our color values by $255.0$.

```python
error = np.linspace(0.0, 1.0, 100)
plt.plot(error, error**2, label='l_2 loss')
plt.plot(error, np.abs(error), label='l_1 loss')
plt.xlabel('error')
plt.ylabel('loss')
plt.legend(loc='lower right')
```

So unlike the $l_2$ loss, the $l_1$ loss is really quickly upset if there is *any* error at all: as soon as error moves away from $0.0$, to $0.1$, the $l_1$ loss is $0.1$.  But the $l_2$ loss is $0.1^2 = 0.01$.  Having a stronger penalty on smaller errors often leads to what the literature calls "sparse" solutions, since it favors activations that try to explain as much of the data as possible, rather than a lot of activations that do a sort of good job,  but when put together, do a great job of explaining the data.  Don't worry about what this means if you are more unfamiliar with Machine Learning.  There is a lot of literature surrounding each of these loss functions that we won't have time to get into, but look them up if they interest you.

During the lecture, we've seen how to create a cost function using Tensorflow.  To create a $l_2$ loss function, you can for instance use tensorflow's `tf.squared_difference` or for an $l_1$ loss function, `tf.abs`.  You'll need to refer to the `Y` and `Y_pred` variables only, and your resulting cost should be a single value.  Try creating the $l_1$ loss to begin with, and come back here after you have trained your network, to compare the performance with a $l_2$ loss.

The equation for computing cost I mentioned above is more succintly written as, for $l_2$ norm:

$$cost(\textbf{Y}, \hat{\textbf{Y}}) = \frac{1}{\text{B}} \displaystyle\sum\limits_{b=0}^{\text{B}} \displaystyle\sum\limits_{c=0}^{\text{C}} (\textbf{Y}_{c} - \hat{\textbf{Y}}_{c})^2$$

For $l_1$ norm, we'd have:

$$cost(\textbf{Y}, \hat{\textbf{Y}}) = \frac{1}{\text{B}} \displaystyle\sum\limits_{b=0}^{\text{B}} \displaystyle\sum\limits_{c=0}^{\text{C}} \text{abs}(\textbf{Y}_{c} - \hat{\textbf{Y}}_{c})$$

Remember, to understand this equation, try to say it out loud: the $cost$ given two variables, $\textbf{Y}$, the actual output we want the network to have, and $\hat{\textbf{Y}}$ the predicted output from the network, is equal to the mean across $\text{B}$ batches, of the sum of $\textbf{C}$ color channels distance between the actual and predicted outputs.  If you're still unsure, refer to the lecture where I've computed this, or scroll down a bit to where I've included the answer.

TODO! COMPLETE THIS SECTION!

```python
# first compute the error, the inner part of the summation.
# This should be the l1-norm or l2-norm of the distance
# between each color channel.
error = tf.squared_difference(Y_pred, Y)
assert(error.get_shape().as_list() == [None, 3])
```

TODO! COMPLETE THIS SECTION!

```python
# Now sum the error for each feature in Y.
# If Y is [Batch, Features], the sum should be [Batch]:
sum_error = tf.reduce_sum(error, 1)
assert(sum_error.get_shape().as_list() == [None])
```

TODO! COMPLETE THIS SECTION!

```python
# Finally, compute the cost, as the mean error of the batch.
# This should be a single value.
cost = tf.reduce_mean(sum_error)
assert(cost.get_shape().as_list() == [])
```

We now need an `optimizer` which will take our `cost` and a `learning_rate`, which says how far along the gradient to move.  This optimizer calculates all the gradients in our network with respect to the `cost` variable and updates all of the weights in our network using backpropagation.  We'll then create mini-batches of our training data and run the `optimizer` using a `session`.

TODO! COMPLETE THIS SECTION!

```python
# Refer to the help for the function GradientDescentOptimizer or AdamOptimizer
optimizer = tf.train.GradientDescentOptimizer(0.004).minimize(cost)

# Create parameters for the number of iterations to run for (< 100)
n_iterations = 2000

# And how much data is in each minibatch (< 500)
batch_size = 600

# Then create a session
sess = tf.Session()
```

We'll now train our network!  The code below should do this for you if you've setup everything else properly.  Please read through this and make sure you understand each step!  Note that this can take a VERY LONG time depending on the size of your image (make it < 100 x 100 pixels), the number of neurons per layer (e.g. < 30), the number of layers (e.g. < 8), and number of iterations (< 1000).  Welcome to Deep Learning :)

```python
# Initialize all your variables and run the operation with your session
sess.run(tf.global_variables_initializer())

# Optimize over a few iterations, each time following the gradient
# a little at a time
imgs = []
costs = []
gif_step = n_iterations // 10
step_i = 0

for it_i in range(n_iterations):

    # Get a random sampling of the dataset
    idxs = np.random.permutation(range(len(xs)))

    # The number of batches we have to iterate over
    n_batches = len(idxs) // batch_size

    # Now iterate over our stochastic minibatches:
    for batch_i in range(n_batches):

        # Get just minibatch amount of data
        idxs_i = idxs[batch_i * batch_size: (batch_i + 1) * batch_size]

        # And optimize, also returning the cost so we can monitor
        # how our optimization is doing.
        training_cost = sess.run(
            [cost, optimizer],
            feed_dict={X: xs[idxs_i], Y: ys[idxs_i]})[0]

    # Also, every 20 iterations, we'll draw the prediction of our
    # input xs, which should try to recreate our image!
    if (it_i + 1) % gif_step == 0:
        costs.append(training_cost / n_batches)
        ys_pred = Y_pred.eval(feed_dict={X: xs}, session=sess)
        img = np.clip(ys_pred.reshape(img.shape), 0, 1)
        imgs.append(img)
        # Plot the cost over time
        fig, ax = plt.subplots(1, 2)
        ax[0].plot(costs)
        ax[0].set_xlabel('Iteration')
        ax[0].set_ylabel('Cost')
        ax[1].imshow(img)
        fig.suptitle('Iteration {}'.format(it_i))
        plt.show()
```

```python
# Save the images as a GIF
_ = gif.build_gif(imgs, saveto='single.gif', show_gif=False)
```

Let's now display the GIF we've just created:

```python
ipyd.Image(url='single.gif?{}'.format(np.random.rand()),
           height=500, width=500)
```

## Explore

Go back over the previous cells and exploring changing different parameters of the network.  I would suggest first trying to change the `learning_rate` parameter to different values and see how the cost curve changes.  What do you notice?  Try exponents of $10$, e.g. $10^1$, $10^2$, $10^3$... and so on.  Also try changing the `batch_size`: $50, 100, 200, 500, ...$ How does it effect how the cost changes over time?

Be sure to explore other manipulations of the network, such as changing the loss function to $l_2$ or $l_1$.  How does it change the resulting learning?  Also try changing the activation functions, the number of layers/neurons, different optimizers, and anything else that you may think of, and try to get a basic understanding on this toy problem of how it effects the network's training.  Also try comparing creating a fairly shallow/wide net (e.g. 1-2 layers with many neurons, e.g. > 100), versus a deep/narrow net (e.g. 6-20 layers with fewer neurons, e.g. < 20).  What do you notice?

## A Note on Crossvalidation

The cost curve plotted above is only showing the cost for our "training" dataset.  Ideally, we should split our dataset into what are called "train", "validation", and "test" sets.  This is done by taking random subsets of the entire dataset.  For instance, we partition our dataset by saying we'll only use 80% of it for training, 10% for validation, and the last 10% for testing.  Then when training as above, you would only use the 80% of the data you had partitioned, and then monitor accuracy on both the data you have used to train, but also that new 10% of unseen validation data.  This gives you a sense of how "general" your network is.  If it is performing just as well on that 10% of data, then you know it is doing a good job.  Finally, once you are done training, you would test one last time on your "test" dataset.  Ideally, you'd do this a number of times, so that every part of the dataset had a chance to be the test set.  This would also give you a measure of the variance of the accuracy on the final test.  If it changes a lot, you know something is wrong.  If it remains fairly stable, then you know that it is a good representation of the model's accuracy on unseen data.

We didn't get a chance to cover this in class, as it is less useful for exploring creative applications, though it is very useful to know and to use in practice, as it avoids overfitting/overgeneralizing your network to all of the data.  Feel free to explore how to do this on the application above!

# Part Three - Learning More than One Image

## Instructions

We're now going to make use of our Dataset from Session 1 and apply what we've just learned to try and paint every single image in our dataset.  How would you guess is the best way to approach this?  We could for instance feed in every possible image by having multiple row, col -> r, g, b values.  So for any given row, col, we'd have 100 possible r, g, b values.  This likely won't work very well as there are many possible values a pixel could take, not just one.  What if we also tell the network *which* image's row and column we wanted painted?  We're going to try and see how that does.

You can execute all of the cells below unchanged to see how this works with the first 100 images of the celeb dataset.  But you should replace the images with your own dataset, and vary the parameters of the network to get the best results!

I've placed the same code for running the previous algorithm into two functions, `build_model` and `train`.  You can directly call the function `train` with a 4-d image shaped as N x H x W x C, and it will collect all of the points of every image and try to predict the output colors of those pixels, just like before.  The only difference now is that you are able to try this with a few images at a time.  There are a few ways we could have tried to handle multiple images.  The way I've shown in the `train` function is to include an additional input neuron for *which* image it is.  So as well as receiving the row and column, the network will also receive as input which image it is as a number.  This should help the network to better distinguish the patterns it uses, as it has knowledge that helps it separates its process based on which image is fed as input.

```python
def build_model(xs, ys, n_neurons, n_layers, activation_fn,
                final_activation_fn, cost_type):

    xs = np.asarray(xs)
    ys = np.asarray(ys)

    if xs.ndim != 2:
        raise ValueError(
            'xs should be a n_observates x n_features, ' +
            'or a 2-dimensional array.')
    if ys.ndim != 2:
        raise ValueError(
            'ys should be a n_observates x n_features, ' +
            'or a 2-dimensional array.')

    n_xs = xs.shape[1]
    n_ys = ys.shape[1]

    X = tf.placeholder(name='X', shape=[None, n_xs],
                       dtype=tf.float32)
    Y = tf.placeholder(name='Y', shape=[None, n_ys],
                       dtype=tf.float32)

    current_input = X
    for layer_i in range(n_layers):
        current_input = utils.linear(
            current_input, n_neurons,
            activation=activation_fn,
            name='layer{}'.format(layer_i))[0]

    Y_pred = utils.linear(
        current_input, n_ys,
        activation=final_activation_fn,
        name='pred')[0]

    if cost_type == 'l1_norm':
        cost = tf.reduce_mean(tf.reduce_sum(
                tf.abs(Y - Y_pred), 1))
    elif cost_type == 'l2_norm':
        cost = tf.reduce_mean(tf.reduce_sum(
                tf.squared_difference(Y, Y_pred), 1))
    else:
        raise ValueError(
            'Unknown cost_type: {}.  '.format(
            cost_type) + 'Use only "l1_norm" or "l2_norm"')

    return {'X': X, 'Y': Y, 'Y_pred': Y_pred, 'cost': cost}
```

```python
def train(imgs,
          learning_rate=0.0001,
          batch_size=200,
          n_iterations=10,
          gif_step=2,
          n_neurons=30,
          n_layers=10,
          activation_fn=tf.nn.relu,
          final_activation_fn=tf.nn.tanh,
          cost_type='l2_norm'):

    N, H, W, C = imgs.shape
    all_xs, all_ys = [], []
    for img_i, img in enumerate(imgs):
        xs, ys = split_image(img)
        all_xs.append(np.c_[xs, np.repeat(img_i, [xs.shape[0]])])
        all_ys.append(ys)
    xs = np.array(all_xs).reshape(-1, 3)
    xs = (xs - np.mean(xs, 0)) / np.std(xs, 0)
    ys = np.array(all_ys).reshape(-1, 3)
    ys = ys / 127.5 - 1

    g = tf.Graph()
    with tf.Session(graph=g) as sess:
        model = build_model(xs, ys, n_neurons, n_layers,
                            activation_fn, final_activation_fn,
                            cost_type)
        optimizer = tf.train.AdamOptimizer(
            learning_rate=learning_rate).minimize(model['cost'])
        sess.run(tf.global_variables_initializer())
        gifs = []
        costs = []
        step_i = 0
        for it_i in range(n_iterations):
            # Get a random sampling of the dataset
            idxs = np.random.permutation(range(len(xs)))

            # The number of batches we have to iterate over
            n_batches = len(idxs) // batch_size
            training_cost = 0

            # Now iterate over our stochastic minibatches:
            for batch_i in range(n_batches):

                # Get just minibatch amount of data
                idxs_i = idxs[batch_i * batch_size:
                              (batch_i + 1) * batch_size]

                # And optimize, also returning the cost so we can monitor
                # how our optimization is doing.
                cost = sess.run(
                    [model['cost'], optimizer],
                    feed_dict={model['X']: xs[idxs_i],
                               model['Y']: ys[idxs_i]})[0]
                training_cost += cost

            print('iteration {}/{}: cost {}'.format(
                    it_i + 1, n_iterations, training_cost / n_batches))

            # Also, every 20 iterations, we'll draw the prediction of our
            # input xs, which should try to recreate our image!
            if (it_i + 1) % gif_step == 0:
                costs.append(training_cost / n_batches)
                ys_pred = model['Y_pred'].eval(
                    feed_dict={model['X']: xs}, session=sess)
                img = ys_pred.reshape(imgs.shape)
                gifs.append(img)
        return gifs
```

## Code

Below, I've shown code for loading the first 100 celeb files.  Run through the next few cells to see how this works with the celeb dataset, and then come back here and replace the `imgs` variable with your own set of images.  For instance, you can try your entire sorted dataset from Session 1 as an N x H x W x C array.  Explore!

TODO! COMPLETE THIS SECTION!

```python
celeb_imgs = utils.get_celeb_imgs()
plt.figure(figsize=(10, 10))
plt.imshow(utils.montage(celeb_imgs).astype(np.uint8))
# It doesn't have to be 100 images, explore!
imgs = np.array(celeb_imgs).copy()
```

Explore changing the parameters of the `train` function and your own dataset of images.  Note, you do not have to use the dataset from the last assignment!  Explore different numbers of images, whatever you prefer.

TODO! COMPLETE THIS SECTION!

```python
# Change the parameters of the train function and
# explore changing the dataset
gifs = train(imgs=imgs)
```

Now we'll create a gif out of the training process.  Be sure to call this 'multiple.gif' for your homework submission:

```python
montage_gifs = [np.clip(utils.montage(
            (m * 127.5) + 127.5), 0, 255).astype(np.uint8)
                for m in gifs]
_ = gif.build_gif(montage_gifs, saveto='multiple.gif')
```

And show it in the notebook

```python
ipyd.Image(url='multiple.gif?{}'.format(np.random.rand()),
           height=500, width=500)
```

What we're seeing is the training process over time.  We feed in our `xs`, which consist of the pixel values of each of our 100 images, it goes through the neural network, and out come predicted color values for every possible input value.  We visualize it above as a gif by seeing how at each iteration the network has predicted the entire space of the inputs.  We can visualize just the last iteration as a "latent" space, going from the first image (the top left image in the montage), to the last image, (the bottom right image).

```python
final = gifs[-1]
final_gif = [np.clip(((m * 127.5) + 127.5), 0, 255).astype(np.uint8) for m in final]
gif.build_gif(final_gif, saveto='final.gif')
```

```python
ipyd.Image(url='final.gif?{}'.format(np.random.rand()),
           height=200, width=200)
```

# Part Four - Open Exploration (Extra Credit)

I now what you to explore what other possible manipulations of the network and/or dataset you could imagine.  Perhaps a process that does the reverse, tries to guess where a given color should be painted?  What if it was only taught a certain palette, and had to reason about other colors, how it would interpret those colors?  Or what if you fed it pixel locations that weren't part of the training set, or outside the frame of what it was trained on?  Or what happens with different activation functions, number of layers, increasing number of neurons or lesser number of neurons?  I leave any of these as an open exploration for you.

Try exploring this process with your own ideas, materials, and networks, and submit something you've created as a gif!   To aid exploration, be sure to scale the image down quite a bit or it will require a much larger machine, and much more time to train.  Then whenever you think you may be happy with the process you've created, try scaling up the resolution and leave the training to happen over a few hours/overnight to produce something truly stunning!

Make sure to name the result of your gif: "explore.gif", and be sure to include it in your zip file.

TODO! COMPLETE THIS SECTION!

```python
# Train a network to produce something, storing every few
# iterations in the variable gifs, then export the training
# over time as a gif.
...

gif.build_gif(montage_gifs, saveto='explore.gif')
```

```python
ipyd.Image(url='explore.gif?{}'.format(np.random.rand()),
           height=500, width=500)
```

# Assignment Submission

After you've completed the notebook, create a zip file of the current directory using the code below.  This code will make sure you have included this completed ipython notebook and the following files named exactly as:

    session-2/
      session-2.ipynb
      single.gif
      multiple.gif
      final.gif
      explore.gif*
      libs/
        utils.py

    * = optional/extra-credit

You'll then submit this zip file for your second assignment on Kadenze for "Assignment 2: Teach a Deep Neural Network to Paint"!  If you have any questions, remember to reach out on the forums and connect with your peers or with me.

To get assessed, you'll need to be a premium student!  This will allow you to build an online portfolio of all of your work and receive grades.  If you aren't already enrolled as a student, register now at http://www.kadenze.com/ and join the [#CADL](https://twitter.com/hashtag/CADL) community to see what your peers are doing! https://www.kadenze.com/courses/creative-applications-of-deep-learning-with-tensorflow/info

Also, if you share any of the GIFs on Facebook/Twitter/Instagram/etc..., be sure to use the #CADL hashtag so that other students can find your work!

```python
utils.build_submission('session-2.zip',
                       ('reference.png',
                        'single.gif',
                        'multiple.gif',
                        'final.gif',
                        'session-2.ipynb'),
                       ('explore.gif'))
```


```python
# %load imports.py
## Local packages:

%matplotlib inline
%load_ext autoreload
%autoreload 2
%config Completer.use_jedi = False  ## (To fix autocomplete)

## External packages:
import pandas as pd
pd.options.display.max_rows = 999
pd.options.display.max_columns = 999
pd.set_option("display.max_columns", None)
import numpy as np
np.set_printoptions(linewidth=150)

import numpy as np
import os
import matplotlib.pyplot as plt
#if os.name == 'nt':
#    plt.style.use('presentation.mplstyle')  # Windows

import plotly.express as px
import plotly.graph_objects as go

import seaborn as sns
import sympy as sp
from sympy.physics.mechanics import (dynamicsymbols, ReferenceFrame,
                                      Particle, Point)
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex
from src.substitute_dynamic_symbols import run, lambdify

import pyro

import sklearn
import pykalman
from statsmodels.sandbox.regression.predstd import wls_prediction_std
import statsmodels.api as sm

from scipy.integrate import solve_ivp

## Local packages:
from src.data import mdl

from src.symbols import *
from src.parameters import *
import src.symbols as symbols
from src import prime_system
from src.models import regression
from src.visualization.regression import show_pred
from src.visualization.plot import track_plot

## Load models:
# (Uncomment these for faster loading):
import src.models.vmm_nonlinear_EOM  as vmm
from src.models.vmm import ModelSimulator
from docs.book.example_1 import ship_parameters, df_parameters

if os.name == 'nt':
    plt.style.use('../docs/book/book.mplstyle')  # Windows

from src.visualization.plot import track_plot, plot
```

```python
from src.extended_kalman_vmm import ExtendedKalman
```

## Load test

```python
id=22773
df, units, meta_data = mdl.load(dir_path = '../data/raw', id=id)
df.index = df.index.total_seconds()
df.index-=df.index[0]
df['x0']-=df.iloc[0]['x0']
df['y0']-=df.iloc[0]['y0']
df['psi']-=df.iloc[0]['psi']

```

```python
fig,ax=plt.subplots()
fig.set_size_inches(10,10)
track_plot(df=df, lpp=meta_data.lpp, x_dataset='x0', y_dataset='y0',  psi_dataset='psi', beam=meta_data.beam, ax=ax);
```

## Filtering

```python
parameters = df_parameters["prime"].copy()
ek = ExtendedKalman(vmm=vmm,
                    parameters=parameters,
                    ship_parameters=ship_parameters)

```

```python
from numpy import cos as cos
from numpy import sin as sin
from src.data.lowpass_filter import lowpass_filter

df_lowpass = df.copy()
t = df_lowpass.index
ts = np.mean(np.diff(t))
fs = 1/ts

position_keys = ['x0','y0','psi']
for key in position_keys:
    df_lowpass[key] = lowpass_filter(data=df_lowpass[key], fs=fs, cutoff=1, order=1)

df_lowpass['x01d_gradient'] = x1d_ = np.gradient(df_lowpass['x0'], t)
df_lowpass['y01d_gradient'] = y1d_ = np.gradient(df_lowpass['y0'], t)
df_lowpass['r'] = r_ = np.gradient(df_lowpass['psi'], t)

psi_ = df_lowpass['psi']

df_lowpass['u'] = x1d_*cos(psi_) + y1d_*sin(psi_)
df_lowpass['v'] = -x1d_*sin(psi_) + y1d_*cos(psi_)

velocity_keys = ['u','v','r']
for key in velocity_keys:
    df_lowpass[key] = lowpass_filter(data=df_lowpass[key], fs=fs, cutoff=1, order=1)
```

```python
data = df.copy()
data['u'] = df_lowpass['u']
data['v'] = df_lowpass['v']
data['r'] = df_lowpass['r']
data=data.iloc[200:-100]
data.index-=data.index[0]

P_prd = np.diag([0.1, 0.1, np.deg2rad(0.01), 0.001, 0.001, np.deg2rad(0.001)])
Qd = np.diag([0.01, 0.01, np.deg2rad(0.1)])  #process variances: u,v,r

Cd = np.array([
    [1, 0, 0, 0, 0, 0],
    [0, 1, 0, 0, 0, 0],
    [0, 0, 1, 0, 0, 0],
])

E = np.array([
    [0,0,0],
    [0,0,0],
    [0,0,0],
    [1,0,0],
    [0,1,0],
    [0,0,1],
    ],
)

ys = data[['x0','y0','psi']].values
h_m = h_ = np.mean(np.diff(data.index))

x0 = np.concatenate((
    data.iloc[0][['x0','y0','psi']].values,
    data.iloc[0][['u','v','r']].values))

us = data['delta'].values

error_max_pos = 0.05
sigma_pos = error_max_pos/3
variance_pos = sigma_pos**2

error_max_psi = np.deg2rad(0.5)
sigma_psi = error_max_psi/3
variance_psi = sigma_psi**2

Rd = np.diag([variance_pos, variance_pos, variance_psi])

h = np.mean(np.diff(data.index))

time_steps = ek.filter(
        data=data, P_prd=P_prd, Qd=Qd, Rd=Rd, E=E, Cd=Cd
    )

x_hats = np.array([time_step["x_hat"].flatten() for time_step in time_steps]).T
time = np.array([time_step["time"] for time_step in time_steps]).T
Ks = np.array([time_step["K"] for time_step in time_steps]).T
variances = np.array([np.diagonal(time_step["P_hat"]) for time_step in time_steps]).T
stds = np.sqrt(variances)
```

```python
df_kalman = pd.DataFrame(data=x_hats.T, index=time, columns=['x0','y0','psi','u','v','r'])
df_kalman['delta'] = us

for key in ['u','v','r']:
    df_kalman[f'{key}1d'] = np.gradient(df_kalman[key], df_kalman.index)
```

```python
smooth_time_steps = ek.smoother()

## Post process rts smoother:
x_hats = np.array(
    [time_step["x_hat"].flatten() for time_step in smooth_time_steps]
).T
time = np.array([time_step["time"] for time_step in smooth_time_steps]).T
df_rts = pd.DataFrame(data=x_hats.T, index=time, columns=['x0','y0','psi','u','v','r'])
df_rts["delta"] = us

for key in ['u','v','r']:
    df_rts[f'{key}1d'] = np.gradient(df_rts[key], df_kalman.index)
```

```python
dataframes = {
    'Mesurement' : data,
    'Kalman filter' : df_kalman,
    'RTS': df_rts,
}

fig,ax=plt.subplots()
styles = {
    'Mesurement' : {
        'linestyle' : '',
        'marker' : '.',
        'ms' : 1,
        'zorder':-10,
    },

    'Kalman filter' : {
        'lw' : 2,
    },

}

for label,df_ in dataframes.items():
    track_plot(
            df=df_,
            lpp=ship_parameters["L"],
            beam=ship_parameters["B"],
            ax=ax,
            label=label,
            plot_boats=False,
            **styles.get(label,{})
        );
ax.legend()

plot(dataframes = dataframes,
     fig_size=(10,15),
     styles = ['r-','g-','b-'],
     keys=['x0','y0','psi','u','v','r','u1d','v1d','r1d']);
```

```python
ek.parameters
```

```python
P_prd = np.diag([0.1, 0.1, np.deg2rad(0.01), 0.01, 0.01, np.deg2rad(0.01)])
Qd = np.diag([0.001, 0.001, np.deg2rad(0.001)])  #process variances: u,v,r

error_max_pos = 0.1
sigma_pos = error_max_pos/3
variance_pos = sigma_pos**2

error_max_psi = np.deg2rad(2)
sigma_psi = error_max_psi/3
variance_psi = sigma_psi**2

Rd = np.diag([variance_pos, variance_pos, variance_psi])

ek.filter(
        data=data, P_prd=P_prd, Qd=Qd, Rd=Rd, E=E, Cd=Cd
    )
ek.smoother();

ek2 = ek.copy()
```

```python
ek2.parameters["Ydelta"] = 3*ek.parameters["Ydelta"]
ek2.parameters["Ndelta"] = 3*ek.parameters["Ndelta"]

ek2.filter(
        data=data, P_prd=P_prd, Qd=Qd, Rd=Rd, E=E, Cd=Cd
    )
ek2.smoother();
```

```python
dataframes = {
    'data': data,
    '1' : ek.df_smooth,
    '2' : ek2.df_smooth,
}
```

```python
plot(dataframes = dataframes,
     fig_size=(10,15),
     styles = ['r-','g-','b-'],
     keys=['x0','y0','psi','u','v','r','u1d','v1d','r1d']);
```

```python
dataframes = {
    'data' : data,
    '1' : ek.simulate(),
    '2' : ek2.simulate(),
}

plot(dataframes = dataframes,
     fig_size=(10,15),
     styles = ['r-','g-','b-'],
     keys=['x0','y0','psi','u','v','r']);
```

```python
ek.variance_smooth.mean()
```

```python
ek2.variance_smooth.mean()
```

```python
ek.simulation_error.abs().sum().sum()
```

```python
ek2.simulation_error.abs().sum().sum()
```

```python
from scipy.stats import multivariate_normal
```

```python
def calculate_likelihood(time_steps):

    likelihood=0
    for time_step in time_steps:

        mean=time_step['x_hat'].flatten()
        rv = multivariate_normal(mean=mean,
                             cov=time_step['P_hat'])

        likelihood+= rv.logpdf(mean)

    return likelihood
```

```python
calculate_likelihood(ek.time_steps_smooth)
```

```python
calculate_likelihood(ek2.time_steps_smooth)
```

```python
from scipy.optimize import minimize
```

```python
def fun(x):
    return x**2

minimize(fun=fun, x0=[3])
```

```python
ek2 = ek.copy()

def fun(x):

    ek2.parameters = pd.Series(data = x, index=ek.parameters.index)

    # E-step:
    ek2.filter(
        data=data, P_prd=P_prd, Qd=Qd, Rd=Rd, E=E, Cd=Cd
    )
    ek2.smoother();

    likelihood = calculate_likelihood(ek2.time_steps_smooth)
    return -likelihood

```

```python
res = minimize(fun=fun, x0=ek.parameters.values)
```

```python
xs = np.linspace(1,20,5)
likelihoods = np.zeros(len(xs))
for i, x in enumerate(xs):
    likelihoods[i] = -fun(x)

```

```python
fig,ax=plt.subplots()
ax.plot(xs, likelihoods)
```

```python
parameters = ek.parameters.copy()
```

```python

```


# ADM Quantities in terms of BSSN Quantities
## Author: Zach Etienne

[comment]: <> (Abstract: TODO)

**Notebook Status:**  Self-Validated

**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented. (TODO)**

### NRPy+ Source Code for this module: [ADM_in_terms_of_BSSN.py](../edit/BSSN/BSSN_in_terms_of_ADM.py)

## Introduction:
This module documents the conversion of ADM variables:

$$\left\{\gamma_{ij}, K_{ij}, \alpha, \beta^i\right\}$$

into BSSN variables

$$\left\{\bar{\gamma}_{i j},\bar{A}_{i j},\phi, K, \bar{\Lambda}^{i}, \alpha, \beta^i, B^i\right\},$$

in the desired curvilinear basis (given by `reference_metric::CoordSystem`). Then it rescales the resulting BSSNCurvilinear variables (as defined in [the covariant BSSN formulation tutorial](Tutorial-BSSN_formulation.ipynb)) into the form needed for solving Einstein's equations with the BSSN formulation:

$$\left\{h_{i j},a_{i j},\phi, K, \lambda^{i}, \alpha, \mathcal{V}^i, \mathcal{B}^i\right\}.$$

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules; set desired output BSSN Curvilinear coordinate system set to Spherical
1. [Step 2](#adm2bssn): Perform the ADM-to-BSSN conversion for 3-metric, extrinsic curvature, and gauge quantities
    1. [Step 2.a](#adm2bssn_gamma): Convert ADM $\gamma_{ij}$ to BSSN $\bar{\gamma}_{ij}$; rescale to get $h_{ij}$
    1. [Step 2.b](#admexcurv_convert): Convert the ADM extrinsic curvature $K_{ij}$ to BSSN $\bar{A}_{ij}$ and $K$; rescale to get $a_{ij}$, $K$.
    1. [Step 2.c](#lambda): Define $\bar{\Lambda}^i$
    1. [Step 2.d](#conformal): Define the conformal factor variable `cf`
1. [Step 3](#code_validation): Code Validation against `BSSN.BSSN_in_terms_of_ADM` NRPy+ module
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize core Python/NRPy+ modules \[Back to [top](#toc)\]
$$\label{initializenrpy}$$

```python
# Step 1: Import needed core NRPy+ modules
from outputC import *             # NRPy+: Core C code output module
import NRPy_param_funcs as par    # NRPy+: Parameter interface
import indexedexp as ixp          # NRPy+: Symbolic indexed expression (e.g., tensors, vectors, etc.) support
import reference_metric as rfm    # NRPy+: Reference metric support
import sys                        # Standard Python modules for multiplatform OS-level functions
import BSSN.BSSN_quantities as Bq # NRPy+: This module depends on the parameter EvolvedConformalFactor_cf,
                                  #        which is defined in BSSN.BSSN_quantities

# Step 1.a: Set DIM=3, as we're using a 3+1 decomposition of Einstein's equations
DIM=3
```

# Step 2: Perform the ADM-to-BSSN conversion for 3-metric, extrinsic curvature, and gauge quantities \[Back to [top](#toc)\]
$$\label{adm2bssn}$$

Here we convert ADM quantities to their BSSN Curvilinear counterparts.

## Step 2.a: Convert ADM $\gamma_{ij}$ to BSSN $\bar{\gamma}_{ij}$; rescale to get $h_{ij}$ \[Back to [top](#toc)\]
$$\label{adm2bssn_gamma}$$

We have (Eqs. 2 and 3 of [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):
$$
\bar{\gamma}_{i j} = \left(\frac{\bar{\gamma}}{\gamma}\right)^{1/3} \gamma_{ij},
$$
where we always make the choice $\bar{\gamma} = \hat{\gamma}$.

After constructing $\bar{\gamma}_{ij}$, we rescale to get $h_{ij}$ according to the prescription described in the [the covariant BSSN formulation tutorial](Tutorial-BSSN_formulation.ipynb) (also [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):

$$
h_{ij} = (\bar{\gamma}_{ij} - \hat{\gamma}_{ij})/\text{ReDD[i][j]}.
$$

```python
# Step 2: All ADM quantities were input into this function in the Spherical or Cartesian
#         basis, as functions of r,th,ph or x,y,z, respectively. In Steps 1 and 2 above,
#         we converted them to the xx0,xx1,xx2 basis, and as functions of xx0,xx1,xx2.
#         Here we convert ADM quantities to their BSSN Curvilinear counterparts:

# Step 2.a: Convert ADM $\gamma_{ij}$ to BSSN $\bar{gamma}_{ij}$:
#           We have (Eqs. 2 and 3 of [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):
def gammabarDD_hDD(gammaDD):
    global gammabarDD,hDD
    if rfm.have_already_called_reference_metric_function == False:
        print("BSSN.BSSN_in_terms_of_ADM.hDD_given_ADM(): Must call reference_metric() first!")
        sys.exit(1)
    # \bar{gamma}_{ij} = (\frac{\bar{gamma}}{gamma})^{1/3}*gamma_{ij}.
    gammaUU, gammaDET = ixp.symm_matrix_inverter3x3(gammaDD)
    gammabarDD = ixp.zerorank2()
    hDD        = ixp.zerorank2()
    for i in range(DIM):
        for j in range(DIM):
            gammabarDD[i][j] = (rfm.detgammahat/gammaDET)**(sp.Rational(1,3))*gammaDD[i][j]
            hDD[i][j] = (gammabarDD[i][j] - rfm.ghatDD[i][j]) / rfm.ReDD[i][j]
```

## Step 2.b: Convert the ADM extrinsic curvature $K_{ij}$ to BSSN quantities $\bar{A}_{ij}$ and $K={\rm tr}(K_{ij})$; rescale $\bar{A}_{ij}$ to get $a_{ij}$ \[Back to [top](#toc)\]
$$\label{admexcurv_convert}$$

Convert the ADM extrinsic curvature $K_{ij}$ to the trace-free extrinsic curvature $\bar{A}_{ij}$, plus the trace of the extrinsic curvature $K$, where (Eq. 3 of [Baumgarte *et al.*](https://arxiv.org/pdf/1211.6632.pdf)):
\begin{align}
K &= \gamma^{ij} K_{ij} \\
\bar{A}_{ij} &= \left(\frac{\bar{\gamma}}{\gamma}\right)^{1/3} \left(K_{ij} - \frac{1}{3} \gamma_{ij} K \right)
\end{align}

After constructing $\bar{A}_{ij}$, we rescale to get $a_{ij}$ according to the prescription described in the [the covariant BSSN formulation tutorial](Tutorial-BSSN_formulation.ipynb) (also [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):

$$
a_{ij} = \bar{A}_{ij}/\text{ReDD[i][j]}.
$$

```python
# Step 2.b: Convert the extrinsic curvature K_{ij} to the trace-free extrinsic
#           curvature \bar{A}_{ij}, plus the trace of the extrinsic curvature K,
#           where (Eq. 3 of [Baumgarte *et al.*](https://arxiv.org/pdf/1211.6632.pdf)):
def trK_AbarDD_aDD(gammaDD,KDD):
    global trK,AbarDD,aDD
    if rfm.have_already_called_reference_metric_function == False:
        print("BSSN.BSSN_in_terms_of_ADM.trK_AbarDD(): Must call reference_metric() first!")
        sys.exit(1)
    # \bar{gamma}_{ij} = (\frac{\bar{gamma}}{gamma})^{1/3}*gamma_{ij}.
    gammaUU, gammaDET = ixp.symm_matrix_inverter3x3(gammaDD)
    # K = gamma^{ij} K_{ij}, and
    # \bar{A}_{ij} &= (\frac{\bar{gamma}}{gamma})^{1/3}*(K_{ij} - \frac{1}{3}*gamma_{ij}*K)
    trK = sp.sympify(0)
    for i in range(DIM):
        for j in range(DIM):
            trK += gammaUU[i][j]*KDD[i][j]

    AbarDD = ixp.zerorank2()
    aDD    = ixp.zerorank2()
    for i in range(DIM):
        for j in range(DIM):
            AbarDD[i][j] = (rfm.detgammahat/gammaDET)**(sp.Rational(1,3))*(KDD[i][j] - sp.Rational(1,3)*gammaDD[i][j]*trK)
            aDD[i][j]    = AbarDD[i][j] / rfm.ReDD[i][j]
```

## Step 2.c: Assuming the ADM 3-metric $\gamma_{ij}$ is given as an explicit function of `(xx0,xx1,xx2)`, convert to BSSN $\bar{\Lambda}^i$; rescale to compute $\lambda^i$ \[Back to [top](#toc)\]
$$\label{lambda}$$

To define $\bar{\Lambda}^i$ we implement Eqs. 4 and 5 of [Baumgarte *et al.*](https://arxiv.org/pdf/1211.6632.pdf):
$$
\bar{\Lambda}^i = \bar{\gamma}^{jk}\left(\bar{\Gamma}^i_{jk} - \hat{\Gamma}^i_{jk}\right).
$$

The [reference_metric.py](../edit/reference_metric.py) module provides us with exact, closed-form expressions for $\hat{\Gamma}^i_{jk}$, so here we need only compute exact expressions for $\bar{\Gamma}^i_{jk}$, based on $\gamma_{ij}$ given as an explicit function of `(xx0,xx1,xx2)`. This is particularly useful when setting up initial data.

After constructing \bar{\Lambda}^i$, we rescale to get $\lambda^i$ according to the prescription described in the [the covariant BSSN formulation tutorial](Tutorial-BSSN_formulation.ipynb) (also [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):

$$
\lambda^i = \bar{\Lambda}^i/\text{ReU[i]}.
$$

```python
# Step 2.c: Define \bar{Lambda}^i (Eqs. 4 and 5 of [Baumgarte *et al.*](https://arxiv.org/pdf/1211.6632.pdf)):
def LambdabarU_lambdaU__exact_gammaDD(gammaDD):
    global LambdabarU,lambdaU

    # \bar{Lambda}^i = \bar{gamma}^{jk}(\bar{Gamma}^i_{jk} - \hat{Gamma}^i_{jk}).
    gammabarDD_hDD(gammaDD)
    gammabarUU, gammabarDET = ixp.symm_matrix_inverter3x3(gammabarDD)

    # First compute Christoffel symbols \bar{Gamma}^i_{jk}, with respect to barred metric:
    GammabarUDD = ixp.zerorank3()
    for i in range(DIM):
        for j in range(DIM):
            for k in range(DIM):
                for l in range(DIM):
                    GammabarUDD[i][j][k] += sp.Rational(1,2)*gammabarUU[i][l]*( sp.diff(gammabarDD[l][j],rfm.xx[k]) +
                                                                                sp.diff(gammabarDD[l][k],rfm.xx[j]) -
                                                                                sp.diff(gammabarDD[j][k],rfm.xx[l]) )
    # Next evaluate \bar{Lambda}^i, based on GammabarUDD above and GammahatUDD
    #       (from the reference metric):
    LambdabarU = ixp.zerorank1()
    for i in range(DIM):
        for j in range(DIM):
            for k in range(DIM):
                LambdabarU[i] += gammabarUU[j][k] * (GammabarUDD[i][j][k] - rfm.GammahatUDD[i][j][k])
    lambdaU    = ixp.zerorank1()
    for i in range(DIM):
        lambdaU[i] = LambdabarU[i] / rfm.ReU[i]
```

## Step 2.d: Define the conformal factor variable `cf` \[Back to [top](#toc)\]
$$\label{conformal}$$

We define the conformal factor variable `cf` based on the setting of the `"BSSN_quantities::EvolvedConformalFactor_cf"` parameter.

For example if `"BSSN_quantities::EvolvedConformalFactor_cf"` is set to `"phi"`, we can use Eq. 3 of [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf), which in arbitrary coordinates is written:

$$
\phi = \frac{1}{12} \log\left(\frac{\gamma}{\bar{\gamma}}\right).
$$

Alternatively if `"BSSN_quantities::EvolvedConformalFactor_cf"` is set to `"chi"`, then
$$
\chi = e^{-4 \phi} = \exp\left(-4 \frac{1}{12} \left(\frac{\gamma}{\bar{\gamma}}\right)\right)
= \exp\left(-\frac{1}{3} \log\left(\frac{\gamma}{\bar{\gamma}}\right)\right) = \left(\frac{\gamma}{\bar{\gamma}}\right)^{-1/3}.
$$

Finally if `"BSSN_quantities::EvolvedConformalFactor_cf"` is set to `"W"`, then
$$
W = e^{-2 \phi} = \exp\left(-2 \frac{1}{12} \log\left(\frac{\gamma}{\bar{\gamma}}\right)\right) =
\exp\left(-\frac{1}{6} \log\left(\frac{\gamma}{\bar{\gamma}}\right)\right) =
\left(\frac{\gamma}{\bar{\gamma}}\right)^{-1/6}.
$$

```python
# Step 2.d: Set the conformal factor variable cf, which is set
#           by the "BSSN_quantities::EvolvedConformalFactor_cf" parameter. For example if
#           "EvolvedConformalFactor_cf" is set to "phi", we can use Eq. 3 of
#           [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf),
#           which in arbitrary coordinates is written:
def cf_from_gammaDD(gammaDD):
    global cf

    # \bar{Lambda}^i = \bar{gamma}^{jk}(\bar{Gamma}^i_{jk} - \hat{Gamma}^i_{jk}).
    gammabarDD_hDD(gammaDD)
    gammabarUU, gammabarDET = ixp.symm_matrix_inverter3x3(gammabarDD)
    gammaUU, gammaDET       = ixp.symm_matrix_inverter3x3(gammaDD)

    cf = sp.sympify(0)

    if par.parval_from_str("EvolvedConformalFactor_cf") == "phi":
        # phi = \frac{1}{12} log(\frac{gamma}{\bar{gamma}}).
        cf = sp.Rational(1,12)*sp.log(gammaDET/gammabarDET)
    elif par.parval_from_str("EvolvedConformalFactor_cf") == "chi":
        # chi = exp(-4*phi) = exp(-4*\frac{1}{12}*(\frac{gamma}{\bar{gamma}}))
        #      = exp(-\frac{1}{3}*log(\frac{gamma}{\bar{gamma}})) = (\frac{gamma}{\bar{gamma}})^{-1/3}.
        #
        cf = (gammaDET/gammabarDET)**(-sp.Rational(1,3))
    elif par.parval_from_str("EvolvedConformalFactor_cf") == "W":
        # W = exp(-2*phi) = exp(-2*\frac{1}{12}*log(\frac{gamma}{\bar{gamma}}))
        #   = exp(-\frac{1}{6}*log(\frac{gamma}{\bar{gamma}})) = (\frac{gamma}{bar{gamma}})^{-1/6}.
        cf = (gammaDET/gammabarDET)**(-sp.Rational(1,6))
    else:
        print("Error EvolvedConformalFactor_cf type = \""+par.parval_from_str("EvolvedConformalFactor_cf")+"\" unknown.")
        sys.exit(1)
```

## Step 2.e: Rescale $\beta^i$ and $B^i$ to compute $\mathcal{V}^i={\rm vet}^i$ and $\mathcal{B}^i={\rm bet}^i$, respectively \[Back to [top](#toc)\]
$$\label{betvet}$$

We rescale $\beta^i$ and $B^i$ according to the prescription described in the [the covariant BSSN formulation tutorial](Tutorial-BSSN_formulation.ipynb) (also [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):
\begin{align}
\mathcal{V}^i &= \beta^i/\text{ReU[i]}\\
\mathcal{B}^i &= B^i/\text{ReU[i]}.
\end{align}

```python
# Step 2.e: Rescale beta^i and B^i according to the prescription described in
#         the [BSSN in curvilinear coordinates tutorial notebook](Tutorial-BSSNCurvilinear.ipynb)
#         (also [Ruchlin *et al.*](https://arxiv.org/pdf/1712.07658.pdf)):
#
# \mathcal{V}^i &= beta^i/(ReU[i])
# \mathcal{B}^i &= B^i/(ReU[i])
def betU_vetU(betaU,BU):
    global vetU,betU

    if rfm.have_already_called_reference_metric_function == False:
        print("BSSN.BSSN_in_terms_of_ADM.bet_vet(): Must call reference_metric() first!")
        sys.exit(1)
    vetU    = ixp.zerorank1()
    betU    = ixp.zerorank1()
    for i in range(DIM):
        vetU[i]    =      betaU[i] / rfm.ReU[i]
        betU[i]    =         BU[i] / rfm.ReU[i]
```

# Step 3: Code Validation against `BSSN.BSSN_in_terms_of_ADM` module \[Back to [top](#toc)\]
$$\label{code_validation}$$

Here, as a code validation check, we verify agreement in the SymPy expressions for [UIUC initial data](Tutorial-ADM_Initial_Data-UIUC_BlackHole.ipynb) between
1. this tutorial and
2. the NRPy+ [BSSN.BSSN_in_terms_of_ADM](../edit/BSSN/BSSN_in_terms_of_ADM.py) module.

As no basis transformation is performed, we analyze these expressions in their native, Spherical coordinates.

```python
# Step 3.a: Set the desired *output* coordinate system to Spherical:
par.set_parval_from_str("reference_metric::CoordSystem","Spherical")
rfm.reference_metric()

# Step 3.b: Set up initial data; assume UIUC spinning black hole initial data
import BSSN.UIUCBlackHole as uibh
uibh.UIUCBlackHole(ComputeADMGlobalsOnly=True)

# Step 3.c: Call above functions to convert ADM to BSSN curvilinear
gammabarDD_hDD(                   uibh.gammaSphDD)
trK_AbarDD_aDD(                   uibh.gammaSphDD,uibh.KSphDD)
LambdabarU_lambdaU__exact_gammaDD(uibh.gammaSphDD)
cf_from_gammaDD(                  uibh.gammaSphDD)
betU_vetU(                        uibh.betaSphU,uibh.BSphU)

# Step 3.d: Now load the BSSN_in_terms_of_ADM module and perform the same conversion
import BSSN.BSSN_in_terms_of_ADM as BitoA
BitoA.gammabarDD_hDD(                   uibh.gammaSphDD)
BitoA.trK_AbarDD_aDD(                   uibh.gammaSphDD,uibh.KSphDD)
BitoA.LambdabarU_lambdaU__exact_gammaDD(uibh.gammaSphDD)
BitoA.cf_from_gammaDD(                  uibh.gammaSphDD)
BitoA.betU_vetU(                        uibh.betaSphU,uibh.BSphU)

# Step 3.e: Perform the consistency check
print("Consistency check between this tutorial notebook and BSSN.BSSN_in_terms_of_ADM NRPy+ module: ALL SHOULD BE ZERO.")

print("cf - BitoA.cf = " + str(cf - BitoA.cf))
print("trK - BitoA.trK = " + str(trK - BitoA.trK))
# alpha is the only variable that remains unchanged:
# print("alpha - BitoA.alpha = " + str(alpha - BitoA.alpha))

for i in range(DIM):
    print("vetU["+str(i)+"] - BitoA.vetU["+str(i)+"] = " + str(vetU[i] - BitoA.vetU[i]))
    print("betU["+str(i)+"] - BitoA.betU["+str(i)+"] = " + str(betU[i] - BitoA.betU[i]))
    print("lambdaU["+str(i)+"] - BitoA.lambdaU["+str(i)+"] = " + str(lambdaU[i] - BitoA.lambdaU[i]))
    for j in range(DIM):
        print("hDD["+str(i)+"]["+str(j)+"] - BitoA.hDD["+str(i)+"]["+str(j)+"] = "
              + str(hDD[i][j] - BitoA.hDD[i][j]))
        print("aDD["+str(i)+"]["+str(j)+"] - BitoA.aDD["+str(i)+"]["+str(j)+"] = "
              + str(aDD[i][j] - BitoA.aDD[i][j]))
```

    Consistency check between this tutorial notebook and BSSN.BSSN_in_terms_of_ADM NRPy+ module: ALL SHOULD BE ZERO.
    cf - BitoA.cf = 0
    trK - BitoA.trK = 0
    vetU[0] - BitoA.vetU[0] = 0
    betU[0] - BitoA.betU[0] = 0
    lambdaU[0] - BitoA.lambdaU[0] = 0
    hDD[0][0] - BitoA.hDD[0][0] = 0
    aDD[0][0] - BitoA.aDD[0][0] = 0
    hDD[0][1] - BitoA.hDD[0][1] = 0
    aDD[0][1] - BitoA.aDD[0][1] = 0
    hDD[0][2] - BitoA.hDD[0][2] = 0
    aDD[0][2] - BitoA.aDD[0][2] = 0
    vetU[1] - BitoA.vetU[1] = 0
    betU[1] - BitoA.betU[1] = 0
    lambdaU[1] - BitoA.lambdaU[1] = 0
    hDD[1][0] - BitoA.hDD[1][0] = 0
    aDD[1][0] - BitoA.aDD[1][0] = 0
    hDD[1][1] - BitoA.hDD[1][1] = 0
    aDD[1][1] - BitoA.aDD[1][1] = 0
    hDD[1][2] - BitoA.hDD[1][2] = 0
    aDD[1][2] - BitoA.aDD[1][2] = 0
    vetU[2] - BitoA.vetU[2] = 0
    betU[2] - BitoA.betU[2] = 0
    lambdaU[2] - BitoA.lambdaU[2] = 0
    hDD[2][0] - BitoA.hDD[2][0] = 0
    aDD[2][0] - BitoA.aDD[2][0] = 0
    hDD[2][1] - BitoA.hDD[2][1] = 0
    aDD[2][1] - BitoA.aDD[2][1] = 0
    hDD[2][2] - BitoA.hDD[2][2] = 0
    aDD[2][2] - BitoA.aDD[2][2] = 0

# Step 5: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename [Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.pdf](Tutorial-ADM_Initial_Data-Converting_Exact_ADM_Spherical_or_Cartesian_to_BSSNCurvilinear.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
!jupyter nbconvert --to latex --template latex_nrpy_style.tplx --log-level='WARN' Tutorial-BSSN_in_terms_of_ADM.ipynb
!pdflatex -interaction=batchmode Tutorial-BSSN_in_terms_of_ADM.tex
!pdflatex -interaction=batchmode Tutorial-BSSN_in_terms_of_ADM.tex
!pdflatex -interaction=batchmode Tutorial-BSSN_in_terms_of_ADM.tex
!rm -f Tut*.out Tut*.aux Tut*.log
```

    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode
    This is pdfTeX, Version 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) (preloaded format=pdflatex)
     restricted \write18 enabled.
    entering extended mode