# Generating C Code to implement Method of Lines Timestepping for Explicit Runge Kutta Methods

## Authors: Zach Etienne & Brandon Clark

## This tutorial notebook generates three blocks of C Code in order to perform Method of Lines timestepping.

**Notebook Status:**  Validated

**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). All Runge-Kutta Butcher tables were validated using truncated Taylor series in [a separate module](Tutorial-RK_Butcher_Table_Validation.ipynb). Finally, C-code implementation of RK4 was validated against a trusted version. C-code implementations of other RK methods seem to work as expected in the context of solving the scalar wave equation in Cartesian coordinates.

### NRPy+ Source Code for this module:
* [MoLtimestepping/C_Code_Generation.py](../edit/MoLtimestepping/C_Code_Generation.py)
* [MoLtimestepping/RK_Butcher_Table_Dictionary.py](../edit/MoLtimestepping/RK_Butcher_Table_Dictionary.py) ([**Tutorial**](Tutorial-RK_Butcher_Table_Dictionary.ipynb)) Stores the Butcher tables for the explicit Runge Kutta methods

## Introduction:

When numerically solving a partial differential equation initial-value problem, subject to suitable boundary conditions,  we implement Method of Lines to "integrate" the solution forward in time.

### The Method of Lines:

Once we have the initial data for a PDE, we "evolve it forward in time", using the [Method of Lines](https://reference.wolfram.com/language/tutorial/NDSolveMethodOfLines.html). In short, the Method of Lines enables us to handle
1. the **spatial derivatives** of an initial value problem PDE using **standard finite difference approaches**, and
2. the **temporal derivatives** of an initial value problem PDE using **standard strategies for solving ordinary differential equations (ODEs), like Runge Kutta methods** so long as the initial value problem PDE can be written in the first-order-in-time form
$$\partial_t \vec{f} = \mathbf{M}\ \vec{f},$$
where $\mathbf{M}$ is an $N\times N$ matrix containing only *spatial* differential operators that act on the $N$-element column vector $\vec{f}$. $\mathbf{M}$ may not contain $t$ or time derivatives explicitly; only *spatial* partial derivatives are allowed to appear inside $\mathbf{M}$.

You may find the next module [Tutorial-ScalarWave](Tutorial-ScalarWave.ipynb) extremely helpful as an example for implementing the Method of Lines for solving the Scalar Wave equation in Cartesian coordinates.

### Generating the C code:
This module describes how three C code blocks are written to implement Method of Lines timestepping for a specified RK method. The first block is dedicated to allocating memory for the appropriate number of grid function lists needed for the given RK method. The second block will implement the Runge Kutta numerical scheme based on the corresponding Butcher table. The third block will free up the previously allocated memory after the Method of Lines run is complete. These blocks of code are stored within the following three header files respectively

1. `MoLtimestepping/RK_Allocate_Memory.h`
1. `MoLtimestepping/RK_MoL.h`
1. `MoLtimestepping/RK_Free_Memory.h`

The generated code is then included in future Start-to-Finish example tutorial notebooks when solving PDEs numerically.

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#initializenrpy): Initialize needed Python/NRPy+ modules
1. [Step 2](#diagonal): Checking if Butcher Table is Diagonal
1. [Step 3](#ccode): Generating the C Code
    1. [Step 3.a](#allocate): Allocating Memory, `MoLtimestepping/RK_Allocate_Memory.h`
    1. [Step 3.b](#rkmol): Implementing the Runge Kutta Scheme for Method of Lines Timestepping,  `MoLtimestepping/RK_MoL.h`
    1. [Step 3.c](#free): Freeing Allocated Memory, `MoLtimestepping/RK_Free_Memory.h`
1. [Step 4](#code_validation): Code Validation against `MoLtimestepping.RK_Butcher_Table_Generating_C_Code` NRPy+ module
1. [Step 5](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Initialize needed Python/NRPy+ modules [Back to [top](#toc)\]
$$\label{initializenrpy}$$

Let's start by importing all the needed modules from Python/NRPy+:

```python
import sympy as sp  # Import SymPy, a computer algebra system written entirely in Python
import os, shutil   # Standard Python modules for multiplatform OS-level functions
from MoLtimestepping.RK_Butcher_Table_Dictionary import Butcher_dict
```

# Step 2: Checking if a Butcher table is Diagonal [Back to [top](#toc)\]
$$\label{diagonal}$$

A diagonal Butcher table takes the form

$$\begin{array}{c|cccccc}
    0 & \\
    a_1 & a_1 & \\
    a_2 & 0 & a_2 & \\
    a_3 & 0 & 0 & a_3 & \\
    \vdots & \vdots & \ddots & \ddots & \ddots \\
    a_s & 0 & 0 & 0 & \cdots & a_s \\ \hline
     & b_1 & b_2 & b_3 & \cdots & b_{s-1} & b_s
\end{array}$$

where $s$ is the number of required predictor-corrector steps for a given RK method (see [Butcher, John C. (2008)](https://onlinelibrary.wiley.com/doi/book/10.1002/9780470753767)). One known diagonal RK method is the classic RK4 represented in Butcher table form as:

$$\begin{array}{c|cccc}
    0 & \\
    1/2 & 1/2 & \\
    1/2 & 0 & 1/2 & \\
    1 & 0 & 0 & 1 & \\ \hline
     & 1/6 & 1/3 & 1/3 & 1/6
\end{array} $$

Diagonal Butcher tables are nice when it comes to saving required memory space. Each new step for a diagonal RK method, when computing the new $k_i$, does not depend on the previous calculation, and so there are ways to save memory. Signifcantly so in large three-dimensional spatial grid spaces.

```python
def diagonal(key):
    diagonal = True #  Start with the Butcher table is diagonal
    Butcher = Butcher_dict[key][0]
    L = len(Butcher)-1 # Establish the number of rows to check for diagonal trait, all bust last row
    row_idx = 0 # Initialize the Butcher table row index
    for i in range(L): # Check all the desired rows
        for j in range(1,row_idx): # Check each element before the diagonal element in a row
            if Butcher[i][j] != sp.sympify(0): # If any element is non-zero, then the table is not diagonal
                diagonal = False
                break
        row_idx += 1 # Update to check the next row
    return diagonal

# State whether each Butcher table is diagonal or not
for key, value in Butcher_dict.items():
    if diagonal(key) == True:
        print("The RK method "+str(key)+" is diagonal!")
    else:
        print("The RK method "+str(key)+" is NOT diagonal!")
```

    The RK method Euler is diagonal!
    The RK method RK2 Heun is diagonal!
    The RK method RK2 MP is diagonal!
    The RK method RK2 Ralston is diagonal!
    The RK method RK3 is NOT diagonal!
    The RK method RK3 Heun is diagonal!
    The RK method RK3 Ralston is diagonal!
    The RK method SSPRK3 is NOT diagonal!
    The RK method RK4 is diagonal!
    The RK method DP5 is NOT diagonal!
    The RK method DP5alt is NOT diagonal!
    The RK method CK5 is NOT diagonal!
    The RK method DP6 is NOT diagonal!
    The RK method L6 is NOT diagonal!
    The RK method DP8 is NOT diagonal!

# Step 3: Generating the C Code [Back to [top](#toc)\]
$$\label{ccode}$$

The following sections build up the C code for implementing the Method of Lines timestepping algorithm for solving PDEs. To see what the C code looks like for a particular method, simply change the `RK_method` below, otherwise it will default to `"RK4"`.

```python
# Choose a method to see the C code print out for
RK_method = "RK3 Ralston"
```

## Step 3.a: Freeing Allocated Memory, `MoLtimestepping/RK_Free_Memory.h` [Back to [top](#toc)\]
$$\label{free}$$

We define the function `RK_free()` which generates the C code for freeing the memory that was being occupied by the grid functions lists that had been allocated. The function writes the C code to the header file `MoLtimestepping/RK_Free_Memory.h`

```python
# Step 3.a: When allocating memory, we populate a list malloced_gridfunctions,
#           which is used here to determine which gridfunctions need memory freed,
#           via the free() command. Free the mallocs!
def free_allocated_memory(outdir,RK_method,malloced_gridfunctions):
    # This step is made extremely easy, as we had to
    with open(os.path.join(outdir, "RK_Free_Memory.h"), "w") as file:
        file.write("// Code snippet freeing gridfunction memory for \"" + RK_method + "\" method:\n")

        for gridfunction in malloced_gridfunctions:
            file.write("free(" + gridfunction + ");\n")
```

## Step 3.b: Implementing the Runge Kutta Scheme for Method of Lines Timestepping,  `MoLtimestepping/RK_MoL.h` [Back to [top](#toc)\]
$$\label{rkmol}$$

We define the function `RK_MoL()` which generates the C code for implementing Method of Lines using a specified Runge Kutta scheme. The function writes the C code to the header file `MoLtimestepping/RK_MoL.h`.

```python
# Step 3.b: Main driver function for outputting all the MoL C Code
def MoL_C_Code_Generation(RK_method = "RK4", RHS_string = "", post_RHS_string = "",outdir="MoLtimestepping/",
                          MemAllocOnly=False):

####### Step 3.b.i: Allocating Memory
    malloc_str = "// Code snippet allocating gridfunction memory for \"" + RK_method + "\" method:\n"

    # Loop over grids
    malloced_gridfunctions = []
    # Set gridfunction type
    type_str = "REAL *restrict "
    # Define a couple useful functions for outputting the needed C code for allocating memory
    def malloc_gfs_str(varname):
        malloced_gridfunctions.append(varname)
        memory_alloc_str = " = (REAL *)malloc(sizeof(REAL) * NUM_EVOL_GFS * Nxx_plus_2NGHOSTS_tot"+")"
        return type_str + varname + memory_alloc_str + ";\n"
    def diagnostic_output_gfs_equal_to(gfs):
        return type_str + "diagnostic_output_gfs"+" = "+gfs + ";\n"

    # No matter the method we define gridfunctions "y_n_gfs" to store the initial data
    malloc_str += malloc_gfs_str("y_n_gfs")
    if diagonal(RK_method) == True and "RK3" in RK_method:
        malloc_str += malloc_gfs_str("k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs")
        malloc_str += malloc_gfs_str("k2_or_y_nplus_a32_k2_gfs")
        malloc_str += diagnostic_output_gfs_equal_to("k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs")
    else:
        if diagonal(RK_method) == False: # Allocate memory for non-diagonal Butcher tables
            # Determine the number of k_i steps based on length of Butcher Table
            num_k = len(Butcher_dict[RK_method][0])-1
            # For non-diagonal tables an intermediate gridfunction "next_y_input" is used for rhs evaluations
            malloc_str += malloc_gfs_str("next_y_input_gfs")
            for i in range(num_k): # Need to allocate all k_i steps for a given method
                malloc_str += malloc_gfs_str("k"+str(i+1)+"_gfs")
            malloc_str += diagnostic_output_gfs_equal_to("k1_gfs")
        else: # Allocate memory for diagonal Butcher tables, which use a "y_nplus1_running_total gridfunction"
            malloc_str += malloc_gfs_str("y_nplus1_running_total_gfs")
            if RK_method != 'Euler': # Allocate memory for diagonal Butcher tables that aren't Euler
                # Need k_odd for k_1,3,5... and k_even for k_2,4,6...
                malloc_str += malloc_gfs_str("k_odd_gfs")
                malloc_str += malloc_gfs_str("k_even_gfs")
            malloc_str += diagnostic_output_gfs_equal_to("y_nplus1_running_total_gfs")
    with open(os.path.join(outdir,"RK_Allocate_Memory.h"), "w") as file:
        file.write(malloc_str)

    if MemAllocOnly:
        free_allocated_memory(outdir,RK_method,malloced_gridfunctions)
        return
########################################################################################################################
# EXAMPLE
# ODE: y' = f(t,y), y(t_0) = y_0
# Starting at time t_n with solution having value y_n and trying to update to y_nplus1 with timestep dt

# Example of scheme for RK4 with k_1, k_2, k_3, k_4 (Using non-diagonal algortihm) Notice this requires storage of
# y_n, y_nplus1, k_1 through k_4

# k_1      = dt*f(t_n, y_n)
# k_2      = dt*f(t_n + 1/2*dt, y_n + 1/2*k_1)
# k_3      = dt*f(t_n + 1/2*dt, y_n + 1/2*k_2)
# k_4      = dt*f(t_n + dt, y_n + k_3)
# y_nplus1 = y_n + 1/3k_1 + 1/6k_2 + 1/6k_3 + 1/3k_4

# Example of scheme RK4 using only k_odd and k_even (Diagonal algroithm) Notice that this only requires storage

# k_odd     = dt*f(t_n, y_n)
# y_nplus1  = 1/3*k_odd
# k_even    = dt*f(t_n + 1/2*dt, y_n + 1/2*k_odd)
# y_nplus1 += 1/6*k_even
# k_odd     = dt*f(t_n + 1/2*dt, y_n + 1/2*k_even)
# y_nplus1 += 1/6*k_odd
# k_even    = dt*f(t_n + dt, y_n + k_odd)
# y_nplus1 += 1/3*k_even
########################################################################################################################

####### Step 3.b.ii: Implementing the Runge Kutta Scheme for Method of Lines Timestepping
    Butcher = Butcher_dict[RK_method][0] # Get the desired Butcher table from the dictionary
    num_steps = len(Butcher)-1 # Specify the number of required steps to update solution
    # Diagonal RK3 only!!!

    def single_RK_substep(commentblock, RHS_str, RHS_input_str, RHS_output_str, RK_lhss_list, RK_rhss_list,
                          post_RHS_list, post_RHS_output_list, indent = "  "):
        return_str  = commentblock + "\n"
        if not isinstance(RK_lhss_list,list):
            RK_lhss_list = [RK_lhss_list]
        if not isinstance(RK_rhss_list,list):
            RK_rhss_list = [RK_rhss_list]

        if not isinstance(post_RHS_list,list):
            post_RHS_list = [post_RHS_list]
        if not isinstance(post_RHS_output_list,list):
            post_RHS_output_list = [post_RHS_output_list]

        # Part 1: RHS evaluation:
        return_str += RHS_str.replace("RK_INPUT_GFS", RHS_input_str).\
                              replace("RK_OUTPUT_GFS",RHS_output_str)+"\n"

        # Part 2: RK update
        return_str += "LOOP_ALL_GFS_GPS"+"(i) {\n"
        for lhs,rhs in zip(RK_lhss_list,RK_rhss_list):
            return_str += indent + lhs + "[i] = " + rhs.replace("_gfs","_gfs") + ";\n"
        return_str += "}\n"

        # Part 3: Call post-RHS functions
        for post_RHS,post_RHS_output in zip(post_RHS_list,post_RHS_output_list):
            return_str += post_RHS.replace("RK_OUTPUT_GFS",post_RHS_output)+"\n"

        return return_str+"\n"

    RK_str = "// C code implementation of " + RK_method + " Method of Lines timestepping.\n"

    if diagonal(RK_method) == True and "RK3" in RK_method:
        #  In a diagonal RK3 method, only 3 gridfunctions need be defined. Below implements this approach.

        # k_1
        RK_str += """
// In a diagonal RK3 method like this one, only 3 gridfunctions need be defined. Below implements this approach.
// Using y_n_gfs as input, k1 and apply boundary conditions\n"""

        RK_str += single_RK_substep(
            commentblock = """
// ***k1 substep:***
//  1. We will store k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs now as
//     ...  the update for the next rhs evaluation y_n + a21*k1*dt
// Post-RHS evaluation:
//  1. Apply post-RHS to y_n + a21*k1*dt""",
            RHS_str = RHS_string,
            RHS_input_str = "y_n_gfs", RHS_output_str = "k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs",
            RK_lhss_list = ["k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs"],
            RK_rhss_list = ["("+sp.ccode(Butcher[1][1]).replace("L","")+")*k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i]*dt + y_n_gfs[i]"],
            post_RHS_list = [post_RHS_string], post_RHS_output_list = ["k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs"])

        # k_2
        RK_str += single_RK_substep(
            commentblock="""
// ***k2 substep:***
//    1. Reassign k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs to be the running total y_{n+1}; a32*k2*dt to the running total
//    2. Store k2_or_y_nplus_a32_k2_gfs now as y_n + a32*k2*dt

// Post-RHS evaluation:
//    1. Apply post-RHS to both y_n + a32*k2 (stored in k2_or_y_nplus_a32_k2_gfs)
//       ... and the y_{n+1} running total, as they have not been applied yet to k2-related gridfunctions""",
            RHS_str=RHS_string,
            RHS_input_str="k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs", RHS_output_str="k2_or_y_nplus_a32_k2_gfs",
            RK_lhss_list=["k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs","k2_or_y_nplus_a32_k2_gfs"],
            RK_rhss_list=["("+sp.ccode(Butcher[3][1]).replace("L","")+")*(k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i] - y_n_gfs[i])/("+sp.ccode(Butcher[1][1]).replace("L","")+") + y_n_gfs[i] + ("+sp.ccode(Butcher[3][2]).replace("L","")+")*k2_or_y_nplus_a32_k2_gfs[i]*dt",
                          "("+sp.ccode(Butcher[2][2]).replace("L","")+")*k2_or_y_nplus_a32_k2_gfs[i]*dt + y_n_gfs[i]"],
            post_RHS_list=[post_RHS_string,post_RHS_string],
            post_RHS_output_list=["k2_or_y_nplus_a32_k2_gfs","k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs"])

        # k_3
        RK_str += single_RK_substep(
            commentblock="""
// ***k3 substep:***
//    1. Add k3 to the running total and save to y_n

// Post-RHS evaluation:
//    1. Apply post-RHS to y_n""",
            RHS_str=RHS_string,
            RHS_input_str="k2_or_y_nplus_a32_k2_gfs", RHS_output_str="y_n_gfs",
            RK_lhss_list=["y_n_gfs","k2_or_y_nplus_a32_k2_gfs"],
            RK_rhss_list=["k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i] + ("+sp.ccode(Butcher[3][3]).replace("L","")+")*y_n_gfs[i]*dt"],
            post_RHS_list=[post_RHS_string],
            post_RHS_output_list=["y_n_gfs"])
    else:
        y_n           = "y_n_gfs"
        if diagonal(RK_method) == False:
            for s in range(num_steps):
                next_y_input  = "next_y_input_gfs"

                # If we're on the first step (s=0), we use y_n gridfunction as input.
                #      Otherwise next_y_input is input. Output is just the reverse.
                if s==0: # If on first step:
                    RHS_input = y_n
                else:    # If on second step or later:
                    RHS_input = next_y_input
                RHS_output = "k" + str(s + 1) + "_gfs"
                if s == num_steps-1: # If on final step:
                    RK_lhs = y_n
                    RK_rhs = y_n + "[i] + dt*("
                else:                # If on anything but the final step:
                    RK_lhs = next_y_input
                    RK_rhs = y_n + "[i] + dt*("
                for m in range(s+1):
                    if Butcher[s+1][m+1] != 0:
                        if Butcher[s+1][m+1] != 1:
                            RK_rhs += " + k"+str(m+1)+"_gfs[i]*("+sp.ccode(Butcher[s+1][m+1]).replace("L","")+")"
                        else:
                            RK_rhs += " + k"+str(m+1)+"_gfs[i]"
                RK_rhs += " )"

                post_RHS = post_RHS_string
                if s == num_steps-1: # If on final step:
                    post_RHS_output = y_n
                else:                # If on anything but the final step:
                    post_RHS_output = next_y_input

                RK_str += single_RK_substep(
                    commentblock="// ***k" + str(s + 1) + " substep:***",
                    RHS_str=RHS_string,
                    RHS_input_str=RHS_input, RHS_output_str=RHS_output,
                    RK_lhss_list=[RK_lhs],   RK_rhss_list=[RK_rhs],
                    post_RHS_list=[post_RHS],
                    post_RHS_output_list=[post_RHS_output])
        else:
            y_nplus1_running_total = "y_nplus1_running_total_gfs"
            if RK_method == 'Euler': # Euler's method doesn't require any k_i, and gets its own unique algorithm
                RK_str += single_RK_substep(
                    commentblock="// ***Euler timestepping only requires one RHS evaluation***",
                    RHS_str=RHS_string,
                    RHS_input_str=y_n, RHS_output_str=y_nplus1_running_total,
                    RK_lhss_list=[y_n],   RK_rhss_list=[y_n+"[i] + "+y_nplus1_running_total+"[i]*dt"],
                    post_RHS_list=[post_RHS_string],
                    post_RHS_output_list=[y_n])
            else:
                for s in range(num_steps):
                    # If we're on the first step (s=0), we use y_n gridfunction as input.
                    # and k_odd as output.
                    if s == 0:
                        RHS_input  = "y_n_gfs"
                        RHS_output = "k_odd_gfs"
                    # For the remaining steps the inputs and ouputs alternate between k_odd and k_even
                    elif s%2 == 0:
                        RHS_input = "k_even_gfs"
                        RHS_output = "k_odd_gfs"
                    else:
                        RHS_input = "k_odd_gfs"
                        RHS_output = "k_even_gfs"

                    RK_lhs_list = []
                    RK_rhs_list = []
                    if s != num_steps-1: # For anything besides the final step
                        if s == 0: # The first RK step
                            RK_lhs_list.append(y_nplus1_running_total)
                            RK_rhs_list.append(RHS_output+"[i]*dt*("+sp.ccode(Butcher[num_steps][s+1]).replace("L","")+")")

                            RK_lhs_list.append(RHS_output)
                            RK_rhs_list.append(y_n+"[i] + "+RHS_output+"[i]*dt*("+sp.ccode(Butcher[s+1][s+1]).replace("L","")+")")
                        else:
                            if Butcher[num_steps][s+1] !=0:
                                RK_lhs_list.append(y_nplus1_running_total)
                                if Butcher[num_steps][s+1] !=1:
                                    RK_rhs_list.append(y_nplus1_running_total+"[i] + "+RHS_output+"[i]*dt*("+sp.ccode(Butcher[num_steps][s+1]).replace("L","")+")")
                                else:
                                    RK_rhs_list.append(y_nplus1_running_total+"[i] + "+RHS_output+"[i]*dt")
                            if Butcher[s+1][s+1] !=0:
                                RK_lhs_list.append(RHS_output)
                                if Butcher[s+1][s+1] !=1:
                                    RK_rhs_list.append(y_n+"[i] + "+RHS_output+"[i]*dt*("+sp.ccode(Butcher[s+1][s+1]).replace("L","")+")")
                                else:
                                    RK_rhs_list.append(y_n+"[i] + "+RHS_output+"[i]*dt")
                        post_RHS_output = RHS_output
                    if s == num_steps-1: # If on the final step
                        if Butcher[num_steps][s+1] != 0:
                            RK_lhs_list.append(y_n)
                            if Butcher[num_steps][s+1] != 1:
                                RK_rhs_list.append(y_n+"[i] + "+y_nplus1_running_total+"[i] + "+RHS_output+"[i]*dt*("+sp.ccode(Butcher[num_steps][s+1]).replace("L","")+")")
                            else:
                                RK_rhs_list.append(y_n+"[i] + "+y_nplus1_running_total+"[i] + "+RHS_output+"[i]*dt)")
                        post_RHS_output = y_n

                    RK_str += single_RK_substep(
                        commentblock="// ***k" + str(s + 1) + " substep:***",
                        RHS_str=RHS_string,
                        RHS_input_str=RHS_input, RHS_output_str=RHS_output,
                        RK_lhss_list=RK_lhs_list, RK_rhss_list=RK_rhs_list,
                        post_RHS_list=[post_RHS_string],
                        post_RHS_output_list=[post_RHS_output])

    with open(os.path.join(outdir,"RK_MoL.h"), "w") as file:
        file.write(RK_str)

####### Step 3.b.iii: Freeing Allocated Memory
    free_allocated_memory(outdir,RK_method,malloced_gridfunctions)

MoL_C_Code_Generation(RK_method,"rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, RK_INPUT_GFS, RK_OUTPUT_GFS);",
                                "apply_bcs(Nxx,Nxx_plus_2NGHOSTS, RK_OUTPUT_GFS);")
print("This is the MoL timestepping RK scheme C code for the "+str(RK_method)+" method: \n")
with open(os.path.join("MoLtimestepping/","RK_MoL.h"), "r") as file:
    print(file.read())
```

    This is the MoL timestepping RK scheme C code for the RK3 Ralston method:

    // C code implementation of RK3 Ralston Method of Lines timestepping.

    // In a diagonal RK3 method like this one, only 3 gridfunctions need be defined. Below implements this approach.
    // Using y_n_gfs as input, k1 and apply boundary conditions

    // ***k1 substep:***
    //  1. We will store k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs now as
    //     ...  the update for the next rhs evaluation y_n + a21*k1*dt
    // Post-RHS evaluation:
    //  1. Apply post-RHS to y_n + a21*k1*dt
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, y_n_gfs, k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs);
    LOOP_ALL_GFS_GPS(i) {
      k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i] = (1.0/2.0)*k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i]*dt + y_n_gfs[i];
    }
    apply_bcs(Nxx,Nxx_plus_2NGHOSTS, k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs);

    // ***k2 substep:***
    //    1. Reassign k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs to be the running total y_{n+1}; a32*k2*dt to the running total
    //    2. Store k2_or_y_nplus_a32_k2_gfs now as y_n + a32*k2*dt

    // Post-RHS evaluation:
    //    1. Apply post-RHS to both y_n + a32*k2 (stored in k2_or_y_nplus_a32_k2_gfs)
    //       ... and the y_{n+1} running total, as they have not been applied yet to k2-related gridfunctions
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs, k2_or_y_nplus_a32_k2_gfs);
    LOOP_ALL_GFS_GPS(i) {
      k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i] = (2.0/9.0)*(k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i] - y_n_gfs[i])/(1.0/2.0) + y_n_gfs[i] + (1.0/3.0)*k2_or_y_nplus_a32_k2_gfs[i]*dt;
      k2_or_y_nplus_a32_k2_gfs[i] = (3.0/4.0)*k2_or_y_nplus_a32_k2_gfs[i]*dt + y_n_gfs[i];
    }
    apply_bcs(Nxx,Nxx_plus_2NGHOSTS, k2_or_y_nplus_a32_k2_gfs);
    apply_bcs(Nxx,Nxx_plus_2NGHOSTS, k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs);

    // ***k3 substep:***
    //    1. Add k3 to the running total and save to y_n

    // Post-RHS evaluation:
    //    1. Apply post-RHS to y_n
    rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, k2_or_y_nplus_a32_k2_gfs, y_n_gfs);
    LOOP_ALL_GFS_GPS(i) {
      y_n_gfs[i] = k1_or_y_nplus_a21_k1_or_y_nplus1_running_total_gfs[i] + (4.0/9.0)*y_n_gfs[i]*dt;
    }
    apply_bcs(Nxx,Nxx_plus_2NGHOSTS, y_n_gfs);

# Step 4: Code Validation against `MoLtimestepping.RK_Butcher_Table_Generating_C_Code` NRPy+ module [Back to [top](#toc)\]
$$\label{code_validation}$$

As a code validation check, we verify agreement in the dictionary of Butcher tables between

1. this tutorial and
2. the NRPy+ [MoLtimestepping.RK_Butcher_Table_Generating_C_Code](../edit/MoLtimestepping/RK_Butcher_Table_Generating_C_Code.py) module.

We generate the header files for each RK method and check for agreement with the NRPY+ module.

```python
import sys
import MoLtimestepping.C_Code_Generation as MoLC

print("\n\n ### BEGIN VALIDATION TESTS ###")
import filecmp
for key, value in Butcher_dict.items():
    MoL_C_Code_Generation(key,"rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, RK_INPUT_GFS, RK_OUTPUT_GFS);",
                              "apply_bcs(Nxx,Nxx_plus_2NGHOSTS, RK_OUTPUT_GFS);")
    for filename in ["RK_Allocate_Memory.h","RK_MoL.h","RK_Free_Memory.h"]:
        shutil.copy(os.path.join("MoLtimestepping/",filename), os.path.join("MoLtimestepping/",filename+key+".h"))

    MoLC.MoL_C_Code_Generation(key, "rhs_eval(Nxx,Nxx_plus_2NGHOSTS,dxx, RK_INPUT_GFS, RK_OUTPUT_GFS);",
                                    "apply_bcs(Nxx,Nxx_plus_2NGHOSTS, RK_OUTPUT_GFS);")
    for filename in ["RK_Allocate_Memory.h","RK_MoL.h","RK_Free_Memory.h"]:
        if filecmp.cmp(os.path.join("MoLtimestepping/",filename), os.path.join("MoLtimestepping/",filename+key+".h")) == False:
            print("VALIDATION TEST FAILED ON files: "+os.path.join("MoLtimestepping/",filename)+" and "+
                  os.path.join("MoLtimestepping/",filename+key+".h"))
            sys.exit(1)
    print("VALIDATION TEST PASSED on all files from "+str(key)+" method")
print("### END VALIDATION TESTS ###")
```

     ### BEGIN VALIDATION TESTS ###
    VALIDATION TEST PASSED on all files from Euler method
    VALIDATION TEST PASSED on all files from RK2 Heun method
    VALIDATION TEST PASSED on all files from RK2 MP method
    VALIDATION TEST PASSED on all files from RK2 Ralston method
    VALIDATION TEST PASSED on all files from RK3 method
    VALIDATION TEST PASSED on all files from RK3 Heun method
    VALIDATION TEST PASSED on all files from RK3 Ralston method
    VALIDATION TEST PASSED on all files from SSPRK3 method
    VALIDATION TEST PASSED on all files from RK4 method
    VALIDATION TEST PASSED on all files from DP5 method
    VALIDATION TEST PASSED on all files from DP5alt method
    VALIDATION TEST PASSED on all files from CK5 method
    VALIDATION TEST PASSED on all files from DP6 method
    VALIDATION TEST PASSED on all files from L6 method
    VALIDATION TEST PASSED on all files from DP8 method
    ### END VALIDATION TESTS ###

# Step 5: Output this notebook to $\LaTeX$-formatted PDF \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-RK_Butcher_Table_Generating_C_Code.pdf](Tutorial-RK_Butcher_Table_Generating_C_Code.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-Method_of_Lines-C_Code_Generation")
```

    Created Tutorial-Method_of_Lines-C_Code_Generation.tex, and compiled LaTeX
        file to PDF file Tutorial-Method_of_Lines-C_Code_Generation.pdf


# Soldering pen design

```python
#Initialization of iPython, some helper functions.
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
plt.style.use('ggplot')
from sympy import *
from IPython.display import display, Math, Latex
init_printing(use_latex="mathjax")

_Omega=u'\u03A9'

def readCsvData(fileName):
    return np.genfromtxt(fileName, delimiter=';')

def plotData(data, timebase=1, xlabel="", ylabel="",names=[]):
    fig=plt.figure(figsize=[10,5])
    l = np.size(data,0)
    time = np.linspace(0,l-1,l)*timebase
    if np.ndim(data)>1: # If more than one vector to plot
        count = np.size(data,1)
        for i in range(count):
            plt.plot(time, data[:,i], label=names[i])
    else:
        plt.plot(time, data, label=names[0])
    plt.xlabel(xlabel)
    plt.ylabel(ylabel)
    plt.legend()
    plt.plot()

def plotCsv(fileName, timebase=1, xlabel="", ylabel="",names=[]):
    data = readCsvData(fileName)
    plotData(data, timebase, xlabel, ylabel,names)
```

## Basic assumptions

### Block diagram

### Power supply
Power requirements are:

```python
# Tip rated power [W]
Ptip = 40
# Supply voltage [minimum, typical, maximum] [V]
Vsupply = [6, 12, 20]
```

Rated tip power (40 W) is defined for typical supply voltage (12 V). For lower supply voltage maximum power is lower than nominal. For minimum power supply:

```python
Ptip_min=(Vsupply[0]**2)/(Vsupply[1]**2)*40
print(Ptip_min,"W")
```

    10.0 W

Device withstands power supply voltage in range of -30..+30 V but is non-operational outside Vsupply range.

### Temperature
Operational range of ambient temperature is:

```python
Tambient = [0, 40] # [minimum, maximum] *C
```

Working range of tip temperature is:

```python
Ttip = [200, 350] # [minimum, maximum] *C
```

## Power supply circuits

## Heater driver
Heater driver is selected to be BTS5012SDA. Lets calculate maximum current of the driver.

```python
# Nominal resistance of the tip heater
Rtip = Vsupply[1]**2/Ptip; print("Rtip = %1.2f %s" % (Rtip, _Omega))
# Nominal heater current at nominal voltage
Itip_nominal = Vsupply[1]/Rtip; print("Itip_nominal = %1.2f A" % Itip_nominal)
# Safety factor
k_safety = 1.1
# Minimum tip current - at minimum voltage, including safety factor
Itip_min = Vsupply[0]/Rtip/k_safety; print("Itip_min = %1.2f A" % Itip_min)
# Peak current - at maximum voltage, including safety factor
Itip_max = Vsupply[2]/Rtip*k_safety; print("Itip_max = %1.2f A" % Itip_max)
```

    Rtip = 3.60 Ω
    Itip_nominal = 3.33 A
    Itip_min = 1.52 A
    Itip_max = 6.11 A

BTS5012SDA has built-in diagnosis and current sense circuits. It can be used to detect that proper heater is connected.

Value of current sense ratio ($k_{ilis}$) can be read from dataqsheet graph (actual points read using http://arohatgi.info/WebPlotDigitizer/). For simplification, only two points are used.

```python
# Supply voltage for use as x axis in plots
Vs = np.array([Vsupply[0], Vsupply[2]])
# Minimum k_ilis valuses for supply voltage range
k_ilis_min = np.array([6400, 7900])
# Maximum k_ilis valuses for supply voltage range
k_ilis_max = np.array([16300, 12200])
# Minimum nominal current sense, mA
Is_min = Vs/Rtip/k_safety/k_ilis_max*1000
# Maximum nominal current sense, mA
Is_max = Vs/Rtip*k_safety/k_ilis_min*1000
# Plot graph
plt.rcParams['figure.figsize'] = 12, 5
plt.figure
plt.plot(Vs, Is_min,"k")
plt.plot(Vs, Is_max,"k")
plt.fill_between(Vs, Is_min, Is_max, facecolor='lime')
plt.ylim(0); plt.xlabel("Vsupply [V]"); plt.ylabel("Is [mA]"); plt.show()
```

To calculate ADC readings from current sense circuit sense resistor and ADC errors must be taken into account.

```python
# Sense resistor nominal value.
Rs = 470
# Sense resistor tolerance, including thermal coefficient.
Rs_tol = 0.06
# uC power supply voltage range, V
Vcc = 3.3; Vcc_min = 2.97; Vcc_max = 3.63
# Input voltage, min and max
Vs_min = Is_min * Rs / 1000 / (1+Rs_tol)
Vs_max = Is_max * Rs / 1000 * (1+Rs_tol)
# ADC readings, min and max
ADCs_min = Vs_min / Vcc_max * 4096
ADCs_max = Vs_max / Vcc_min * 4096
# Limits are chosen to provide some additional safety
ADC_lim_min = np.floor(ADCs_min * 0.85 - 5)
ADC_lim_max = np.ceil(ADCs_max * 1.15 + 15)
# print limits
print(ADC_lim_min, ADC_lim_max)
# Plot graph
plt.figure
plt.plot(Vs, ADCs_min,"k")
plt.plot(Vs, ADCs_max,"k")
plt.plot(Vs, ADC_lim_min, "r", linewidth=3)
plt.plot(Vs, ADC_lim_max, "r", linewidth=3)
plt.fill_between(Vs, ADCs_min, ADCs_max, facecolor='lime')
plt.text(12, 200, "Nominal")
plt.fill_between(Vs, 0, ADC_lim_min, facecolor='aqua')
plt.text(14, 50, "Open load")
plt.fill_between(Vs, ADC_lim_max, 700, facecolor='orange')
plt.text(10, 500, "Overload")
plt.ylim(0,700); plt.xlabel("Vsupply [V]"); plt.ylabel("ADC reading [LSB]"); plt.show()
```

Vsupply is measured by Vin_sense ADC input. Current sense thresholds related to Vin_sense readings are calculated below. Power supply sense resistors tolerances are neglected. At the end, limits are represented as $ADC_{lim}=a*ADC_{Vin}+b$.

```python
# ADC readings for Vin_sense input, only nominal values
ADCvin = np.floor(Vs * 10 / (100 + 10) / Vcc * 4095)
display(ADCvin)
# Coefficient for integer multiply
int_coef = 65536
# Calculate coefficients for lim = a*Vin+b
a_min = int((ADC_lim_min[1] - ADC_lim_min[0])/(ADCvin[1] - ADCvin[0])*int_coef)
b_min = int(ADC_lim_min[1] - a_min/int_coef * ADCvin[1])
a_max = int((ADC_lim_max[1] - ADC_lim_max[0])/(ADCvin[1] - ADCvin[0])*int_coef)
b_max = int(ADC_lim_max[1] - a_max/int_coef * ADCvin[1])
# Display as en equations
display(Math(r"ADClim_{min}=\frac{%d}{%d}*ADC_{Vin}+%d" %(a_min, int_coef, b_min)))
display(Math(r"ADClim_{max}=\frac{%d}{%d}*ADC_{Vin}+%d" %(a_max, int_coef, b_max)))
# Check
display(Math(r"ADC\ limits\ assumed\begin{pmatrix} %d & %d \\ %d & %d \end{pmatrix}" %
            (ADC_lim_max[0], ADC_lim_max[1], ADC_lim_min[0], ADC_lim_min[1]) +
            r"calculated\begin{pmatrix} %d & %d \\ %d & %d \end{pmatrix}" %
            (ADCvin[0] * a_max / int_coef + b_max, ADCvin[1] * a_max / int_coef + b_max,
            ADCvin[0] * a_min / int_coef + b_min, ADCvin[1] * a_min / int_coef + b_min)))
```

    array([  676.,  2256.])

$$ADClim_{min}=\frac{5682}{65536}*ADC_{Vin}+-24$$

$$ADClim_{max}=\frac{15969}{65536}*ADC_{Vin}+77$$

$$ADC\ limits\ assumed\begin{pmatrix} 242 & 627 \\ 34 & 171 \end{pmatrix}calculated\begin{pmatrix} 241 & 626 \\ 34 & 171 \end{pmatrix}$$

```python
display(Math(r'F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx'))
```

$$F(k) = \int_{-\infty}^{\infty} f(x) e^{2\pi i k} dx$$

## Sensor amplifier

## Microcontroller

## Magnetometer

Permanent neodymium magnet is mounted in soldering iron stand. On board MEMS magnetometer is used to sense its' proximity. If magnet is detected, soldering iron goes to sleep mode after programmed time.
MAG3110 has full scale range of $\pm 1000 \mu T$. Zero flux offset is typically $\pm 100 \mu T$. It's large, compared to typical geomagnetic field strength of $25\ldots60 \mu T$. Zero offset calibration is desirable.

### Magnitude calculation
For calculating magnitude of magnetic field flux density from X, Y, Z readings, following formula should be used:

```python
x, y, z, a, b, c, m_real, m_appr, z_1 = symbols("x y z a b c m_real m_appr z_1")
m_real_ = Equality(m_real, sqrt(x**2 + y**2 + z**2)); display(m_real_)
```

$$m_{real} = \sqrt{x^{2} + y^{2} + z^{2}}$$

To minimize calculations, approximation will be used. Derivative of alpha max plus beta min algorithm will be used.

```python
m_appr_ = Equality(m_appr, a*abs(x) + b*abs(y) + c*abs(z)); display(m_appr_)
```

$$m_{appr} = a \left\lvert{x}\right\rvert + b \left\lvert{y}\right\rvert + c \left\lvert{z}\right\rvert$$

Simple values for a, b, c are selected:

```python
m_appr_ = m_appr_.subs(a,1).subs(b, 1/2).subs(c, 1/4); display(m_appr_)
```

$$m_{appr} = \left\lvert{x}\right\rvert + 0.5 \left\lvert{y}\right\rvert + 0.25 \left\lvert{z}\right\rvert$$

Let's calculate error of this formula.

To simplify calculations and visualization, three parameters are simplified to two. For every $\frac{y}{x}=const$ and $\frac{z}{x}=const$ error will be the same. $x=const=1$ is selected for this calculation. x>=y>=z, so maximum value of y is 1. As y>=z another variable is substituted instead of z:

```python
Equality(z_1,z/y)
```

$$z_{1} = \frac{z}{y}$$

```python
m_real_ = Equality(m_real, sqrt(1 + y**2 + (z_1*y)**2)); display(m_real_)
m_appr_ = m_appr_.subs(x,1).subs(z,z_1*y); display(m_appr_)
```

$$m_{real} = \sqrt{y^{2} z_{1}^{2} + y^{2} + 1}$$

$$m_{appr} = 0.5 \left\lvert{y}\right\rvert + 0.25 \left\lvert{y z_{1}}\right\rvert + 1$$

```python
plt.rcParams['figure.figsize'] = 15, 10
plotting.plot3d(m_appr_.rhs/m_real_.rhs-1,(y,0,1),(z_1,0,1));plt.show()
```

~15% error is perfectly acceptable. It's even better, because error is always >=0, so relative ripples are +-7.5%.


+ This notebook is part of lecture 26 *Complex matrices and the fast Fourier transform* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols, I, sqrt, Rational
from IPython.display import Image
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Complex vectors, matrices
# Fast Fourier transform

## Complex vectors

* Consider the following vector with complex entries (from this point on I will not use the underscore to indicate a vector, so as not to create confusion with the bar, noting complex conjugate, instead, inferring from context)
$$ {z} = \begin{bmatrix} {z}_{1} \\ {z}_{2} \\ \vdots \\ {z}_{n} \end{bmatrix} $$

* The length (actually length squared) of this vector is *no good*, since it should be positive
$$ {z}^{T}{z} $$
* Instead we consider the following
$$ z\bar { z } ={ \left| { z } \right|  }^{ 2 }\\ \therefore \quad \bar { z } ^{ T }z\\ \left[ { \bar { z }  }_{ 1 },{ \bar { z }  }_{ 2 },\dots ,{ \bar { z }  }_{ n } \right] \begin{bmatrix} { z }_{ 1 } \\ { z }_{ 2 } \\ \vdots  \\ { z }_{ n } \end{bmatrix} $$

```python
z = Matrix([1, I]) # I is the sympy symbol for the imaginary number i
z
```

* Let's calculate this manually

```python
z.norm() # The length of a vector
```

```python
z_cc = Matrix([1, -I])
z_cc
```

```python
sqrt(z_cc.transpose() * z)
```

* Taking the transpose of the complex conjugate is called the Hermitian
$$ {z}^{H}{z} $$

* We can use the Hermitian for non-complex (or mixed complex) vectors **u** and **v** too
$$ \bar{y}^{T}{x} \\ {y}^{H}{x} $$

```python
from sympy.physics.quantum.dagger import Dagger # A fun way to quickly get the Hermitian
```

```python
Dagger(z)
```

```python
sqrt(Dagger(z) * z)
```

## Complex symmetric matrices

### The transpose

* If the symmetric matrix has complex entries then AT=A is *no good*

```python
A = Matrix([[2, 3 + I], [3 - I, 5]])
A # A Hermitian matrix
```

```python
A.transpose() == A
```

```python
Dagger(A)
```

```python
Dagger(A) == A
```

* This will work for real-values symmetric matrices as well

```python
A = Matrix([[3, 4], [4, 2]])
A
```

```python
A.transpose() == A
```

```python
Dagger(A) == A
```

### The eigenvalues and eigenvectors

* Back to the complex matrix A

```python
A = Matrix([[2, 3 + I], [3 - I, 5]])
A
```

```python
A.eigenvals()
```

$$ A=\begin{bmatrix} 2 & 3+i \\ 3-i & 5 \end{bmatrix}\\ A-\lambda I=\underline { 0 } \\ \left| \begin{bmatrix} 2 & 3+i \\ 3-i & 5 \end{bmatrix}-\begin{bmatrix} \lambda  & 0 \\ 0 & \lambda  \end{bmatrix} \right| =0\\ \begin{vmatrix} 2-\lambda  & 3+i \\ 3-i & 5-\lambda  \end{vmatrix}=0\\ \left( 2-\lambda  \right) \left( 5-\lambda  \right) -\left( 3+i \right) \left( 3-i \right) =0\\ 10-7\lambda +{ \lambda  }^{ 2 }-\left( 9+1 \right) =0\\ { \lambda  }^{ 2 }-7\lambda =0\\ { \lambda  }_{ 1 }=0\\ { \lambda  }_{ 2 }=7 $$

```python
A.eigenvects()
```

```python
S, D = A.diagonalize()
```

```python
S
```

```python
D
```

* What about S now?
* We have to use its transpose, but it is complex, so we have to take the Hermitian

```python
Dagger(S)
```

```python
S == Dagger(S) # Don't get confused here, S is not symmetric
```

* Remember that for a symmetric matrix the column vectors in S (usually called Q, the matrix of eigenvectors) are orthogonal, with QTQ=I
* With complex entries we have to consider the Hermitian here, not just the simple transpose
* Here we call Q *unitary*

## The fast Fourier transform

* Look at this special matrix (where we start counting rows and columns at zero)

$$ { F }_{ n }=\begin{bmatrix} W^{ \left( 0 \right) \left( 0 \right)  } & { W }^{ \left( 0 \right) \left( 1 \right)  } & { W }^{ \left( 0 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( 0 \right) \left( n-1 \right)  } \\ W^{ \left( 1 \right) \left( 0 \right)  } & { W }^{ \left( 1 \right) \left( 1 \right)  } & { W }^{ \left( 1 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( 1 \right) \left( n-1 \right)  } \\ { W }^{ \left( 2 \right) \left( 0 \right)  } & { W }^{ \left( 2 \right) \left( 1 \right)  } & { W }^{ \left( 2 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( 2 \right) \left( n-1 \right)  } \\ \vdots  & \vdots  & \vdots  & \dots  & \vdots  \\ { W }^{ \left( n-1 \right) \left( 0 \right)  } & { W }^{ \left( n-1 \right) \left( 1 \right)  } & { W }^{ \left( n-1 \right) \left( 2 \right)  } & \dots  & { W }^{ \left( n-1 \right) \left( n-1 \right)  } \end{bmatrix} \\ \left({F}_{n}\right)_{ij}={W}^{ij}; i,j=0,1,2,\dots,n-1 $$

* W is a special number whose *n*th power equals 1
$$ {W}^{n}=1 \\ W={ e }^{ \frac { i2\pi  }{ n }  }=\cos { \frac { 2\pi  }{ n } +i\sin { \frac { 2\pi  }{ n }  }  }  $$
* It is in the complex plane of course (as written in *sin* and *cos* above)

* Remember than *n* here refers to the size the matrix
* Here it also refers to the *n*th *n* roots (if that makes any sense, else look at the image below)

```python
Image(filename = 'W.png')
```

* So for *n*=4 we will have the following
$$ { F }_{ 4 }=\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 1 } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 2 } } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 3 } } \\ 1 & \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 2 } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 4 } } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 6 } } \\ 1 & \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 3 } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 6 } } & { \left( { e }^{ \frac { 2\pi i }{ 4 }  } \right) ^{ 9 } } \end{bmatrix} $$

* We note that a quarter of the way around is *i*
$$ {e}^{\frac{2\pi{i}}{4}}={i} $$
* We thus have the following
$$ { F }_{ 4 }=\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & i & { i }^{ 2 } & { i }^{ 3 } \\ 1 & { i }^{ 2 } & { i }^{ 4 } & { i }^{ 6 } \\ 1 & { i }^{ 3 } & { i }^{ 6 } & { i }^{ 9 } \end{bmatrix}\\ { F }_{ 4 }=\begin{bmatrix} 1 & 1 & 1 & 1 \\ 1 & i & -1 & -i \\ 1 & -1 & 1 & -1 \\ 1 & -i & -1 & i \end{bmatrix} $$

* Note how the columns are orthogonal

```python
F = Matrix([[1, 1, 1, 1], [1, I, -1, -I], [1, -1, 1, -1], [1, -I, -1, I]])
F
```

```python
F.col(0) # Calling only the selected column (counting starts at 0)
```

* The columns are supposed to be orthogonal, i.e. inner (dot) product should be zero
* Clearly below it is not

```python
F.col(1).dot(F.col(3))
```

* Remember, though, that this is a complex matrix and we have to use the Hermitian

```python
col1 = F.col(1)
col3 = F.col(3)
col1, col3
```

```python
Dagger(col3), col1
```

```python
Dagger(col3) * col1 # Another way to do the dot product
```

* So, these columns are all orthogonal, but they are not orthonormal
* Note, though that the are all of length 2, so we can normalize each

```python
Rational(1, 2) * F
```

* We also note the following
$$ {F}_{n}^{H}{F}_{n}={I} $$
* Just remember to normalize them

```python
Dagger(Rational(1, 2) * F)
```

```python
Dagger(Rational(1, 2) * F) * ((Rational(1, 2) * F))
```

* Now why do we call it *fast* Fourier transform
* Note the following
$$ { W }_{ n }={ e }^{ \frac { 2\pi i }{ n }  }\\ { \left( { W }_{ n } \right)  }^{ p }={ \left( { e }^{ \frac { 2\pi i }{ n }  } \right)  }^{ p }\\ { \left( { W }_{ 64 } \right)  }^{ 2 }={ \left( { e }^{ \frac { 2\pi i }{ 64 }  } \right)  }^{ 2 };\quad n=64,\quad p=2\\ \therefore \quad { \left( { W }_{ 64 } \right)  }^{ 2 }={ W }_{ 32 } $$

* Now we have the following connection between the two
$$ \left[ { F }_{ 64 } \right] =\begin{bmatrix} I & D \\ I & -D \end{bmatrix}\begin{bmatrix} { F }_{ 32 } & 0 \\ 0 & { F }_{ 32 } \end{bmatrix}\left[ P \right] \\ D=\begin{bmatrix} 1 & 0 & 0 & \dots  & 0 \\ 0 & W & 0 & \dots  & 0 \\ 0 & 0 & { W }^{ 2 } & \dots  & 0 \\ \vdots  & \vdots  & \vdots  & \dots  & \vdots  \\ 0 & 0 & 0 & \dots  & { W }^{ 31 } \end{bmatrix}$$
* P is a permutation matrix

* Going down to 16 will include the following
$$ \begin{bmatrix} I & D & 0 & 0 \\ I & -D & 0 & 0 \\ 0 & 0 & I & D \\ 0 & 0 & I & -D \end{bmatrix}\begin{bmatrix} { F }_{ 16 } & 0 & 0 & 0 \\ 0 & { F }_{ 16 } & 0 & 0 \\ 0 & 0 & { F }_{ 16 } & 0 \\ 0 & 0 & 0 & { F }_{ 16 } \end{bmatrix}\left[ P \right]  $$

* The recursive work above leads to decreasing the work that is required for working with these problems

```python

```


# Week 7 - GAN Part 3 and Evaluation Metrics Notebook

In this notebook, we will solve questions on GANs and evaluation metrics of generative models in general.

 - This notebook is prepared using PyTorch. However, you can use any Python package you want to implement the necessary functions in questions.
 - If the question asks you to implement a specific function, please do not use its readily available version from a package and implement it yourself.

## Question 1

Please answer the questions below:

1. Please explain the concept of batch normalization. What problem does it solve and how?
2. What is spectral normalization and why do we use it?
3. What is the meaning of class-conditional GAN? How do we make GANs class conditional?
4. What are the main differences between StyleGAN-v1 and StyleGAN-v2?
5. Why is it so hard to quantitatively evaluate generative models?
6. What assumptions are we making on the data/model distribution when using Inception Score and Frechet Inception distance?

You can write your answer for each question in the markdown cell below:

**Please write your answer for each question here**

## Question 2

**Part 1**: Implement regular batch normalization for convolutional layers. Definition of the function and the input to test is given.

For reference, you can use the batch normalization paper given [here](https://arxiv.org/abs/1502.03167). Also, you can refer to the lecture slides.

Please note that we will implement a simple version of batch normalization, and in this simple version we will not be using *running mean*, *running variance*, and *momentum* parameters.

```python
import torch
torch.manual_seed(0)

batch_size = 16
num_channels = 12

input_tensor = torch.normal(3, 10, size=(batch_size, num_channels, 24, 24))
```

```python
def batch_normalization_conv(X, gamma, beta, eps=1e-8):
  """Performs batch normalization operation for convolutional layer output
    Args:
        X: input tensor (N x C x H x W)
        gamma: scale variable
        beta: offset variable
        eps: epsilon value
    Returns:
        the resulting tensor of the batch norm operation
  """
  #######################
  # Write code here
  #######################
  mean = X.mean(dim=(0, 2, 3), keepdim=True)
  var = ((X - mean)**2).mean(dim=(0, 2, 3), keepdim=True)

  X_hat = (X - mean) / torch.sqrt(var + eps)
  return gamma * X_hat + beta
```

```python
print("Expected Output:")
print(batch_normalization_conv(input_tensor, gamma=1, beta=0)[0,0,0,:])
```

    Expected Output:
    tensor([-1.1380, -1.1645, -0.2613, -0.4449,  0.8398,  0.6829, -0.3268, -2.1290,
             0.3125, -1.2757,  0.3403,  0.2984,  0.1098,  1.2294,  1.1083, -0.2580,
            -1.3651, -1.7090,  0.5573,  0.7845,  0.5895, -1.5679, -0.3522,  1.8458])

```python
input_tensor[0,0,0,:]
```

    tensor([ -8.2584,  -8.5236,   0.4942,  -1.3388,  11.4871,   9.9201,  -0.1601,
            -18.1522,   6.2227,  -9.6333,   6.4998,   6.0813,   4.1984,  15.3766,
             14.1678,   0.5272, -10.5265, -13.9593,   8.6665,  10.9351,   8.9884,
            -12.5510,  -0.4136,  21.5301])

**Part 2**: Implement class-conditional batch normalization for convolutional layers. You can copy-paste and modify your code from part 1 and use the same input above with the given **Y** vector below. You can refer to the lecture slides for the pseudocode.

This part is a bit tricky since we cannot directly use the class labels as inputs to a feed-forward neural network.

We therefore use the embeddings of the classes instead. We define 10-dimensional embeddings to represent our $y \in {0, 1}$ classes as float vectors.

We then randomly generate 0 and 1 values with the amount of **batch_size** and get their embeddings.

In our function, we will imitate a feed-forward neural network to implement class-conditional batch normalization, so we also define the weights and biases of this very simple perceptron as *gamma_w*, *gamma_b*, *beta_w*, and *beta_b*.

```python
import torch.nn as nn

# Assuming binary classification (binary labels)
num_classes = 2
embedding_dim = 10

# 10-dimensional embeddings for two classes: 2 x 10
class_embeddings = nn.Embedding(num_classes, embedding_dim)

# 16 random labels of 0 and 1
input_labels = torch.randint(0, 2, size=(batch_size,))

# Get class embeddings
input_label_embeddings = class_embeddings(input_labels)

gamma_w = torch.randn(embedding_dim, num_channels)
gamma_b = torch.zeros(1, num_channels)
beta_w = torch.randn(embedding_dim, num_channels)
beta_b = torch.zeros(1, num_channels)
```

```python
def cond_batch_normalization_conv(X, Y, gamma_w, gamma_b, beta_w, beta_b, eps=1e-8):
  """Performs conditional batch normalization operation for convolutional layer output
    Args:
        X: input tensor (N x C x H x W)
        Y: input labels (N x emb_dim)
        gamma_w: scale weights (emb_dim x C)
        gamma_b: scale bias (1 x C)
        beta_w: offset weights (emb_dim x C)
        beta_b: offset bias (1 x C)
        eps: epsilon value
    Returns:
        the resulting tensor of the batch norm operation
  """
  #######################
  # Write code here
  #######################
  mean = X.mean(dim=(0, 2, 3), keepdim=True)
  var = ((X - mean)**2).mean(dim=(0, 2, 3), keepdim=True)

  X_hat = (X - mean) / torch.sqrt(var + eps)

  gamma = torch.matmul(Y, gamma_w) + gamma_b
  beta = torch.matmul(Y, beta_w) + beta_b

  gamma = gamma.unsqueeze(2).unsqueeze(2)
  beta = beta.unsqueeze(2).unsqueeze(2)

  return gamma * X_hat + beta
```

```python
print("Expected Output:")
print(cond_batch_normalization_conv(input_tensor, input_label_embeddings, gamma_w, gamma_b, beta_w, beta_b)[0, 0, 0, :].data)
```

    Expected Output:
    tensor([-4.8654, -4.9883, -0.8110, -1.6601,  4.2812,  3.5554, -1.1141, -9.4485,
             1.8426, -5.5023,  1.9710,  1.7771,  0.9049,  6.0829,  5.5230, -0.7957,
            -5.9161, -7.5062,  2.9747,  4.0255,  3.1238, -6.8539, -1.2315,  8.9334])

## Question 3

Implement the adaptive instance normalization (AdaIN) from StyleGAN. You can refer to the lecture slides or the StyleGAN paper [here](https://arxiv.org/abs/1812.04948).

Adaptive instance normalization is used in StyleGAN to incorporate the *style* information to the network through combining learned affine transformations and feature maps produced by convolutions.

AdaIN operation is defined mathemtically with the following equation:

\begin{equation}
\text{AdaIN}(\mathbf{x}_i, \mathbf{y}) = \mathbf{y}_{s, i}\frac{\mathbf{x}_i - \mu(\mathbf{x}_i)}{\sigma(\mathbf{x}_i)} + \mathbf{y}_{b,i}
\end{equation}

which takes the feature map $\mathbf{x}_i$ and the style vector $\mathbf{y}$ as parameters. Essentially, the operation normalizes the feature maps, scales it with half of the style vector and shifts it with the other half. Representations $\mathbf{y}_s$ and $\mathbf{y}_b$ correspond to *shift* and *bias* and they are simply two halves of the style vector $\mathbf{y} = (\mathbf{y}_s, \mathbf{y}_b)$

```python
input_feature_map = torch.randn(batch_size, num_channels, 24, 24)
style_vector = torch.randn(batch_size, 2 * num_channels)
```

```python
def adaptive_instance_normalization(X, y, eps=1e-8):
  """Performs adaptive instance normalization on the given feature map X with the
  style input y
    Args:
      X: Feature map (N x C x W x H)
      y: Style vector (N x 2C)
    Returns:
      The resulting tensor from the operation
  """
  mean = X.mean(dim=(0, 2, 3), keepdim=True)
  var = ((X - mean)**2).mean(dim=(0, 2, 3), keepdim=True)
  X_hat = (X - mean) / torch.sqrt(var + eps)

  factor, bias = y.chunk(2, 1)

  factor = factor.unsqueeze(2).unsqueeze(2)
  bias = bias.unsqueeze(2).unsqueeze(2)

  return X_hat * factor + bias
```

```python
print(adaptive_instance_normalization(input_feature_map, style_vector)[0,0,0,:])
```

    tensor([9.5492e-01, 3.1419e-01, 3.7104e+00, 1.3578e+00, 1.9867e+00, 2.0013e+00,
            3.8199e+00, 3.2714e+00, 3.1662e-04, 2.9241e+00, 1.2787e+00, 3.0599e+00,
            3.0669e+00, 4.9342e-01, 5.8750e-01, 2.1606e+00, 2.4507e+00, 2.4924e+00,
            2.1146e+00, 2.1017e+00, 2.4752e+00, 2.3877e+00, 3.2252e+00, 4.0635e+00])

## Question 4

Implement a function that calculates the Frechet Inception Distance score from given real examples and fake examples.

You can refer to its original paper [here](https://arxiv.org/abs/1706.08500).

\begin{equation}
\text{FID} = ||\mu_1 – \mu_2||^2 + \text{Tr}(C_1 + C_2 – 2\sqrt{C_1C_2})
\end{equation}

where $\mu_1$ and $\mu_2$ are the feature-wise means of the real and generated samples, respectively. In addition, $C_1$ and $C_2$ are the covariance matrices of the real and generated samples, sometimes also referred as sigma ($\Sigma$).

```python
import torch
torch.manual_seed(0)
import torch.nn as nn
from torchvision.models import inception_v3
from torchvision.datasets import MNIST
from torchvision.transforms import Resize

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
mnist_dataset = MNIST(root=".", download=True)

# Get 200 MNIST examples
mnist_examples = mnist_dataset.data[:32].type(torch.FloatTensor)
mnist_examples /= 255

resizer = Resize(299)
mnist_examples = resizer(mnist_examples)

plt.imshow(mnist_examples[2].numpy(), interpolation='nearest', cmap='gray')

# Reshape the images to 3D to give them as inputs to the Inception network
mnist_examples = mnist_examples.unsqueeze(dim=3).repeat((1, 1, 1, 3))
```

```python
# Create 200 noisy images as the same sizes with MNIST
noisy_examples = torch.randn(32, 299, 299, 1).clip(0, 1).repeat((1, 1, 1, 3))

plt.imshow(noisy_examples[2].numpy(), interpolation='nearest', cmap='gray')
```

```python
# Download the pretrained inception v3 model
inception_model = inception_v3(pretrained=True)
# Replace the classification layer with an identity layer to get the activations
inception_model.fc = nn.Identity()
# Evaluation mode
inception_model.eval()

with torch.no_grad():
  mnist_features = inception_model(mnist_examples.permute(0, 3, 1, 2))
  noisy_features = inception_model(noisy_examples.permute(0, 3, 1, 2))
```

    Downloading: "https://download.pytorch.org/models/inception_v3_google-0cc3c7bd.pth" to /root/.cache/torch/hub/checkpoints/inception_v3_google-0cc3c7bd.pth

      0%|          | 0.00/104M [00:00<?, ?B/s]

    /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
      return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)

```python
mnist_features.shape == noisy_features.shape
```

    True

```python
from scipy.linalg import sqrtm

# Solution taken from: https://machinelearningmastery.com/how-to-implement-the-frechet-inception-distance-fid-from-scratch/
def calculate_fid(real_features, fake_features):
  """Calculates the Frechet Inception Distance of the given real and fake features
  to measure the similarity of two data distributions

  Args:
    real_features: Features taken from the real images (N x D)
    fake_features: Features taken from the fake images (N x D)
  Returns:
    A scalar number as the distance between two data distributions
  """
  # calculate mean and covariance statistics
  mu1, sigma1 = real_features.mean(axis=0), np.cov(real_features, rowvar=False)
  mu2, sigma2 = fake_features.mean(axis=0), np.cov(fake_features, rowvar=False)
  # calculate sum squared difference between means
  ssdiff = np.sum((mu1 - mu2)**2.0)
  # calculate sqrt of product between cov
  covmean = sqrtm(sigma1.dot(sigma2))
  # check and correct imaginary numbers from sqrt
  if np.iscomplexobj(covmean):
    covmean = covmean.real
  # calculate score
  fid = ssdiff + np.trace(sigma1 + sigma2 - 2.0 * covmean)
  return fid
```

```python
calculate_fid(mnist_features.numpy(), noisy_features.numpy())
```

    396.46000186190406

```python
calculate_fid(mnist_features.numpy()[:16], mnist_features.numpy()[16:])
```

    95.29507726881653

inzva x METU ImageLab Joint Program/Week 7 - GANs 3 and Evaluation Metrics/Week_7_GAN_Part_3_and_Evaluation_Metrics_Solutions.ipynb


# Derivation of SGPR equations

*James Hensman, March 2016. Corrections by Alex Matthews, December 2016*

This notebook contains a derivation of the form of the equations for the marginal likelihood bound and predictions for the sparse Gaussian process regression model in GPflow, `gpflow.models.SGPR`.

The primary reference for this work is Titsias 2009 [1], though other works (Hensman et al. 2013 [2], Matthews et al. 2016 [3]) are useful for clarifying the prediction density.

## Marginal likelihood bound
The bound on the marginal likelihood (Titsias 2009) is:

\begin{equation}
\log p(\mathbf y) \geq \log \mathcal N(\mathbf y\,|\,\mathbf 0,\, \mathbf Q_{ff} + \sigma^2 \mathbf I) - \tfrac{1}{2} \sigma^{-2}\textrm{tr}(\mathbf K_{ff} - \mathbf Q_{ff}) \triangleq \mathcal L
\end{equation}
where:
\begin{equation}
\mathbf Q_{ff} = \mathbf K_{fu}\mathbf K_{uu}^{-1}\mathbf K_{uf}
\end{equation}

The kernel matrices $\mathbf K_{ff}$, $\mathbf K_{uu}$, $\mathbf K_{fu}$ represent the kernel evaluated at the data points $\mathbf X$, the inducing input points $\mathbf Z$, and between the data and inducing points respectively. We refer to the value of the GP at the data points $\mathbf X$ as $\mathbf f$, at the inducing points $\mathbf Z$ as $\mathbf u$, and at the remainder of the function as $f^\star$.

To obtain an efficient and stable evaluation on the bound $\mathcal L$, we first apply the Woodbury identity to the effective covariance matrix:

\begin{equation}
[\mathbf Q_{ff} + \sigma^2 \mathbf I ]^{-1} = \sigma^{-2} \mathbf I - \sigma^{-4} \mathbf K_{fu}[\mathbf K_{uu} + \mathbf K_{uf}\mathbf K_{fu}\sigma^{-2}]^{-1}\mathbf K_{uf}
\end{equation}

Now, to obtain a better conditioned matrix for inversion, we rotate by $\mathbf L$, where $\mathbf L\mathbf L^\top = \mathbf K_{uu}$:

\begin{equation}
[\mathbf Q_{ff} + \sigma^2 \mathbf I ]^{-1} = \sigma^{-2} \mathbf I - \sigma^{-4} \mathbf K_{fu}\color{red}{\mathbf L^{-\top} \mathbf L^\top}[\mathbf K_{uu} + \mathbf K_{uf}K_{fu}\sigma^{-2}]^{-1}\color{red}{\mathbf L \mathbf L^{-1}}\mathbf K_{uf}
\end{equation}

This matrix is better conditioned because, for many kernels, it has eigenvalues bounded above and below. For more details, see section 3.4.3 of [Gaussian Processes for Machine Learning](http://www.gaussianprocess.org/gpml/chapters/RW.pdf).

\begin{equation}
\phantom{[\mathbf Q_{ff} + \sigma^2 \mathbf I ]^{-1}} = \sigma^{-2} \mathbf I - \sigma^{-4} \mathbf K_{fu}\color{red}{\mathbf L^{-\top}} [\color{red}{\mathbf L^{-1}}\mathbf (K_{uu} + \mathbf K_{uf}K_{fu}\sigma^{-2})\color{red}{\mathbf L^{-\top}}]^{-1}\color{red}{ \mathbf L^{-1}}\mathbf K_{uf}
\end{equation}

\begin{equation}
\phantom{[\mathbf Q_{ff} + \sigma^2 \mathbf I ]^{-1} }= \sigma^{-2} \mathbf I - \sigma^{-4} \mathbf K_{fu}\color{red}{\mathbf L^{-\top}} [\mathbf I + \color{red}{\mathbf L^{-1}}\mathbf (\mathbf K_{uf}K_{fu})\color{red}{\mathbf L^{-\top}}\sigma^{-2}]^{-1}\color{red}{ \mathbf L^{-1}}\mathbf K_{uf}
\end{equation}

For notational convenience, we'll define $\mathbf L^{-1}\mathbf K_{uf}\sigma^{-1} \triangleq \mathbf A$, and  $[\mathbf I + \mathbf A\mathbf A^\top]\triangleq \mathbf B$:

\begin{equation}
\phantom{[\mathbf Q_{ff} + \sigma^2 \mathbf I ]^{-1} }= \sigma^{-2} \mathbf I - \sigma^{-2} \mathbf A^{\top} [\mathbf I + \mathbf A\mathbf A^\top]^{-1}\mathbf A
\end{equation}

\begin{equation}
\phantom{[\mathbf Q_{ff} + \sigma^2 \mathbf I ]^{-1}}= \sigma^{-2} \mathbf I - \sigma^{-2} \mathbf A^{\top} \mathbf B^{-1}\mathbf A
\end{equation}

We also apply the [matrix determinant lemma](https://en.wikipedia.org/wiki/Matrix_determinant_lemma) to the same:

\begin{equation}
|{\mathbf Q_{ff}} + \sigma^2 {\mathbf I}| = |{\mathbf K_{uu}} +
 \mathbf K_{uf}\mathbf K_{fu}\sigma^{-2}| \, |\mathbf K_{uu}^{-1}| \, |\sigma^{2}\mathbf I|
\end{equation}

Substituting $\mathbf K_{uu} = {\mathbf {L L}^\top}$:
\begin{equation}
|{\mathbf Q_{ff}} + \sigma^2 {\mathbf I}| = |{\mathbf {L L}^\top} +
 \mathbf K_{uf}\mathbf K_{fu}\sigma^{-2}| \, |\mathbf L^{-\top}|\,| \mathbf L^{-1}| \, |\sigma^{2}\mathbf I|
\end{equation}

\begin{equation}
|{\mathbf Q_{ff}} + \sigma^2 {\mathbf I}| = |\mathbf I +
 \mathbf L^{-1}\mathbf K_{uf}\mathbf K_{fu} \mathbf L^{-\top}\sigma^{-2}| \, |\sigma^{2}\mathbf I|
\end{equation}

\begin{equation}
|{\mathbf Q_{ff}} + \sigma^2 {\mathbf I}| = |\mathbf B| \, |\sigma^{2}\mathbf I|
\end{equation}

With these two definitions, we're ready to expand the bound:

\begin{equation}
\mathcal L = \log \mathcal N(\mathbf y\,|\,\mathbf 0,\, \mathbf Q_{ff} + \sigma^2 \mathbf I) - \tfrac{1}{2} \sigma^{-2}\textrm{tr}(\mathbf K_{ff} - \mathbf Q_{ff})
\end{equation}

\begin{equation}
= -\tfrac{N}{2}\log{2\pi} -\tfrac{1}{2}\log|\mathbf Q_{ff}+\sigma^2\mathbf I| - \tfrac{1}{2}\mathbf y^\top [ \mathbf Q_{ff} + \sigma^2 \mathbf I]^{-1}\mathbf y - \tfrac{1}{2} \sigma^{-2}\textrm{tr}(\mathbf K_{ff} - \mathbf Q_{ff})
\end{equation}

\begin{equation}
= -\tfrac{N}{2}\log{2\pi} -\tfrac{1}{2}\log(|\mathbf B||\sigma^{2}\mathbf I|)  - \tfrac{1}{2}\sigma^{-2}\mathbf y^\top (\mathbf I - \sigma^{-2} \mathbf A^{\top} \mathbf B^{-1}\mathbf A)\mathbf y - \tfrac{1}{2} \sigma^{-2}\textrm{tr}(\mathbf K_{ff} - \mathbf Q_{ff})
\end{equation}

\begin{equation}
= -\tfrac{N}{2}\log{2\pi}
-\tfrac{1}{2}\log|\mathbf B|
-\tfrac{N}{2}\log\sigma^{2}
-\tfrac{1}{2}\sigma^{-2}\mathbf y^\top\mathbf y
+\tfrac{1}{2}\sigma^{-2}\mathbf y^\top\mathbf A^{\top} \mathbf B^{-1}\mathbf A\mathbf y
-\tfrac{1}{2}\sigma^{-2}\textrm{tr}(\mathbf K_{ff})
+ \tfrac{1}{2}\textrm{tr}(\mathbf {AA}^\top)
\end{equation}

where $\sigma^{-2}\textrm{tr}(\mathbf Q) = \textrm{tr}(\mathbf {AA}^\top)$.

Finally, we define $\mathbf c \triangleq \mathbf L_{\mathbf B}^{-1}\mathbf A\mathbf y \sigma^{-1}$, with $\mathbf {L_BL_B}^\top = \mathbf B$, so that:

\begin{equation}
\sigma^{-2}\mathbf y^\top\mathbf A^{\top} \mathbf B^{-1}\mathbf A\mathbf y =
\mathbf c^\top \mathbf c
\end{equation}

The `SGPR` code implements this equation with small changes for multiple concurrent outputs (columns of the data matrix Y), and also a prior mean function.

## Prediction
At prediction time, we need to compute the mean and variance of the variational approximation at some new points $\mathbf X^\star$.

Following Hensman et al. (2013), we know that all the information in the posterior approximation is contained in the Gaussian distribution $q(\mathbf u)$, which represents the distribution of function values at the inducing points $\mathbf Z$. Remember that:

\begin{equation}
q(\mathbf u) = \mathcal N(\mathbf u\,|\,  \mathbf m, \mathbf \Lambda)
\end{equation}

with:

\begin{equation}
\mathbf \Lambda = \mathbf K_{uu}^{-1} + \mathbf K_{uu}^{-1}\mathbf K_{uf}\mathbf K_{fu}\mathbf K_{uu}^{-1} \sigma^{-2}
\end{equation}

\begin{equation}
\mathbf m = \mathbf \Lambda^{-1} \mathbf K_{uu}^{-1}\mathbf K_{uf}\mathbf y\sigma^{-2}
\end{equation}

To make a prediction, we need to integrate:

\begin{equation}
p(\mathbf f^\star) = \int p(\mathbf f^\star \,|\, \mathbf u) q (\mathbf u) \textrm d \mathbf u
\end{equation}

with:

\begin{equation}
p(\mathbf f^\star \,|\, \mathbf u) = \mathcal N(\mathbf f^\star\,|\, \mathbf K_{\star u}\mathbf K_{uu}^{-1}\mathbf u, \,\mathbf K_{\star\star} - \mathbf K_{\star u}\mathbf K_{uu}^{-1}\mathbf K_{u\star})
\end{equation}

The integral results in:

\begin{equation}
p(\mathbf f^\star) = \mathcal N(\mathbf f^\star\,|\, \mathbf K_{\star u}\mathbf K_{uu}^{-1}\mathbf m, \,\mathbf K_{\star\star} - \mathbf K_{\star u}\mathbf K_{uu}^{-1}\mathbf K_{u\star} + \mathbf K_{\star u}\mathbf K_{uu}^{-1}\mathbf \Lambda \mathbf K_{uu}^{-1}\mathbf K_{u\star})
\end{equation}

Note from our above definitions we have:

\begin{equation}
\mathbf K_{uu}^{-1}\mathbf \Lambda \mathbf K_{uu}^{-1} =
\mathbf L^{-\top}\mathbf B^{-1}\mathbf L^{-1}
\end{equation}

and further:

\begin{equation}
\mathbf K_{uu}^{-1}\mathbf m = \mathbf L^{-\top}\mathbf L_{\mathbf B}^{-\top}\mathbf c
\end{equation}

substituting:

\begin{equation}
p(\mathbf f^\star) = \mathcal N(\mathbf f^\star\,|\, \mathbf K_{\star u}\mathbf L^{-\top}\mathbf L_{\mathbf B}^{-\top}\mathbf c, \,\mathbf K_{\star\star} - \mathbf K_{\star u}\mathbf L^{-1}(\mathbf I - \mathbf B^{-1})\mathbf L^{-1}\mathbf K_{u\star})
\end{equation}

The code in `SGPR` implements this equation, with an additional switch depending on whether the full covariance matrix is required.

## References

[1] Titsias, M: Variational Learning of Inducing Variables in Sparse Gaussian Processes, PMLR 5:567-574, 2009

[2] Hensman et al: Gaussian Processes for Big Data, UAI, 2013

[3] Matthews et al: On Sparse Variational Methods and the Kullback-Leibler Divergence between Stochastic Processes, AISTATS, 2016


# Introducción a `SymPy`

Esta introducción está basada en las [transparencias de SymPy](https://minireference.com/static/tutorials/sympy_tutorial.pdf) de [J. Gross](http://www.jonathangross.de)

Véase también [Taming math and physics using SymPy](https://minireference.com/static/tutorials/sympy_tutorial.pdf), basado en [No bullshit guide to Math and Physics](https://minireference.com)

Una forma fácil de usar `SymPy` en línea es en la página de [SymPy Live](http://live.sympy.org) o bien utilizando los servicios de [sage en la nube](https://cloud.sagemath.com)

# Vamos a aprender a guarrear celdas

## Primeros pasos

### Listas

1. Una cosa

3. Se me olvidó

1. Otra cosa

Otro lista

- uno

- dos

### Fórmulas

Una ecuación en línea $\begin{pmatrix} 1 & 2 \\ 3 & 4 \end{pmatrix}^2+1=0$, y centrada $$\int x^2+1 \mathrm d x$$ en $\mathbb R$ y un ideal $\mathfrak u$

Para que los gráficos se incrusten en este bloc necesitamos usar el siguiente comando

```python
%matplotlib inline
```

Con las siguientes instrucciones importamos todas las variables y métodos de `SymPy`(en algunos casos sólo importaremos los métodos y variables que nos interesen)

También se puede ejecutar en línea con `isympy`

```python
~$ isympy
IPython console for SymPy 0.7.6.1 (Python 3.5.1-64-bit) (ground types: python)

These commands were executed:
>>> from __future__ import division
>>> from sympy import *
>>> x, y, z, t = symbols('x y z t')
>>> k, m, n = symbols('k m n', integer=True)
>>> f, g, h = symbols('f g h', cls=Function)
>>> init_printing()

Documentation can be found at http://www.sympy.org

In [1]:
```

A partir de `Python 3` la divisón se denota por `/` y la división entera por `//`; de ahí que se use
```
from __future__ import division
```
para evitar confusiones si usamos `Python 2`

```python
from sympy import *
```

```python
init_printing(use_latex=true)
```

## Enteros, racionales, reales y complejos

División entera

```python
1//7
```

División real

```python
1/7
```

```python
type(1/7)
```

    float

```python
r=S("1/7")
s=S("2/3")
```

```python
r+s
```

```python
type(_)
```

    sympy.core.numbers.Rational

```python
2**50
```

Si queremos trabajar con un racional $a/b$, podemos "sympificar" la expresión "a/b"

```python
a=S("1/7")
```

```python
a.evalf(20)
```

```python
a==Rational(1,7)
```

    True

```python
float(a)
```

```python
a.n()
```

```python
1+2*I
```

```python
(I)**4
```

```python
pi.n(150)
```

### Símbolos

Si intentamos evaluar alguna expresión con `x` o `y` dará un error de nombre no definido; tenemos primero que declararlos como símbolos

Esto lo hacemos con `Symbol` para una variable, o `symbols` para varios

```python
x,y =symbols("x, y")
```

```python
p = (x+y)**10
```

```python
p
```

```python
expand(p)
```

```python
factor(x**2-y**2)
```

```python
_.args
```

```python
simplify(x** 2+2*x+y-2*x)
```

```python
simplify(cos(x)**2+sin(x)**2)
```

```python
collect(x**2+2*x*y+2*x,x)
```

```python
(ln(x)+y).subs({x:2,y:1})
```

```python
_.n()
```

```python
a,b,c,d = symbols("a b c d")
```

```python
together(a/b+c/d)
```

```python
apart((a**2+1)/a)
```

```python
sin(x)==cos(x-pi/2)
```

    True

```python
expand(sin(2*x))
```

```python
expand_trig(sin(2*x))
```

```python
trigsimp(_)
```

## Resolución de ecuaciones

```python
solve(x**2+2*x-1)
```

```python
l=solve(a*x**2+b*x+c,x)
```

```python
solve(a*x**2+b*x+c,x)
```

```python
l[1].subs({a:1,b:2,c:-1})
```

```python
solve([x**2+y**2-1, y-x**2],[x,y])
```

```python
plot_implicit(Eq(x**2+y**2-1,0) | Eq(x**2-y,0),(x,-2,2),(y,-2,2))
```

## Límites

El infinito se denota por `oo` (dos oes en minúscula seguidos)

```python
oo+1
```

```python
n=Symbol("n", integer=true)
```

```python
limit((1+1/n)**n, n, oo)
```

```python
limit(1/x,x,0,dir="-")
```

```python
limit(1/x,x,0,dir="+")
```

```python
plot((1/x,(x,.1,10)),(1/x,(x,-10,-.1)))
```

```python
a_n=1/n
```

```python
limit(1+a_n,n,oo)
```

```python
[a_n.subs({n:i}) for i in range(1,10)]
```

## Derivadas

```python
diff((x**3*sin(x**2))/cos(x))
```

```python
diff((x**3*sin(x**2))/cos(x),y)
```

```python
f=symbols("f", cls=Function)
```

```python
dsolve(f(x)-diff(f(x),x),f(x))
```

## Integrales

```python
integrate(cos(x))
```

```python
integrate(cos(x),(x,-1,1))
```

## Series

```python
a_n = 1/n
```

```python
summation(a_n,(n,1,oo))
```

```python
b_n=1/factorial(n)
```

```python
summation(b_n,(n,0,oo))
```

```python
exp = x**n/factorial(n)
```

```python
summation(exp.subs({x:1}),(n,0,10))
```

```python
_.n()
```

```python
summation(exp.subs({x:5}),(n,0,oo))
```

Series de Taylor (en 0, MacLaurin)

```python
series(sin(x),x,0,8)
```

O bien en el 1

```python
series(sin(x+1),x,0,8)
```

## Matrices (y vectores)

Las matrices se introducen for filas como una lista de listas con el comando `Matrix`
Los vectores se consideran matrices con una sola fila

```python
a=Matrix([[1,1,2],[1,-1,0]])
```

```python
a
```

$$\left[\begin{matrix}1 & 1 & 2\\1 & -1 & 0\end{matrix}\right]$$

```python
a=a[0:2,0:2]
```

```python
a.det()
```

```python
a.transpose()
```

$$\left[\begin{matrix}1 & 1\\1 & -1\end{matrix}\right]$$

```python
pretty_print(a)
```

    ⎡1  1 ⎤
    ⎢     ⎥
    ⎣1  -1⎦

```python
latex(a)
```

    '\\left[\\begin{matrix}1 & 1\\\\1 & -1\\end{matrix}\\right]'

```python
a[0,1]
```

```python
a[0,0]
```

```python
a.inv()
```

$$\left[\begin{matrix}\frac{1}{2} & \frac{1}{2}\\\frac{1}{2} & - \frac{1}{2}\end{matrix}\right]$$

```python
a.inv()*a
```

$$\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right]$$

```python
a.row(1)
```

$$\left[\begin{matrix}1 & -1\end{matrix}\right]$$

```python
a.rref()
```

$$\left ( \left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right], \quad \left [ 0, \quad 1\right ]\right )$$

Valores propios, vectores propios y diagonalización

```python
a.eigenvals()
```

```python
a.eigenvects()
```

$$\left [ \left ( - \sqrt{2}, \quad 1, \quad \left [ \left[\begin{matrix}- \frac{1}{1 + \sqrt{2}}\\1\end{matrix}\right]\right ]\right ), \quad \left ( \sqrt{2}, \quad 1, \quad \left [ \left[\begin{matrix}- \frac{1}{- \sqrt{2} + 1}\\1\end{matrix}\right]\right ]\right )\right ]$$

```python
(a-sqrt(2)*eye(2)).nullspace()
```

$$\left [ \left[\begin{matrix}- \frac{1}{- \sqrt{2} + 1}\\1\end{matrix}\right]\right ]$$

```python
Q,D=a.diagonalize()
```

```python
D
```

$$\left[\begin{matrix}- \sqrt{2} & 0\\0 & \sqrt{2}\end{matrix}\right]$$

```python
Q
```

$$\left[\begin{matrix}- \frac{1}{1 + \sqrt{2}} & - \frac{1}{- \sqrt{2} + 1}\\1 & 1\end{matrix}\right]$$

```python
(Q*D*Q.inv())
```

$$\left[\begin{matrix}\frac{\sqrt{2}}{1 + \sqrt{2}} \left(\frac{1}{- \sqrt{2} + 1} - \frac{1}{\left(- \sqrt{2} + 1\right)^{3} \left(2 \sqrt{2} + 4\right)}\right) + \frac{\sqrt{2}}{\left(- \sqrt{2} + 1\right)^{2} \left(2 \sqrt{2} + 4\right)} & - \frac{\sqrt{2}}{\left(- \sqrt{2} + 1\right) \left(2 \sqrt{2} + 4\right)} + \frac{\sqrt{2}}{\left(1 + \sqrt{2}\right) \left(- \sqrt{2} + 1\right)^{2} \left(2 \sqrt{2} + 4\right)}\\- \frac{\sqrt{2}}{\left(- \sqrt{2} + 1\right) \left(2 \sqrt{2} + 4\right)} - \sqrt{2} \left(\frac{1}{- \sqrt{2} + 1} - \frac{1}{\left(- \sqrt{2} + 1\right)^{3} \left(2 \sqrt{2} + 4\right)}\right) & - \frac{\sqrt{2}}{\left(- \sqrt{2} + 1\right)^{2} \left(2 \sqrt{2} + 4\right)} + \frac{\sqrt{2}}{2 \sqrt{2} + 4}\end{matrix}\right]$$

```python
simplify(_)
```

$$\left[\begin{matrix}1 & 1\\1 & -1\end{matrix}\right]$$

```python
b=Matrix([[1,3],[0,1]])
```

```python
b.diagonalize()
```

```python
b.eigenvects()
```

$$\left [ \left ( 1, \quad 2, \quad \left [ \left[\begin{matrix}1\\0\end{matrix}\right]\right ]\right )\right ]$$

```python
b.jordan_form()
```

$$\left ( \left[\begin{matrix}3 & 0\\0 & 1\end{matrix}\right], \quad \left[\begin{matrix}1 & 1\\0 & 1\end{matrix}\right]\right )$$

La matriz indentidad tiene un función propia

```python
eye(3)
```

$$\left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\end{matrix}\right]$$

Vamos a discutir el siguiente sistema en función del parámetro $a$.

$$ \begin{array}{c} x+y+z=a,\\  2x+ay+z=-2,\\  -2x-2y+az=4.  \end{array} $$

```python
x,y,z,a=symbols("x y z a")
C=Matrix([[1,1,1],[2,a,1],[-2,-2,a]])
b=Matrix([a,-2,4])
```

```python
sol=solve(C.det())
```

```python
sol
```

```python
Ca=C.col_insert(-1,b)
```

```python
C.subs({a:2}).rank()
```

```python
Ca.subs({a:2}).rank()
```

```python
linsolve((C.subs({a:2}),b.subs({a:2})),[x,y,z])
```

```python
C.subs({a:-2}).rank()
```

```python
Ca.subs({a:-2}).rank()
```

```python
linsolve((C.subs({a:-2}),b.subs({a:-2})),[x,y,z])
```

```python
linsolve((C,b),[x,y,z])
```


# What would today's Asian American population be if Japanese Americans were deported rather than interned in 1942?

Colin Takeda and Nathan Weil
ModSim Project 1

```python
# Configure Jupyter so figures appear in the notebook
%matplotlib inline

# Configure Jupyter to display the assigned value after an assignment
%config InteractiveShell.ast_node_interactivity='last_expr_or_assign'

# import functions from the modsim library
from modsim import *

# import everything from SymPy.
from sympy import *

# Set up Jupyter notebook to display math.
init_printing()

from pandas import read_html

import csv
```

## Question

What would today's Asian American population be if Japanese Americans were deported rather than interned in 1942?

What would be today's percent population of Asian Americans be if all other races increased naturally as well?

## Model

Inputs:
- Asian American population in 1942 - total number of Japanese Americans interned
- Populations of other races in 1942

Data:
- Populations of other races over time

Actions:
- Simulate change in population over time per race and plot

Metrics:
- Growth over time of each subset of population
- Percent change for Asian Americans over time

```python
'''This section is for Data input
the idea is to take a .csv file from excel and import the data that way
'''

with open('data/racedemographics.csv', newline='') as csvfile:
    data = csv.reader(csvfile, delimiter=' ', quotechar='|')
    for row in data:
        print(', '.join(row))

data_time = [1940, 1950, 1960, 1970, 1980, 1990, 2000, 2010]

# Population Data
api = [254918, 321033, 980337, 1526401, 3500439, 7273662, 10641833, 15214265]
wh = []
bl = []
aiea = []
his = []
nhw = []
```

```python
def plot_results(api, data_time, title):

    plot(api, data_time)

    decorate(xlabel='Year',
             ylabel='Population',
             title=title)
```

```python
'''Data imported and made into var to pass to plot'''
empty = TimeSeries()
plot_results(api, data_time, 'World population estimates')
```

## Results

We expect a much smaller Asian American percentage of the population over time if the interned Japanese Americans had ben deported. This data can be verified by projecting it to the 2010 census data.

## Interpretation

plot(results)