# Spice Up Thy Jupyter Notebooks With mypy

I use [Jupyter](http://jupyter.org/) notebooks all the time. I use them to author the content for articles like the one you're reading. I use them to explore data. And I use them whenever I want to prototype standalone modules or learn new programming concepts, since notebooks allow us to quickly iterate on new ideas and provide powerful ways introspect, profile, and debug code - and more.

I've also recently become interested in Python's new typing module and mypy, and I wanted to be able to use them seamlessly in the context of my Jupyter notebooks.

Before we continue, I should mention that Jupyter is an evolution of the original IPython project. I bring this up because I'm going to reference the documentation for IPython, which can be thought of as the the code that enables Python to work with Jupyter. Think of Jupyter as the tooling that allows for us to do all the fancy code editing and visualization browser, and IPython as the compatibility layer between Jupyter and Python. This is a bit simplistic, but hopefully avoids confusion for anyone who isn't intimately familiar with both projects and the genesis of the Jupyter project.

Jupyter has a concept of cell "magics" which allow us to perform helpful functions in the context of a Jupyter shell. For example, when using IPython, the `%pastebin` magic allows us to easily upload bits of code to Github's Gist pastebin, `%pdoc [object]` prints the docstring for a particular object, and `%%latex` renders a block of text as latex.

Magics that start with  '%' apply only to the line that it is written on, while
magics that start with '%%' apply to a block of code proceeding that line.

Jupyter allows us to write our own custom magics, apart from the built-in magics provided. Detailed documentation on how to do that can be found [here](http://ipython.readthedocs.io/en/stable/config/custommagics.html).

Since the custom magics you use depend on how you have IPython configured and what profile you're using, I'm providing the following code in the form of a gist, since distributing it as a package would be finicky as the package would need to attempt to figure out your IPython configuration, a task which seems best left to you, the human.

I'm going to assume that you have Jupyter installed and that you have at least a default profile created. If not `pip install -U jupyter` will install Jupyter and `ipython profile create` will create a default profile for you if you don't yet have one. This will allow you to use custom magics and configure the way IPython works in various other ways.

For example, my IPython configuration for my default profile, 'ipython_config.py' has the following lines

    c.TerminalInteractiveShell.ast_node_interactivity = 'all'
    c.InteractiveShell.ast_node_interactivity = 'all'

which tells IPython to print the the repr for any object which has it's own line.

For example:

```python
foo = 'bar'
bar = 'baz'

foo

bar # normally, only this object is printed
```

    'bar'

    'baz'

The configuration for your current IPython profile can be found using `ipython locate profile`

I wanted my magic to do 4 things, primarily.

1. Run the block of code through mypy and print its output
2. Allow me to alter mypy's behavior, as with the mypy cli
2. Not conflict with the normal behavior of the block of code
3. Run the block of code in the context of my current profile

Though the code it took to do this is really straightforward, it took some digging around in the Jupyter and IPython docs to get it to work as I wanted. Hopefully you find it useful and elucidating.

Without further ado, {{<gist knowsuchagency f7b2203dd613756a45f816d6809f01a6 >}}

```python
%%typecheck --ignore-missing-imports

from sympy import sympify, init_printing
init_printing()

integer: int

integer = "mypy won't like this"

sympify('(e**3)**(1/2)')

"Notice the previous line was also printed out"
```

    <string>:8: error: Incompatible types in assignment (expression has type "str", variable has type "int")

$$\sqrt{e^{3}}$$

    'Notice the previous line was also printed out'

```python

```


```python
from sympy.matrices import Matrix
import sympy as sp
import numpy as np
from Exercise import Exercise, MarkdownBlock

from process_latex import process_sympy

try:
    from config import URL, TOKEN
except:
    None

# TODO: replace with supplied strings
Exercise.URL = URL
Exercise.TOKEN = TOKEN
```

```python
def hello2():
    return "2"
x = None

s ="""
def hello(name: str):
    print(x)
    return f"Hello"
print(hello("x"))
"""

# import inspect
# lines = inspect.getsource(hello)
# print(lines)
# exec?

# Note: if any helper functions are used, these should be available in the serverless function file as well!

exec(s)
```

    None
    Hello

```python

```

```python

```


Dependencies:
 - whoosh
 - yattag
 - hurry.filesize

```python
import nbformat
```

nbformat doc: http://nbformat.readthedocs.org/en/latest/api.html

```python
from whoosh.index import create_in
from whoosh.fields import *
from whoosh.qparser import QueryParser
```

whoosh doc: http://whoosh.readthedocs.org/en/latest/quickstart.html

```python
import os, fnmatch
import stat
import datetime
```

```python
schema = Schema(
    title=TEXT(stored=True),
    markdown=TEXT(stored=True),
    code=TEXT(stored=True),
    path=ID(stored=True),
    user=KEYWORD(stored=True),
    tags=KEYWORD(stored=True),
    modified=DATETIME(stored=True),
    accessed=DATETIME(stored=True),
    size=NUMERIC(stored=True)
)
```

```python
ix = create_in("indexdir", schema)
writer = ix.writer()
```

```python
def get_file_info(path):
    info = dict()
    try:
        st = os.stat(path)
        info['uid'] = st.st_uid
        info['gid'] = st.st_gid
        info['size'] = st.st_size
        info['atime'] = st.st_atime
        info['mtime'] = st.st_mtime
        info['ctime'] = st.st_ctime
    except IOError:
        print("Failed to get information about", file)
    else:
        try:
            import pwd # not available on all platforms
            userinfo = pwd.getpwuid(st[stat.ST_UID])
        except (ImportError, KeyError):
            print("Failed to get the owner name for", file)
        else:
            info['owner'] = userinfo.pw_name
            info['complete_owner'] = userinfo.pw_gecos
    return info
```

```python
def add_notebook(writer, path):
    info = get_file_info(path)
    note = nbformat.read(path, nbformat.NO_CONVERT)

    markdown = ''
    code = ''
    tags = ''

    for cell in note['cells']:
        if cell['cell_type'] == 'markdown':
            markdown += cell['source']
            markdown += '\n\n'
        if cell['cell_type'] == 'code':
            code += cell['source']
            code += '\n\n'

    writer.add_document(
        title=''.join(path.split('/')[-1].split('.')[:-1]),
        path=path,
        markdown=markdown,
        code=code,
        user=info['owner'],
        tags=tags,
        modified=datetime.datetime.fromtimestamp(info['mtime']),
        accessed=datetime.datetime.fromtimestamp(info['atime']),
        size=info['size']
    )
```

```python
def find(path, pattern, antipattern):
    result = []
    for root, dirs, files in os.walk(path):
        for name in files:
            if fnmatch.fnmatch(name, pattern) and not fnmatch.fnmatch(name, antipattern):
                result.append(os.path.join(root, name))
    return result
file_path = find('/Users/lukas/Projects', '*.ipynb', '*.ipynb_checkpoints')
```

```python
for path in file_path:
    add_notebook(writer, path)
```

```python
writer.commit()
```

```python
from IPython.html import widgets
from IPython.display import display, clear_output
from IPython.core.display import HTML
from yattag import Doc
from hurry.filesize import size
```

```python
display(HTML('''
<style>
.rendered_html tr, .rendered_html th, .rendered_html td {
border-collapse: collapse;
margin: 1em 2em;
}
.rendered_html table {
margin-left: auto;
margin-right: auto;
border: none;
border-collapse: collapse;
}
</style>
'''
))
```

widget discussion: http://stackoverflow.com/questions/26352555/how-does-one-get-widget-values-with-a-button-in-ipython

```python
def format_output(results):
    doc, tag, text = Doc().tagtext()
    doc.text('Number of hits: {}\n'.format(len(results)))
    with tag('table', klass='table table-striped'):
        with tag('thead'):
            with tag('tr'):
                with tag('th'):
                    doc.text('#')
                with tag('th'):
                    doc.text('Title')
        with tag('tbody'):
            for idx, result in enumerate(results):
                with tag('tr'):
                    with tag('td'):
                        doc.text(idx)
                    with tag('td'):
                        with tag('table'):
                            with tag('tbody'):
                                with tag('tr'):
                                    with tag('td', klass='col-md-6'):
                                        with tag('a', href=result['path']):
                                            doc.text(result['title'])
                                    with tag('td', klass='col-md-6'):
                                        doc.text(' asdf({})'.format(result['user']))
    display(HTML(doc.getvalue()))
```

```python
import whoosh.highlight as highlight
class BracketFormatter(highlight.Formatter):
    def format_token(self, text, token, replace=False):
        # Use the get_text function to get the text corresponding to the
        # token
        tokentext = highlight.get_text(text, token, replace=True)

        # Return the text as you want it to appear in the highlighted
        # string
        return "<mark>%s</mark>" % tokentext
```

```python
def format_output(results):
    doc, tag, text = Doc().tagtext()
    doc.text('Number of hits: {}\n'.format(len(results)))
    for idx, result in enumerate(results):
        with tag('div', klass='row'):
            with tag('div', klass='col-md-1'):
                doc.text(idx)
            with tag('div', klass='col-md-11'):
                with tag('strong'):
                    doc.text(result['title'])
        with tag('div', klass='row'):
            with tag('div', klass='col-md-1'):
                pass
            with tag('div', klass='col-md-11'):
                with tag('a'):
                    doc.text(result['path'])
        with tag('div', klass='row'):
            with tag('div', klass='col-md-1'):
                pass
            with tag('div', klass='col-md-2'):
                doc.text('User: {}'.format(result['user']))
            with tag('div', klass='col-md-2'):
                doc.text('Size: {}B'.format(size(result['size'])))
            with tag('div', klass='col-md-7'):
                doc.text('Modified: {}'.format(result['modified']))
        with tag('div', klass='row'):
            with tag('div', klass='col-md-1'):
                pass
            with tag('div', klass='col-md-11'):
                doc.asis(result.highlights('markdown'))
        with tag('br'):
            pass
    display(HTML(doc.getvalue()))
```

```python
def search(query_string, field='markdown'):
    limit = 200
    with ix.searcher() as searcher:
        query = QueryParser(field, ix.schema).parse(query_string)
        results = searcher.search(query, limit=limit)
        brf = BracketFormatter()
        results.formatter = brf
        results.fragmenter.maxchars = 300
        results.fragmenter.surround = 50
        format_output(results)
```

```python
def button_callback(btn):
    clear_output()
    search(query_string=container.children[0].value)

button = widgets.ButtonWidget(description="Click me!")
button.on_click(button_callback)
text_box = widgets.Text(value='exa*')
container = widgets.HBox(children=(text_box, button))
display(container)
```

Number of hits: 7

0

2015-09-01 Complex derivatives

/Users/lukas/Projects/notes/2015-09-01 Complex derivatives.ipynb

User: lukas

Size: 4KB

Modified: 2015-09-02 08:32:30

are satisfied, the above equations yield an exact derviative.

If not, the above equations contradict each...when far away from that critical point?

## Simple example

\begin{align}
f(z) &= |z|^2 &
u(x, y) &= x^2 + y^2 &
v(x, y) &= 0 &
\frac{\partial u}{\partial...magnitudes and to compare phases. These are examples:
 - Exponential distribution to compare magnitudes

1

2015-10-01 First complex Chainer tests

/Users/lukas/Projects/notes/2015-10-01 First complex Chainer tests.ipynb

User: lukas

Size: 105KB

Modified: 2015-10-02 10:44:27

example functions to generate training data

# Define network architecture

# Training

Until now, just works with vanilla stochastic

2

2015-09-01 Complex derivatives-checkpoint

/Users/lukas/Projects/notes/.ipynb_checkpoints/2015-09-01 Complex derivatives-checkpoint.ipynb

User: lukas

Size: 4KB

Modified: 2015-09-02 08:32:30

are satisfied, the above equations yield an exact derviative.

If not, the above equations contradict each...when far away from that critical point?

## Simple example

\begin{align}
f(z) &= |z|^2 &
u(x, y) &= x^2 + y^2 &
v(x, y) &= 0 &
\frac{\partial u}{\partial...magnitudes and to compare phases. These are examples:
 - Exponential distribution to compare magnitudes

3

2015-10-01 First complex Chainer tests-checkpoint

/Users/lukas/Projects/notes/.ipynb_checkpoints/2015-10-01 First complex Chainer tests-checkpoint.ipynb

User: lukas

Size: 105KB

Modified: 2015-10-02 10:44:27

example functions to generate training data

# Define network architecture

# Training

Until now, just works with vanilla stochastic

4

Einsum

/Users/lukas/Projects/nt_notebooks/basics/Einsum.ipynb

User: lukas

Size: 11KB

Modified: 2015-10-02 10:43:56

Einsum examples

- There are some examples here: http://docs.scipy.org/doc/numpy/reference/generated

5

MNIST_Latent_Autoencoder

/Users/lukas/Projects/nt_notebooks/toolbox_examples/nn/MNIST_Latent_Autoencoder.ipynb

User: lukas

Size: 604KB

Modified: 2015-10-02 10:43:56

factors of variation in deep networks

This is an example of how to implement the autoencoder architecture from...and output data
- plot cost evolution and one test example

## Define function to plot input images, output images

6

MNIST_Latent_Autoencoder

/Users/lukas/Projects/toolbox/nt/nn/examples/MNIST_Latent_Autoencoder.ipynb

User: lukas

Size: 18KB

Modified: 2015-08-20 08:46:05

factors of variation in deep networks

This is an example of how to implement the autoencoder architecture from...and output data
- plot cost evolution and one test example

## Define function to plot input images, output images

```python

```


## Initial setup

```python
from google.colab import drive
drive.mount('/content/drive')
```

    Mounted at /content/drive

```python
import tensorflow as tf
print(tf.__version__)
import torch
print(torch.__version__)
import matplotlib
print(matplotlib.__version__)
```

    2.8.0
    1.10.0+cu111
    3.2.2

```python
!nvidia-smi
```

    Fri Jan 21 19:41:32 2022
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 495.46       Driver Version: 460.32.03    CUDA Version: 11.2     |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |                               |                      |               MIG M. |
    |===============================+======================+======================|
    |   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
    | N/A   34C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
    |                               |                      |                  N/A |
    +-------------------------------+----------------------+----------------------+

    +-----------------------------------------------------------------------------+
    | Processes:                                                                  |
    |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
    |        ID   ID                                                   Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+

```python
# Other imports
! pip install tensorflow_addons
! pip install tensorflow_io

import os
from tensorflow.keras.layers import *
from tensorflow.keras.models import *
from keras.callbacks import Callback, EarlyStopping, ModelCheckpoint
from tensorflow.keras.applications.resnet50 import preprocess_input
from tensorflow.keras.preprocessing.image import load_img

import matplotlib.pyplot as plt
import matplotlib.cm as cm
from matplotlib.offsetbox import OffsetImage, AnnotationBbox
from matplotlib.ticker import MultipleLocator, FormatStrFormatter, AutoMinorLocator
from imutils import paths
from tqdm import tqdm
import tensorflow as tf
import tensorflow_addons as tfa
import tensorflow_datasets as tfds
import tensorflow_io as tfio
import tensorflow_hub as hub
import numpy as np
import cv2
import pandas as pd
import seaborn as sns
from scipy.stats import mannwhitneyu
from sklearn.preprocessing import LabelEncoder
from sklearn.cluster import KMeans
import sklearn.manifold
from sklearn.metrics.pairwise import cosine_similarity as cos
from sympy.utilities.iterables import multiset_permutations
from sklearn.metrics import accuracy_score, f1_score,precision_score, recall_score, roc_auc_score, confusion_matrix
from sklearn.model_selection import *
from sklearn.preprocessing import StandardScaler
from IPython.display import Image, display

import zipfile
import concurrent.futures

# Random seed fix
random_seed = 42
tf.random.set_seed(random_seed)
np.random.seed(random_seed)
```

    Collecting tensorflow_addons
      Downloading tensorflow_addons-0.16.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
         |████████████████████████████████| 1.1 MB 6.4 MB/s
    Requirement already satisfied: typeguard>=2.7 in /usr/local/lib/python3.7/dist-packages (from tensorflow_addons) (2.7.1)
    Installing collected packages: tensorflow-addons
    Successfully installed tensorflow-addons-0.16.1
    Collecting tensorflow_io
      Downloading tensorflow_io-0.24.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (23.4 MB)
         |████████████████████████████████| 23.4 MB 1.8 MB/s
    Requirement already satisfied: tensorflow-io-gcs-filesystem==0.24.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_io) (0.24.0)
    Installing collected packages: tensorflow-io
    Successfully installed tensorflow-io-0.24.0

## Dataset gathering and preparation

```python
%cd /content/drive/MyDrive/nanowire-morphology-classification-project
```

```python
training_batch_size = 4

BATCH_SIZE = training_batch_size

imageSize = 224

category_names = ['bundle', 'dispersed', 'network', 'singular']
color_method = ['C0', 'C1', 'C2', 'C3', 'C4']
color = ['black', 'magenta', 'cyan', 'yellow']
marker = ['o', 's', '<', '>', '^']
seaborn_palette = sns.color_palette("colorblind")
```

```python
# Image preprocessing utils
@tf.function
def parse_images(image_path):
    image_string = tf.io.read_file(image_path)
    image = tf.image.decode_jpeg(image_string, channels=3)
    # image = tfio.experimental.image.decode_tiff(image_string)[:, :, :3]   # in the doc, it transforms tiff to 4 channels, with additional channel of opacity which is not needed.
    image = tf.image.convert_image_dtype(image, tf.float32)
    image = tf.image.resize(image, size=[imageSize, imageSize])

    return image
```

## clustering and percolation analysis of the segmentation ground truth

```python
np.random.seed(random_seed)
peptide_morph_seglabel_train_path = "/content/drive/MyDrive/TEM image datasets/2022-nanowire-morphology"
peptide_morph_images_train_seglabel = list(paths.list_files(basePath=peptide_morph_seglabel_train_path, validExts='png'))
peptide_morph_images_train_seglabel = np.random.choice(np.array(peptide_morph_images_train_seglabel), len(peptide_morph_images_train_seglabel), replace=False)
print(len(peptide_morph_images_train_seglabel))
```

    200

```python
def generate_ground_truth_images(image, resolution):
  image_bool = np.ones((resolution, resolution))
  for i in range(image.shape[0]):
    for j in range(image.shape[1]):
      if image[i, j, 1] == image[i, j, 2]:
        image_bool[i, j] = 0            # background is black with code of 0
      else:
        image_bool[i, j] = 1            # nanowire is white with code of 1
  return image_bool
```

```python
segmentation_class_labels = []
for i in range(peptide_morph_images_train_seglabel.shape[0]):
  seg_class_label = peptide_morph_images_train_seglabel[i].split("/")[-2]
  segmentation_class_labels.append(seg_class_label)
le = LabelEncoder()
peptide_morph_train_seg_enc = le.fit_transform(segmentation_class_labels)
```

```python
# image_mask = np.zeros((len(peptide_morph_images_train_seglabel), imageSize, imageSize))
images_no_annotation_directory = []
images_no_annotation = np.zeros((len(peptide_morph_images_train_seglabel), imageSize, imageSize, 3))

for i in range(len(peptide_morph_images_train_seglabel)):
  images_no_annotation_directory.append('%s.jpg' % (peptide_morph_images_train_seglabel[i].split("_")[0] + "_" + peptide_morph_images_train_seglabel[i].split("_")[1]))
  images_no_annotation[i] = parse_images(images_no_annotation_directory[i])

  # these were used to create the ground truth grayscale images from the manual segmentation labels.
  # image_string = tf.io.read_file(peptide_morph_images_train_seglabel[i])
  # image = tf.image.decode_image(image_string, channels=3) / 255
  # image = tf.image.resize(image, (imageSize, imageSize))
  # image = tf.image.convert_image_dtype(image, tf.float32)
  # trans_nd_image_array = image.numpy()
  # image_mask[i] = generate_ground_truth_images(trans_nd_image_array, imageSize)

# np.savez_compressed('seg_mask_res512.npz', mask=image_mask)

# once we have the seg_mask saved, we can directly load from npz file
image_mask = np.load('seg_mask_res%i.npz' % (imageSize), allow_pickle=True)['mask']
```

    WARNING:tensorflow:5 out of the last 5 calls to <function parse_images at 0x7f9490054b90> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.
    WARNING:tensorflow:6 out of the last 6 calls to <function parse_images at 0x7f9490054b90> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

```python
# quantify percolation or not for each segmentation mask from the two morphologies
from sklearn.cluster import DBSCAN
nanowire_perc = np.zeros((image_mask.shape[0]))
for k in range(image_mask.shape[0]):
  nanowire_pix = np.transpose(np.vstack((np.where(image_mask[k] == 1)[0], np.where(image_mask[k] == 1)[1])))
  cluster = DBSCAN(eps=2, min_samples=2, n_jobs=-1).fit(nanowire_pix)
  cluster_size = np.zeros(np.unique(cluster.labels_).shape)
  labels = cluster.labels_

  for i in np.unique(cluster.labels_):
    if i == -1:            # i == -1 is the default label for points not in any cluster, these are excluded, as they are likely artifacts during resizing the image to lower resolution
      labels = np.delete(labels, np.where(labels == -1)[0])
      continue             # because the segmentation image masks practically don't have single pixels, i.e. they should have close neighbors because they are hand drawn

    cluster_size[i] = np.where(labels == i)[0].shape[0]
    right = np.max(cluster.components_[np.where(labels == i)[0], 0])
    top = np.max(cluster.components_[np.where(labels == i)[0], 1])
    left = np.min(cluster.components_[np.where(labels == i)[0], 0])
    bottom = np.min(cluster.components_[np.where(labels == i)[0], 1])

    count_horizontal = right - left
    count_vertical = top - bottom
    if count_horizontal >= imageSize - 2 and count_vertical >= imageSize - 2:
      nanowire_perc[k] = 1

print('number of dispersed morphology images: %i' % len(np.where(nanowire_perc == 0)[0]))
print('number of percolated network morphology images: %i' % int(sum(nanowire_perc)))

accuracy_score(nanowire_perc, peptide_morph_train_seg_enc)
# if the accuracy_score is 1.0, that means all the dispersed morphology images are validated as not percolated,
# all the network morphology images are validated as percolated.
```

    number of dispersed morphology images: 100
    number of percolated network morphology images: 100

    1.0


#  4 Trajectory Analysis

***

** Important Note **

Before proceeding to the rest of the analysis, it is a good time to define a path that points to the location of the MD simulation data, which we will analyze here.

If you successfully ran the MD simulation, the correct path should be:

```python
# Execute the cell to define the path to your simulation files
path="OxCompBio-Datafiles/run"
```

If however, you need/want to use the data from the simulation that has been already performed, uncomment the command below to instead define the path that points to the prerun simulation.

```python
# Uncomment the line below and execute the cell to define the path to the prerun simulation files
#path="OxCompBio-Datafiles/prerun/run"
```

##  4.1 Visualize the simulation

The simplest and easiest type of analysis you should always do is to look at it with your eyes! Your eyes will tell you if something strange is happening immediately. A numerical analysis may not.

###   4.1.1 VMD

*Note: Again, this step is optional. If you don't have VMD, go to section 4.1.2 below to visualize the trajectory with NGLView instead.*

Let us look at the simulations on VMD.

Open your vmd, by typing on your terminal:

`% vmd`

When it has finished placing all the windows on the screen. Click on `File` in the VMD main menu window and select `New Molecule`. The Molecule File Browser window should appear. Click on `Browse...` then select the `OxCompBio-Datafiles` and then the `run` directory and finally select `em.gro` (i.e. the file you made that has protein system energy minimized). Click `OK` and then click `Load`. It should load up the starting coordinates into the main window. Then click `Browse...` in the Molecule File Browser window. Select again the `OxCompBio-Datafiles`, then the `run` directory and then `md.xtc`. Select `OK` and then hit `Load`. The trajectory should start loading into the main VMD window.

Although things will be moving, you can see that it is quite difficult to visualize the individual components. That is one of the problems with simulating such large and complicated systems. VMD makes it quite easy to look at individual components of a system. For example, let us consider the protein only. On the VMD Main menu, left-click on Graphics and select `Representations`. A new menu will appear (`Graphical Representations`). In the box entitled `Selected Atoms` type protein and hit enter. Only those atoms that form part of the protein are now selected. Various other selections and drawing methods will help to visualize different aspects of the simulation.

 **Questions**

* How would you say the protein behaves?

* Is it doing anything unexpected? What would you consider unexpected behaviour?

* Can you add representations for the ASP and the ARG residues only? Can you observe if they form any inter-subunit interactions? We will quantify those later on.

###  4.1.2 NGLView

You have already tested NGLView at the Python tutorial (Notebook `12_ProteinAnalysis`) and at the beginning of this tutorial. This time however, you can visualize the trajectory you generated after carrying out the MD simulation.

You should also be familiar now with the MDAnalysis Python library that we will use to analyze the MD trajectory. We will also use it below, to create a Universe and load it on NGLView.

```python
# Import MDAnalysis and NGLView
import MDAnalysis
import nglview

# Load the protein structure and the trajectory as a universe named protein
protein=MDAnalysis.Universe(f"{path}/em.gro", f"{path}/md_fit.xtc")
protein_view = nglview.show_mdanalysis(protein)
protein_view.gui_style = 'ngl'

#Color the protein based on its secondary structure
protein_view.update_cartoon(color='sstruc')
protein_view
```

    ThemeManager()

    NGLWidget(gui_style='ngl', max_frame=100)

 **Questions**

* How would you say the protein behaves?

* Is it doing anything unexpected? What would you consider unexpected behaviour?

* Can you add representations for the ASP and the ARG residues only? Can you observe if they form any inter-subunit interactions? We will quantify those later on.

Now that we are sure the simulation is not doing anything ridiculous, we can start to ask questions about the simulation. The first thing to establish is whether the simulation has equilibrated to some state. So what are some measures of the system
being equilibrated? And what can we use to test the reliability of the simulation?

##  4.2 System Equilibration

###  4.2.1 Temperature fluctuation

The system temperature as a function of time was calculated in the previous section, with the built-in GROMACS tool `gmx energy`, but we still have not looked at it. It is now time to plot the temperature *vs* time and assess the results.

 **Questions**

* Does the temperature fluctuate around an equilibrium  value?

* Does this value correspond to the temperature that we predefined in the `md.mdp` input file?

Import numpy and pyplot from matplotlib, required to read and plot the data, respectively.

```python
# We declare matplotlib inline to make sure it plots properly
%matplotlib inline
# We need to import numpy
import numpy as np
# We need pyplot from matplotlib to generate our plots
from matplotlib import pyplot
```

Now, using numpy, we can read the data from the `1hsg_temperature.xvg` file; the first column is the time (in ps) and the secong is the system temperature (in K).

```python
# Read the file that contains the system temperature for each frame
time=np.loadtxt(f"{path}/1hsg_temperature.xvg", comments=['#','@'])[:, 0]
temperature=np.loadtxt(f"{path}/1hsg_temperature.xvg", comments=['#','@'])[:, 1]
```

You can use numpy again to compute the average temperature and its standard deviation.

```python
# Calculate and print the mean temperature and the standard deviation
# Keep only two decimal points
mean_temperature=round(np.mean(temperature), 2)
std_temperature=round(np.std(temperature), 2)
print(f"The mean temperature is {mean_temperature} ± {std_temperature} K")
```

    The mean temperature is 300.29 ± 1.67 K

Finally, you can plot the temperature *vs* simulation time.

```python
# Plot the temperature
pyplot.plot(time, temperature, color='darkred')
pyplot.title("Temperature over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("Temperature [K]")
pyplot.show()
```

###  4.2.2 Energy of the system

Another set of properties that is quite useful to examine is the various energetic contributions to the energy. The total
energy should be constant. but the various contributions can change and this can sometimes indicate something
interesting or strange happening in your simulation. Let us look at some energetic properties of the simulation.

We have already exctracted the Lennard-Jones energy, the Coulomb energy and the potential energy using again the GROMACS built-in tool `gmx energy`. The data of these three energetic components are saved in the same file called `1hsg_energies.xvg`; the first column contains the time (in ps) and the columns that follow contain the energies (in kJ/mol), in the same order as they were generated.

We can now read the data from the `1hsg_energies.xvg` file using numpy.

```python
# Read the file that contains the various energetic components for each frame
time=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 0]
lennard_jones=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 1]
coulomb=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 2]
potential=np.loadtxt(f"{path}/1hsg_energies.xvg", comments=['#','@'])[:, 3]
```

And now that we read the data file, we can plot the energetic components *vs* simulation time in separate plots using matplotlib.

```python
# Plot the Lennard-Jones energy
pyplot.plot(time, lennard_jones, color='blue')
pyplot.title("Lennard Jones energy over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("LJ energy [kJ/mol]")
pyplot.show()

# Plot the electrostatic energy
pyplot.plot(time, coulomb, color='purple')
pyplot.title("Electrostatic energy over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("Coulomb energy [kJ/mol]")
pyplot.show()

# Plot the potential energy
pyplot.plot(time, potential, color='green')
pyplot.title("Potential energy over time")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("Potential energy [kJ/mol]")
pyplot.show()
```

 **Questions**

* Can you plot the Coulomb energy and the potential energy, following the same steps as above?

* Is the total energy stable in this simulation?

* What is the dominant contribution to the potential energy?

##  4.3 Analysis of Protein

*Hint:* Ideally, you should visualize the trajectory with either VMD or NGLView and combine the information you extract from the plots with what you observe visually. That applies to all the structural analysis that you perform. So feel free at any point to go back to VMD or NGLView to verify the results you get from the analysis below.

###  4.3.1 Root mean square deviation (RMSD) of 1HSG

The RMSD gives us an idea of how 'stable' our protein is when compared to our starting, static, structure. The lower the RMSD is, the more stable we can say our protein is.

The RMSD as a function of time, $\rho (t)$, can be defined by the following equation:

\begin{equation}
\\
\rho (t) = \sqrt{\frac{1}{N}\sum^N_{i=1}w_i\big(\mathbf{x}_i(t) - \mathbf{x}^{\text{ref}}_i\big)^2}
\end{equation}

Luckily MDAnalysis has its own built-in function to calculate this and we can import it.

```python
# Import built-in MDAnalysis tools for alignment and RMSD.
from MDAnalysis.analysis import align
from MDAnalysis.analysis.rms import RMSD as rmsd

# Define the simulation universe and the reference structure (protein structure at first frame)
protein = MDAnalysis.Universe(f"{path}/md.gro", f"{path}/md_fit.xtc")
protein_ref = MDAnalysis.Universe(f"{path}/em.gro", f"{path}/md_fit.xtc")
protein_ref.trajectory[0]

# Call the MDAnalysis align function to align the MD simulation universe to the reference (first frame) universe
align_strucs = align.AlignTraj(protein, protein_ref, select="backbone", weights="mass", in_memory=True, verbose=True)

R = align_strucs.run()
rmsd_data = R.rmsd

# Plot the RMSD
pyplot.plot(rmsd_data)
pyplot.title("RMSD over time")
pyplot.xlabel("Frame number")
pyplot.ylabel("RMSD (Angstrom)")
pyplot.show()

```

 **Questions**

* What does this tell you about the stability of the protein? Is it in a state of equilibrium and if so why and at what time?

* Can you think of a situation where this approach might not be a very good indication of stability?

###  4.3.2 Root mean square fluctuation (RMSF) of 1HSG

A similar property that is particularly useful is the root mean square fluctuation (RMSF), which shows how each residue flucuates over its average position.

The RMSF for an atom, $\rho_i$, is given by:

\begin{equation}
\rho_i = \sqrt{\sum^N_{i=1} \big\langle(\mathbf{x}_i - \langle \mathbf{x}_i \rangle )^2 \big\rangle }
\end{equation}

```python
from MDAnalysis.analysis.rms import RMSF as rmsf

# Define again the simulation universe, using however the renumbered .gro file that you had generated earlier
protein = MDAnalysis.Universe(f"{path}/em.gro", f"{path}/md_fit.xtc")

# Reset the trajectory to the first frame
protein.trajectory[0]

# We will need to select the alpha Carbons only
calphas = protein.select_atoms("name CA")

# Compute the RMSF of alpha carbons. Omit the first 20 frames,
# assuming that the system needs this amount of time (200 ps) to equilibrate
rmsf_calc = rmsf(calphas, verbose=True).run(start=20)

# Plot the RMSF
pyplot.plot(calphas.resindices+1, rmsf_calc.rmsf, color='darkorange' )
pyplot.title("Per-Residue Alpha Carbon RMSF")
pyplot.xlabel("Residue Number")
pyplot.ylabel("RMSF (Angstrom)")
pyplot.show()
```

 **Questions**

* Can you identify structural regions alone from this plot and does that fit in with the structure?

* Residues 43-58 form part of the flexible flap that covers the binding site. How does this region behave in the simulation? Remember that you have a homodimer, so chain A contains residues 1-99 and chain B contains residues 100 to 198.

###  4.3.3 Hydrogen Bond Formation

We can also use the simulation to monitor the formation of any hydrogen bonds that may be of interest.

In the case of HIV-1 protease, the hydrogen bonds (HB) that are formed between the ARG8', the ASP29 and the ARG87 amino acids at the interface of the two subunits act in stabilising the dimer.

We can analyse the trajectory and monitor the stability of these interactions *vs* simulation time.

```python
# Import the MDAnalysis built-in tool for HB Analysis
from MDAnalysis.analysis.hydrogenbonds.hbond_analysis import HydrogenBondAnalysis as HBA

# Define the protein universe
# Note that when using this tool, it is recommended to include the .tpr file instead of the .gro file,
# because it contains bond information, required for the identification of donors and acceptors.
protein = MDAnalysis.Universe(f"{path}/md.tpr", f"{path}/md.xtc")

# Define the atom selections for the HB calculation.
# In this case, the ARG hydrogens and the ASP oxygens, which act as the HB acceptors are specifically defined.
hbonds = HBA(universe=protein, hydrogens_sel='resname ARG and name HH21 HH22', acceptors_sel='resname ASP and name OD1 OD2')

# Perform the HB calculation
hbonds.run()

# Plot the total number of ASP-ARG HBs vs time
hbonds_time=hbonds.times
hbonds_data=hbonds.count_by_time()

pyplot.plot(hbonds_time, hbonds_data, color='cyan')
pyplot.title("ASP-ARG Hydrogen Bonds")
pyplot.xlabel("Time [ps]")
pyplot.ylabel("# Hydrogen Bonds")
pyplot.show()

# Compute and print the average number of HBs and the standard deviation
aver_hbonds=round(np.mean(hbonds_data), 2)
std_hbonds=round(np.std(hbonds_data), 2)
print(f"The average number of ASP-ARG HBs is {aver_hbonds} ± {std_hbonds}")
```

 **Questions**

* How much variation is there in the number of hydrogen bonds?

* Do any break and not reform?

* Using VMD, can you observe the HB formation and breakage throughout the simulation?

***

This concludes the analysis section, but the aim was only to give you an idea of the numerous information that we can gain when analysing an MD trajectory. Feel free to ask and attempt to answer your own questions, utilising the tools that you were introduced to during the tutorial.

##  4.4 Further Reading

The texts recommended here are the same as those mentioned in the lecture:
* "Molecular Modelling. Principles and Applications". Andrew Leach. Publisher: Prentice Hall. ISBN: 0582382106. This book has rapidly become the defacto introductory text for all aspects of simulation.
* "Computer simulation of liquids". Allen, Michael P., and Dominic J. Tildesley. Oxford university press, 2017.
* "Molecular Dynamics Simulation: Elementary Methods". J.M. Haile. Publisher: Wiley. ISBN: 047118439X. This text provides a more focus but slightly more old-fashioned view of simulation. It has some nice simple examples of how to code (in fortran) some of the algorithms though.

##  End of Tutorial

Congratulations, you have finished the MD tutorial. Hopefully, by you are familiar with the molecular dynamics method, the GROMACS software that we have used to perform the simulation, and you have an idea of the type of information that we can extract from an MD run and how we can visualize, compute and plot the data of interest.


# **Assignment 1: Rendering systems**

## ECSE 446/546: Realistic/Advanced Image Synthesis, Fall 2021

#### Due: Wednesday, October 13, 2021 at 11:59pm EST on [myCourses](https://mycourses2.mcgill.ca/)

#### Weight: 25% (ECSE 446) / 20% (ECSE 546)

*In this assignment, you will implement the techniques discussed in the 'Rendering Systems' module. You will implement key components of **rasterization** and **ray tracing** pipelines.*

Since evaluation for this assignment is partly automated, it is important that you fill in your code **only in the cells indicated**. If needed, extra cells may be added, but only at the end of the notebook. Any cells added in between will very likely present issues with the autograder.

The *bonus* questions in the end are for extra credit and are optional.

---

## **Assignment Submission Process**

Modify the `ipynb` notebook we provide on myCourses and submit your modified file, renamed according to your student ID, as

A1-[YourStudentID].ipynb

For example, if your student ID is **123456789**, then your submission filename should be **A1-123456789.ipynb**.

> Note that every time you submit a new file, your previous submission will be overwritten.
**All submissions must be made using myCourses.**
You can submit as many times as you like, but we will only grade the last submission.

This assignment is to be completed *individually*.

### Late policy

Failure to submit a (valid) assignment on time will result in a score of **zero (0%)** on the assignment.
**We will not grant extensions.**
Exceptional circumstances will be treated as specified in [McGill's Policies on Student Rights and Responsibilities](https://www.mcgill.ca/students/srr/).

### Plagiarism and cheating
Plagiarism is an academic offense of misrepresenting authorship. This can result in penalties up to expulsion. It is also possible to plagiarise your own work, e.g., by submitting work from another course without proper attribution. When in doubt, attribute!

We expect you to submit your own work. Assignments are individual tasks. That said, we want to promote an environment where you are comfortable discussing ideas together. A good rule to follow: fully understand every solution you submit and only submit code that was written by you.

McGill values academic integrity and students should take the time to fully understand the meaning and consequences of cheating, plagiarism and other academic offenses (as defined in the Code of Student Conduct and Disciplinary Procedures — see these two links [[1]](https://www.mcgill.ca/integrity) [[2]](https://www.mcgill.ca/students/srr/honest)).

In accordance with article 15 of the Charter of Students' Rights, students may submit any written or programming components in either French or English.

### Code comparison

We will use code similarity checkers to detect and deal with cases of plagiarism and cheating.

### Python `import` statements

We provide all import statements required to complete the assignment.
You must not use any `import`s other than the ones we provide.
Doing so will result in a score of **zero (0%)** on the assignment.

---

## **Notes and Tips**

### Debugging in Jupyter Notebook / Google Colab

[Here](https://colab.research.google.com/github/jakevdp/PythonDataScienceHandbook/blob/master/notebooks/01.06-Errors-and-Debugging.ipynb) is a quick tutorial by Jake VanderPlas on how to debug in Jupyter Notebook / Google Colab using `ipdb`.
For a complete list of commands and their usage, check out `pdb`'s [official documentation](https://docs.python.org/3/library/pdb.html).

### OpenEXR Format and EXR Viewer

[OpenEXR](http://www.openexr.com/) is a standardized file format for storing high dynamic range (HDR) images.
Unlike low dynamic range (LDR) image formats — such as JPG and PNG — HDR image pixel color values are not restricted to the interval $[0, 1]$.
This allows us to represent luminance ranges similar to the range we experience through the human visual system in the real-world.

**For this assignment though, all output pixel color values in your rendered image should still be clipped to $[0, 1]$.**

We recommend using [tev](https://github.com/Tom94/tev), a lightweight yet powerful EXR viewer that supports various tonemapping operators and pixel-level image inspection and comparison.
To obtain `tev`, follow the OS-specific instruction below:

- Windows / macOS: Download the [pre-built binaries](https://github.com/Tom94/tev/releases).
If your Mac comes with the Apple M1 chip and you want to to run `tev` natively (not via Rosetta 2), [build it from source](https://github.com/Tom94/tev#macos--linux).
- Linux: Read this [step-by-step guide](https://github.com/Tom94/tev#macos--linux) to build from source.
For Arch Linux and other Arch-based distros (Manjaro, EndeavourOS, etc.), `tev` is available in [AUR](https://aur.archlinux.org/packages/tev/).

### Render time

For each experiment, the default image height and width `imgWidth` and `imgHeight` have been set to `64`. This enables faster debugging.

**Once you think your implementation works, render images of width and height `512` instead of `64`**.

---

## **Preparation**

Let us first install and/or import all necessary modules for this assignment. Note that you are not allowed to import anything apart from the modules in this cell, as they will not be available in our autograding environment.

Run the cell below to install and import all necessary modules.

```python
!pip install pyexr

%matplotlib inline

import math
import time

import numpy as np
import matplotlib.pyplot as plt
from PIL import Image as im
import pyexr
```

    Collecting pyexr
      Downloading PyEXR-0.3.10.tar.gz (4.6 kB)
    Collecting OpenEXR
      Downloading OpenEXR-1.3.2.tar.gz (10 kB)
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from pyexr) (1.19.5)
    Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from pyexr) (0.16.0)
    Building wheels for collected packages: pyexr, OpenEXR
      Building wheel for pyexr (setup.py) ... done
      Created wheel for pyexr: filename=PyEXR-0.3.10-py2.py3-none-any.whl size=4518 sha256=d70e177ca95629de80e6a2de2c19fb1ec70245f7a4130ecc373b729b082ac448
      Stored in directory: /root/.cache/pip/wheels/da/c8/fb/f00811aa6cb7b74d76ab5941aea24ecf85fd8ca26f1a53ee39
      Building wheel for OpenEXR (setup.py) ... done
      Created wheel for OpenEXR: filename=OpenEXR-1.3.2-cp37-cp37m-linux_x86_64.whl size=189355 sha256=d53d41cc06e35a0584faaa63016bfee7cf05c866d4aeb2d6261b45071ef2694a
      Stored in directory: /root/.cache/pip/wheels/d1/c6/6c/dcf5d042ed2d547da41021f47957b16bebe9cadfd095765d44
    Successfully built pyexr OpenEXR
    Installing collected packages: OpenEXR, pyexr
    Successfully installed OpenEXR-1.3.2 pyexr-0.3.10

### Helper Functions

Next, we define some helper functions below that will be useful across the whole assignment.

> **Note**: You are not _required_ to use these helper function implementations. These are only provided to reduce the amount of boilerplate code you need to write.

Specifically,
- `normalize(v)`: Normalizes a vector `v`.
- `distance(a, b)`: Computes the distance between two points `a` and `b`.
- `squaredDistance(a, b)`: Computes the *squared* distance between two points `a` and `b`.
- `perspective(fovy, aspect, near, far)`: Computes the perspective projection matrix.
- `lookAt(eye, at, up)`: Computes the viewing transformation matrix.
- `image2NDC(x, y, width, height)`: Maps the given image coordinate $(x, y)$ into the canonical view volume (or normalized device coordinates, NDC).
- `NDC2Image(x, y, width, height)`: Maps the given coordinate $(x, y)$ in canonical view volume into the screen space.

*Note that you may also want to define your own helper functions in this cell.*

```python
cam_near = 0.1
cam_far = 2000
p = 1

def interpolate_q4(triangle, attr, p):
    """
    Accurate perspective interpolation.
    Parameters:
    triangle: 2D numpy array containing triangle's vertex positions (homogeneous
              coordinates) in NDC space.
        Note that 'triangle' here is in the form of
        [[v0.x, v0.y, v0.z, v0.w], [v1.x, v1.y, v1.z, v1.w], [v2.x, v2.y, v2.z, v2.w]]
        and NOT a Triangle object.
    attr: Attribute to interpolate across the triangle (e.g., position)
    p: Point (in NDC space) at which to compute interpolated attribute
    """
    (alpha, beta, gamma) = barycentric_q4(triangle[0], triangle[1], triangle[2], p)
    return (alpha * attr[0] / triangle[0][3] + beta * attr[1] / triangle[1][3] + gamma * attr[2] / triangle[2][3]) / (
            alpha / triangle[0][3] + beta / triangle[1][3] + gamma / triangle[2][3])

def barycentric_q4(a, b, c, p):
    """
    Compute Barycentric coordinates of point p w.r.t triangle a, b, c.
    Parameters:
    a, b, c: vertices coordinates of the triangle.
    p: the point you want to know its  barycentric coordinates
    """
    a = a / a[-1]
    b = b / b[-1]
    c = c / c[-1]
    area = signedArea(a, b, c)
    alpha = signedArea(b, c, p) / area
    beta = signedArea(c, a, p) / area
    gamma = 1 - alpha - beta
    return alpha, beta, gamma

def rasterize_q4(scene, triangle, img, zbuf):
    """
        Rasterize a single triangle in the scene.
        Parameters:
        scene: an instance of Scene class.
        triangle: the triangle to be rasterized. It is an instance of Triangle class, not vertices in np.array.
    """

    # Scene Configs
    width = scene.config["width"]
    height = scene.config["height"]
    fov = scene.config["fov"] * np.pi / 180.

    # Transformation Matrices
    transform_Cam = lookAt(scene.cam.eye, scene.cam.at, scene.cam.up)
    transform_Pers = perspective(fov, width / height, cam_near, cam_far) @ transform_Cam

    # Triangle Related Vertices
    tri_Wrd3d = np.array(([triangle.v0, triangle.v1, triangle.v2]))
    # 4d versions below
    tri_Wrd = np.array(([np.append(v, 1) for v in tri_Wrd3d]))

    tri_Cam = np.transpose(transform_Cam @ np.transpose(tri_Wrd))
    tri_NDC = np.transpose(transform_Pers @ np.transpose(tri_Wrd))
    v0_NDC4d, v1_NDC4d, v2_NDC4d = tri_NDC

    # Compute the bounding box and map to Image Space
    ws = np.stack((tri_NDC[:, 3], tri_NDC[:, 3]), axis=-1)
    xmin, xmax, ymin, ymax = bbox(tri_NDC[:, :2] / ws)

    # The corresponding minimum and maximum points of the Bounding box of the triangle in the Image Space
    min_Img, max_Img = np.array(NDC2Image(xmin, ymin, width, height)), np.array(
        NDC2Image(xmax, ymax, width, height))

    # Loop over pixels in the bbox
    for i in range(min_Img[0], max_Img[0] + 1):
        for j in range(max_Img[1], min_Img[1] + 1):
            p_NDC = np.append(image2NDC(i, j, width, height), [1, 1])

            # Compute the Bary Coord for the pixel in NDC Space
            (a, b, c) = barycentric(v0_NDC4d / v0_NDC4d[3], v1_NDC4d / v1_NDC4d[3], v2_NDC4d / v2_NDC4d[3], p_NDC)

            # the pixel is in the triangle
            if 0 <= a and 0 <= b  and 0 <= c:
                # Interpolate to compute the World Space Coord for the pixel
                # tri_NDC = tri_NDC / np.stack(([tri_NDC[:, -1],tri_NDC[:, -1], tri_NDC[:, -1], tri_NDC[:, -1]]), axis=-1)
                p_Wrd = interpolate_q4(tri_NDC, tri_Wrd, p_NDC)
                p_Wrd /= p_Wrd[-1]
                p_depth2Cam = p_Wrd[-2] - scene.cam.eye[-1]

                # Update if closer to the camera
                if p_depth2Cam < zbuf[j, i]:
                    # compute the world coord
                  #  p_World = np.linalg.inv(transform_Cam) @ p_Cam
                  #  p_World /= p_World[-1]
                    zbuf[j, i] = p_depth2Cam
                    img[j, i] = shade(scene, p_Wrd[:-1], triangle)
    return img

def normalize(v, eps=1e-8):
    """
    Normalize a vector. Add a tiny eps to the denominator to prevent
    divide-by-zero errors.
    """
    dist = np.linalg.norm(v)
    return v / (dist + eps)

def distance(a, b):
    return np.linalg.norm(b - a)

def squaredDistance(a, b):
    v = b - a
    return np.dot(v.T, v)

def perspective(fovy, aspect, near, far):
    """
    Perspective projection transformation
    Parameters:
        fovy (float): vertical field of view (in radians).
        aspect (float): aspect ratio (width/height).
        near (float): distance to near plane.
        far (float): distance to far plane.
    """
    n = near
    f = far
    t = near * math.tan(fovy / 2)
    r = t * aspect
    return np.array([
        [n / r, 0, 0, 0],
        [0, n / t, 0, 0],
        [0, 0, - (f + n) / (f - n), -2 * f * n / (f - n)],
        [0, 0, -1, 0]
    ])

def lookAt(eye, at, up):
    """
    Viewing transformation.
    Parameters:
        eye (np.array): eye postion
        at (np.array): the point the eye is looking at (usually, the center of
          an object of interest)
        up (np.array): up vector (vertically upward direction)
    """
    z = normalize(eye - at)
    x = normalize(np.cross(up, z))
    y = normalize(np.cross(z, x))
    A = np.column_stack((x, y, z, eye))
    A = np.row_stack((A, np.array([0, 0, 0, 1])))
    return np.linalg.inv(A)

def image2NDC(x, y, width, height):
    """
    Image space (-0.5..width-0.5, -0.5..height-0.5) to NDC (normalized
    device coordinates) space (-1..1, -1..1) transform.
    Note that (0, 0) in image space is the TOP left of the image.
    Parameters:
        x (float): x coordinate in image space.
        y (float): y coordinate in image space.
    """
    return ((2 * x - width + 1) / width, -(2 * y - height + 1) / height)

def NDC2Image(x, y, width, height):
    """
    NDC space to image space transform.
    Parameters:
        x (float): x coordinate in NDC space.
        y (float): y coordinate in NDC space.
    """
    return (round((width * x + width - 1) / 2), round((height * (-y) + height - 1) / 2))
```

### Rendering data structures

We now define essential data structures for our _scene description_. All these data structures (except for *Triangle*) **must be used as is, without modification**. These classes will then be employed to define rasterization and ray tracing routines. (these cells have a "DO NOT TOUCH" warning.

> **Note**: The autograder will not pick these cells up from your submission, but will rather use ones from the handout notebook -- so any modifications you make to the rendering data structures will not be available during evaluation.

In particular, we define the *Scene*, *Camera*, *PointLight*, *Ray*, *Triangle* and *Diffuse* classes. There are two blank methods to be completed in the *Triangle* class. These methods are to be completed for the ray tracing part of the assignment. (For now, if you choose to do rasterization first, you may safely ignore the incomplete methods in the *Triangle* class)

All other classes except *Ray* are shared across rasterization and ray tracing.

```python
# DO NOT TOUCH
class Scene:
    """
    A class used to represent a rendering scene
    Attributes:
        geometry: objects in the scene.
        light: light info
        camera: camera info
        config: store width, height and fov
    """
    def __init__(self, geometry, light, camera, config):
        self.geometry = geometry
        self.light = light
        self.cam = camera
        # config is a dictionary with keys: width, height, fov
        self.config = config
```

```python
# DO NOT TOUCH
class Camera:
    """
    A class to represent a camera
    Attributes:
        eye: position of the camera
        at: position where the camera looks at
        up: up direction of the camera
    """
    def __init__(self, eye, at, up):
        self.eye = eye
        self.at = at
        self.up = up
```

```python
# DO NOT TOUCH
class PointLight:
    """
    A point light source
    Attributes:
        p: position of the light source
        e: emmisive flux (power) of the light source
    """
    def __init__(self, position, emission):
        self.p = position
        self.e = emission
```

```python
# DO NOT TOUCH
class Ray:
    """
    A class to represent a ray
    Attributes:
        o: origin of the ray
        d: direction of the ray
        max: max distance at which the ray is capped (to avoid evaluating
          indefinitely far points along a ray, we cap it to a large distance)
    """
    def __init__(self, origin, dir, max=1e5):
        self.o = origin
        self.d = dir
        self.max = max
```

```python
# DO NOT TOUCH
class Diffuse:
    """
    Diffuse material
    Attributes:
        c: diffuse color
    """
    def __init__(self, color):
        self.c = np.asarray(color)
```

The following `Triangle` class has two methods you will need to complete for the ray tracing component of this assignment. Instructions for completing this are found below, in the ray tracing section. You may safely ignore these for the first question (rasterization).

```python
class Triangle:
    """
    A class to represent a triangle
    Attributes:
        v0, v1, v2: vertices of the triangle
        mat: material of the triangle
    """
    def __init__(self, v0, v1, v2, mat):
        self.v0 = np.asarray(v0)
        self.v1 = np.asarray(v1)
        self.v2 = np.asarray(v2)
        self.mat = mat

    def intersect(self, r):
        """
      Intersect this triangle with a ray
      Parameters:
          r: ray to intersect
      Returns:
          0 if no hit, otherwise the distance to the hit point
      """
        # TODO: implement this
        A = np.transpose(np.array([-r.d, self.v1 - self.v0, self.v2 - self.v0]))
        b = r.o - self.v0
        eps = 1e-5
        # Solve Ax = b
        try:
            x = np.linalg.solve(A, b)
            t = x[0]
            dist = distance(r.o + t * r.d, r.o)
            if 0 <= x[1] <= 1 and 0 <= x[2] <= 1 and 0 <= x[1] + x[2] <= 1 and t > 0 and (dist + eps) < r.max:
                return dist
            else:
                return 0
        except np.linalg.LinAlgError:
            return 0

    def normal(self, hit):
        # TODO: implement this
        # get 2 edges
        e1 = self.v1 - self.v0
        e2 = self.v2 - self.v0
        return normalize(np.cross(e1, e2))
```

## Question 1 - Rasterization - Single primitive (25 points)

We will implement a minimal rasterization pipeline as a smoketest. The goal of this question will be to rasterize a single triangle and display it on the screen.

Recall the basic triangle rasterization algorithm you learned in class:
* compute _edge functions_ from projected vertices
* check if the edge function values evaluated at a given pixel are all non-negative.

We will take an alternative (but equivalent) approach that is more convenient when interpolating vertex positions and $z$-buffer values. You may recall this is the Barycentric coordinate approach which we discussed, albeit in the context of raytracing (ray-triangle intersection test).

#### Rasterization using Barycentric coordinates

Suppose a point $P$ is represented in terms of its barycentric coordinates $(\alpha, \beta, \gamma)$ with respect to a triangle $\triangle ABC$.
We know that, $P$ lies inside $\triangle ABC$ if $\alpha \geq 0, \beta \geq 0, \gamma \geq 0$.
So, given Barycentric coordinates $P$, inside-outside tests simply amount to checking the non-negativity of $\alpha, \beta, \gamma$.
This reduces the problem to computing the exact barycentric coordinates of $P$, which may be done using the following formulae:

$$
\begin{align*}
    \alpha &= \frac{S_{\triangle BCP}}{S_{\triangle ABC}}, \\
    \beta &= \frac{S_{\triangle CAP}}{S_{\triangle ABC}}, \\
    \gamma &= 1 - \alpha - \beta,
\end{align*}
$$

where the signed area of a triangle $S_{\triangle v_0 v_1 v_2}$ is defined as

$$
S_{\triangle v_0 v_1 v_2} = \frac{1}{2} [ (v_{1x} - v_{0x}) (v_{2y} - v_{0y}) - (v_{1y} - v_{0y}) (v_{2x} - v_{0x}) ].
$$

Interpolation of parameters across the triangle is a tricky problem and requires perspective correction.
For technical details, please consult [The OpenGL Graphics System: A Specification](https://www.khronos.org/registry/OpenGL/specs/gl/glspec44.core.pdf), page 427.

We provide in the following cell a complete implementation of all helper functions desribed above.

```python
def cross(a, b):
    """ Cross product of vectors in x-y plane. The result is the the z component (a scalar). """
    return a[0]*b[1] - a[1]*b[0]

def signedArea(a, b, c):
    """
    Signed area of the triangle times 2.
    Parameters:
    a, b, c: vertices coordinates of the triangle.
    """
    return cross(b-a, c-a)

def barycentric(a, b, c, p):
    """
    Compute Barycentric coordinates of point p w.r.t triangle a, b, c.
    Parameters:
    a, b, c: vertices coordinates of the triangle.
    p: the point you want to know its  barycentric coordinates
    """
    area = signedArea(a, b, c)
    alpha = signedArea(b, c, p) / area
    beta = signedArea(c, a, p) / area
    gamma = 1 - alpha - beta
    return (alpha, beta, gamma)

def interpolate(triangle, attr, p):
    """
    Accurate perspective interpolation.
    Parameters:
    triangle: 2D numpy array containing triangle's vertex positions (homogeneous
              coordinates) in NDC space.
        Note that 'triangle' here is in the form of
        [[v0.x, v0.y, v0.z, v0.w], [v1.x, v1.y, v1.z, v1.w], [v2.x, v2.y, v2.z, v2.w]]
        and NOT a Triangle object.
    attr: Attribute to interpolate across the triangle (e.g., position)
    p: Point (in NDC space) at which to compute interpolated attribute
    """
    (alpha, beta, gamma) = barycentric(triangle[0], triangle[1], triangle[2], p)
    return (alpha * attr[0] / triangle[0][3] + beta * attr[1] / triangle[1][3] + gamma * attr[2] / triangle[2][3]) / (alpha / triangle[0][3] + beta / triangle[1][3] + gamma / triangle[2][3])
```

We are now ready to implement the rasterization loop for a single triangle. In effect, we will do the following.

1. compute the viewing and perspective transformation matrices; this enables us to project the triangle vertices from world coordinates into a canonical view volume,
2. compute the bounding box of the triangle and map its vertices to image coordinates, and
3. for each pixel in the bounding box, check whether or not it lies inside the triangle; if the pixel lies inside the triangle, assign the color of the triangle (diffuse color) to that pixel

Using this information, complete the empty functions `bbox()` and `rasterize()` below.

**Note**: the bellow cells must only contain the `bbox()` and `rasterize()` functions respectively. You may NOT have any statements in these cells that are outside the scope of the functions. You may NOT modify the function call (arguments, return statements) as they interfere with the autograder.

```python
def bbox(triangle, eps=1e-6):
    """
        The bounding box of the triangle.
        Parameters:
        triangle: triangle vertices in np.array. E.g., triangle[0], triangle[1], triangle[3] are 3 vertices (coordinates).
        eps: a tiny number (epsilon) for clipping the result. See the comments below.
    """
    # TODO: Compute the bounding box of a griven triangle
    # The triangle is assumed to be in 2D (normalized view volume) and each
    # vertex of the triangle may be accessed by indexing as follows:
    # First dim indexes the triangle vertex (0 -> first vertex,
    # 1 -> second vertex, 2 -> third vertex)
    # Second dim indexes the coordinate (0 -> x, 1 -> y)
    # E.g., the y coordinate of the first vertex is triangle[0][1]
    # Note that xmin, xmax, ymin, xmax should be in the NDC space. E.g., -1 < xmin < 1, etc.
    # That is, you need to clamp them in [-1+eps, 1-eps].
    s_tri = np.clip(np.sort(triangle, axis=0), -1 + eps, 1 - eps)
    xs, ys = s_tri[:, 0], s_tri[:, 1]

    xmin, xmax, ymin, ymax = xs[0], xs[-1], ys[0], ys[-1]

    return xmin, xmax, ymin, ymax
```

```python
def rasterize(scene, triangle):
    """
        Rasterize a single triangle in the scene.
        Parameters:
        scene: an instance of Scene class.
        triangle: the triangle to be rasterized. It is an instance of Triangle class, not vertices in np.array.
    """
    # DO NOT modify these three lines
    width = scene.config["width"]
    height = scene.config["height"]
    img = np.zeros((height, width, 3))

    # TODO: Implement the rasterization loop here
    # Recall that a rasterization loop comprises the following steps
    # 1. transform primitives (here, triangles) from world space to NDC space
    # 2. Compute the bounding box enclosing the triangle (this assumes the
    #    triangle to be in NDC space)
    #         `(xmin, xmax, ymin, ymax) = bbox(triangle_ndc)`
    # 3. Loop over all pixels in the bounding box, and implement the Barycentric
    #    coordinate-based rasterization scheme described in the question.

    # DO NOT modify this return statement
    cam = scene.cam
    eye, at, up = cam.eye, cam.at, cam.up
    near, far = cam_near, cam_far

    config = scene.config
    fov = config["fov"] * np.pi / 180.
    aspect = config["width"] / config["height"]

    # transform from world space to NDC space

    tri_World3d = np.hstack((np.stack((triangle.v0, triangle.v1), axis=-1), triangle.v2[:, np.newaxis]))
    tri_Cam4d = lookAt(eye, at, up) @ np.vstack((tri_World3d, np.ones((3,), int)))
    tri_NDC4d_t = np.transpose(perspective(fov, aspect, near, far) @ tri_Cam4d)

    v0_NDC4d, v1_NDC4d, v2_NDC4d = tri_NDC4d_t[0, :], tri_NDC4d_t[1, :], tri_NDC4d_t[2, :]
    tri_Attr = np.array([triangle.mat.c, triangle.mat.c, triangle.mat.c])

    # Compute the bounding box and map to Image Space
    xmin, xmax, ymin, ymax = bbox(tri_NDC4d_t[:, :2] / np.transpose(tri_Cam4d[3, np.newaxis]))

    min_Img, max_Img = np.array(NDC2Image(xmin, ymin, width, height)), np.array(
        NDC2Image(xmax, ymax, width, height))

    # Loop over pixels in the bbox
    diff_Img = max_Img - min_Img
    for p in np.ndindex(diff_Img[0], -diff_Img[1]):
        p_real = (p[0] + min_Img[0], -p[1] + min_Img[1])
        p_NDC = image2NDC(p_real[0], p_real[1], width, height)
        p_NDC = np.append(p_NDC, [1, 1])
        (a, b, c) = barycentric(v0_NDC4d / v0_NDC4d[3], v1_NDC4d / v1_NDC4d[3], v2_NDC4d / v2_NDC4d[3], p_NDC)
        if 0 <= a <= 1 and 0 <= b <= 1 and 0 <= c <= 1:
            img[p_real[1], p_real[0]] = interpolate(tri_NDC4d_t, tri_Attr, p_NDC)
            # print(img[p_real[1], p_real[0]])
    # print(img)
    return img

```

The following code loads a scene with only one triangle for testing.
It will be reused later by your ray tracer.
Simply run the cell before generating your rasterized image.

```python
def loadTriangleScene(width, height):
    objects = [Triangle([5, -2.5, 0], [-5, -2.5, 0], [0, 2.5, 0], Diffuse([1, 0, 0]))]
    light = PointLight([0, 15, -15], [36000, 36000, 36000])
    camera = Camera(np.array([0, 0, -15]), np.array([0, 0, 0]), np.array([0, 1, 0]))
    config = { "width": width, "height": height, "fov": 60 }
    return Scene(objects, light, camera, config)
```

Now render your rasterized triangle and see if you can successfully obtain a red-colored, upward-pointing triangle.
The rendered result will also be written to an EXR image.

For faster debugging, we currently set `imgWidth` and `imgHeight` to `64`. **For the eventual solution, you are required to render a `512 x 512` image**. (Note that rendering at this resolution may take several minutes, so please plan ahead of time)

```python
imgWidth = 64
imgHeight = 64
```

```python
scene = loadTriangleScene(imgWidth, imgHeight)
print(f"Start rendering in resolution: {imgWidth}x{imgHeight}, fov: {scene.config['fov']}")
startTime = time.monotonic()
img = rasterize(scene, scene.geometry[0])
duration = time.monotonic() - startTime
print("Finished rendering, time taken:", duration, "seconds")

plt.imshow(img)

print("Writing PNG...")
plt.imsave(f"triangle-rasterization.png", img)

print("Writing EXR...")
output = np.power(img, 2.2)
pyexr.write(f"triangle-rasterization.exr", output)
```

---

## Question 2 - Ray Tracing - Single primitive (35 points)

We will now implement a barebones ray tracer to render the same scene (i.e., a single triangle).

**Hint:** Remember that rasterization and ray tracing are two different techniques, but solve the same problem (the primary/eye-visibility problem). You should get identical results regardless of the technique used (helpful when debugging ray tracing implementations).

In the `Triangle` class above, the following two routines need to be implemented:

- `intersect(ray)`: Check if ray intersects the current triangle, and if so, the distance to the hit point.
- `normal(hit):` Computes the triangle normal at a hit point.

After completing the `Triangle` class, implement the following ray-scene intersection method: given a `Ray` object, intersect it with all objects (here, triangles) in the scene. Determine which objects intersect this ray, and return the distance of the closest _hit_ object.

**Note**: There are often multiple objects in the scene that a ray might hit. If so, you must remember to line-search for the closest _hit_.

```python
def intersect(scene, ray):
    """
    Method to intersect all objects in the scene
    Parameters:
        scene: scene to render.
        ray: ray to trace.
    Returns:
      (has_hit, hit_object_id, distance_to_object)
      has_hit is 0 if there is not hit, else it is 1
      id is the idx of the scene object hit
      t is the ray parameter (recall a ray is o + td) at which
        a hit is found
    """
    # TODO: implement this
    primitives = scene.geometry

    # Initialize return values
    has_hit, hit_id, t = False, -1, ray.max

    # Set up distance range
    t_min, t_max = 0., ray.max

    # loop
    for (index, primitive) in zip(range(len(primitives)), primitives):
        t = primitive.intersect(ray)
        if t_min < t < t_max:
            t_max = t
            hit_id = index
            has_hit = True
    dist = t_max
    return has_hit, hit_id, dist
```

Recall for a point light with intensity $\Phi$, the radiance of diffusely reflected light $L_d$ at a shading point is computed as

\begin{equation}
    L_d = \frac{\rho}{\pi} \left( \frac{\Phi}{4 \pi r^2} \right) \max\left( 0, \mathbf{n} \cdot \mathbf{l} \right),
\end{equation}

where $\rho$ is the diffuse albedo, $r$ is the distance between the shading point and the light, $\mathbf{n}$ is the surface normal and $\mathbf{l}$ is the light direction.

Implement the following `shade()` function according to the above formula.
Assume $\rho = 1$ in your code.

```python
def shade(scene, hit, triangle):
    """
    Method to shade a hit point
    Parameters:
      scene: scene to render.
      hit: hit point to shade.
      triangle: hit triangle object
    Returns:
      radiance at the hit point
    """
    # TODO: implement this
    light = scene.light
    light_pos, light_flux = light.p, np.asarray(light.e)

    cam = scene.cam
    cam_pos = cam.eye

    diff_l = light_pos - hit
    r = distance(light_pos, hit)
    li = normalize(diff_l)

    diff_c = normalize(cam_pos - hit)
    normal = triangle.normal(hit)
    tmp = 2 * (np.dot(diff_c, normal) > 0)

    n = normal * (tmp - 1)

    cosine = max(0, np.dot(n, li))
    a = light_flux / (4 * np.pi * (r ** 2))
    k = p / np.pi
    b = k * a * cosine
    radiance = b * triangle.mat.c
    radiance = np.array([min(1, radiance[0]), min(1, radiance[1]), min(1, radiance[2])])
    # radiance = triangle.mat.c
    return radiance  # Replace this with your return statement, when implementation is ready

```

Now trace a ray into the scene and collect the hit information.
Evaluate and return the outgoing radiance using your `shade()` method.

```python
def trace(scene, ray):
    """
    Trace a ray in the scene, and shade the hit point
    Parameters:
      scene: scene to render
      ray: ray to trace
    Returns:
      light radiance at the hit point
    """
    intersection = intersect(scene, ray)
    has_hit = intersection[0]
    if has_hit is False:
      return np.zeros(3)
    hit_id = intersection[1]  # object id
    t = intersection[2] # distance

    return shade(scene, ray.o + ray.d * t, scene.geometry[hit_id])
```

### Putting Everything Together ###

Now we have all the pieces of the ray tracer, let's put it together and implement the ray tracing rendering loop.
The `renderRaytracing()` function below is currently empty: complete it by looping over all pixels on the image plane and computing their color.
At a high-level, this is done as follows:

1. calculate the camera perspective, the camera-to-world transformation matrix and the aspect ratio,
2. loop over all pixels on the image plane, and
3. generate a ray through each pixel and *splat* its contribution onto the image plane.

The last step can be broken down into four (4) substeps:

1. retrieve the *pixel* center $(x, y)$,
2. construct a ray with its origin at the camera's center and its direction pointing through the pixel center,
3. transform this ray to world coordinates, taking the FoV and aspect ratio into account,
4. trace this ray into the scene and output the returned radiance contribution to the image buffer.

```python
def renderRaytracing(scene):
    """
    Ray tracer main rendering function
    Parameters:
      scene: scene to render
    Returns:
      rendered image
    """
    print("Rendering the scene")

    # TODO: implement this
    # Hints:
    # 1. Set up the model view projection transforms (use the `width`, `height`,
    #    `fov` parameters from the `scene.config` dictionary for this).
    # 2. Loop over all pixels in a ray, and follow the instructions from the
    #    question to complete the ray tracing routine.
    width, height, fovy = scene.config["width"], scene.config["height"], scene.config["fov"] * np.pi / 180
    aspect = width / height
    n = cam_near
    t = n * math.tan(fovy / 2)
    r = t * aspect

    cam = scene.cam
    cam2World = np.linalg.inv(lookAt(cam.eye, cam.at, cam.up))
    x, y, z, eye = cam2World[:-1, 0], cam2World[:-1, 1], cam2World[:-1, 2], cam2World[:-1, 3]

    imgBuffer = np.zeros(shape=(height, width, 3))
    # pixelBuffer = np.zeros(shape=(width, height, 2))

    # set the pivot point at the top left
    for i in range(0, width):
        for j in range(0, height):
            u, v = -r + ((i + 0.5) * 2 * r / width), t - ((j + 0.5) * 2 * t / height)
            # pixel_Cam= np.asarray([u, v, d])
            # dir_Cam = hp.normalize(pixel_Cam)
            dir = normalize(u * x + v * y - n * z)
            ray = Ray(eye, dir)
            imgBuffer[j, i] = trace(scene, ray)

    # Return your image
    return imgBuffer
```

Finally, it's time to check your solution.
Run the two cells below to render and display the same triangle scene and output the result to an EXR image.

For faster debugging, we currently set `imgWidth` and `imgHeight` to `64`. **For the eventual solution, you are required to render a `512 x 512` image**. (Note that rendering at this resolution may take several minutes, so please plan ahead of time)

```python
imgWidth = 64
imgHeight = 64
```

```python
scene = loadTriangleScene(imgWidth, imgHeight)
print(f"Start rendering in resolution: {scene.config['width']}x{scene.config['height']}, fov: {scene.config['fov']}")
startTime = time.monotonic()
img = renderRaytracing(scene)
duration = time.monotonic() - startTime
print("Finished rendering, time taken:", duration, "seconds")

plt.imshow(img)

print("Writing PNG...")
plt.imsave(f"triangle-raytracing.png", img)

print("Writing EXR...")
output = np.power(img, 2.2)
pyexr.write(f"triangle-raytracing.exr", output)
```

## Question 3 - Ray tracing the Cornell Box (15 points)

Having successfully rendered a triangle, we are now ready to render the famous [*Cornell Box* scene](https://www.graphics.cornell.edu/online/box/data.html).

In this question, we will test our ray tracer code from above, on a more complex scene. If your above implementation was correct, it will NOT require any changes and should seamlessly run on the Cornell Box scene.

The code cell below loads the scene description into memory.

```python
def loadCornellBox(width, height):
    objects = [
    # Floor
    Triangle([552.8, 0, 0], [0, 0, 0], [0, 0, 559.2], Diffuse([1, 1, 1])),
    Triangle([552.8, 0, 0], [0, 0, 559.2], [552.8, 0, 559.2], Diffuse([1, 1, 1])),
    # Ceiling
    Triangle([556.0, 548.8, 0.0], [556.0, 548.8, 559.2],[0.0, 548.8, 559.2], Diffuse([1, 1, 1])),
    Triangle([556.0, 548.8, 0.0], [0.0, 548.8, 559.2], [0.0, 548.8, 0.0], Diffuse([1, 1, 1])),
    # Back Wall
    Triangle([549.6, 0.0, 559.2], [0.0, 0.0, 559.2], [0.0, 548.8, 559.2], Diffuse([1, 1, 1])),
    Triangle([549.6, 0.0, 559.2], [0.0, 548.8, 559.2], [556.0, 548.8, 559.2], Diffuse([1, 1, 1])),
    # Right Wall
    Triangle([0.0, 0.0, 559.2 ], [0.0, 0.0, 0.0], [0.0, 548.8,   0.0], Diffuse([0, 1, 0])),
    Triangle([0.0, 0.0, 559.2], [0.0, 548.8, 0.0], [0.0, 548.8, 559.2], Diffuse([0, 1, 0])),
    # Left Wall
    Triangle([552.8, 0.0, 0.0], [549.6, 0.0, 559.2], [556.0, 548.8, 559.2], Diffuse([1, 0, 0])),
    Triangle([552.8, 0.0, 0.0], [556.0, 548.8, 559.2], [556.0, 548.8, 0.0], Diffuse([1, 0, 0])),
    # Short Block
    Triangle([130.0, 165.0,  65.0], [82.0, 165.0, 225.0], [240.0, 165.0, 272.0],  Diffuse([1, 1, 1])),
    Triangle([130.0, 165.0,  65.0], [240.0 ,165.0, 272.0], [290.0, 165.0, 114.0], Diffuse([1, 1, 1])),
    Triangle([290.0, 0.0, 114.0], [290.0, 165.0, 114.0], [240.0, 165.0, 272.0], Diffuse([1, 1, 1])),
    Triangle([290.0, 0.0, 114.0], [240.0, 165.0, 272.0], [240.0, 0.0, 272.0], Diffuse([1, 1, 1])),
    Triangle([130.0, 0.0, 65.0], [130.0, 165.0, 65.0], [290.0, 165.0, 114.0], Diffuse([1, 1, 1])),
    Triangle([130.0, 0.0, 65.0], [290.0, 165.0, 114.0], [290.0, 0.0, 114.0], Diffuse([1, 1, 1])),
    Triangle([82.0, 0.0, 225.0], [82.0, 165.0, 225.0], [130.0, 165.0, 65.0], Diffuse([1, 1, 1])),
    Triangle([82.0, 0.0, 225.0], [130.0, 165.0, 65.0], [130.0, 0.0, 65.0], Diffuse([1, 1, 1])),
    Triangle([240.0, 0.0, 272.0], [240.0, 165.0, 272.0], [82.0, 165.0, 225.0], Diffuse([1, 1, 1])),
    Triangle([240.0, 0.0, 272.0], [82.0, 165.0, 225.0], [82.0, 0.0, 225.0], Diffuse([1, 1, 1])),
    # Tall Block
    Triangle([423.0, 330.0, 247.0], [265.0, 330.0, 296.0], [314.0, 330.0, 456.0], Diffuse([1, 1, 1])),
    Triangle([423.0, 330.0, 247.0], [314.0, 330.0, 456.0], [472.0, 330.0, 406.0], Diffuse([1, 1, 1])),
    Triangle([423.0, 0.0, 247.0], [423.0, 330.0, 247.0], [472.0, 330.0, 406.0], Diffuse([1, 1, 1])),
    Triangle([423.0, 0.0, 247.0], [472.0, 330.0,  406.0], [472.0, 0.0, 406.0], Diffuse([1, 1, 1])),
    Triangle([472.0, 0.0, 406.0], [472.0, 330.0,406.0], [314.0, 330.0, 456.0], Diffuse([1, 1, 1])),
    Triangle([472.0, 0.0, 406.0], [314.0, 330.0, 456.0], [314.0, 0.0, 456.0], Diffuse([1, 1, 1])),
    Triangle([314.0, 0.0, 456.0], [314.0, 330.0, 456.0], [265.0, 330.0, 296.0], Diffuse([1, 1, 1])),
    Triangle([314.0, 0.0, 456.0], [265.0,  330.0, 296.0], [265.0, 0.0, 296.0], Diffuse([1, 1, 1])),
    Triangle([265.0, 0.0, 296.0], [265.0, 330.0, 296.0], [423.0, 330.0, 247.0], Diffuse([1, 1, 1])),
    Triangle([265.0, 0.0, 296.0], [423.0, 330.0, 247.0], [423.0, 0.0, 247.0], Diffuse([1, 1, 1])),
    ]
    light = PointLight([278, 508, 279.5], [5e6, 5e6, 5e6])
    camera = Camera(np.array([278, 273 , -800]),np.array([278, 273, 0]), np.array([0, 1, 0]))
    config = { "width": width, "height": height, "fov": 38 }
    return Scene(objects, light, camera, config)
```

If all of your ray tracing code is correctly set up, the following code cell should produce an image of the Cornell Box scene. Note that the orignal image may be accessed at [this webpage](https://www.graphics.cornell.edu/online/box/data.html).

Since we currently use a low resolution output image (64 x 64), rendering should take about 20 seconds (on a Google Colab CPU-only instance). Once you are satisfied with the output, **render a 512 x 512 image**. This image will be the deliverable for this question.

(Note that rendering at a `512 x 512` resolution may take several minutes, so please plan ahead of time)

```python
imgWidth = 64
imgHeight = 64
```

```python
scene = loadCornellBox(imgWidth, imgHeight)
print(f"Start rendering in resolution: {imgWidth}x{imgHeight}, fov: {scene.config['fov']}")
startTime = time.monotonic()
img = renderRaytracing(scene)
duration = time.monotonic() - startTime
print("Finished rendering, time taken:", duration, "seconds")

plt.imshow(img)

print("Writing PNG...")
plt.imsave(f"cbox-raytracing.png", img)

print("Writing EXR...")
output = np.power(img, 2.2)
pyexr.write(f"cbox-raytracing.exr", output)
```

## Question 4 - Rasterizing the Cornell Box (25 points)

For rasterizing the Cornell Box scene, we will need to implement a Z-buffer, to ensure only the object closest to the camera at each pixel is rendered. Recall that, for a triangle, we only had a single primitive; hence we did not need to perform Z-buffering.

In this question, you will modify the `rasterize()` method you implemented in Question 1, and will implement a more general `renderRasterization()` method that includes a visibility test by means of a Z-buffer. In particular:
* Factor in the position and intensity of the scene's position light
* Initialize a Z-buffer (an array of size equal to that of pixels in the output image). Initialize depth for each pixel to a large value or infinity (`np.inf`).
* Loop over all triangles, and rasterize them (keeping in mind the bullet points below)
* Compute the depth with respect to the camera ($z^{'}$) by interpolating the pixel location within each triangle's vertices.
* If the new depth (w.r.t. the camera) is smaller than the value stored in the Z-buffer for the pixel, shade the pixel using the diffuse reflection of this new triangle at the shade point.

Hint: keep visualizing intermediate stages of your Z-buffer to get a better sense for debugging.

```python
def renderRasterization(scene):
    # DO NOT modify these three lines
    width = scene.config["width"]
    height = scene.config["height"]
    img = np.zeros((height, width, 3))

    zbuf = np.full((height, width), np.inf)
    # TODO: Implement the rasterization loop here

    # transformation matrix from world space to NDC space
    fovy = scene.config["fov"]
    PVM = perspective(fovy * math.pi / 180, width / height, 0.1, 10) @ lookAt(scene.cam.eye, scene.cam.at,
                                                                                scene.cam.up)

    # set up point light
    light_pos = np.asarray(scene.light.p)
    light_em = np.asarray(scene.light.e)

    for obj in scene.geometry:
        rasterize_q4(scene, obj, img, zbuf)
    return img
```

Finally, rasterize the Cornell Box by running the two cells below.
The rendered result should look identical to what you get with ray tracing.

For faster debugging, we currently set `imgWidth` and `imgHeight` to `64`. **For the eventual solution, you are required to render a `512 x 512` image**. (Note that rendering at this resolution may take several minutes, so please plan ahead of time).

```python
imgWidth = 64
imgHeight = 64
```

```python
scene = loadCornellBox(imgWidth, imgHeight)
print(f"Start rendering in resolution: {imgWidth}x{imgHeight}, fov: {scene.config['fov']}")
startTime = time.monotonic()
img = renderRasterization(scene)
duration = time.monotonic() - startTime
print("Finished rendering, time taken:", duration, "seconds")

plt.imshow(img)

print("Writing PNG...")
plt.imsave(f"cbox-rasterization.png", img)

print("Writing EXR...")
output = np.power(img, 2.2)
pyexr.write(f"cbox-rasterization.exr", output)
```

---

## **Bonus Questions**

The following questions are for **extra credit**, and are not mandatory.

**Only attempt this part if you have finished all required questions**. These questions build atop Questions 1 through 4.

**No partial points will be awarded on these questions.**

**You may choose to attempt one or more bonus questions, and in no particular order**.

The bonus questions are not autograded -- you are free to use as many cells and define functions as needed. You may not, however, import libraries other than those approved for this assignment. If in doubt, please talk to the TAs or the instructor.

## Bonus Question 1 - Shading normals (15 points)

This question involves computing the shading normals for a triangle mesh.

Download the Stanford bunny `.obj` file available [here](https://graphics.stanford.edu/~mdfisher/Data/Meshes/bunny.obj). Read it into a `numpy` array (you may use the `ObjLoader` class provided [here](https://inareous.github.io/posts/opening-obj-using-py)).

Note how the bunny obj file only has vertex and face information, but no normals? The objective of this bonus question is to compute the _shading normals_ for this triangle mesh and render it using your choice of rasterization / ray tracing implementation from above. With proper shading normals, the bunny should look smooth (except at the silhouettes), instead of faceted).

(Hint: you will find the discussion on shading normals from the end of the rasterization module slides useful.)

## Bonus Question 2 - Phong illumination (15 points)

For this bonus question, you will implement a Phong BRDF / reflection model. Recall, from the lectures on phenomenological shading and appearance models how Phong specular reflection models work. Augment the ray tracer you just implemented by adding a Phong illumination model to get realistic reflections of the light source on object surfaces.

```python
def shade_phong(scene, hit, triangle, kd = 0.6, ks = 0.5, p = 8):
    """
        Method to shade a hit point
        Parameters:
          scene: scene to render.
          hit: hit point to shade.
          triangle: hit triangle object
          kd: diffuse coefficient
          ks: specular coefficient
        Returns:
          radiance at the hit point
        """
    # TODO: implement this
    light = scene.light
    light_pos, light_flux = light.p, np.asarray(light.e)

    cam = scene.cam

    r_i = distance(light_pos, hit)
    l_i = normalize(light_pos - hit)

    r_cam = distance(cam.eye, hit)
    l_cam =  normalize(cam.eye - hit)

    h = normalize(l_cam + l_i)

    n_tmp = triangle.normal(hit)
    n = (2 * (np.dot(l_cam, n_tmp) > 0) - 1) * n_tmp

    # Compute Diffuse Reflection
    I = light_flux / (4 * np.pi ** 2 * (r_i ** 2))
    L_diffuse = I * max([0, np.dot(n, l_i)])

    # Compute Phong Reflection
    lambertian = max(np.dot(-n, l_i), 0.0)
    L_specular = 0.
    if lambertian > 0.:
        r_r = normalize(-l_i - 2* np.dot(-l_i, n) * n)
        L_specular = math.pow(max(0, np.dot(r_r, l_cam)), p)

    # Add together
    radiance_unclip = kd * L_diffuse * triangle.mat.c + ks * L_specular * np.array(([1,1,1]))
    radiance = np.array(([min(1, v) for v in radiance_unclip]))
    return radiance # Replace this with your return statement, when implementation is ready

def trace_phong(scene, ray):
    """
        Trace a ray in the scene, and shade the hit point
        Parameters:
          scene: scene to render
          ray: ray to trace
        Returns:
          light radiance at the hit point
        """
    intersection = intersect(scene, ray)
    has_hit = intersection[0]
    if has_hit is False:
        return np.zeros(3)
    hit_id = intersection[1]  # object id
    t = intersection[2]  # distance

    return shade_phong(scene, ray.o + ray.d * t, scene.geometry[hit_id])

def renderRaytracing_phong(scene):
    """
        Ray tracer main rendering function
        Parameters:
          scene: scene to render
        Returns:
          rendered image
        """
    print("Rendering the scene")

    # TODO: implement this
    # Hints:
    # 1. Set up the model view projection transforms (use the `width`, `height`,
    #    `fov` parameters from the `scene.config` dictionary for this).
    # 2. Loop over all pixels in a ray, and follow the instructions from the
    #    question to complete the ray tracing routine.
    width, height, fovy = scene.config["width"], scene.config["height"], scene.config["fov"] * np.pi / 180
    aspect = width / height
    n = cam_near
    t = n * math.tan(fovy / 2)
    r = t * aspect

    cam = scene.cam
    cam2World = np.linalg.inv(lookAt(cam.eye, cam.at, cam.up))
    x, y, z, eye = cam2World[:-1, 0], cam2World[:-1, 1], cam2World[:-1, 2], cam2World[:-1, 3]

    imgBuffer = np.zeros(shape=(height, width, 3))
    # pixelBuffer = np.zeros(shape=(width, height, 2))

    # set the pivot point at the top left
    for i in range(0, width):
        for j in range(0, height):
            u, v = -r + ((i + 0.5) * 2 * r / width), t - ((j + 0.5) * 2 * t / height)
            # pixel_Cam= np.asarray([u, v, d])
            # dir_Cam = hp.normalize(pixel_Cam)
            dir = normalize(u * x + v * y - n * z)
            ray = Ray(eye, dir)
            imgBuffer[j, i] = trace_phong(scene, ray)

    # Return your image
    return imgBuffer
```

```python
imgWidth = 512
imgHeight = 512
scene = loadCornellBox(imgWidth, imgHeight)
startTime = time.monotonic()
img = renderRaytracing_phong(scene)
duration = time.monotonic() - startTime
print("Finished rendering, time taken:", duration, "seconds")
plt.imshow(img)
plt.show()
print("Writing PNG...")
plt.imsave(f"cbox-raytracing-phong.png", img)
print("Writing EXR...")
output = np.power(img, 2.2)
```

## Bonus Question 3 - Ray tracing with hard shadows (25 points)

Notice how your ray tracing outputs do not contain any shadows? In this bonus question, you will perform ray tracing with additional _shadow rays_. In the phenomenological shading module, we saw how to extend the `trace()` routine to cast an additional shadow ray to determine whether a shade point is in  shadow w.r.t. a light source. We recommend you do this on the Cornell Box scene, but remember to use a low resolution image for debugging.

```python
def shade_shadow(scene, hit, triangle):
    """
        Method to shade a hit point
        Parameters:
          scene: scene to render.
          hit: hit point to shade.
          triangle: hit triangle object
        Returns:
          radiance at the hit point
        """
    # TODO: implement this
    light = scene.light
    light_pos, light_flux = light.p, np.asarray(light.e)

    cam = scene.cam
    cam_pos = cam.eye

    diff_l = light_pos - hit
    r = distance(light_pos, hit)
    li = normalize(diff_l)

    ray = Ray(light_pos, -li, r)
    hasblocked = any(tri.intersect(ray) != 0 for tri in scene.geometry)
    if not hasblocked:
        diff_c = normalize(cam_pos - hit)
        normal = triangle.normal(hit)
        tmp = 2 * (np.dot(diff_c, normal) > 0)

        n = normal * (tmp - 1)

        cosine = max(0, np.dot(n, li))
        a = light_flux / (4 * np.pi * (r ** 2))
        k = p / np.pi
        b = k * a * cosine
        radiance = b * triangle.mat.c
        radiance = np.array([min(1, radiance[0]), min(1, radiance[1]), min(1, radiance[2])])
        # radiance = triangle.mat.c
        return radiance  # Replace this with your return statement, when implementation is ready
    else:
        return np.zeros(3)

def trace_shadow(scene, ray):
    """
        Trace a ray in the scene, and shade the hit point
        Parameters:
          scene: scene to render
          ray: ray to trace
        Returns:
          light radiance at the hit point
        """
    intersection = intersect(scene, ray)
    has_hit = intersection[0]
    if has_hit is False:
        return np.zeros(3)
    hit_id = intersection[1]  # object id
    t = intersection[2]  # distance

    return shade_shadow(scene, ray.o + ray.d * t, scene.geometry[hit_id])

def renderRaytracing_shadow(scene):
    """
        Ray tracer main rendering function
        Parameters:
          scene: scene to render
        Returns:
          rendered image
        """
    print("Rendering the scene")

    # TODO: implement this
    # Hints:
    # 1. Set up the model view projection transforms (use the `width`, `height`,
    #    `fov` parameters from the `scene.config` dictionary for this).
    # 2. Loop over all pixels in a ray, and follow the instructions from the
    #    question to complete the ray tracing routine.
    width, height, fovy = scene.config["width"], scene.config["height"], scene.config["fov"] * np.pi / 180
    aspect = width / height
    n = cam_near
    t = n * math.tan(fovy / 2)
    r = t * aspect

    cam = scene.cam
    cam2World = np.linalg.inv(lookAt(cam.eye, cam.at, cam.up))
    x, y, z, eye = cam2World[:-1, 0], cam2World[:-1, 1], cam2World[:-1, 2], cam2World[:-1, 3]

    imgBuffer = np.zeros(shape=(height, width, 3))
    # pixelBuffer = np.zeros(shape=(width, height, 2))

    # set the pivot point at the top left
    for i in range(0, width):
        for j in range(0, height):
            u, v = -r + ((i + 0.5) * 2 * r / width), t - ((j + 0.5) * 2 * t / height)
            # pixel_Cam= np.asarray([u, v, d])
            # dir_Cam = hp.normalize(pixel_Cam)
            dir = normalize(u * x + v * y - n * z)
            ray = Ray(eye, dir)
            imgBuffer[j, i] = trace_shadow(scene, ray)

    # Return your image
    return imgBuffer

imgWidth = 64
imgHeight = 64
scene = loadCornellBox(imgWidth, imgHeight)
startTime = time.monotonic()
img = renderRaytracing_shadow(scene)
duration = time.monotonic() - startTime
print("Finished rendering, time taken:", duration, "seconds")

plt.imshow(img)
plt.show()
print("Writing PNG...")
plt.imsave(f"cbox-rasterization.png", img)

print("Writing EXR...")
output = np.power(img, 2.2)
```