# Percentages of flights belonging to a distance group that halved a delay

In this notebook we compute the percentage of flights belonging to a given "distance group" that were able to halve their departure delays by the time they arrived at their destinations. Distance groups assort flights by their total distance in miles. Flights with distances that are less than 200 miles belong in group 1, flights with distances that are between 200 and 399 miles belong in group 2, flights with distances that are between 400 and 599 miles belong in group 3, and so on. The last group contains flights whose distances are between 2400 and 2599 miles.

So basically:

- By the _Distance_ attribute we can infer the flight group:

\begin{equation}
FlightGroup = \left( Distance // 200 \right) + 1
\end{equation}

- For each flight we can say if it has halved or not its delay:

\begin{equation}
    DelayHalved=
    \begin{cases}
      yes, & \text{if}\ \left(DepDealy \neq 0\right) \quad \textrm{and} \quad \left(ArrDelay <= \frac{DepDealy}{2} \right)\\
      no, & \text{otherwise}
    \end{cases}
\end{equation}

- So, for each flight group we can create two sets of flight: those that have halved their delays and the others

### Assumptions

- The last group, as the assignment states, is group ( 2599 // 200 ) + 1 = 13
- Each flight not belonging to a valid group is not considered

#### Execution mode

If development mode is turned on, all the calculations will be performed on a reduced instance of the data set.

This allows to *save computation time* while developing.

```python
# development mode
# False when performing real analytics
DEV = False

### threads to be used to run spark worker nodes locally
spark_local_threads = 4

### max number of flights groups
max_flight_group = 13

# default transformed dataset path
transformed_dataset_path = '../dataset/03_cache_distance_groups_percentages_halved_delay.parquet'
missings_dataframe_path = '../dataset/03_cache_distance_groups_percentages_halved_delay_missing.pkl'
dropped_info_path = '../dataset/03_cache_dropped_info.pkl'
```

#### Find and import Apache Spark on current machine

```python
import findspark
findspark.init()
```

```python
# imports
import pickle
from copy import copy

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from pyspark.sql.types import *

import numpy as np
import pandas as pd

from pathlib import Path

from missing_values_utils import *
```

#### Create a new session

```python
# Build a Spark SQL Session for DataFrames
master = 'local[{}]'.format(spark_local_threads)
appName = 'Airport distance groups percentages'
spark = SparkSession  \
    .builder          \
    .appName(appName) \
    .master(master)   \
    .getOrCreate()
```

### Load data

If there exists a cached version of the already transformed dataset, just load it and skip all the unnecessary computation.

```python
do_computation = True

if not DEV and Path(transformed_dataset_path).is_dir():
    df = spark.read.load(transformed_dataset_path)
    do_computation = False
```

Try to load the optimized parquet format data set. If parquet data set is not found, load full compressed data sets, reduce and save them.

```python
from preprocessing_utils import *

if do_computation:
    if DEV:
        # DEV preprocessing
        perform_DEV_dataset_preprocessing(spark)
    else:
        # Production preprocessing
        perform_dataset_preprocessing(spark)
```

    Starting preprocessing of ../dataset/*.csv.bz2
    Preprocessing NOT performed.
    Preprocessed dataset already exists: ../dataset/preprocessed_dataset.parquet

```python
# Load the parquet dataset
if do_computation:
    if DEV:
        # Load DEV dataset
        df = load_DEV_preprocessed_dataset(spark)
    else:
        # Load production dataset
        df = load_preprocessed_dataset(spark)
```

    Peprocessed dataset loaded.
    ../dataset/preprocessed_dataset.parquet

### Attributes selection

```python
if do_computation:
    # Keep only the dimensions we need
    df = df.select('Year', 'ArrDelay', 'DepDelay','Distance')
    # Explore the schema
    df.printSchema()
```

    root
     |-- Year: integer (nullable = true)
     |-- ArrDelay: string (nullable = true)
     |-- DepDelay: string (nullable = true)
     |-- Distance: string (nullable = true)

### Missing values exploration and exclusion

```python
dropped_info = {}

dropped_info['total'] = N_before = df.count()
N_before
```

    91469371

```python
df, df_missing = get_df_missing_and_drop_all_null(df, do_computation, missings_dataframe_path)
```

    Years examined:
     [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008]

    Missing values inspection:
    0  null values in:  Year
    2151526  null values in:  ArrDelay
    1949479  null values in:  DepDelay
    28936  null values in:  Distance

            Count  MissingsCount  MissingsPercentage
    1994  5180048         101637            0.019621
    1995  5327435         108295            0.020328
    1996  5351983         142657            0.026655
    1997  5411843         109844            0.020297
    1998  5384721         157670            0.029281
    1999  5527884         167866            0.030367
    2000  5683047         201744            0.035499
    2001  5967780         244107            0.040904
    2002  5271359          73499            0.013943
    2003  6488540         112851            0.017392
    2004  7129270         141541            0.019854
    2005  7140596         147758            0.020693
    2006  7141922         138120            0.019339
    2007  7453215         177927            0.023873
    2008  7009728         154699            0.022069

```python
N_after = df.count()
N_after
```

    89289156

```python
dropped_info['due_null']  = N_before - N_after

print("# of rows \n\t BEFORE: {:,}\n\t  AFTER: {:,}\n\tDROPPED: {:,}".format(N_before, N_after, dropped_info['due_null'] ))

df.show(5)
```

    # of rows
    	 BEFORE: 91,469,371
    	  AFTER: 89,289,156
    	DROPPED: 2,180,215
    +----+--------+--------+--------+
    |Year|ArrDelay|DepDelay|Distance|
    +----+--------+--------+--------+
    |2007|       1|       7|     389|
    |2007|       8|      13|     479|
    |2007|      34|      36|     479|
    |2007|      26|      30|     479|
    |2007|      -3|       1|     479|
    +----+--------+--------+--------+
    only showing top 5 rows

```python
if do_computation:
    # Keep only the dimensions we need
    df = df.select('ArrDelay', 'DepDelay','Distance')

    # cast delays to integer
    df = df.withColumn('ArrDelay', df['ArrDelay'].cast(IntegerType()))
    df = df.withColumn('DepDelay', df['DepDelay'].cast(IntegerType()))
    df = df.withColumn('Distance', df['Distance'].cast(IntegerType()))

    df.printSchema()
```

    root
     |-- ArrDelay: integer (nullable = true)
     |-- DepDelay: integer (nullable = true)
     |-- Distance: integer (nullable = true)

### Compute the flight group

#### Which is the maximum Distance?

```python
if DEV:
    max_row = df.rdd.max(key=lambda x: int(x["Distance"]))

    print(max_row) # Row(ArrDelay='-13', DepDelay='6', Distance='4502')
```

```python
if do_computation:
    # Calculate group
    df = df.withColumn("Group", F.floor(df.Distance / 200) + 1)

#df.show(100)
#df.show()
```

#### Cut out all the groups above the considered limit

All flights that were categorized in groups above 13, will be dropped.

```python
if do_computation:
    N_before = df.count()
    # cut out all the groups above the considered limit
    df = df.filter(df.Group <= max_flight_group)
    N_after = df.count()

    dropped_info['due_group_limit'] = N_before - N_after

    print("# of rows \n\t BEFORE: {:,}\n\t  AFTER: {:,}\n\tDROPPED: {:,}".format(N_before, N_after, dropped_info['due_group_limit']))
```

    # of rows
    	 BEFORE: 89,289,156
    	  AFTER: 88,845,727
    	DROPPED: 443,429

### Delay was halved or not?

```python
if do_computation:
    def delay_halved(dep_delay, arr_delay):
        # case in which the flight was departed on time
        # it cannot happen that it is able to halve its delay. There's no delay!
        if dep_delay == 0:
            return 0

        if arr_delay <=  (dep_delay / 2):
            return 1

        # delay was not halved: arr_delay > dep_delay / 2
        return 0

    delay_halved = F.udf(delay_halved, IntegerType())

    # Calculate if the delay was halved or not
    df = df.withColumn('DelayHalved', delay_halved(df['DepDelay'], df['ArrDelay']))

    #df.show()
```

```python
if do_computation:
    # create count support column for total number of flights per group
    df = df.withColumn('Count', F.lit(1))

    df = df.groupBy('Group')                                      \
            .sum('Count', 'DelayHalved')                          \
            .withColumnRenamed('sum(Count)', 'Count')             \
            .withColumnRenamed('sum(DelayHalved)', 'DelayHalvedCount')

    #df.show()
```

```python
if do_computation:
    df.write.mode('overwrite').save(transformed_dataset_path, format='parquet')
```

```python
if Path(dropped_info_path).is_file() and not do_computation:
    # load from cache
    with open(r'' + dropped_info_path, 'rb') as pf:
        dropped_info = pickle.load(pf)
else:
    # cache it
    with open(r'' + dropped_info_path, 'wb') as pf:
        pickle.dump(dropped_info, pf)
```

```python
# data frame rows should be only 13 rows, feasible to retrieve as pandas.DataFrame
pdf = df.toPandas()
```

```python
pdf
```

      Group
      Count
      DelayHalvedCount

      0
      7
      2835992
      1311225

      1
      6
      6493682
      2947424

      2
      9
      2359585
      1084546

      3
      5
      10031980
      4549856

      4
      1
      8279746
      3571480

      5
      10
      1472944
      679377

      6
      3
      15961771
      6632157

      7
      12
      898501
      432107

      8
      8
      3318934
      1487152

      9
      11
      1036110
      485696

      10
      2
      22984521
      8688902

      11
      4
      11634173
      5040541

      12
      13
      1537788
      750274

```python
pdf = pdf.sort_values(by='Group').reset_index(drop=True)
pdf
```

      Group
      Count
      DelayHalvedCount

      0
      1
      8279746
      3571480

      1
      2
      22984521
      8688902

      2
      3
      15961771
      6632157

      3
      4
      11634173
      5040541

      4
      5
      10031980
      4549856

      5
      6
      6493682
      2947424

      6
      7
      2835992
      1311225

      7
      8
      3318934
      1487152

      8
      9
      2359585
      1084546

      9
      10
      1472944
      679377

      10
      11
      1036110
      485696

      11
      12
      898501
      432107

      12
      13
      1537788
      750274

```python
from pprint import pprint
from collections import OrderedDict
halved_delays = OrderedDict()

# Check sum of yes and no for halved delay correspond to the total number of flights of the group
for i, r in pdf.iterrows():
    halved_delays[i+1] = {
        'group_no': i+1,
        'distance_from': 200*i,
        'distance_to': (200*(i+1)-1),
        'total_n_flights': r['Count'],
        'halved_yes': r['DelayHalvedCount'],
        'halved_no': r['Count'] - r['DelayHalvedCount'],
        'perc_do_halved_delay': r['DelayHalvedCount'] / r['Count']
    }

pprint(halved_delays)
```

    OrderedDict([(1,
                  {'distance_from': 0,
                   'distance_to': 199,
                   'group_no': 1,
                   'halved_no': 4708266,
                   'halved_yes': 3571480,
                   'perc_do_halved_delay': 0.4313513965283476,
                   'total_n_flights': 8279746}),
                 (2,
                  {'distance_from': 200,
                   'distance_to': 399,
                   'group_no': 2,
                   'halved_no': 14295619,
                   'halved_yes': 8688902,
                   'perc_do_halved_delay': 0.3780327638761756,
                   'total_n_flights': 22984521}),
                 (3,
                  {'distance_from': 400,
                   'distance_to': 599,
                   'group_no': 3,
                   'halved_no': 9329614,
                   'halved_yes': 6632157,
                   'perc_do_halved_delay': 0.4155025780034058,
                   'total_n_flights': 15961771}),
                 (4,
                  {'distance_from': 600,
                   'distance_to': 799,
                   'group_no': 4,
                   'halved_no': 6593632,
                   'halved_yes': 5040541,
                   'perc_do_halved_delay': 0.43325305545997983,
                   'total_n_flights': 11634173}),
                 (5,
                  {'distance_from': 800,
                   'distance_to': 999,
                   'group_no': 5,
                   'halved_no': 5482124,
                   'halved_yes': 4549856,
                   'perc_do_halved_delay': 0.4535351944481548,
                   'total_n_flights': 10031980}),
                 (6,
                  {'distance_from': 1000,
                   'distance_to': 1199,
                   'group_no': 6,
                   'halved_no': 3546258,
                   'halved_yes': 2947424,
                   'perc_do_halved_delay': 0.4538910282332889,
                   'total_n_flights': 6493682}),
                 (7,
                  {'distance_from': 1200,
                   'distance_to': 1399,
                   'group_no': 7,
                   'halved_no': 1524767,
                   'halved_yes': 1311225,
                   'perc_do_halved_delay': 0.4623514452791122,
                   'total_n_flights': 2835992}),
                 (8,
                  {'distance_from': 1400,
                   'distance_to': 1599,
                   'group_no': 8,
                   'halved_no': 1831782,
                   'halved_yes': 1487152,
                   'perc_do_halved_delay': 0.4480812212595972,
                   'total_n_flights': 3318934}),
                 (9,
                  {'distance_from': 1600,
                   'distance_to': 1799,
                   'group_no': 9,
                   'halved_no': 1275039,
                   'halved_yes': 1084546,
                   'perc_do_halved_delay': 0.45963421533871424,
                   'total_n_flights': 2359585}),
                 (10,
                  {'distance_from': 1800,
                   'distance_to': 1999,
                   'group_no': 10,
                   'halved_no': 793567,
                   'halved_yes': 679377,
                   'perc_do_halved_delay': 0.461237494432918,
                   'total_n_flights': 1472944}),
                 (11,
                  {'distance_from': 2000,
                   'distance_to': 2199,
                   'group_no': 11,
                   'halved_no': 550414,
                   'halved_yes': 485696,
                   'perc_do_halved_delay': 0.46876876007373736,
                   'total_n_flights': 1036110}),
                 (12,
                  {'distance_from': 2200,
                   'distance_to': 2399,
                   'group_no': 12,
                   'halved_no': 466394,
                   'halved_yes': 432107,
                   'perc_do_halved_delay': 0.48091988767959076,
                   'total_n_flights': 898501}),
                 (13,
                  {'distance_from': 2400,
                   'distance_to': 2599,
                   'group_no': 13,
                   'halved_no': 787514,
                   'halved_yes': 750274,
                   'perc_do_halved_delay': 0.48789169898581597,
                   'total_n_flights': 1537788})])

##### Data visualization functions

Scroll down to see the charts

```python
from matplotlib.font_manager import FontProperties
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

def show_missing_pie_chart(fig_size=[7,7], title_size=20):
    """
    Show a pie char representing the data excluded due to missing values
    """
    global dropped_info

    plt.rcParams['figure.figsize'] = fig_size
    plt.rcParams['axes.titlesize'] = title_size

    total_minus_missing = dropped_info['total'] - dropped_info['due_null']

    slices = [total_minus_missing, dropped_info['due_null']]
    labels = [
        'Valid data [{:,} records]'.format(slices[0]),
        'Dropped due to missing values [{:,} records]'.format(slices[1])
    ]

    fig, ax = plt.subplots();
    wedges, texts, autotexts = ax.pie( slices, labels=labels,
                                       colors=['#009900','#e60000'],
                                       autopct='%1.2f%%',
                                       startangle=90,
                                       textprops=dict(color="w", fontproperties=FontProperties(size=14)));
    ax.set_title("Missing values");

    ax.axis('equal');

    ax.legend(wedges, labels,
              loc="center left",
              fontsize=14,
              bbox_to_anchor=(1, 0, 0.5, 1))

def show_group_invalid_pie_chart(fig_size=[7,7], title_size=20):
    """
    Show a pie char representing the data excluded due to group number > 13
    """
    global dropped_info

    plt.rcParams['figure.figsize'] = fig_size
    plt.rcParams['axes.titlesize'] = title_size

    total_minus_missing = dropped_info['total'] - dropped_info['due_null']
    total_minus_missing_minus_groups = total_minus_missing - dropped_info['due_group_limit']

    slices = [total_minus_missing_minus_groups, dropped_info['due_group_limit']]
    labels = [
        'Valid group value [{:,} records]'.format(slices[0]),
        'Dropped due to group > 13 [{:,} records]'.format(slices[1])
    ]

    fig, ax = plt.subplots();
    wedges, texts, autotexts = ax.pie( slices, labels=labels,
                                       colors=['#009900','#990073'],
                                       autopct='%1.2f%%',
                                       startangle=90,
                                       textprops=dict(color="w", fontproperties=FontProperties(size=14)));
    ax.set_title("Dropped due to invalud group");

    ax.axis('equal');

    ax.legend(wedges, labels,
              loc="center left",
              fontsize=14,
              bbox_to_anchor=(1, 0, 0.5, 1));

def show_dropped_summary_pie_chart(fig_size=[7,7], title_size=20):
    """
    Show a pie chart that summarize and compare the excluded data.
    """
    global dropped_info

    plt.rcParams['figure.figsize'] = fig_size
    plt.rcParams['axes.titlesize'] = title_size

    total_minus_missing = dropped_info['total'] - dropped_info['due_null']
    total_minus_missing_minus_groups = total_minus_missing - dropped_info['due_group_limit']

    slices = [total_minus_missing_minus_groups, dropped_info['due_null'], dropped_info['due_group_limit']]
    labels = [
        'Orinal total data [{:,} records]'.format(dropped_info['total']),
        'Dropped due to missing [{:,} records]'.format(slices[1]),
        'Dropped due to group > 13 [{:,} records]'.format(slices[2]),
        'Totally dropped [{:,} records]'.format(dropped_info['total'] - total_minus_missing_minus_groups),
        'Valid data [{:,} records]'.format(slices[0])
    ]

    radius=1.2
    size = 0.2

    fig, ax = plt.subplots()
    wedges, texts, autotexts = ax.pie(slices,
                                      colors=['#009900', '#e60000', '#990073'],
                                      radius=radius,
                                      autopct='%1.2f%%',
                                      pctdistance=radius,
                                      wedgeprops=dict(width=size, edgecolor='w'),
                                      textprops=dict(fontproperties=FontProperties(size=14)))

    #------------------------------------------
    # Weges hack: since the legends of the first pie
    # chart will be only of the slices passed, here
    # below add a copy of the wedge ad first element
    # in the weges
    w_orig_data = copy(wedges[0])
    w_orig_data.set_color('#3385ff')

    w_totally_dropped = copy(wedges[0])
    w_totally_dropped.set_color('#ffffff')
    #rearrange order according to labels
    wedges = [w_orig_data, wedges[1], wedges[2], w_totally_dropped, wedges[0]]

    ax.legend(wedges,
              labels,
              loc="center left",
              fontsize=14,
              bbox_to_anchor=(1.2, 0, 0.5, 1))
    #------------------------------------------

    wedges, texts, autotexts = ax.pie([dropped_info['total']],
                                      colors=['#3385ff'],
                                      radius=radius-size,
                                      autopct='',
                                      wedgeprops=dict(width=0.1, edgecolor='w'))

    ax.set(aspect="equal", title='Dropped values summary')

    plt.show()

def show_group_halved_delays_pie_chart(group):
    """
    Show a pie chart representing the percentage of halved delays for a group.

        Example of group structure
        group = {
            'distance_from': 0,
            'distance_to': 199,
            'group_no': 1,
            'halved_no': 338014,
            'halved_yes': 199997,
            'perc_do_halved_delay': 0.3717340351777194,
            'total_n_flights': 538011
        }
    """
    plt.rcParams['figure.figsize'] = [7,7]
    plt.rcParams['axes.titlesize'] = 20

    slices = [group['halved_yes'], group['halved_no']]
    labels = [
        'Group {} [{:,} flights]'.format(group['group_no'], sum(slices)),
        'Halved delay [{:,} flights]'.format(slices[0]),
        'Not halved delay [{:,} flights]'.format(slices[1]),
    ]

    fig, ax = plt.subplots();
    wedges, texts, autotexts = ax.pie( slices,
                                       colors=['#009900','#990073'],
                                       autopct='%1.2f%%',
                                       startangle=90,
                                       textprops=dict(color="w", fontproperties=FontProperties(size=14)))
    ax.set_title("Group {}".format(group['group_no']))

    ax.axis('equal')

    w_tot_flights = copy(wedges[0])
    w_tot_flights.set_color('#ffffff')
    wedges = [w_tot_flights] + wedges

    ax.legend(wedges, labels,
              loc="center left",
              fontsize=14,
              bbox_to_anchor=(1, 0, 0.5, 1))
    plt.show()

def show_percetages_comparison_horizontal_stack_chart(fig_size=[12,10], title_size=20):
    """
    Show an horizontal stacked bar chart that compares the percetages of halved delays.
    """
    global halved_delays

    plt.rcParams['figure.figsize'] = fig_size
    plt.rcParams['axes.titlesize'] = title_size

    fig, ax = plt.subplots()
    #ax = plt.figure().add_subplot(111)

    patch_handles = []
    for group_no, g in halved_delays.items():
        percentages = [
            g['perc_do_halved_delay']*100,
            (1-g['perc_do_halved_delay'])*100
        ]

        width=percentages
        left=[0, width[0]]

        patch_handles.append(
            ax.barh(group_no, width=width, align='center', left=left, color=['#009900','#990073'])
        )
        for i, patch in enumerate(patch_handles[-1]):
            bl = patch.get_xy()
            x = 0.5*patch.get_width() + bl[0]
            y = 0.5*patch.get_height() + bl[1]
            ax.text(x, y, "{0:.2f}%".format(round(percentages[i],2)), ha='center',va='center', color='w')

    y_pos = range(1, len(halved_delays)+1)
    ax.set_yticks(y_pos)
    ax.set_yticklabels(halved_delays.keys())
    ax.set_ylabel('Group', fontsize=16)
    ax.set_xlabel('Percentage', fontsize=16)

    legend_elements = [
        matplotlib.lines.Line2D([0], [0], color='#009900', lw=4, label='Halved delay'),
        matplotlib.lines.Line2D([0], [0], color='#990073', lw=4, label='Not halved delay')
    ]

    ax.legend(handles=legend_elements,
              fontsize=14,
              loc="center left",
              bbox_to_anchor=(1, 0, 0.5, 1))
    ax.invert_yaxis()
    ax.set_title("Comparison of halved delay percentages per group", pad=20)
    plt.show()
```

## Data visualization

Analytics for the percentage of flights belonging to a given "distance group" that were able to halve their departure delays are reported below.

## Data exluded from visualization

### Missing values and valid data

```python
plot_missing_values_stacked_bar(df_missing)
```

The following chart show the percentage of dropped rows due to missing values.

```python
show_missing_pie_chart()
```

### Invalid group value

The following chart show the percentage of dropped rows due to group value exceeding the last valid group (13).

```python
show_group_invalid_pie_chart()
```

### Summary of excluded data

The following chart show the summary of dropped rows

```python
show_dropped_summary_pie_chart()
```

## Percentages of flights belonging to a distance group that halved a delay

### Delay per group

The following chart show the percentage of halved delay per group

```python
def ui_callback(group):
    global halved_delays
    show_group_halved_delays_pie_chart(halved_delays[group])

groups_w = widgets.Dropdown(options=halved_delays.keys(),
                                  value=list(halved_delays.keys())[0],
                                  description='Group')

ui = widgets.HBox([groups_w])
out = widgets.interactive_output(ui_callback, {'group': groups_w})

display(ui, out)
```

    HBox(children=(Dropdown(description='Group', options=(1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13), value=1),))

    Output()

## Comparison of halved delay percentages

The following stack bar chart compares the percentages of halved delay for all groups

```python
show_percetages_comparison_horizontal_stack_chart()
```


 Image Interpolation

 INF-285 Computación Científica

 Autor: Francisco Andrades

Lenguaje: Python

Temas:

    - Image Interpolation
    - Interpolación Bicúbica
    - Lagrange, Newton, Spline

```python
import numpy as np
import sympy as sp
from PIL import Image
from scipy import interpolate
import matplotlib.pyplot as plt
```

## Introducción

En la siguiente tarea estudiaremos un método de interpolación denominado **Interpolación Bicúbica**, utilizada frecuentemente sobre imágenes. Aplicaremos el método para aumentar la resolución de una imagen intentando preservar las propiedades de la versión original.

## Contexto

Supongamos que usted conoce $f$ y las derivadas $f_x$, $f_y$ y $f_{xy}$ dentro de las coordenadas $(0,0),(0,1),(1,0)$ y $(1,1)$ de un cuadrado unitario. La superficie que interpola estos 4 puntos es:

$$
p(x,y) = \sum\limits_{i=0}^3 \sum_{j=0}^3 a_{ij} x^i y^j.
$$

Como se puede observar el problema de interpolación se resume en determinar los 16 coeficientes $a_{ij}$ y para esto se genera un total de $16$ ecuaciones utilizando los valores conocidos de $f$,$f_x$,$f_y$ y $f_{xy}$. Por ejemplo, las primeras $4$ ecuaciones son:

$$
\begin{aligned}
f(0,0)&=p(0,0)=a_{00},\\
f(1,0)&=p(1,0)=a_{00}+a_{10}+a_{20}+a_{30},\\
f(0,1)&=p(0,1)=a_{00}+a_{01}+a_{02}+a_{03},\\
f(1,1)&=p(1,1)=\textstyle \sum \limits _{i=0}^{3}\sum \limits _{j=0}^{3}a_{ij}.
\end{aligned}
$$

Para las $12$ ecuaciones restantes se debe utilizar:

$$
\begin{aligned}
f_{x}(x,y)&=p_{x}(x,y)=\textstyle \sum \limits _{i=1}^{3}\sum \limits _{j=0}^{3}a_{ij}ix^{i-1}y^{j},\\
f_{y}(x,y)&=p_{y}(x,y)=\textstyle \sum \limits _{i=0}^{3}\sum \limits _{j=1}^{3}a_{ij}x^{i}jy^{j-1},\\
f_{xy}(x,y)&=p_{xy}(x,y)=\textstyle \sum \limits _{i=1}^{3}\sum \limits _{j=1}^{3}a_{ij}ix^{i-1}jy^{j-1}.
\end{aligned}
$$

Una vez planteadas las ecuaciones, los coeficientes se pueden obtener al resolver el problema $A\alpha=x$, donde $\alpha=\left[\begin{smallmatrix}a_{00}&a_{10}&a_{20}&a_{30}&a_{01}&a_{11}&a_{21}&a_{31}&a_{02}&a_{12}&a_{22}&a_{32}&a_{03}&a_{13}&a_{23}&a_{33}\end{smallmatrix}\right]^T$ y ${\displaystyle x=\left[{\begin{smallmatrix}f(0,0)&f(1,0)&f(0,1)&f(1,1)&f_{x}(0,0)&f_{x}(1,0)&f_{x}(0,1)&f_{x}(1,1)&f_{y}(0,0)&f_{y}(1,0)&f_{y}(0,1)&f_{y}(1,1)&f_{xy}(0,0)&f_{xy}(1,0)&f_{xy}(0,1)&f_{xy}(1,1)\end{smallmatrix}}\right]^{T}}$.

En un contexto más aplicado, podemos hacer uso de la interpolación bicúbica para aumentar la resolución de una imagen. Supongamos que tenemos la siguiente imagen de tamaño $5 \times 5$:

Podemos ir tomando segmentos de la imagen de tamaño $2 \times 2$ de la siguiente forma:

Por cada segmento podemos generar una superficie interpoladora mediante el algoritmo de interpolación cubica. Para el ejemplo anterior estariamos generando $16$ superficies interpoladoras distintas. La idea es hacer uso de estas superficies para estimar los valores de los pixeles correspondienets a una imagen más grande. Por ejemplo, la imagen $5 \times 5$ la podemos convertir a una imagen de $9 \times 9$ agregando un pixel entre cada par de pixeles originales además de uno en el centro para que no quede un hueco.

Aca los pixeles verdes son los mismos que la imagen original y los azules son obtenidos de evaluar cada superficie interpoladora. Notar que existen pixeles azules que se pueden obtener a partir de dos superficies interpoladoras distintas, en esos casos se puede promediar el valor de los pixeles o simplemente dejar uno de los dos.

Para trabajar con la interpolación bicubica necesitamos conocer los valores de $f_x$, $f_y$ y $f_{xy}$. En el caso de las imagenes solo tenemos acceso al valor de cada pixel por lo que deberemos estimar cual es el valor de estos. Para estimar $f_x$ haremos lo siguiente:

Para estimar el valor de $f_x$ en cada pixel haremos una interpolación con los algoritmos conocidos, usando tres pixels en dirección de las filas, luego derivaremos el polinomio obtenido y finalmente evaluaremos en la posición de interes. La misma idea aplica para $f_y$ solo que ahora interpolaremos en dirección de las columnas.

Por ejemplo si queremos obtener el valor de $f_x$ en la posición $(0,0)$ (imagen de la izquierda) entonces haremos una interpolación de Lagrange utilizando los pixeles $(0,0),(0,1)$ y $(0,2)$. Derivaremos el polinomio interpolador y evaluaremos en $(0,0)$. Por otro lado si queremos obtener el valor de $f_y$ en la posición $(0,0)$ (imagen de la derecha) entonces interpolaremos los pixeles $(0,0),(1,0)$ y $(2,0)$. Luego derivaremos el polinomio interpolador y evaluaremos en $(0,0)$.

Para obtener $f_{xy}$ seguiremos la idea anterior. Solo que esta vez se utilizaran los valores de $f_y$ y se interpolaran estos en dirección de las filas.

# Preguntas

## 1. Interpolación bicubica

### 1.1  Obtener derivadas (30 puntos)

Implemente la función `derivativeValues` que reciba como input un arreglo con valores, el método de interpolación y si es que se considera el uso de  los puntos de chebyshev . La función debe retornar un arreglo de igual dimensión con los valores de las derivadas de los puntos obtenidas

Los métodos de interpolación serán representados por los siguientes valores

* Interpolación de lagrange: `'lagrange'`
* Diferencias divididas de Newton: `'newton'`
* Spline cubica: `'spline3'`

```python
def chebyshevNodes(n):
    i = np.arange(1, n+1)
    t = (2*i - 1) * np.pi / (2 * n)
    return np.cos(t)

def newtonDD(x_i, y_i):
    n = x_i.shape[-1]
    pyramid = np.zeros((n, n)) # Create a square matrix to hold pyramid
    pyramid[:,0] = y_i # first column is y
    for j in range(1,n):
        for i in range(n-j):
            # create pyramid by updating other columns
            pyramid[i][j] = (pyramid[i+1][j-1] - pyramid[i][j-1]) / (x_i[i+j] - x_i[i])
    a = pyramid[0] # f[ ... ] coefficients
    N = lambda x: a[0] + np.dot(a[1:], np.array([np.prod(x - x_i[:i]) for i in range(1, n)]))
    return N

def calcular(values1,values2,values3, method, cheb,number):
    y = np.array((values1,values2,values3))
    x = np.array((0,1,2))
    if cheb:
        x = chebyshevNodes(3)
        x.sort()
    xS = sp.symbols('x', reals=True)
    if(method == 'lagrange'):
        L = interpolate.lagrange(x,y)
        deriv = np.polyder(L)
        return deriv(x[number])
    if(method == 'newton'):
        Pn = newtonDD(x, y)
        L = Pn(xS)
        deriv = sp.diff(L,xS)
    if(method=='spline3'):
        deriv = interpolate.CubicSpline(x, y)
        deriv = deriv.derivative()
        return deriv(x[number])
    return deriv.evalf(subs = {xS : x[number]})

calcular_v = np.vectorize(calcular)

#recibe fila de 1 dimension
def derivativeValues(fila,method,cheb):
    """
    Parameters
    ----------
    values:	(int array) points values
    method:	(string)	interpolation method
    cheb:	(boolean)	if chebyshev points are used

    Returns
    -------
    d:	(float array) derivative value of interpolated points

    """
    shape = fila.shape
    nuevo = np.zeros(shape)
    nuevo[1:shape[0]-1] = calcular_v(fila[0:shape[0]-2],fila[1:shape[0]-1],fila[2:shape[0]],method,cheb,1)
    nuevo[0] = calcular_v(fila[0],fila[1],fila[2], method, cheb,0)
    nuevo[shape[0]-1] = calcular_v(fila[shape[0]-3],fila[shape[0]-2],fila[shape[0]-1], method, cheb,2)
    return nuevo
```

### 1.2 Interpolación de imagen (50 puntos)
Implemente la función `bicubicInterpolation` que reciba como input la matriz de la imagen y cuantos píxeles extra se quiere agregar entre los píxeles originales y el algoritmo de interpolación a utilizar. La función debe retornar la matriz con la imagen de dimensión nueva. Considere que se debe aplicar el método de interpolación en cada canal RGB por separado.

```python
def obtain_all_derivatives(image,method,cheb):
        shape = image.shape
        nuevo_x = np.zeros(shape)
        nuevo_y = np.zeros(shape)
        nuevo_xy = np.zeros(shape)
        for i in range(shape[2]):
            nuevo_y[:,:,i] = np.array([derivativeValues(n, method, cheb) for n in image[:,:,i].T]).T
            nuevo_x[:,:,i] = np.array([derivativeValues(n, method, cheb) for n in image[:,:,i]])
            nuevo_xy[:,:,i] = np.array([derivativeValues(n, method, cheb) for n in nuevo_y[:,:,i]])

        return nuevo_x,nuevo_y,nuevo_xy

def bicubicInterpolation(image, interiorPixels, method,cheb):
    """
    Parameters
    ----------
    image:	(nxnx3 array) image array in RGB format
    interiorPixels:	(int)	interpolation method
    method:	(string)	interpolation method
    cheb:	(boolean)	if chebyshev points are used

    Returns
    -------
    newImage:	(nxnx3 array) image array in RGB format

    """
    matriz = np.array(((1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0),(0,0,0,0,1,0,0,0,0,0,0,0,0,0,0,0),(-3,3,0,0,-2,-1,0,0,0,0,0,0,0,0,0,0),(2,-2,0,0,1,1,0,0,0,0,0,0,0,0,0,0),
                      (0,0,0,0,0,0,0,0,1,0,0,0,0,0,0,0),(0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0),(0,0,0,0,0,0,0,0,-3,3,0,0,-2,-1,0,0),
                      (0,0,0,0,0,0,0,0,2,-2,0,0,1,1,0,0),(-3,0,3,0,0,0,0,0,-2,0,-1,0,0,0,0,0),(0,0,0,0,-3,0,3,0,0,0,0,0,-2,0,-1,0),
                      (9,-9,-9,9,6,3,-6,-3,6,-6,3,-3,4,2,2,1),(-6,6,6,-6,-3,-3,3,3,-4,4,-2,2,-2,-2,-1,-1),(2,0,-2,0,0,0,0,0,1,0,1,0,0,0,0,0),
                      (0,0,0,0,2,0,-2,0,0,0,0,0,1,0,1,0),(-6,6,6,-6,-4,-2,4,2,-3,3,-3,3,-2,-1,-2,-1),(4,-4,-4,4,2,2,-2,-2,2,-2,2,-2,1,1,1,1)))

    shape = image.shape
    nueva_imagen = np.zeros((shape[0]*(interiorPixels+1)-interiorPixels,shape[1]*(interiorPixels+1)-interiorPixels,shape[2]),dtype=image.dtype)
    nuevo_x, nuevo_y, nuevo_xy = obtain_all_derivatives(image,method,cheb)

    for j in range(shape[0]-1):
        for i in range(shape[0]-1):
            for rgb in range(shape[2]):
                array = np.array((image[i,j,rgb],image[i+1,j,rgb],image[i,j+1,rgb]
                                  ,image[i+1,j+1,rgb],nuevo_x[i,j,rgb],nuevo_x[i+1,j,rgb]
                                  ,nuevo_x[i,j+1,rgb],nuevo_x[i+1,j+1,rgb],nuevo_y[i,j,rgb]
                                  ,nuevo_y[i+1,j,rgb],nuevo_y[i,j+1,rgb],nuevo_y[i+1,j+1,rgb]
                                  ,nuevo_xy[i,j,rgb],nuevo_xy[i+1,j,rgb],nuevo_xy[i,j+1,rgb],nuevo_xy[i+1,j+1,rgb]))
                a = matriz.dot(array.T)
                P = lambda x,y: np.sum([a[i]*(x**(i%4))*y**(int(i/4)) for i in range(16)])

                numero_fila = (interiorPixels + 1)*i
                numero_columna = (interiorPixels+1)*j

                #rellenar
                for cont in range(interiorPixels+2):
                    for cont1 in range(interiorPixels+2):
                        value = P(cont1/(interiorPixels+1),cont/(interiorPixels+1))
                        if(value > 255):
                            value = 255
                        if(value < 0):
                            value = 0
                        if(nueva_imagen[numero_fila+cont1,numero_columna+cont,rgb] != 0):
                            value = (nueva_imagen[numero_fila+cont1,numero_columna+cont,rgb]+value)/2
                        nueva_imagen[numero_fila+cont1,numero_columna+cont,rgb] = value

    return nueva_imagen

img = Image.open('sunset.png')
img = img.convert('RGB')
array=np.array(img)
array_nuevo = bicubicInterpolation(array, 4, 'spline3',False)
#original
plt.imshow(img)
plt.show()
#interpolada
plt.imshow(array_nuevo)
plt.show()
```

```python
print("Tamaño Original: ",array.shape)
print("Interpolada: ", array_nuevo.shape)
```

    Tamaño Original:  (100, 100, 3)
    Interpolada:  (496, 496, 3)

## 2. Evaluacion de algoritmos

### 2.1 Tiempo de ejecucion
Implemente la funcion `timeInterpolation` que mida el tiempo de interpolacion de una imagen dado el algoritmo de interpolacion , en segundos.(5 puntos)

```python
import time
def timeInterpolation(image, interiorPixels, method,cheb):
    """
    Parameters
    ----------
    image:	(nxnx3 array) image array in RGB format
    interiorPixels:	(int)	interpolation method
    method:	(string)	interpolation method
    cheb:	(boolean)	if chebyshev points are used

    Returns
    -------
    time:	(float) time in seconds

    """
    time1 = time.time()
    bicubicInterpolation(image, interiorPixels, method,cheb)
    time2 = time.time()
    return time2-time1
```

***Pregunta: ¿Cual es el metodo que presenta mayor velocidad en general? (5 puntos)***

'spline3' es el método con mayor velocidad

### 2.2 Calculo de error
Implemente la funcion `errorInterpolation` la cual debe obtener el error de la imagen obtenida comparandola con una de referencia. El error debe ser calculado utilizando el indice SSIM (Structural similarity) (5 puntos)

```python
from skimage import metrics
def errorInterpolation(original,new):
    """
    Parameters
    ----------
    image:	(nxn array) original image array in RGB format
    new:	(nxn array) new image array in RGB format obtained from interpolation

    Returns
    -------
    error:	(float) difference between images

    """

    s = metrics.structural_similarity(original, new, multichannel = True)
    return 1-s
```

***Pregunta: ¿Cual metodo presenta menor error? (5 puntos)***

Depende.
Para gradient con 1 pixel, 'lagrange'.
Para gradient con 4 pixel, 'spline3'.
Para sunset con 1 pixel, 'lagrange'.
Para sunset con 2 pixel, 'lagrange'.

Cabe destacar que los errores son muy parecidos, con una diferencia entre 10^-5 y 10^-6.

Referencias:
chebyshevNodes(), NewtonDD() sacados del jupiter del curso.


# Alternating Direction Method of Multipliers

Although gradient descent is a reliable algorithm that is guaranteed to converge, it is still slow. If we want to process larger sets of data (e.g. 3D imaging), have a live feed of DiffuserCam, or just want to process images more quickly, we need to tailor the algorithm more closely to the optical system involved. While this introduces more tuning parameters ("knobs" to turn), speed of reconstruction can be drastically improved. Here we present (without proof) the result of using alternating direction method of multipiers (ADMM) to  reconstruct the image.

For background on ADMM, please refer to sections 2 and 3 of: http://stanford.edu/~boyd/papers/pdf/admm_distr_stats.pdf. To understand this document, background knowledge from Chapters 5 (Duality) and 9 (Unconstrained minimization) from https://web.stanford.edu/~boyd/cvxbook/bv_cvxbook.pdf may be necessary.

For a detailed derivation of the update steps as they apply to our system, refer to the companion documents to this notebook (specific sections will be referenced).

#### Generic setup code (load psf, etc.)

```python
%matplotlib inline
import numpy as np
import numpy.fft as fft
from PIL import Image
import matplotlib.pyplot as plt
from IPython import display
```

The code takes in two grayscale images: a point spread function (PSF) $\texttt{psfname}$ and a sensor measurement $\texttt{imgname}$. The images can be downsampled by a factor $f$, which is a power of $\frac{1}{2}$

```python
"""Stuff that is normally in the config file"""
psfname = "./psf_sample.tif"
imgname = "./rawdata_hand_sample.tif"

# Downsampling factor (used to shrink images)
f = 0.25

# Hyper-parameters in the ADMM implementation (like step size in GD)
mu1 = 1e-6
mu2 = 1e-5
mu3 = 4e-5
tau = 0.0001

# Number of iterations
iters = 5
```

```python
def loadData(show_im=True):
    psf = Image.open(psfname)
    psf = np.array(psf, dtype='float32')
    data = Image.open(imgname)
    data = np.array(data, dtype='float32')

    """In the picamera, there is a non-trivial background
    (even in the dark) that must be subtracted"""
    bg = np.mean(psf[5:15,5:15])
    psf -= bg
    data -= bg

    """Resize to a more manageable size to do reconstruction on.
    Because resizing is downsampling, it is subject to aliasing
    (artifacts produced by the periodic nature of sampling). Demosaicing is an attempt
    to account for/reduce the aliasing caused. In this application, we do the simplest
    possible demosaicing algorithm: smoothing/blurring the image with a box filter"""

    def resize(img, factor):
        num = int(-np.log2(factor))
        for i in range(num):
            img = 0.25*(img[::2,::2,...]+img[1::2,::2,...]+img[::2,1::2,...]+img[1::2,1::2,...])
        return img

    psf = resize(psf, f)
    data = resize(data, f)

    """Now we normalize the images so they have the same total power. Technically not a
    necessary step, but the optimal hyperparameters are a function of the total power in
    the PSF (among other things), so it makes sense to standardize it"""

    psf /= np.linalg.norm(psf.ravel())
    data /= np.linalg.norm(data.ravel())

    if show_im:
        fig1 = plt.figure()
        plt.imshow(psf, cmap='gray')
        plt.title('PSF')
#         display.display(fig1)
        fig2 = plt.figure()
        plt.imshow(data, cmap='gray')
        plt.title('Raw data')
#         display.display(fig2)
    return psf, data
```

```python
"""The "uncropped" size of the image. As with the gradient descent, we pad the images
so that convolution is linear instead of circular"""
psf, data = loadData(True)
sensor_size = np.array(psf.shape)
full_size = 2*sensor_size
```

$\newcommand\sensorco{\mathbf{x}} % vector containing (x,y) sensor coordinates$
$\newcommand\objectco{\bm \xi} % vector containing (x,y) object coordinates$
$\newcommand\measurementvec{\mathbf{b}}$
$\newcommand\measurementmtx{\mathbf{M}}$
$\newcommand\full{\mathbf{A}}$
$\newcommand\imagevec{\mathbf{v}}$
$\newcommand{\crop}{\mathbf{C}}$
$\newcommand{\rhcomment}[1]{{\bf{{\blue{{RH --- #1}}}}}}$
$\newcommand{\argmin}{\text{argmin }}$

Recall the objective function:

\begin{equation} \begin{aligned} \hat{\imagevec} = \underset{w\geq 0,u,x}{\argmin}&\frac{1}{2}\|\measurementvec - \crop x\|_2^2  + \tau \|u\|_1\\ \mbox{s.t. } &x = \measurementmtx \imagevec, w = \imagevec\text{,} \end{aligned} \end{equation}

For reference, the corresponding ADMM update steps derived in the companion document are:
\begin{align*}
    u_{k+1} &\leftarrow \mathcal{T}_{\frac{\tau}{\mu_2}} \left(\Psi \imagevec_k + \frac{\eta_k}{\mu_2}\right) \\
    x_{k+1} &\leftarrow \left(\crop^H \crop + \mu_1 I\right)^{-1}
\left(\xi_k + \mu_1 \mathbf{M}\imagevec_k + \crop^H \measurementvec\right) \\
w_{k+1} &\leftarrow \max(\rho_k/\mu_3 +\imagevec_k, 0) \\
\imagevec_{k+1} &\leftarrow (\mu_1 \mathbf{M}^H \mathbf{M} + \mu_2 \Psi^H \Psi + \mu_3 I)^{-1} r_k, \\
    \xi_{k+1} & \leftarrow \xi_k + \mu_1(\measurementmtx \imagevec_k - x_{k+1}) \\
    \eta_{k+1} & \leftarrow \eta_k + \mu_2(\Psi \imagevec_{k+1} - u_{k+1}) \\
    \rho_{k+1} & \leftarrow \rho_k + \mu_3(\imagevec_{k+1} - w_{k+1})
\end{align*}
where $$r_k = (\mu_3 w_{k+1} - \rho_k) + \Psi^H(\mu_2 u_{k+1} - \eta_k) + \mathbf{M}^H (\mu_1 x_{k+1} - \xi_k).$$

## We now demonstrate how each of these updates is implemented.

### The $u$-update (total variation update)
\begin{equation}
u_{k+1} \leftarrow \mathcal{T}_{\frac{\tau}{\mu_2}} \left(\Psi \imagevec_k + \eta_k/\mu_2\right)
\end{equation}

```python
def U_update(eta, image_est, tau):
    return SoftThresh(Psi(image_est) + eta/mu2, tau/mu2)
```

#### Implementing $\mathcal{T}$
$\mathcal{T}$ is a soft-thresholding operator: we zero out all values within $\tau/\mu_2$ of $0$, and all other components of the vector are decreased in magnitude by $\tau/\mu_2$. Note that $\tau$ and $\mu_2$ are regularization "hyperparameters" -- constants we can tune to increase or decrease the effect of different terms in the objective function. For example, increasing $\tau$ increases the contribution of $\|u\|_1$ to the objective function, so the solution has lower total variation and more "defined" edges.

A clean way of writing soft-thresholding component-wise is:

$$\left[\mathcal{T}_{\tau} (x)\right]_i = \mathrm{sgn}(x_i)\max(0,|x_i| - \tau)$$

```python
def SoftThresh(x, tau):
    # numpy automatically applies functions to each element of the array
    return np.sign(x)*np.maximum(0, np.abs(x) - tau)
```

#### Implementing $\Psi$
$\Psi \imagevec_k$ is the gradient of the image estimate (whose norm is the "total variation" of the image). Because images are made of discrete pixels, we approximate this gradient by a finite-difference. Here we choose to use the 2D forward-difference with a circular boundary condition, defined by the following relation. See Section 2.2 of the Appendix for an explanation of the boundary condition:
$$\Psi v_{ij} = \begin{bmatrix}v_{i+1,j} - v_{i,j} \\ v_{i,j+1}-v_{i,j} \end{bmatrix}$$

The choice of finite difference does not have a large effect on the final result, as long as the corresponding adjoint (see the $\imagevec$-update) is calculated correctly.

We implement the pixel-wise definition above in parallel using numpy operations: $\texttt{np.roll}$ shifts the image circularly, so if we subtract $\texttt{v}$ from $\texttt{np.roll(v,1,axis=0)}$ we perform all the row-wise finite differences in a single matrix subtraction. We can do the same for the column differences.

$\Psi$ maps a pixel to a length-2 vector, where the first element is the row difference and the second is the column difference. $\texttt{np.stack}$ allows us to do this for every pixel in the image using the matrices calculated with $\texttt{roll}$. For an $n \times m$ image $v$, $\Psi(v)$ returns a $n\times m \times 2$ stack of two images (the horizontal and vertical differences).

```python
def Psi(v):
    return np.stack((np.roll(v,1,axis=0) - v, np.roll(v, 1, axis=1) - v), axis=2)
```

### The $x$-update (uncropped image update)
$$x_{k+1} \leftarrow \left(\crop^H \crop + \mu_1 I\right)^{-1}
\left(\xi_k + \mu_1 \measurementmtx\imagevec_k + \crop^H \measurementvec\right)$$

```python
def X_update(xi, image_est, H_fft, sensor_reading, X_divmat):
    return X_divmat * (xi + mu1*M(image_est, H_fft) + CT(sensor_reading))
```

#### Implementing $\measurementmtx$
First consider the right-hand part of the update, which doesn't require any inversion. $\xi_k$ is one of the dual variables, so we will cover how it is calculated later. $\mu_1$ is a hyperparameter like above.

$\measurementmtx$ is the convolution operator:
$$\measurementmtx \imagevec_k \iff h \ast v_k,$$ where $h$ is the point spread function of the diffuser. We implement this convolution in the same way that we did in the gradient descent algorithm:
$$h \ast v = \mathcal{F}^{-1} \left\{(\mathcal{F}(h))\cdot (\mathcal{F}(v))\right\} = \ \texttt{crop}\left[\ \texttt{DFT}^{-1}\left\{\ \texttt{DFT} [\ \texttt{pad}[h]\ ]\cdot\texttt{DFT}[\ \texttt{pad}[v]\ ]\ \right\} \right]$$

However, there are 2 details that change the implementation slightly. First, due to our variable splitting, we've actually absorbed the "sensor crop" (which crops the image from $\texttt{full_size}$ to $\texttt{sensor_size}$) into the $\crop$ operator. So, we don't need to crop down the convolution at this stage; it will be taken care of separately. Second, we can precalculate the padded DFT of $h$ (denoted $\texttt{H}\_\texttt{fft}$ in code) and our input $v_k$ is already padded (it has the dimensions of $\texttt{full_size}$). So, there is no need to pad them again.

Thus our final form for $\measurementmtx$ is:
$$\texttt{DFT}^{-1} \left[ \texttt{DFT}[v_k] \cdot \texttt{H}\_\texttt{fft} \right]$$

```python
def M(vk, H_fft):
    return np.real(fft.fftshift(fft.ifft2(fft.fft2(fft.ifftshift(vk))*H_fft)))
```

#### Implementing $\crop^H$ and $\crop$
The next term is $\crop^H \measurementvec$. First, as defined in the "Guide to modeling and algorithms", $\crop$ is the operator that accounts for the sensor cropping (it crops from size $\texttt{full_size}$ to size $\texttt{sensor_size}$). The adjoint $\crop^H$ zero-pads an image of size $\texttt{sensor_size}$ to an image of size $\texttt{full_size}$. We have chosen $\texttt{full_size}$ in such a way that it provides enough padding to make circular and linear convolutions look the same after being cropped back down to $\texttt{sensor_size}$.

We implement $\crop^H$ by padding the sensor reading $\measurementvec$ "up" to $\texttt{full_size}$:

```python
def C(M):
    # Image stored as matrix (row-column rather than x-y)
    top = (full_size[0] - sensor_size[0])//2
    bottom = (full_size[0] + sensor_size[0])//2
    left = (full_size[1] - sensor_size[1])//2
    right = (full_size[1] + sensor_size[1])//2
    return M[top:bottom,left:right]

def CT(b):
    v_pad = (full_size[0] - sensor_size[0])//2
    h_pad = (full_size[1] - sensor_size[1])//2
    return np.pad(b, ((v_pad, v_pad), (h_pad, h_pad)), 'constant',constant_values=(0,0))
```

#### Implementing $\left(\crop^H \crop + \mu_1 I\right)^{-1}$
Lastly, the inverse term. While we have largely avoided dealing with the actual matrices used to represent operators such as cropping, convolution, and padding, here we need to consider what they look like, because inverting the operators efficiently will implicitly use properties of the matrix representation.

Let's go through the process with a linear function (or operator) $O$ that operates on images. Let $\mathbf{O}$ be the matrix corresponding to that operator. Recall from the algorithms guide (https://waller-lab.github.io/DiffuserCam/tutorial/algorithm_guide.pdf) that if $O$ operates on $m\times n$ images, then $\mathbf{O}$ is an $mn \times mn$-dimensional matrix, operating on vectorized images of length $mn$.

Now consider the image $y = O(x)$. In matrix vector form, we can write this as $\mathbf{y} = \mathbf{O}\mathbf{x}$. This equation tells us explicitly that every pixel $\mathbf{y}_j$ is a linear combination of the pixels in $\mathbf{x}$: $\mathbf{y}_j = \sum_i\mathbf{O}_{ij} x_i$.

So, if $\mathbf{O}$ is diagonal, then we find $\mathbf{y}_j = \mathbf{O}_{jj} x_j$. In other words, a diagonal operator $O$ scales the $j$th pixel by factor $\mathbf{O}_{jj}$! Importantly, the inverse of a diagonal matrix is also diagonal, with all the entries inverted. So, the inverse operator $O^{-1}$ also scales each pixel individually, now by factor $\dfrac{1}{\mathbf{O}_{jj}}$ (remember that $j$ is an index we give to the pixel based on the vectorized image). The main idea is that we can multiply each pixel by the appropriate weight without regard to other pixels in the image, which can easily be done in parallel.

Now we apply the process for $\mathbf{O} = (\crop^H \crop + \mu_1 I)$:

$\crop^H \crop$ is a crop down to an image of size $\texttt{sensor_size}$ and then a zero-pad back up to $\texttt{full_size}$. In other words, it zeros out any pixels outside of the cropped region, and leaves everything inside the cropped region untouched. We know this corresponds to a diagonal matrix because it is pointwise multiplication of each pixel by either a $0$ or a $1$. So, $\mathbf{O}$ corresponds to a pointwise multiplication of every pixel by either $0 + \mu_1$ or $1 + \mu_1$. The inverse of this operator is thus pointwise multiplication of every pixel by either $\dfrac{1}{\mu_1}$ or $\dfrac{1}{1 + \mu_1}$.

Later, we will see an example of inversion where the operator is diagonalizable in a different, non-standard basis (not pixel-wise) -- the basic idea will remain the same.

Since $\mathbf{O}^{-1}$ doesn't change with each iteration, we precompute the entries. Whenever we need to calculate its action on an image, we use numpy element-wise division with the stored entries.
We create the multiplication mask of $\mu_1$'s and $1+\mu_1$'s by first creating the submask of ones (with size $\texttt{sensor_size}$) using $\texttt{np.ones}$. We can pad to the right size using the zero-pad function $\texttt{CT}$:

```python
def precompute_X_divmat():
    """Only call this function once!
    Store it in a variable and only use that variable
    during every update step"""
    return 1./(CT(np.ones(sensor_size)) + mu1)
```

### The $w$-update (non-negativity update)
$$w_{k+1} \leftarrow \max(\rho_k/\mu_3 +\imagevec_k, 0)$$

Updates for $\rho_k$ and $\imagevec_k$ will be calculated later, so we can assume we have the variables while implementing $w_{k+1}$.

```python
def W_update(rho, image_est):
    return np.maximum(rho/mu3 + image_est, 0)
```

### The $\imagevec$-update (Image estimate update!)
$$\imagevec_{k+1} \leftarrow (\mu_1 \measurementmtx^H \measurementmtx + \mu_2 \Psi^H \Psi + \mu_3 I)^{-1} r_k,$$
where $$r_k = (\mu_3 w_{k+1} - \rho_k) + \Psi^H(\mu_2 u_{k+1} - \eta_k) + \measurementmtx^H (\mu_1 x_{k+1} - \xi_k)$$

While long, most of these terms are expressions involving other variables that have already been calculated. There are three things that require new calculations: $\Psi^H$ (adjoint of finite difference), $\measurementmtx^H$ (adjoint of convolution), and the large inverse.

```python
def r_calc(w, rho, u, eta, x, xi, H_fft):
    return (mu3*w - rho)+PsiT(mu2*u - eta) + MT(mu1*x - xi, H_fft)

def V_update(w, rho, u, eta, x, xi, H_fft, R_divmat):
    freq_space_result = R_divmat*fft.fft2( fft.ifftshift(r_calc(w, rho, u, eta, x, xi, H_fft)) )
    return np.real(fft.fftshift(fft.ifft2(freq_space_result)))
```

####  Implementing $\Psi^H$
We calculate the adjoint of the finite difference in the appendix of the companion document. Because $\Psi$ maps a single image to two images stacked on top of each other, $\Psi^H$ must map a stack of two images to a single image. Let each "pixel" in the stack be $u_{ij} = \begin{bmatrix} u_{ij}^x & u_{ij}^y \end{bmatrix}^H$.

Then, $$\Psi^H u_{ij} = (u_{i-1,j}^x - u_{i,j}^x) + (u_{i,j-1}^y - u_{i,j}^y)$$

In words, the adjoint of the 2D forward difference (i.e. of the form $v_{j+1}- v_j$) is the sum of two backward differences (i.e. of the form $u_{i-1} - u_i$).

Again, since all the boundary conditions are circular, we can use $\texttt{np.roll}$ to help us parallelize the pixel-wise differences. $\texttt{PsiT}$ takes in an $n\times m \times 2$ matrix and outputs a $n\times m$ matrix.

```python
def PsiT(U):
    diff1 = np.roll(U[...,0],-1,axis=0) - U[...,0]
    diff2 = np.roll(U[...,1],-1,axis=1) - U[...,1]
    return diff1 + diff2
```

#### Implementing $\measurementmtx^H$

This is the adjoint of the convolution operator, which we calculated in the implementation of gradient descent as:
$$\measurementmtx^H x = \mathcal{F}^{-1}\left\{ \mathcal{F}(\mathbf{h})^* \cdot \mathcal{F}\left(\text{pad}[x]\right)\right\}$$

Again, our input is already padded and we can pre-calculate the DFT of $h$. So, we get $$\measurementmtx^H x = \texttt{DFT}^{-1} \left[ \texttt{DFT}[x] \cdot \texttt{H}\_\texttt{fft}^* \right]$$

```python
def MT(x, H_fft):
    x_zeroed = fft.ifftshift(x)
    return np.real(fft.fftshift(fft.ifft2(fft.fft2(x_zeroed) * np.conj(H_fft))))
```

#### Implementing $(\mu_1 \measurementmtx^H \measurementmtx + \mu_2 \Psi^H \Psi + \mu_3 I)^{-1}$

If we try to calculate the matrix entries for acting $\mu_1 \measurementmtx^H \measurementmtx$ on a vectorized image like before, we find that the product is not diagonal, because convolutions can depend on many different pixels at once by definition.

In general, large matrix inverses are the main bottleneck in an algorithm; we need to be careful to make sure that all inverses involved in our algorithm are easily calculated via some method that doesn't require instantiating the matrix. In the previous case, we were lucky because both $\crop^H \crop$ and $\mu_1 I$ are diagonalizable in the same standard basis, so multiplying by its inverse is the same as dividing by the summed entries. Our only option here is to find a different basis where all the terms are diagonalizable.

In fact, the matrix corresponding to $\measurementmtx^H \measurementmtx$ is not diagonal in the standard basis precisely because it is diagonal in a different basis! As we showed in the Gradient Descent notebook:
$$\begin{align}
\measurementmtx^H \measurementmtx \imagevec &= \mathbf{F}^{-1} \mathrm{diag}(\mathbf{Fh})^* \ \mathrm{diag}(\mathbf{Fh}) \ \mathbf{F} \imagevec \\
&= \mathcal{F}^{-1} \left\{ \mid\mathcal{F}h \mid^2 \cdot (\mathcal{F}v) \right\},
\end{align}$$

we find that it performs pixel-wise multiplication in the Fourier (frequency) space. In other words, $\measurementmtx^H \measurementmtx$ is diagonal if we take a Fourier transform first. If we wanted to efficiently calculate the inverse $(\mu_1 \measurementmtx^H \measurementmtx)^{-1}$, we would do a DFT to switch to frequency space, do division by the values of $\mid\mathcal{F}h\mid^2$ in that space, and then inverse DFT to get back to standard (pixel) basis.

However, the inverse we wish to calculate has other terms as well, so we can only use this method if $\Psi^H \Psi$ and $I$ are also diagonalizable by a Fourier transform. Since $\mathcal{F}^{-1} I \mathcal{F} = I$, the identity is diagonalizable.

We knew $\measurementmtx^H \measurementmtx$ was diagonalizable by a Fourier Transform because it was a convolution. So, to show $\Psi^H \Psi$ is also diagonalizable by a Fourier transform we must write $\Psi^H \Psi \imagevec = \Psi^H \left(\Psi \imagevec \right)$ as a convolution between $\imagevec$ and a fixed kernel.
To that end, in 2D the forward difference acted on the $i,j$th pixel looks like:
$$\begin{align}
\Psi v_{ij} &= \begin{bmatrix}v_{i+1,j} - v_{i,j} \\ v_{i,j+1}-v_{i,j}\end{bmatrix}\\
&= \begin{bmatrix}
\begin{bmatrix}0 & -1 & 1\end{bmatrix} \cdot \begin{bmatrix}v_{i-1,j} & v_{i,j}& v_{i+1,j}\end{bmatrix}\\
\begin{bmatrix}0 \\ -1 \\ 1 \end{bmatrix} \cdot \begin{bmatrix}v_{i,j-1} \\ v_{i,j} \\ v_{i,j+1}\end{bmatrix}
\end{bmatrix} \\
\end{align}$$
The above formula holds for every pixel in the image (assuming the same circular boundary conditions that are used to derive $\Psi^H$ itself). This formula is also the pointwise definition of cross-correlation with the non-vectorized image $v$:
$$
\Psi \imagevec =
\begin{bmatrix}
\begin{bmatrix}0 & 0 & 0 \\ 0 & -1 & 1 \\ 0 & 0 & 0\end{bmatrix} \star v
\\
\begin{bmatrix} 0 & 0 & 0 \\ 0 & -1 & 0 \\ 0 & 1 & 0 \end{bmatrix} \star v
\end{bmatrix}, \\
$$
where we have written both the row-wise and column-wise pixel differences in terms of a cross-correlation, just with two different "kernels."

Let's call the row (top) kernel $k_{RF}$ and the column (bottom) kernel $k_{CF}$, where the "$F$" stands for forward difference.

Using exactly the same process with $\Psi^H$ (and the stack of two images $u$ defined as above):
$$ \Psi^H u = -k_{RB} \star u^x - k_{CB} \star u^y, $$
where the "B" stands for backwards difference.

So,
\begin{align}
\Psi^H \Psi \imagevec &= \Psi^H
                                 \begin{bmatrix}k_{RF} \star v \\ k_{CF} \star v\end{bmatrix} \\
                              &= -k_{RB} \star \left(k_{RF} \star v\right) - k_{CB} \star \left( k_{CF} \star v \right)
\end{align}
At this point, we cannot simplify the operations because cross-correlations are not associative -- we need to reformulate the expression in terms of _convolutions_. So, we use the property that cross-correlation with a kernel $k$ is equivalent to convolution with the _flipped_ kernel $k'$ where we flip horizontally _and_ vertically. For example, if $k = \begin{bmatrix} 0 & -1 & 1\end{bmatrix}$, then $k' = \begin{bmatrix} 1 & -1 & 0 \end{bmatrix}$. If $k = \begin{bmatrix} 0 \\ -1 \\ 1 \end{bmatrix}$ then $k' = \begin{bmatrix} 1 \\ -1 \\ 0\end{bmatrix}$. Flipping the kernels and using associativity of convolution, we find:
\begin{align}
    \Psi^H \Psi \imagevec &= -k_{RB} \star \left(k_{RF} \star v\right) - k_{CB} \star \left( k_{CF} \star v \right) \\
                                  &= -k_{RB}' \ast \left(k_{RF}' \ast v\right) - k_{CB}' \ast \left( k_{CF}' \ast v\right) \\
                                  &= -\left[(k_{RB}' \ast k_{RF}') + (k_{CB}' \ast k_{CF}')\right] \ast v
\end{align}

Finally, we again use the convolution theorem: for any $k, v$, $ k \ast v = \mathcal{F}^{-1} \{ \mathcal{F}(k) \cdot \mathcal{F}(v) \} $. Thus, setting $k = -(k_{RB}' \ast k_{RF}') - (k_{CB}' \ast k_{CF}')$, we have that
$$
\\
\Psi^H \Psi \imagevec = \mathcal{F}^{-1} \{ \mathcal{F}(k) \cdot \mathcal{F}(v) \},
\\
$$
which corresponds to pixel-wise multiplication by $\mathcal{F}(k)$ in the Fourier space.

To summarize, in matrix form, the inverse term can be written as
$$
\begin{align}
\\
(\mu_1 \measurementmtx^H \measurementmtx + \mu_2 \Psi^H \Psi + \mu_3 I)^{-1} &=
(\mu_1 \mathbf{F}^{-1} D_M \mathbf{F} + \mu_2 \mathbf{F}^{-1} D_{\Psi} \mathbf{F} + \mu_3 \mathbf{F}^{-1} \mathbf{F})^{-1} \\
&= \mathbf{F}^{-1} (\mu_1 D_M + \mu_2 D_{\Psi} + \mu_3)^{-1} \mathbf{F}, \\
\end{align}
$$

where $D_M$ and $D_{\Psi}$ are the diagonal operators corresponding to $\mathbf{M}^H \mathbf{M}$ and $\Psi^H \Psi$ implemented using pixel-wise multiplication in the Fourier space:

\begin{cases}
D_M = \mathrm{diag}(|\mathcal{F}(h)|^2) \\
D_{\Psi} = \mathrm{diag}(\mathcal{F}(\texttt{pad}(k))),
\end{cases}

where the $\texttt{pad}$ is used just to make the kernel the same size as the image.

What is that kernel?
$$
\begin{align}
\\
k &= -[(k_{RB}' \ast k_{RF}') + (k_{CB}' \ast k_{CF}')] \\ \\
  &= -\left(\begin{bmatrix}
                      0 & 0 & 0 \\
                      0 & 1 & -1 \\
                      0 & 0 & 0
                      \end{bmatrix} \ast
             \begin{bmatrix}
                      0 & 0 & 0 \\
                      1 & -1 & 0 \\
                      0 & 0 & 0
                      \end{bmatrix} \right) -
     \left(\begin{bmatrix}
                      0 & 0 & 0 \\
                      0 & 1 & 0 \\
                      0 & -1 & 0
                      \end{bmatrix} \ast
           \begin{bmatrix}
                      0 & 1 & 0 \\
                      0 & -1 & 0 \\
                      0 & 0 & 0
                      \end{bmatrix} \right) \\ \\
   &= -\begin{bmatrix}
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 0 \\
0 & 1 & -2 & 1 & 0 \\
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 0
\end{bmatrix} -
\begin{bmatrix}
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & -2 & 0 & 0 \\
0 & 0 & 1 & 0 & 0 \\
0 & 0 & 0 & 0 & 0
\end{bmatrix}
\\ \\ &=
\begin{bmatrix}
0 & 0 & 0 & 0 & 0 \\
0 & 0 & -1 & 0 & 0 \\
0 & -1 & 4 & -1 & 0 \\
0 & 0 & -1 & 0 & 0 \\
0 & 0 & 0 & 0 & 0
\end{bmatrix}
\end{align}
$$

Following the convention established in the gradient descent notebook, since this is a Fourier space mask, we store it already shifted so that the origin (4) is in the corner:

```python
def precompute_PsiTPsi():
    PsiTPsi = np.zeros(full_size)
    PsiTPsi[0,0] = 4
    PsiTPsi[0,1] = PsiTPsi[1,0] = PsiTPsi[0,-1] = PsiTPsi[-1,0] = -1
    PsiTPsi = fft.fft2(PsiTPsi)
    return PsiTPsi
```

So, to calculate the inverse operator, we do a DFT to convert to Fourier space, divide the $i,j$th pixel by the $i,j$th entry in $\left(\mu_1 |\mathcal{F}(h)|^2 + \mu_2 \mathcal{F}(k)^* + \mu_3\right)$, and inverse DFT to convert back to real space. As usual, the pixel-wise division can be done in parallel.

```python
def precompute_R_divmat(H_fft, PsiTPsi):
    """Only call this function once!
    Store it in a variable and only use that variable
    during every update step"""
    MTM_component = mu1*(np.abs(np.conj(H_fft)*H_fft))
    PsiTPsi_component = mu2*np.abs(PsiTPsi)
    id_component = mu3
    """This matrix is a mask in frequency space. So we will only use
    it on images that have already been transformed via an fft"""
    return 1./(MTM_component + PsiTPsi_component + id_component)
```

### Dual updates:
\begin{align*}
    \xi_{k+1} & \leftarrow \xi_k + \mu_1(\measurementmtx \imagevec_k - x_{k+1}) \\
    \eta_{k+1} & \leftarrow \eta_k + \mu_2(\Psi \imagevec_{k+1} - u_{k+1}) \\
    \rho_{k+1} & \leftarrow \rho_k + \mu_3(\imagevec_{k+1} - w_{k+1})
\end{align*}

```python
def xi_update(xi, V, H_fft, X):
    return xi + mu1*(M(V,H_fft) - X)

def eta_update(eta, V, U):
    return eta + mu2*(Psi(V) - U)

def rho_update(rho, V, W):
    return rho + mu3*(V - W)
```

## Putting it all together

First, we initialize all the primal and dual variables. Note that the dimensions matter -- we use $\texttt{full_size}$ as the "uncropped" image size. The primal variables $x, w, \imagevec$ all take on this dimension. $u$ is the result of a gradient, which (as is shown below) must be the size of two stacked images. The dual variable dimensions can be derived by applying their update rule to the primal variables.

```python
def init_Matrices(H_fft):
    X = np.zeros(full_size)
    U = np.zeros((full_size[0], full_size[1], 2))
    V = np.zeros(full_size)
    W = np.zeros(full_size)

    xi = np.zeros_like(M(V,H_fft))
    eta = np.zeros_like(Psi(V))
    rho = np.zeros_like(W)
    return X,U,V,W,xi,eta,rho
```

```python
def precompute_H_fft(psf):
    return fft.fft2(fft.ifftshift(CT(psf)))
```

Again, the ADMM steps for reference:
\begin{align*}
    u_{k+1} &\leftarrow \mathcal{T}_{\frac{\tau}{\mu_2}} \left(\Psi \imagevec_k + \frac{\eta_k}{\mu_2}\right) \\
    x_{k+1} &\leftarrow \left(\crop^H \crop + \mu_1 I\right)^{-1}
\left(\xi_k + \mu_1 \mathbf{M}\imagevec_k + \crop^H \measurementvec\right) \\
w_{k+1} &\leftarrow \max(\rho_k/\mu_3 +\imagevec_k, 0) \\
\imagevec_{k+1} &\leftarrow (\mu_1 \mathbf{M}^H \mathbf{M} + \mu_2 \Psi^H \Psi + \mu_3 I)^{-1} r_k, \\
    \xi_{k+1} & \leftarrow \xi_k + \mu_1(\measurementmtx \imagevec_k - x_{k+1}) \\
    \eta_{k+1} & \leftarrow \eta_k + \mu_2(\Psi \imagevec_{k+1} - u_{k+1}) \\
    \rho_{k+1} & \leftarrow \rho_k + \mu_3(\imagevec_{k+1} - w_{k+1})
\end{align*}

```python
def ADMMStep(X,U,V,W,xi,eta,rho, precomputed):
    H_fft, data, X_divmat, R_divmat = precomputed
    U = U_update(eta, V, tau)
    X = X_update(xi, V, H_fft, data, X_divmat)
    V = V_update(W, rho, U, eta, X, xi, H_fft, R_divmat)
    W = W_update(rho, V)
    xi = xi_update(xi, V, H_fft, X)
    eta = eta_update(eta, V, U)
    rho = rho_update(rho, V, W)

    return X,U,V,W,xi,eta,rho
```

```python
def runADMM(psf, data):
    H_fft = precompute_H_fft(psf)
    X,U,V,W,xi,eta,rho = init_Matrices(H_fft)
    X_divmat = precompute_X_divmat()
    PsiTPsi = precompute_PsiTPsi()
    R_divmat = precompute_R_divmat(H_fft, PsiTPsi)

    for i in range(iters):
        X,U,V,W,xi,eta,rho = ADMMStep(X,U,V,W,xi,eta,rho, [H_fft, data, X_divmat, R_divmat])
        if i % 1 == 0:
            image = C(V)
            image[image<0] = 0
            f = plt.figure(1)
            plt.imshow(image, cmap='gray')
            plt.title('Reconstruction after iteration {}'.format(i))
            display.display(f)
            display.clear_output(wait=True)
    return image
```

```python
#psf, data = loadData(True)
final_im = runADMM(psf, data)
plt.imshow(final_im, cmap='gray')
plt.title('Final reconstructed image after {} iterations'.format(iters))
display.display()

```

```python

```