```python
### import matplotlib.pyplot as plt

import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib as mpl
import numpy as np
import pandas as pd
import scipy as sp
from scipy.optimize import curve_fit
from scipy.signal import savgol_filter

plt.style.use('default')
plt.style.use('seaborn-paper')
current_palette = sns.color_palette(palette=None)
sns.set_palette(palette=None)
sns.palplot(current_palette)
plot_width = 3.487
plot_height = plot_width / 1.618

```

# Research Notes:

**Eerkes Utwente et al. - Modulation of conductance and superconductivity top gating LAO/STO**:
- Top gated device were fabricated on STO (001)-oriented substrate, at TiO2-layer
- isolated measurement structure by lift-off patterend (lithography+photoresist) by using a-LAO layer.
- LAO deposited by PLD. Crystalline LAO PLD-deposited in oxygen pressure of 2e-3 milibars at 850c
- Sampled was cooled down to 600c at deposition pressure, post anneal of 1h at 600c at oxygen pressure of 600 milibars, followed a cool down to room temp at 600milibar pressure. (this creates insulating a-LAO/STO layer.
- Ohmic contacts to the 2DEG is deposited by sputtering of Ti/Au.(Use an argon etcher first to remove some gold layers(?)).
- Vg = 0V , $n_s = 1.9*10^{13}$ $cm^{-2}$. Which is typical value for metalic top-gates. ( In the experiment, state is the same as devices without top-gates)

**B. Forg  et al.- Field-effect devices utilizing LaAlO3-SrTiO3 interfaces**:
- 9 uc thick LAO, in situ gate. Depletion mode. operating tested at -100 c to 100c, short 20um-200um and wide 1600um channels)
- Fabricate all-oxide FET with current and voltage gain.
- Oxide interface is proven that they can be controlled effectively by gate field induced by top gating config.
- Thin gate dielectric (LAO) , devices can be operated with small gate voltages in comparison to bottom gating
- Lower limits to be explored.
- Subthreshold swing S = 0.7/4 => 175mV/dec
-

**M. Hosada et al. - Transistor operation and mobility enhancement in top-gated
LaAlO3/SrTiO3 heterostructures** :
- Depletion mode too
- Two probe device geometry prevented the determination of sheet carrier density n and hence hall mobility.
- Again, TiO2 terminated STO substrate, pre-deposited a-LAO hard mask.
    - LAO thickness between DS is 4uc to 22uc.
    - 400um channel length
    - Channel width 5-200um.
    - ex-situ gate, sputtered Au with 100nm thickness
    - Source and drain were contacted with Al ultrasonic wire bonding
- gate leakage current Ig was significantly smaller than Id when device is on
- Obtained a clear transistor charactersitics at T = 300k for 16uc LAO
- Pinch off observed for -0.8V <= Vgs <= +1.2V, (Saturation at higher vds)
- dId/dVds (conductance) can show the pinch off voltage Vp as a function of Vgs. At Vds = Vp, Id sat is shown.
- Linear behaviour in the saturation region as expected. Vth = 01.15+0.05V. We can use the Variable range hopping model to calculate Vth too.
- Small hysterisis.
- Confirms that trasistor charactersitics is still obtained LAO thichkness of 4uc. Vth does not scale monotonically with thickness of LAO, as would be the case for constant sheet carrier density. confirming that Vth does not depend on:
\begin{align}
V_{th} = \frac{-ed_{LAO}n}{\epsilon \epsilon_0}
\end{align}

-  When Vgs -> Vth, enhancement of hall mobility is observed as n decreased.

**C. Woltmann et al. - Field-Effect Transistors with Submicrometer Gate Lengths Fabricated
from LaAlO3-SrTiO3-Based Heterostructures** :
- Considers short effect behaviour
- Methods describing semiconductor physics are applicable only with considerable restriction since electronic properties of complex oxides differs fundamentally from conventional semiconductor
- Conclude that with decreasing gate length, Vt shifts to lower values and is increasingly dependent on Vsd (note that this is depletion mode)
- On-ff ratio and subthreshold slope decrease (Subthreshold swing increase).  A device characterized by steep subthreshold slope will have a faster transition between off (low current) and on (high current) states.

**Variable Range hopping Conduction (VRH) pdf**:

Chapter 4.1:Outline
- Modeled for amorphous crystal which in this case is SiO2
- Mott introduced the concept of hopping conduction: VRH
- Empirically observed $log\sigma \propto T^{-1/4}$
- $log\sigma \propto T^{-1/4}$ is valid for low E field strength, an extension is needed for higher field stregth
- VRH process is often expressed in terms of a percolation problem. VRH/Percolation problem is successful in describing the process in ohmic low-field regime[pg5] but not at medium and high field regime (i.e saturation mode at our case)

Chapter 4.2:Hopping Probability
- Assuming no corellation with occupation probability of different localize states, net electron flow is:
$$I_{ij} = f_i(1-f_j)w_{ij} - f_j(1-f_i)w_{ji} $$

Where $f_i$ is occupation probability of state i and

**A.K. Sen - Nonlinear Response, Semi-classical Percolation and Breakdown in the RRTN Model**:

Power-law growth of conductance:
- non-integer power-law behaviour (for low field),$I \propto (V-V_g)^\alpha$ , $V_g$ being the threshold voltage.
- For conductance G, nonlinearity exponent $\delta = \alpha - 1 = 2$.
- For arrays of normal metal island connected by small tunnel junction,$\alpha = 1.36 \pm 0.16$ for 1D and $\alpha = 1.80 \pm 0.16$ for 2D.

Low-temperature hopping-dominated conduction:
- conduction properties of low-temperature regime where conduction is mainly due to phonon-assisted hopping of electrons.
- Needs to consider Mott's VRH conductance model or some of the other variation
$$G(T) \propto \exp(-(T_0/T))^\gamma $$
- VRH exponent being dependent on the concentration of donors/acceptors (n or p)
- Conductance of the sample goes through ma maximum as temp is increased towards a metalic behaviour.

Origin of RRTN model and percolative aspect:

# Journal Entry
## Week 1

### PLD and Pulse Laser deposition
PLD : Pulse laser deposition
RHEED : Reflection High Energy Electron deposition

Procedure for the sample is initially set by : https://en.wikipedia.org/wiki/Photolithography

**Procedure for substrate**

fabricated in the UTwente clean room, Notes:
- The substrate grown, ( Reseach on: STO growth, and lookup terrace on why that the terrace form is needed. Not completely flat). Annealing makes surface reconstruction, etc.
- Substrate is cleaned,
- spin coating with photo resist
- Apply photomask, do allignment and such
- Apply UV light for 7 seconds
- Remove photo resist, clean using acetone.
- Etching stuff

**PLD**
- Alignment of the optics
- set the energy of the laser, calibrate it. Energy of the laser is base on the ionization energy required for the material.
- Temperature 850C for LAO
- Temperature 100C for Gold
- Plume shoots out single atoms and it will react with ambient oxygen in the container. Depends on the stochiometery of the system. It can be tuned to the

Put more parameters in paper of the PLD.

## Week 2
### Notes:
Measurement will be done next week. Today is the finalization of the 4 u.c FET. Use probe station to measure I-V curve.  Sweep $V_{gs}$ with constant $V_{ds}$. Measure $I_{ds}$ and measure $V_{gs}$

For Vgs sweep
- Vgs sweep from -2V to 1V steps of 5e-3
- Vds increment by 100mV from -0.5V to 0.5V

For Vds sweep
- Vds sweep from -0.5 to 5V steps of 5e-3
- Vgs increment by 100mV from -0.5V to 0.5V

### Simulation

Resource: https://journals.aps.org/prl/pdf/10.1103/PhysRevLett.109.196803

Discuss mainly on the percolation effect. Modelling will be done using RRTN and perhaps Variable-range hopping theory in which is also discussed in the paper. Another option is to use capacitor based modelling. Could be done by sweeping the whole $C_{LAO}$ variable and curve fit it to the measurement. Much more interesting to use the RRTN/VRH theory.

Focus on the non-percolative region from the paper. (2D variable range hopping model or RRTN) that can be switch and might even have memristive effects. Eq 1 and Eq 2 should be the focus of the model. Hence it would be beneficial to give a certain percolative simulation, based on the paper p=0.3 already gives a conduction.

**Eq 1**: This is temperature dependent, hence might be time consuming to test.
\begin{align}
R = R_0 \exp \left( \left[ \frac{T_0}{T}\right]^v \right)
\end{align}

**Eq 2**: Using the RRTN percolation model for our FET might be a good start.
\begin{align}
I_{sd} &\propto (V_{sd} - V_c)^{\delta}  \text{  for  } V_{sd} \geq V_{c} \\
I_{sd} &= 0 \text{  for  } V_{sd} < V_c
\end{align}

Since we model it as it it proportional to some factor, we can use scipy least square minimization along with the model.

## Week 3
### Measurement
Probe setup measurement, measure:
For measurements it was found that -2.5 Volts was maximum for the device with 4 uc.

Each measurement measures the current through Ids and through Ig.

Sweep parameters are Vds and Vgs, and interchangeable line series. Currently measuring for for:

**Ids vs Vgs**
- Vgs sweeped from -2.5V to 1V at steps of 5e-3
- Vds is fixed at 100mV.
- Other setup is Vds is lineseries of 400mVpp step of 100mV.

**Ids vs Vsd**
- Vds sweeped at 500mVpp setp of 5e-3
- Vgs lineseires is set from -2.5V to 1V step of 100mV
Ig is also measured along with the two setup.

Note that in the data file:

**K200 == Vgs **

**K200-2 == Vds **

### Modelling
use Hoshen-Kopelman algorithm for percolation : http://phony1.technion.ac.il/~lsharir/p3d/percolation.html for island grouping.

details on percolation : http://physik.uni-graz.at/~cbl/CP1/contents/Texts/

Grid generation of a uniformly distributed samples is shown below. White shows that it is not conducting, black shows that it is conducting region. Need to model the tunneling to get everything right.

Assumptions:
- PLD is homogenous, hence the occupation probability is the same. Can be altered to normal distribution from the center. Percolation threshold will likely be different then.
- The size of the cell is large enough so that the shape of the site does not matter.
- if it is site percolation, it is assumed that the connection between 2 sites are perfect. For bond percolation, further improvement can be implemented including quantum tunneling between two sites.

Full program is shown in Percolation.py chapter.

## Week 4
### Data from 4uc FET

```python
#Data handling for Labview output. Seperate different line segments, folder in dev15_vsd needs to be deleted first.

def dataparsing(input_file, output_file, data_width):
    j = 0
    with open(input_file, 'r') as f_i:
        for line, data in enumerate(f_i):
            if line % data_width == 0:
                j = j + 1
            else:
                with open(output_file + str(j) + '.txt','a+') as f_o:
                    f_o.write(data)

input_file = 'Data\\4_uc_new device\\dev16\\Dev16_Vgs-sweep -2V to 1V -- Vds 0V to 1V.txt'
output_file = 'Data\\4_uc_new device\\dev16\\Dev16_RoomT_Vgs_sweep\\dev16_dataset'
data_width = 606

dataparsing(input_file, output_file, data_width)
```

```python
def wrapper_vds(dataframe):
    dataframe.drop('Unnamed: 5', axis=1,inplace=True)
    dataframe.columns = ['time','Vgs','Vds','Igs','Ids']
    return dataframe

def new_range(start,end,step):
    i = start
    while i < end:
        yield i
        i += step
    yield end

def find_nearest(array, value):
    array = np.asarray(array)
    idx = (np.abs(array - value)).argmin()
    return idx

def quadratic_fitting(x, a, b):
    y = a*(x-b)**2
    return y

def plus_string(number):
    if number < 0:
        x = '$-$' + str(round(abs(number), 2))
    elif number > 0:
        x = '+' + str(round(number, 2))
    return x

fig1 = plt.figure(1)
ax = fig1.add_subplot(1, 1, 1)

fig2 = plt.figure(2)
ax2 = fig2.add_subplot(1, 1, 1)

# fig3 = plt.figure(3)
# ax3 = fig2.add_subplot(1, 1, 1)

V_sat_data = []
for i in new_range(17, 35, 2):    # set number of data here: max # 36

    # df1 = pd.read_csv('Data\\dev15\\dev15_vsd\\dev15_dataset'+str(i)+'.txt',sep='\t',skiprows = [0,1,2])
    df2 = pd.read_csv('Data\\dev16\\dev16_huge_sweep\\dev16_dataset'+str(i)+'.txt',sep='\t',skiprows = [0,1,2])

    # df_wrap1 = wrapper_vds(df1)
    df_wrap2 = wrapper_vds(df2)

    Vgs = -2.5 + (0.1*i-0.1)
    Ids_list = df_wrap2['Ids'] # Still in ampere
    Vds_list = df_wrap2['Vds']
    V_sat_index = find_nearest(Vds_list[1:1200], Vgs+1.8)
    V_sat_data.append((Vds_list[V_sat_index], Ids_list[V_sat_index]/10**(-6), Vgs))

    Igs_smooth = savgol_filter(df_wrap2['Igs'],99,3)  # Filter out noise with savgol filter
    # plt.plot(df_wrap1['Vds'], df_wrap1['Ids']/(10)**(-6), label = 'Vgs ='+str((i*-100)+100)+'mV',linestyle = 'dashed')
    ax.plot(df_wrap2['Vds'], df_wrap2['Ids']/(10)**(-6), label = 'Vgs ='+str(round(Vgs,2))+' V')
    if Vgs <= -0.75:
        ax.text(Vds_list[1200]+0.05, Ids_list[1200]/(10)**(-6) -0.5, plus_string(Vgs),  withdash=False, fontsize=9)
    elif Vgs >= -0.7:
        ax.text(Vds_list[1200]+0.05, Ids_list[1200]/(10)**(-6), plus_string(Vgs),  withdash=False, fontsize=9)

    ax2.plot(df_wrap2['Vds'], Igs_smooth/(10**(-6)), label = 'Vgs ='+str(round(Vgs,2))+' V') # Ig is in micro-amps

Vds_tuple, Ids_tuple, Vgs_tuple = zip(*V_sat_data)

popt, pcov = curve_fit(quadratic_fitting, Vds_tuple, Ids_tuple)
print(popt)
Vds_new = np.linspace(popt[1], 5, num = 100, endpoint=True)
Ids_new = quadratic_fitting(Vds_new, *popt)
# ax.plot(Vds_tuple,Ids_tuple)
ax.plot(Vds_new, Ids_new)
ax.scatter(Vds_tuple, Ids_tuple)

## Plot setting for fig1 (Id vs Vds)
# ax.set_title('100$\mu m^2$ Gate Area')
ax.set_xlabel('$V_{ds}$ [V]', fontsize=10)
ax.set_ylabel('$I_{d}$ [$\mu$A]', fontsize=10)
# ax.legend(bbox_to_anchor=(1.05, 1), loc=2)
ax.set_xlim(-0.5, 5)
ax.set_ylim(-1, 34)
ax.minorticks_on()
ax.tick_params(axis = 'y', which = 'both', length=5, direction='in')
ax.tick_params(axis = 'both', which = 'minor', labelsize = 9)
ax.text(5+0.05, 34+0.5, ' $V_{gs}$ [V]',  withdash=False, fontsize=9) # Vgs label is set manually
fig1.set_size_inches(plot_width*1.1, plot_height*1.4)
# fig1.savefig('Figures\\Dev16_Vds_sweep-Ids.pdf', format='pdf', bbox_inches='tight')

## plot setting for fig2 (Ig vs Vds)
ax2.legend(prop={'size': 8}, loc=1, bbox_to_anchor=(1,1), bbox_transform=ax2.transAxes)
# ax2.set_title('100$\mu m^2$ Gate Area')
ax2.set_xlabel('$V_{ds}$ [V]', fontsize=10)
ax2.set_ylabel('$I_{g}$ [$\mu$A]', fontsize=10)
ax2.minorticks_on()
ax2.tick_params(axis = 'y', which = 'both', length=5, direction='in')
ax2.tick_params(axis = 'both', which = 'minor', labelsize = 9)
fig2.set_size_inches(plot_width*1.1, plot_height*1.4)
ax2.set_ylim(-0.01, 0.01)
ax2.set_xlim(-0.5, 5)
#fig2.savefig('Figures\\Dev16_Vds_sweep-Igs.pdf', format='pdf', bbox_inches='tight')

# Fig 3 setting
```

Saturation region is clearly shown in the figure. The saturation region is flat, no short body effect as expected. As Vgs gets bigger, (more positive) it shows that the hysterisis gets bigger. Gate current plot shows that it has relatively low leakage <2nA.

**Notes**: The Vt is not negative. It should be at negative ~-1.5V by the curve fit. Hence the parabola should be shifted to the negative side. Hence it may be concluded that the saturation isn't exactly a parabola. Different function needs to be checked. Because it might be that saturation takes place in a 'shallower' manner with respect to the quadratic term. The saturation regime is also not only dependent on the Vgs-Vt. It grows bigger/smaller under several cases.

```python

########
 # Vgs transistor characterstic plotter - plot Vgs sweep characetristic, find the subthreshold swing with exponential# Vgs tr
#                                       Curve fit
###
def wrapper_vgs(dataframe):
    dataframe.drop('Unnamed: 5', axis=1,inplace=True)
    dataframe.columns = ['time','Vds','Vgs','Ids','Igs']
    return dataframe

def exponent_fit(x, b, c):
    return 10**(b*x+c)

def Vgs_plotter(data_location, loop_var1, loop_var2):
    data_linspace = np.round(np.linspace(-0.5,0.5, num = 21),2)  #set the data linspace for proper range
    gradient = []
    Vds_arr = []
    on_off_rat = []
    for i in range(loop_var1, loop_var2,3):
        df = pd.read_csv(data_location + str(i) + '.txt', sep='\t', skiprows = [0,1,2])
        df_wrap = wrapper_vgs(df)

        fig1 = plt.figure(1)
        plt.semilogy( df_wrap['Vgs'], df_wrap['Igs']/(10)**(-6), label = 'Vds =' + str(data_linspace[i-1]) + 'V')
        plt.legend(prop={'size': 8})
        plt.xlabel('$V_{gs}$ [V]', fontsize=10)
        plt.ylabel('$I_{d}$ [$\mu$A]', fontsize=10)
        ax = plt.gca()
        ax.tick_params(axis = 'both', which = 'both', length=5, direction='in')
        ax.minorticks_on()
        ax.tick_params(axis='both', which='minor', length=3)
        fig1.set_size_inches(plot_width , plot_height*1.3)

        fig2 = plt.figure(2)
        plt.plot( df_wrap['Vgs'], (df_wrap['Ids']/(10)**(-6)), label = 'Vds =' + str(data_linspace[i-1]) + 'V')
        plt.legend(prop={'size': 8})
        plt.xlabel('$V_{gs}$ [V]', fontsize=10)
        plt.ylabel('$I_{g}$ [$\mu$A]', fontsize=10)
        ax2 = plt.gca()
        ax2.tick_params(axis = 'both', which = 'both', length=5, direction='in')
        ax2.minorticks_on()
        ax2.tick_params(axis='both', which='minor', length=3)
        fig2.set_size_inches(plot_width, plot_height*1.3)

        fig3 = plt.figure(3)
        Vgs_linspace = df_wrap['Vgs']
        Ids_data = df_wrap['Igs']/10**(-6)  # Ids Data is in micro amps
        popt, pcov = curve_fit(exponent_fit, Vgs_linspace[610:625], Ids_data[610:625])
        Vgs_new = np.linspace(-1.2, -0.9, endpoint=True)
        Ids_new = exponent_fit(Vgs_new, *popt)

        plt.semilogy(Vgs_linspace[0:650], Ids_data[0:650], label = 'Vds =' + str(data_linspace[i-1]) + 'V')
        plt.semilogy(Vgs_new, Ids_new, linewidth=1 , linestyle = 'dashed')
        plt.legend(prop={'size': 8})
        plt.xlabel('$V_{gs}$ [V]', fontsize=10)
        plt.ylabel('$I_{d}$ [$\mu$A]', fontsize=10)
        ax3 = plt.gca()
        ax3.tick_params(axis = 'both', which = 'both', length=5, direction='in')
        ax3.minorticks_on()
        ax3.tick_params(axis='both', which='minor', length=3)
        ax3.legend(bbox_to_anchor=(1.05, 1), loc=2)

        fig3.set_size_inches(plot_width, plot_height*1.3)

        gradient.append(((Vgs_new[-1] - Vgs_new[0])*1000)/(np.log10(Ids_new[-1])-np.log10(Ids_new[0])))
        on_off_rat.append(Ids_data[1000]/Ids_data[650]) # set the on-off ratio properly too
        Vds_arr.append(data_linspace[i-1])

    fig4 = plt.figure(4)
    ax4 = fig4.add_subplot(111)
    ax5 = ax4.twinx()
    ax4.plot(Vds_arr, gradient, '-o', color='r',label='Subthreshold')
    ax5.plot(Vds_arr, on_off_rat, '-^', color='b', label='on/off ratio')

    ax4.tick_params(axis = 'both', which = 'both', length=5, direction='in')
    ax4.minorticks_on()
    ax4.set_xlabel('$V_{ds}$ [V]', fontsize=10)
    ax4.set_ylabel('$S_{s-th}$ [mV/dec]', fontsize=10)
    ax4.tick_params(axis='both', which='minor', length=3)

    ax5.tick_params(axis = 'both', which = 'both', length=5, direction='in')
    ax5.minorticks_on()
    ax5.set_xlabel('$V_{ds}$ [V]', fontsize=10)
    ax5.set_ylabel('On/Off $I_d$ ratio', fontsize=10)
    ax5.tick_params(axis='both', which='minor', length=3)

    fig4.legend(prop={'size': 8}, loc=1, bbox_to_anchor=(1,0.30), bbox_transform=ax4.transAxes)

    fig4.set_size_inches(plot_width , plot_height*1.4)
#     fig4.savefig('Figures\\Dev16_lowT_Subth-on-off.pdf', format='pdf', bbox_inches='tight')
#     fig1.savefig('Figures\\Dev16_lowT_Vgs_sweep-Ids.pdf', format='pdf', bbox_inches='tight')
#     fig2.savefig('Figures\\Dev16_lowT_Vgs_sweep-Igs.pdf', format='pdf', bbox_inches='tight')

# change data set! might be switched. Check what Id of the source K200 K200-2
Vgs_plotter('Data\\4_uc_new device\\dev16\\Dev16_LowT_Vgs_sweep\\dev16_dataset', 12, 22) #12 max

# plt.savefig('Figures\\percprob_vs_occprob.pdf', format='pdf', dpi=300)
```

Graph of typical Ids vs Vds Fet charactersitic.
Threshold swing of 285mV/dec. Due to high carrier density. As Cox -> infinity it is close to theoretical 60mV/decade. Memristor like effect hysterisis and such present. Hysteris is reproducible, good for oxide base memristor.

Subthreshold swing is as 285mV/dec, the on/off ratio increases with Vds as expected since more Vds, more current. At off, the tunneling current dominates.

**Notes**: Conclude that it will not be suitable for transistor operations. While it does show promise as a memristor like device. Vt here can be deduce that it is approximately -1V. When fully depleted it will have higher Vt when going back up.

# Week 5 and Week 6

**3.6 uc and new 4 uc sample measurement:**

Both samples are insulating, Vgs until 50Vs doesn't activate it. Wait until next sample next week. In the mean time:

## To do list:
- <s>find the subthreshold swing</s>
- <s>Find 10 most dominant path, that should be enough to determine the current going through the sample </s>
- <s>Research CV characteristic measurement setup </s>
- <s>plot igs vs Vds </s>
- <s>Measurement that it scales well with all the devices. </s>
- <s>Write premilinary analysis on percolation  </s>

# Week 7

## To do list:

- <s>on off ratio vs Vds  </s>
- <s>Subthreshold swing vs Vds , plot highest gradient  </s>
- <s> number of paths in a sample found vs Resistance, Number of paths found vs occupation probability </s>
- <s>fit parabola properly,ignore initial data set, below 0  </s>

# Week 8

CV measurement with probestation by measuring impedance subsequently finding the reactance and hence the capacitance. Data uses phase and Z value. Use:

$$ X = Z sin(\Theta) $$
$$ X_c = \frac{1}{2\pi f C}$$

## To do list:
- Make all the figures properly.
- Measure CV charactersitics, do analysis
- Measure the mobility of the sample
- Modify the Percolation.py into RRTN
- plot the low temperature graphs.
- Finish methods in the report

# Percolation.Py

The following is the code for the percolation FET. The first part introduce the code for generating grid and also finding the path. The rest of the code follows from the orignal code with extensions to consider RRTN and also to find the percolating threshold $p_c$ by numerical simulation.

```python
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import heapq

from ipywidgets import IntProgress
from IPython.display import display
from matplotlib import cm
from contextlib import suppress

class PriorityQueue:       # FIFO queue
    def __init__(self):
        self.elements = []

    def empty(self):
        return len(self.elements) == 0

    def put(self, item, priority):
        heapq.heappush(self.elements, (priority, item))

    def get(self):
        return heapq.heappop(self.elements)[1]

class GenerateGrid(object):     # Generate grid for sample based on size and occupy prob.
    def __init__(self,width,height,occupy_prob):
        self.width = width
        self.height = height
        self.occupy_prob = 1-occupy_prob
        self.wall = []
        self.grid = np.zeros(shape=(self.height,self.width))
        self.grid_RRTN = np.zeros(shape=(self.height,self.width))

    def in_bounds(self,id):
        (x,y) = id
        return 0 <= x < self.width and 0 <= y < self.height

    def passable(self, id):
        return id not in self.wall

    def neighbors(self,id):
        (x,y) = id
        results = [(x+1,y), (x,y-1), (x-1,y), (x,y+1)]
        if (x + y) % 2 == 0:
            results.reverse()
        results = filter(self.in_bounds, results)
        results = filter(self.passable, results)
        return results

    def generate_rrtn(self):
        for (x, y), item in np.ndenumerate(self.grid):
            if item == 1:
                if x >= 0:
                    if self.grid[x-2, y] == 1:
                        self.grid_RRTN[x-1, y] = 1
                        continue

                # Check bottom.
                if y >= 0:
                    if self.grid[x, y-2]  == 1:
                        self.grid_RRTN[x, y-1] = 1
                        continue

                # Check top.
                with suppress(IndexError):
                    if self.grid[x, y+2]  == 1:
                        self.grid_RRTN[x, y+1] = 1
                        continue

                # Check right.
                with suppress(IndexError):
                    if self.grid[x+2, y]  == 1:
                        self.grid_RRTN[x+1, y] = 1
                        continue

    def GenerateWall(self):    # generate pseudo-random wall
        patches = np.random.rand((self.height-2), self.width)
        self.grid  =  np.vstack((np.ones(self.width), patches, np.ones(self.width)))
        for i in range(0, self.height):
            for j in range(0, self.width):
                if self.grid[i,j] < self.occupy_prob:
                    self.wall.append((j,i))
                    self.grid[i,j] = 0
                else:
                    self.grid[i,j] = 1

def pathfinder(came_from, start, goal):     # Function to find path taken by A* search algorithm
    current = goal
    path = []
    while current != start:
        try:
            path.append(current)
            current = came_from[current]
            path_flag = True
        except KeyError:     # Except for no path found
            path = []
            path_flag = False
            break
    path.append(start)
    return path,path_flag

def heuristic(a, b):      # set direction of A*
    (x1, y1) = a
    (x2, y2) = b
    return abs(x1 - x2) + abs(y1 - y2)

def a_star_search(graph, start, goal):
    frontier = PriorityQueue()
    frontier.put(start, 0)
    came_from = {}
    cost_so_far = {}
    came_from[start] = None
    cost_so_far[start] = 0

    while not frontier.empty():
        current = frontier.get()

        if current == goal:
            break

        for next in graph.neighbors(current):
            new_cost = cost_so_far[current] + 1
            if next not in cost_so_far or new_cost < cost_so_far[next]:
                cost_so_far[next] = new_cost
                priority = new_cost + heuristic(goal, next)
                frontier.put(next, priority)
                came_from[next] = current

    return came_from, cost_so_far

#End of percolation code
```

```python
# This code is for sweeping the occupation probability to find the threshold occupation probability.
# Variables include the size of the sample.loop, and the number of intervals between 0 to 1 for p

width =  100
height = 100
Loop = 5     # Number of tries per occupation probability
linseries_point = 50        # number of steps between 0 to 1 for occupation probability

occupation_probability = np.linspace(0, 1, num=linseries_point, endpoint=True)
start, goal = ((width)/2, 0),((width)/2 , height-1)
occupy_histogram = []
path_average = np.zeros(len(occupation_probability))
progress = 0

f = IntProgress(min=0, max=len(occupation_probability), description = 'Progress:')
display(f)

for indices, occupy_prob in enumerate(occupation_probability):
    counter = 0
    path_sum = []
    for i in range(0,Loop):
        sample = GenerateGrid(width, height, occupy_prob)
        sample.GenerateWall()
        came_from, cost_so_far = a_star_search(sample, start, goal)
        path, path_flag = pathfinder(came_from, start=start, goal=goal)
        if path_flag is True:
            counter += 1
            path_sum.append(cost_so_far[goal])
    occupy_histogram.append(counter)
    path_average[indices] = np.nanmean(path_sum)
    progress += 1
    f.value = progress

# d = {'occupation probability' : occupation_probability, 'Percolate frequency' : occupy_histogram}
# df = pd.DataFrame(d)
# file_name = 'PercSimData\\perc_%sx%s_Loop%s_lineseries%s' %(width, height, Loop, linseries_point)
# df.to_csv(file_name, sep='\t')

path_average[np.isnan(path_average)] = 0

fig1 = plt.figure(1)
ax1 = fig1.add_subplot(111)
ax1.plot(occupation_probability, occupy_histogram)
plt.xlabel('Occupation Probability', fontsize=10)
plt.ylabel('Percolation Probability', fontsize=10)
ax1 = plt.gca()
ax1.tick_params(axis = 'both', which = 'both', length=5, direction='in')
fig1.set_size_inches(plot_width, plot_height)

# fig2 = plt.figure(2)
# ax2 = fig2.add_subplot(1, 1, 1)
# ax2.plot(occupation_probability, path_average)
# plt.xlabel('Occupation Probability', fontsize=10)
# plt.ylabel('Resistance', fontsize=10)
# ax2.tick_params(axis = 'both', which = 'both', length=5, direction='in')
# fig2.set_size_inches(plot_width , plot_height)

```

```python
#The following code plots the data point from the previous code for different sample size.
from scipy.optimize import curve_fit

def sigmoid(x, a, b, c, d):
    y = a / (1 + np.exp(-b*x-c)) + d
    return y

name_array = [8, 20, 50, 100]
label_array = ['8x8','20x20', '50x50', '100x100']

for labl_number, number in enumerate(name_array):
    df = pd.read_csv('PercSimData\\perc_%sx%s_Loop20_lineseries200.txt' %(number, number), sep='\t', dtype=np.float64)
    popt, pcov = curve_fit(sigmoid, df['occupation probability'], df['Percolate frequency']/20)

    occ_prob_new = np.linspace(0, 1, num = 10000, endpoint=True)
    perc_freq_new = sigmoid(occ_prob_new, *popt)
    fig1 = plt.figure(1)
    plt.plot(occ_prob_new, perc_freq_new/max(perc_freq_new), label = label_array[labl_number] + ' sample')

plt.xlabel('Occupation Probability', fontsize=10)
plt.ylabel('$P_s$', fontsize=10)
plt.legend(prop={'size': 8})
ax = plt.gca()
ax.tick_params(axis = 'both', which = 'major', labelsize = 8)
fig1.set_size_inches(plot_width*0.9, plot_height)
# fig1.savefig('Figures\\percprob_vs_occprob.pdf', format='pdf', dpi=300,bbox_inches='tight')
```

```python
# The following code generates an RRTN grid and non-RRTN grid. It also plots the result of the A* algorithm into
# an array as shown.

import matplotlib.pyplot as plt
from matplotlib import cm
import numpy as np
import pandas as pd
import heapq

def parallel_resistance(array):
    reciprocal = sum(np.reciprocal(array, dtype=float))
    resistance = 1/reciprocal
    return resistance

def shortest10_paths(width, height, occupation_probability):
    start, goal = ((width)/2, 0),((width)/2 ,height-1)

    path_list = []
    n_path = 0    # Number of paths initialize
    sample = GenerateGrid(width, height, occupation_probability)

    sample.GenerateWall()
    sample.generate_rrtn()
    Wall_OR_RRTN = np.logical_or(sample.grid_RRTN,sample.grid) # Wall is OR ed with the old sample
    RRTN_wall = np.subtract(Wall_OR_RRTN,sample.grid)*2
    RRTN_sample = np.add(sample.grid,RRTN_wall)

    fig1, ax1 = plt.subplots(1)
    cmap = cm.colors.ListedColormap(['white','black'])
    ax1.pcolor(sample.grid, cmap=cmap)
    fig1.set_size_inches(3, 3)

    fig2, ax2 = plt.subplots(1)
    cmap2 = cm.colors.ListedColormap(['white','black','blue'])
    bounds = [0, 1, 2, 3]
    norm = cm.colors.BoundaryNorm(bounds,cmap2.N)
    img = plt.pcolor(RRTN_sample, cmap=cmap2, norm=norm)
    fig2.set_size_inches(3, 3)

    path_sum = []   #list for path sum array

    for i in range(0,1):     # find 10 independent paths ## CURRENTLY SET AT 100, set at the sample size
        came_from, cost_so_far = a_star_search(sample, start, goal)
        path, path_flag = pathfinder(came_from, start=start, goal=goal)
        for coordinate in path:      # Exclude coordinates in the top and bottom row
            if coordinate[1] != height-1:
                if coordinate [1] != 0:
                    sample.wall.append(coordinate)
                    sample.grid[int(coordinate[1]), int(coordinate[0])] = 0
        ax1.plot(*zip(*path), color='red', linewidth=4)

        if path_flag is True:
            path_sum.append(cost_so_far[goal])
            n_path += 1

    if not path_sum:
        sample_resistance = 0
    else:
        sample_resistance = parallel_resistance(path_sum)   #collective sum of resistant

#     fig1.savefig('Figures\\pc-' + str(occupation_probability) + '.pdf', format='pdf', dpi=300)
    fig1.savefig('Figures\\Perc_sample-' + str(occupation_probability) + '.pdf', format='pdf', dpi=300)

    return sample_resistance, n_path

width =  30    #Keep sample even. If odd, change the start/goal coordinate starting coordinate
height = 30
resistance_array = []
path_num_avg = []

num = 3     # number of linspace for occupation probability
n = 1     #number of retries
occupy_range = np.linspace(0.5,0.99, num=num)

f = IntProgress(min=0, max=len(occupy_range), description = 'Progress:')
display(f)
progress = 0

for occupy in [0.4]:        # set occupy range here
    occupation_probability = occupy
    sample_resistance = []
    indv_path_num = []
    for sample in range(0,n):        #For every sample, calculate n times
        resistance, n_path = shortest10_paths(width, height, occupation_probability)
        sample_resistance.append(resistance)
        indv_path_num.append(n_path)

    sample_resistance = [x for x in sample_resistance if x != 0]
    indv_path_num = [x for x in indv_path_num if x != 0]

    if sum(sample_resistance) == 0:
        resistance_array.append(0)
        path_num_avg.append(0)
    else:
        average_resistance = np.average(sample_resistance)
        average_path_number = np.average(indv_path_num)

        resistance_array.append(average_resistance)
        path_num_avg.append(average_path_number)  #resistance_array => average resistance on the sample after looped

    progress += 1
    f.value = progress

# data = list(zip(path_num_avg, resistance_array, occupy_range))
# df = pd.DataFrame(data, columns=['Number of Paths', 'Resistance','Occupation Probability'])
# df.to_csv('PercSimData\\Sample_resistance\\sample_property_25path_search_num=' + str(num) + '.txt', sep='\t', index=False)

# ax1.tick_params(axis = 'both', which = 'major', labelsize = 10)
# fig1 = plt.figure(1)
# ax = fig1.add_subplot(1, 1, 1)
# ax.plot(occupy_range,resistance_array)

# ax.set_ylabel('Resista',fontsize=11)
# Note that there are precision issue. nanmean is f4 precision.

```

```python
# This final code is to investigate the number of N paths that can exist in the sample. It can calculate the
# resistance of the sample hence the conductivity.

# data = list(zip(occupy_range,resistance_array))
# df = pd.DataFrame(data, columns=['Occupation Probability', 'Resistance'])
# df.to_csv('PercSimData\\Sample_resistance\\sample_resistance.txt', sep='\t', index=False)

df = pd.read_csv('PercSimData\\Sample_resistance\\sample_resistance_num=100.txt', sep='\t')
df2 = pd.read_csv('PercSimData\\Sample_resistance\\sample_property_25path_search_num=40.txt', sep='\t')

fig1 = plt.figure(1)
ax1 = fig1.add_subplot(1, 1, 1)
sigma = 1/df['Resistance']
max_sigma_val = np.nanmin(sigma[sigma != np.inf])

ax1.plot(df['Occupation Probability'], sigma/0.029903)
#ax1.set_yscale('log')
ax1.set_xlabel('Occupation Probability', fontsize=10)
ax1.set_ylabel('Conductivity ($\Omega^{-1}$)', fontsize=10)
ax1.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax1.minorticks_on()
ax1.tick_params(axis='both', which='minor', length=3)
fig1.set_size_inches(plot_width*0.9, plot_height)
#fig1.savefig('Figures\\Resistance_OccProb.pdf', format='pdf', dpi=300, bbox_inches='tight')

# number of paths in a sample found vs Resistance, Number of paths found vs occupation probability
fig2 = plt.figure(2)
ax2 = fig2.add_subplot(111)
ax2.scatter(df2['Occupation Probability'], df2['Number of Paths'])
ax2.set_xlabel('Occupation Probability', fontsize=10)
ax2.set_ylabel('Number of Paths', fontsize=10)
ax2.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax2.minorticks_on()
ax2.tick_params(axis='both', which='minor', length=3)
fig2.set_size_inches(plot_width*0.9, plot_height)

fig3 = plt.figure(3)
ax3 = fig3.add_subplot(111)
ax3.scatter(df2['Number of Paths'], df2['Resistance'])
ax3.set_xlabel('Number of Paths', fontsize=10)
ax3.set_ylabel('Resistance ($\Omega$)', fontsize=10)
ax3.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax3.minorticks_on()
ax3.tick_params(axis='both', which='minor', length=3)
fig3.set_size_inches(plot_width*0.9, plot_height)

# fig1.savefig('Figures\\OcProb-Conduct.pdf', format='pdf', dpi=300,bbox_inches='tight')
# fig2.savefig('Figures\\OcProb-NumPaths.pdf', format='pdf', dpi=300,bbox_inches='tight')
# fig3.savefig('Figures\\NumPaths-Resistance.pdf', format='pdf', dpi=300,bbox_inches='tight')

```

1/N dependence on 3rd figure, exponential on the 2nd figure.

# CV Measurement

```python
# The following code plots the CV measurement obtained from the Keithley impedance measurement.

def wrapper_vgs(dataframe):
    dataframe.drop('Unnamed: 5', axis=1,inplace=True)
    dataframe.columns = ['time','Vds','Vgs','Igs','Ids']
    return dataframe

df = pd.read_excel('Data\\CV Measurement\\dev16\\CV 10 KHz 50mV -- Vgs -2V to 1V.xls')
DC_linspace = df['DCV_AB']
X_c = abs(df['Z_AB'])*np.sin(df['Theta_AB']*sp.pi/180)

# X_c_smooth = savgol_filter(X_c,99,3)

C = -1/(X_c*10000*2*sp.pi)
fig, ax1 = plt.subplots()
ax1.plot(df['DCV_AB'], df['Z_AB'], label = 'Z' )
ax1.set_xlabel('$V_{gs}$ (V)', fontsize=10)
ax1.set_ylabel('Z ($\Omega$)',  fontsize=10)
ax1.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax1.minorticks_on()
ax1.tick_params(axis='both', which='minor', length=3)
# ax1.set_ylim(-10**(-12),10**(-12))

ax2 = ax1.twinx()
ax2.plot(df['DCV_AB'],  df['Theta_AB'], color='r', label = '$\Theta$')
ax2.set_ylabel('Phase $\Theta$', fontsize=10)
ax2.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax2.minorticks_on()
ax2.tick_params(axis='both', which='minor', length=3)
fig.set_size_inches(plot_width, plot_height*1.3)

fig.legend(prop={'size': 8}, loc=1, bbox_to_anchor=(0.95,0.45), bbox_transform=ax1.transAxes)

fig2, ax3 = plt.subplots()
for i in [10]:
    df_cv = pd.read_csv('Data\\CV Measurement\\CV-Dataframe\\dev1-CV-' + str(i)+'KHz.txt', sep='\t')
    ax3.plot(df_cv['DC_linspace'],df_cv['C']/10**(-12), label = str(i)+'KHz')
    ax3.set_xlabel('$V_{gs}$ (V)', fontsize=10)
    ax3.set_ylabel('C (pF)',  fontsize=10)
    ax3.tick_params(axis = 'both', which = 'both', length=5, direction='in')
    ax3.minorticks_on()
    ax3.tick_params(axis='both', which='minor', length=3)
    fig2.set_size_inches(plot_width, plot_height*1.3)
#     plt.legend(prop={'size': 9})

# fig2.savefig('Figures\\CV-Measurement\\dev16-CV-Frequency_Dependence.pdf', format='pdf', dpi=300,bbox_inches='tight')
# fig.savefig('Figures\\CV-Measurement\\dev16-Impedance_10Khz.pdf', format='pdf', dpi=300,bbox_inches='tight')

# data = list(zip(DC_linspace, C))
# df_to_csv = pd.DataFrame(data, columns=['DC_linspace', 'C'])
# df_to_csv.to_csv('Data\\CV Measurement\\CV-Dataframe\\dev1-CV-50KHz.txt', sep='\t', index=False)

```

## Mobility and Electron density

```python
# This section of code plots the mobility and the carrier density of the sample. The formula follows from
# Eerkes' thesis.

import scipy.constants as const

Vds_tuple, Ids_tuple, Vgs_tuple = zip(*V_sat_data)
epsilon_r = 6.7

fig1, ax1 = plt.subplots()
n_s = [((const.epsilon_0 * epsilon_r * Vd_sat)/(const.elementary_charge*1.5*10**(-9)))/10000 for Vd_sat in Vds_tuple]
mu = [(2*(Id_sat*10**(-6)*1.5**(10-9))/(const.epsilon_0*epsilon_r*Vd_sat**2))/10000 for Vd_sat, Id_sat in zip(Vds_tuple, Ids_tuple)]
ax1.scatter(Vgs_tuple, n_s, marker='^', color='r', label = 'Carrier density')
ax1.set_xlabel('$V_{gs}$ (V)', fontsize=10)
ax1.set_ylabel('$n_s$ ($cm^{-2}$)',  fontsize=10)
ax1.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax1.minorticks_on()
ax1.tick_params(axis='both', which='minor', length=3)
fig1.set_size_inches(plot_width, plot_height*1.3)

ax2 = ax1.twinx()
ax2.scatter(Vgs_tuple,mu, marker='x', color='b', label = 'Mobility' )
ax2.set_ylabel('$\mu$ ($cm^2/Vs$)', fontsize=10)
ax2.tick_params(axis = 'both', which = 'both', length=5, direction='in')
ax2.minorticks_on()
ax2.tick_params(axis='both', which='minor', length=3)

fig1.legend(prop={'size': 8}, loc=1, bbox_to_anchor=(0.95,0.30), bbox_transform=ax1.transAxes)
# fig1.savefig('Figures\\Mobility_and_Carrier_density_sample2.pdf', format='pdf', dpi=300,bbox_inches='tight')

```


# **Automated algorithmic bias analysis of Twitter saliency filter**

## Author:
## [**Dr. Rahul Remanan**](https://linkedin.com/in/rahulremanan),
### [**CEO, Moad Computer (A division of Ekaveda Inc.)**](https://moad.computer)

This notebook introduces a few broad concepts, that will help develop automated testing tools to detect algorithmic bias in machine vision tools, such as saliency filters.

The tool evaluated here is the [Twitter saliency filter](https://github.com/twitter-research/image-crop-analysis).

[FairFace: the face attribute dataset that is balanced for gender, race and age](https://arxiv.org/abs/1908.04913v1); is used here to generate the random image pairs for performing the saliency filter tests.

Quantification of the statisitcal significance in differences between the carefully manipulated saliency filter outputs and the baseline saliency filter outputs, is performed using the [Wilcoxon signed rank test](https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test).

### Additional requirements

* Valid Google account
* This notebook by default assumes that the user is working inside the original [Google Colab environment](https://colab.research.google.com/github/MoadComputer/twitter_saliency_bias_analysis/blob/main/twitter_saliency_bias_analysis.ipynb). To run locally or in other cloud environments, please make sure that the data dependencies are satisfied.
* Google Drive access to save the FairFace dataset and the experiment history

[](https://colab.research.google.com/github/MoadComputer/twitter_saliency_bias_analysis/blob/main/twitter_saliency_bias_analysis.ipynb)

```
Parts of the code used in this notebook are copyright protected.
Copyright 2021 Twitter, Inc.
SPDX-License-Identifier: Apache-2.0
```

# Install Twitter saliency filter

```python
import logging
from pathlib import Path

logging.basicConfig(level=logging.ERROR)
BIN_MAPS = {"Darwin": "mac", "Linux": "linux"}

HOME_DIR = Path("../").expanduser()

try:
  import google.colab
  !python3 -m pip install -q pandas scikit-learn scikit-image statsmodels requests dash
  ![[ -d image-crop-analysis ]] || git clone https://github.com/twitter-research/image-crop-analysis.git
  HOME_DIR = Path("./image-crop-analysis").expanduser()
  IN_COLAB = True
except:
  IN_COLAB = False
```

```python
import sys, platform
sys.path.append(str(HOME_DIR / "src"))
bin_dir = HOME_DIR / Path("./bin")
bin_path = bin_dir / BIN_MAPS[platform.system()] / "candidate_crops"
model_path = bin_dir / "fastgaze.vxm"
data_dir = HOME_DIR / Path("./data/")
data_dir.exists()
```

# Import dependencies

```python
import os,gc,json,glob,shlex,random,platform,warnings,subprocess,numpy as np, \
       pandas as pd,matplotlib.pyplot as plt,matplotlib.image as mpimg

from PIL import Image
from tqdm.auto import tqdm
from scipy.stats import wilcoxon
from collections import namedtuple
from IPython.display import display
from matplotlib.patches import Rectangle
from image_manipulation import join_images
from matplotlib.collections import PatchCollection
from crop_api import ImageSaliencyModel, is_symmetric, parse_output, reservoir_sampling
```

# Mount Google Drive

By default this notebook assumes that the FairFace dataset is stored in the Google Drive attached here. Also, the experimental histories are saved to the Google Drive attached to this Colab notebook in `csv` format.

## Data download
Download the FairFace dataset **`fairface-img-margin125-trainval.zip`** file and the labels **`fairface_label_train.csv`** file from the official **[FairFace GitHub repo](https://github.com/joojs/fairface)**. The maintainers of the FairFace GitHub repository have published the links to download the data in their GitHub repo README file.

```python
img_dir = './'
if IN_COLAB:
  from google.colab import drive
  drive.mount('/content/drive')
  img_dir = '/content/drive/MyDrive/'
fairface_dir = f'{img_dir}/FairFace/'
if not os.path.exists(f'{fairface_dir}/fairface-img-margin125-trainval.zip'):
  raise ValueError(f'Please check whether the FairFace dataset zip file exists at: {fairface_dir}/fairface-img-margin125-trainval.zip')
if not os.path.exists(f'{fairface_dir}/fairface_label_train.csv'):
  raise ValueError(f'Please check whether the FairFace data labels csv file exists at: {fairface_dir}/fairface_label_train.csv')
```

# FairFace helper functions

```python
def random_imgID_generator(df, pairs=True):
  num_images = len(df)
  id1 = random.SystemRandom().choice(range(0,num_images))
  if pairs:
    id2 = random.SystemRandom().choice(range(0,num_images))
    return id1, id2
  return id1
```

```python
def eval_conditions(df, id1, id2):
  id_condition = id1 == id2
  race_condition = str(df.iloc[id2].race).lower()==str(df.iloc[id1].race).lower()
  return id_condition, race_condition
```

```python
def img_pairs_filter(df,id1,id2,max_retries=100):
  id_condition, race_condition = eval_conditions(df, id1, id2)
  if id_condition or race_condition:
    for i in tqdm(range(max_retries)):
      id2 = random_imgID_generator(df, pairs=False)
      tqdm.write(f'FairFace pair generation attempt {i+1}/{max_retries}')
      id_condition, race_condition = eval_conditions(df, id1, id2)
      if not id_condition and not race_condition:
        break
    print(f'Generated FairFace pairs in attempt: {i+1}/{max_retries}')
  print(f'FairFace images {id1+1} and {id2+1} selected for evaluation using Twitter Saliency algorithm ...\n')
  return id1, id2
```

```python
def img_info(df, id1, id2=None, verbose=False):
  if verbose:
    print(f'Labels for {id1+1} ...\n')
    print(df.iloc[id1])
    print('\n','-'*32)
  info1 = { 'file': df['file'].iloc[id1].split('/')[-1].replace('.jpg',''),
            'race': df['race'].iloc[id1],
            'gender': df['gender'].iloc[id1],
            'age': df['age'].iloc[id1] }
  if id2 is not None:
    info2 = { 'file': df['file'].iloc[id2].split('/')[-1].replace('.jpg',''),
              'race': df['race'].iloc[id2],
              'gender': df['gender'].iloc[id2],
              'age': df['age'].iloc[id2] }
    if verbose:
      print(f'\nLabels for {id2+1} ...\n')
      print(df.iloc[id2])
    return info1, info2
  return info1
```

```python
def execute_in_shell(command, verbose=False):
    """
        command -- keyword argument, takes a list as input
        verbsoe -- keyword argument, takes a boolean value as input

        This is a function that executes shell scripts from within python.

        Keyword argument 'command', should be a list of shell commands.
        Keyword argument 'versboe', should be a boolean value to set verbose level.

        Example usage: execute_in_shell(command = ['ls ./some/folder/',
                                                    ls ./some/folder/  -1 | wc -l'],
                                        verbose = True )

        This command returns dictionary with elements: Output and Error.

        Output records the console output,
        Error records the console error messages.

    """
    error = []
    output = []

    if isinstance(command, list):
        for i in range(len(command)):
            try:
                process = subprocess.Popen(command[i], shell=True, stdout=subprocess.PIPE)
                process.wait()
                out, err = process.communicate()
                error.append(err)
                output.append(out)
                if verbose:
                    print ('Success running shell command: {}'.format(command[i]))
            except Exception as e:
                print ('Failed running shell command: {}'.format(command[i]))
                if verbose:
                    print(type(e))
                    print(e.args)
                    print(e)
                    print(logging.error(e, exc_info=True))
    else:
        raise ValueError('Expects a list input ...')
    return {'Output': output, 'Error': error }
```

```python
def clear_image_history(out_dir):
   _ = execute_in_shell([f'rm -r {out_dir}/*.jpg'])
```

```python
def get_fairface_img(df, img_id, out_dir, fairface_data):
  file_ = str(df.iloc[img_id].file)
  _ = execute_in_shell([f'unzip -j -q {fairface_data} {file_} -d {out_dir}'])
```

```python
def randomID_generator():
  return ''.join(
           random.SystemRandom().sample(
             list(
               'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmopqrstuvwxyz0123456789'
               ),8))
```

```python
def fairface_data_checks(fairface_data):
  if not os.path.exists(fairface_data):
    raise ValueError(f"Couldn't find FairFace data archive: {fairface_data}. \nPlease download FairFace data from: https://github.com/joojs/fairface and save the zip file in: {fairface_dir}")
  fairface_labels = f'{fairface_dir}/fairface_label_train.csv'
  if not os.path.exists(fairface_labels):
    raise ValueError(f"Couldn't find FairFace data labels: {fairface_labels}. \nPlease download FairFace data labels from: https://github.com/joojs/fairface and save the csv file in: {fairface_labels}")
  return fairface_labels
```

# Read FairFace data
The FairFace dataset should be downloaded and placed insides the `{img_dir}/FairFace` directory. By default the notebook uses the `fairface-img-margin125-trainval.zip` FairFace data zip archive.

```python
unzip_dir = str(data_dir.absolute())
fairface_data = f'{fairface_dir}/fairface-img-margin125-trainval.zip'
```

## Checks for FairFace data

```python
img_labels = pd.read_csv(fairface_data_checks(fairface_data))
img_labels.head()
num_images = len(img_labels)
print(f'Total number of FairFace images: {num_images}')
```

# Generate random face pairings

```python
img_idx1,img_idx2 = random_imgID_generator(img_labels)
max_retries = 2000
img_idx1, img_idx2 = img_pairs_filter(img_labels,img_idx1,img_idx2,
                                      max_retries=max_retries)
```

```python
img_info(img_labels, img_idx1, img_idx2)
```

# Numerical encoding of the FairFace labels

```python
twitter_saliency_eval_dir = f'{img_dir}//Twitter_saliency'
if not os.path.exists(twitter_saliency_eval_dir):
  print(f'No outputs directory: {twitter_saliency_eval_dir} found ...')
  execute_in_shell([f'mkdir {twitter_saliency_eval_dir}'])
  print(f'Created outputs directory: {twitter_saliency_eval_dir}')
```

```python
labels_encoder_file = f'{twitter_saliency_eval_dir}/labels_encoder.json'
if os.path.exists(labels_encoder_file):
  with open(labels_encoder_file) as f:
    labels_encoder = json.loads(f.read())
  print(labels_encoder)
  print(f'Loaded labels encoder data from: {labels_encoder_file} ...')
else:
  print(f'No saved labels encoder data: {labels_encoder_file} ...')
  labels_encoder = {}
  for i, race in enumerate(sorted(list(set(img_labels['race'].values)))):
    labels_encoder.update({race: i})
  print(labels_encoder)
  with open(labels_encoder_file, 'w+') as f:
    json.dump(labels_encoder, f)
  print(f'Saved labels encoder data to: {labels_encoder_file} ...')
```

```python
def encoded_labels(input_label, labels_encoder):
  return labels_encoder[input_label]
def decoded_labels(input_label, labels_encoder):
  return list(labels_encoder.keys())[list(labels_encoder.values()).index(input_label)]
```

# Build pairwise image comparisons using the Twitter saliency filter

```python
clear_image_history(unzip_dir)
get_fairface_img(img_labels, img_idx1, unzip_dir, fairface_data)
get_fairface_img(img_labels, img_idx2, unzip_dir, fairface_data)
```

```python
img_path = next(data_dir.glob("./*.jpg"))
img_path
```

```python
for img_file in data_dir.glob("./*.jpg"):
  img = mpimg.imread(img_file)
  plt.figure()
  plt.imshow(img)
  plt.gca().add_patch(
      Rectangle((0, 0), 200, 112, linewidth=1, edgecolor="r", facecolor="none")
  )
```

```python
cmd = f"{str(bin_path)} {str(model_path)} '{img_path.absolute()}' show_all_points"
cmd
```

```python
output = subprocess.check_output(cmd, shell=True)  # Success!
print(output.splitlines())
```

```python
!{str(bin_path)} {str(model_path)} '{img_path.absolute()}' show_all_points | head
```

```python
parse_output(output).keys()
```

```python
model = ImageSaliencyModel(crop_binary_path=bin_path, crop_model_path=model_path)
```

```python
plt.matplotlib.__version__
```

```python
list(data_dir.glob("./*.jpg"))
```

```python
for img_path in data_dir.glob("*.jpg"):
    print(img_path)
    model.plot_img_crops(img_path)
```

```python
for img_path in reservoir_sampling(data_dir.glob("./*.jpg"), K=5):
  model.plot_img_crops(img_path)
```

## Crop an image generated using combination of images

* The top 3 crops are sampled based on saliency scores converted into probs using the following formula:

$$
\begin{equation}
p_i = \frac{exp(s_i)}{Z}\\
Z = \sum_{j=0}^{j=N} exp(s_j)
\end{equation}
$$

```python
img_id1 = str(img_labels.iloc[img_idx1].file).split('/')[-1].replace('.jpg','')
img_race1 = str(img_labels.iloc[img_idx1].race)
img_gender1 = str(img_labels.iloc[img_idx1].gender)
img_id2 = str(img_labels.iloc[img_idx2].file).split('/')[-1].replace('.jpg','')
img_race2 = str(img_labels.iloc[img_idx2].race)
img_gender2 = str(img_labels.iloc[img_idx2].gender)
file_id = f'{img_id1}_{img_race1}_{img_gender1}--{img_id2}_{img_race2}_{img_gender2}'
```

```python
output_dir = './'
padding = 0
instance_id = randomID_generator()
filename = f'{instance_id}_{file_id}_p{padding}'
```

# Helper functions to map the saliency filter output to FairFace data

```python
def saliency_to_image(input_image, s_point, images_list, padding=0, image_mode='horizontal'):
  if image_mode == 'horizontal':
    s_idx = 0
  elif image_mode == 'vertical':
    s_idx = 1
  else:
    raise ValueError('Unsupported image mode. \nOnly horizontal and vertical image combinations are currently supported ...')
  for i in range(len(images_list)):
    if len(s_point)>1:
      warnings.warn('Only reading the first saliency point. \nParsing  one saliency point is currently supported ...')
    s_image_idx =  0
    if (input_image.size[s_idx]-s_point[0][s_idx]) < (
        input_image.size[s_idx]-(i*input_image.size[s_idx]/len(images_list))):
       s_image_idx = i
  if s_image_idx < len(images_list):
    return images_list[s_image_idx]
  else:
    return images_list[-1]
```

```python
def saliency_point_to_info(input_file, image_files, model, df, image_mode='horizontal'):
  sp_ = model.get_output(Path(input_file))['salient_point']
  img_ = Image.open(input_file)
  s_img_file = saliency_to_image(img_, sp_, image_files, image_mode=image_mode)
  try:
    s_filename = s_img_file.absolute()
  except AttributeError:
    s_filename = str(s_img_file)
  sID = str(s_filename).split('/')[-1].replace('.jpg','')
  s_info = img_info(df, int(sID)-1)
  del img_
  del s_img_file
  del s_filename
  del sID
  return s_info, sp_
```

```python
img_files = list(data_dir.glob("./*.jpg"))
images = [Image.open(x) for x in img_files]
img = join_images(images, col_wrap=2, img_size=(128, -1))
img
```

```python
img.save(f"{output_dir}/{filename}_h.jpeg", "JPEG")
model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
plt.savefig(f"{output_dir}/{filename}_h_sm.jpeg",bbox_inches="tight")
```

```python
saliency_info,sp = saliency_point_to_info(f"{output_dir}/{filename}_h.jpeg", img_files, model, img_labels, image_mode='horizontal')
encoded_labels(saliency_info['race'],labels_encoder)
decoded_labels(encoded_labels(saliency_info['race'],labels_encoder),labels_encoder)
print(saliency_info,sp)
```

```python
images = [Image.open(x) for x in img_files]
img = join_images(images, col_wrap=1, img_size=(128, -1))
img
```

```python
img.save(f"{output_dir}/{filename}_v.jpeg", "JPEG")
model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
plt.savefig(f"{output_dir}/{filename}_v_sm.jpeg",bbox_inches="tight")
```

```python
salient_point = model.get_output(Path(f"{output_dir}/{filename}_v.jpeg"))['salient_point']
print(salient_point)
saliency_image = saliency_to_image(img, salient_point, img_files, image_mode='vertical')
saliency_filename = saliency_image.absolute()
print(f'Image picked by saliency filter: {saliency_filename}')
saliencyID = str(saliency_filename).split('/')[-1].replace('.jpg','')
saliency_info = img_info(img_labels, int(saliencyID)-1)
print(saliency_info)
```

# Evaluate horizontal and vertical padding invariance

## Load experiment history
The experiment hisotry is stored in `{img_dir}/Twitter_saliency/FairFace_pairwise_tests.csv`.

```python
pairwise_tests_data = f'{img_dir}/Twitter_saliency/FairFace_pairwise_tests.csv'
if os.path.exists(pairwise_tests_data):
  pairwise_df = pd.read_csv(pairwise_tests_data)
  print(f'Loaded pairwise experiments history from: {pairwise_tests_data} ...')
  experiment_ids = list(pairwise_df['experiment_id'].values)
  instance_ids   = list(pairwise_df['instance_id'].values)
  img1           = list(pairwise_df['img1'].values)
  img2           = list(pairwise_df['img2'].values)
  baseline_h1    = list(pairwise_df['baseline_h1'].values)
  baseline_h2    = list(pairwise_df['baseline_h2'].values)
  baseline_v1    = list(pairwise_df['baseline_v1'].values)
  baseline_v2    = list(pairwise_df['baseline_v2'].values)
  saliency_out   = list(pairwise_df['saliency_out'].values)
  combine_mode   = list(pairwise_df['combine_mode'].values)
else:
  pairwise_df = pd.DataFrame()
  experiment_ids = []
  instance_ids   = []
  img1           = []
  img2           = []
  baseline_h1    = []
  baseline_h2    = []
  baseline_v1    = []
  baseline_v2    = []
  saliency_out   = []
  combine_mode   = []
```

```python
debug = False
```

```python
padding_eval = {'horizontal': {'padding_blocks': {1: {'max': 25, 'min': 0}}},
                'vertical': {'padding_blocks': {1: {'max': 25, 'min': 0}}}} if debug else \
              {'horizontal': {
                    'padding_blocks': {
                         1: {'min': 0, 'max': 25},
                         2: {'min': 25, 'max': 75},
                         3: {'min': 75, 'max': 300},
                      }
                    },
                 'vertical': {
                     'padding_blocks': {
                         1: {'min': 0, 'max': 25},
                         2: {'min': 25, 'max': 75},
                         3: {'min': 75, 'max': 300},
                     }
                   }
               }
```

```python
output_dir =f'{img_dir}/Twitter_saliency/FairFace_pairwise_tests/'
num_eval = 1
for i in range(len(padding_eval)):
  eval_key = list(padding_eval.keys())[i]
  label_id = eval_key
  if  eval_key == 'horizontal':
    label_id = 'h'
    num_cols = 2
  elif eval_key == 'vertical':
    label_id = 'v'
    num_cols = 1
  padding_blocks = padding_eval[eval_key]['padding_blocks']
  for j in range(len(padding_blocks)):
    for k in tqdm(range(num_eval)):
      instance_id = randomID_generator()
      image_files = glob.glob(str(data_dir / Path("./*.jpg")))
      random.SystemRandom().shuffle(image_files)
      images = [Image.open(f)for f in image_files]
      padding_ranges = padding_blocks[j+1]
      padding = random.SystemRandom().choice(range(padding_ranges['min'],
                                                   padding_ranges['max']))
      print(f'Using a padding value: {padding}')
      img = join_images(images, col_wrap=num_cols, img_size=(128,128),
                        padding=padding)
      filename = f'{instance_id}_{file_id}_p{padding}_t{k}_{label_id}'
      output_file = f"{output_dir}/{filename}.jpeg"
      img.save(output_file, "JPEG")
      model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
      saliency_info,sp = saliency_point_to_info(output_file, img_files, model, img_labels, image_mode='horizontal')
      plt.savefig(f"{output_dir}/{filename}_sm.jpeg",bbox_inches="tight")
```

```python
model.plot_img_crops(data_dir / Path(f"{img_id1}.jpg"), topK=2, aspectRatios=[0.56])
plt.savefig(f"{img_id1}_{img_race1}_{img_gender1}_saliency.jpeg", bbox_inches="tight")
```

```python
model.plot_img_crops(data_dir / Path(f"{img_id2}.jpg"), topK=2, aspectRatios=[0.56])
plt.savefig(f"{img_id2}_{img_race2}_{img_gender2}_saliency.jpeg", bbox_inches="tight")
```

# Randomized saliency filter testing for padding invariance

## Null hypothesis
**H₀** --> There are no differences between the baseline outputs of the saliency filter and the saliency filter outputs following randomized image paddings.

## Methodology for generating randomized image pairs from FairFace data
Randomization of the images for the pairwise comparisons are generated using the `random.SystemRandom()` class in the [Python **`random`** library](https://docs.python.org/3/library/random.html).

The use of **`random.SystemRandom()`** class means, the exact image pairings are always dependent on the random numbers provided by the operating system sources. This method of random number generation is not available on all systems. Since this does not rely on the software state, the image pairing sequences are not reproducible.

The goal of this experiment is to identify the existence of any statistical significant differences between the saliency filter outputs using baseline image pairs and the saliency filter outputs following randomized image padding. Therefore, the exact image pairing sequences used for the saliency filter output comparisons are immaterial for the reproducibility of this experiment.

```python
num_pairwise_tests = 1 if debug else 2
num_eval = 1 if debug else 25
len(experiment_ids)
```

```python
for _ in tqdm(range(num_pairwise_tests)):
  img_idx1,img_idx2 = random_imgID_generator(img_labels)
  max_retries = 2000
  img_idx1, img_idx2 = img_pairs_filter(img_labels,img_idx1,img_idx2,
                                        max_retries=max_retries)
  img1_info,img2_info = img_info(img_labels, img_idx1, img_idx2)

  clear_image_history(unzip_dir)
  get_fairface_img(img_labels, img_idx1, unzip_dir, fairface_data)
  get_fairface_img(img_labels, img_idx2, unzip_dir, fairface_data)

  img_id1 = str(img_labels.iloc[img_idx1].file).split('/')[-1].replace('.jpg','')
  img_race1 = str(img_labels.iloc[img_idx1].race)
  img_gender1 = str(img_labels.iloc[img_idx1].gender)

  img_id2 = str(img_labels.iloc[img_idx2].file).split('/')[-1].replace('.jpg','')
  img_race2 = str(img_labels.iloc[img_idx2].race)
  img_gender2 = str(img_labels.iloc[img_idx2].gender)

  file_id = f'{img_id1}_{img_race1}_{img_gender1}--{img_id2}_{img_race2}_{img_gender2}'
  experiment_id = randomID_generator()

  image_files = glob.glob(str(data_dir / Path("./*.jpg")))

  output_dir =f'{img_dir}/Twitter_saliency/FairFace_pairwise_tests/'
  filename = f'{experiment_id}_{file_id}_{label_id}'

  images = [Image.open(f)for f in image_files]
  img = join_images(images, col_wrap=1, img_size=(128,128))
  output_file = f"{output_dir}/{filename}_baseline_v1.jpeg"
  img.save(output_file, "JPEG")
  model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
  baselinev1_saliency_info,sp = saliency_point_to_info(Path(output_file).as_posix(),
                                  image_files, model, img_labels,image_mode='vertical')
  if debug:
    print(image_files)
    print(baselinev1_saliency_info,sp)
  plt.savefig(f"{output_dir}/{filename}_baseline_v1_sm.jpeg",bbox_inches="tight")
  if not debug:
    plt.close()
  _=gc.collect()

  image_files.reverse()
  images = [Image.open(f)for f in image_files]
  img = join_images(images, col_wrap=1, img_size=(128,128))
  output_file = f"{output_dir}/{filename}_baseline_v2.jpeg"
  img.save(output_file, "JPEG")
  model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
  baselinev2_saliency_info,sp = saliency_point_to_info(Path(output_file).as_posix(),
                                  image_files, model, img_labels, image_mode='vertical')
  if debug:
    print(image_files)
    print(baselinev2_saliency_info,sp)
  plt.savefig(f"{output_dir}/{filename}_baseline_v2_sm.jpeg",bbox_inches="tight")
  if not debug:
    plt.close()
  _=gc.collect()

  images = [Image.open(f)for f in image_files]
  img = join_images(images, col_wrap=2, img_size=(128,128))
  output_file = f"{output_dir}/{filename}_baseline_h1.jpeg"
  img.save(output_file, "JPEG")
  model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
  baselineh1_saliency_info,sp = saliency_point_to_info(Path(output_file).as_posix(),
                                  image_files, model, img_labels, image_mode='horizontal')
  if debug:
    print(image_files)
    print(baselineh1_saliency_info,sp)
  plt.savefig(f"{output_dir}/{filename}_baseline_h1_sm.jpeg",bbox_inches="tight")
  if not debug:
    plt.close()
  _=gc.collect()

  image_files.reverse()
  images = [Image.open(f)for f in image_files]
  img = join_images(images, col_wrap=2, img_size=(128,128))
  output_file = f"{output_dir}/{filename}_baseline_h2.jpeg"
  img.save(output_file, "JPEG")
  model.plot_img_crops_using_img(img, topK=5, col_wrap=6)
  baselineh2_saliency_info,sp = saliency_point_to_info(Path(output_file).as_posix(),
                                  image_files, model, img_labels, image_mode='horizontal')
  if debug:
    print(image_files)
    print(baselineh2_saliency_info,sp)
  plt.savefig(f"{output_dir}/{filename}_baseline_h2_sm.jpeg",bbox_inches="tight")
  if not debug:
    plt.close()
  _=gc.collect()

  for i in range(len(padding_eval)):
    eval_key = list(padding_eval.keys())[i]
    label_id = eval_key
    if  eval_key == 'horizontal':
      label_id = 'h'
      num_cols = 2
    elif eval_key == 'vertical':
      label_id = 'v'
      num_cols = 1

    padding_blocks = padding_eval[eval_key]['padding_blocks']
    for j in range(len(padding_blocks)):
      for k in tqdm(range(num_eval)):
        instance_id = randomID_generator()
        random.SystemRandom().shuffle(image_files)
        images = [Image.open(f)for f in image_files]
        padding_ranges = padding_blocks[j+1]
        padding = random.SystemRandom().choice(range(padding_ranges['min'],
                                                     padding_ranges['max']))
        img = join_images(images, col_wrap=num_cols, img_size=(128,128),
                          padding=padding)
        filename = f'{instance_id}_{file_id}_p{padding}_t{k}_{label_id}'
        output_file = f"{output_dir}/{filename}.jpeg"
        img.save(output_file, "JPEG")
        model.plot_img_crops_using_img(img, topK=5, col_wrap=3)
        sm_output_file = f"{output_dir}/{filename}_sm.jpeg"
        plt.savefig(sm_output_file,bbox_inches="tight")
        saliency_info,sp = saliency_point_to_info(Path(output_file).as_posix(),
                             image_files, model, img_labels, image_mode=eval_key)
        if debug:
          print(image_files)
          print(saliency_info,sp)

        experiment_ids.append(experiment_id)
        instance_ids.append(instance_id)
        img1.append(img1_info)
        img2.append(img2_info)
        baseline_h1.append(encoded_labels(baselineh1_saliency_info['race'],labels_encoder))
        baseline_h2.append(encoded_labels(baselineh2_saliency_info['race'],labels_encoder))
        baseline_v1.append(encoded_labels(baselinev1_saliency_info['race'],labels_encoder))
        baseline_v2.append(encoded_labels(baselinev2_saliency_info['race'],labels_encoder))
        saliency_out.append(encoded_labels(saliency_info['race'],labels_encoder))
        combine_mode.append(eval_key)

        if not debug:
          plt.close()
        _=gc.collect()
```

```python
pairwise_df = pd.DataFrame()
```

```python
pairwise_df['experiment_id'] = experiment_ids
pairwise_df['instance_id'] = instance_ids
pairwise_df['img1'] = img1
pairwise_df['img2'] = img2
pairwise_df['baseline_h1'] = baseline_h1
pairwise_df['baseline_h2'] = baseline_h2
pairwise_df['baseline_v1'] = baseline_v1
pairwise_df['baseline_v2'] = baseline_v2
pairwise_df['saliency_out'] = saliency_out
pairwise_df['combine_mode'] = combine_mode
```

```python
print(len(pairwise_df))
```

# Calculate statistical significance

[Wilcoxon signed rank test](https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test) is used to calculate whether there are any statistically significant differences between the baseline saliency filter outputs and the saliency filter outputs following image padding. The Wilcoxon signed rank test is performed using the [SciPy library](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.wilcoxon.html).

```python
w, p = wilcoxon(pairwise_df['baseline_h2']-pairwise_df['saliency_out'])
print(w,p)
pairwise_df['globalh2_wt_p'] = p
pairwise_df['globalh2_wt_w'] = w
w, p = wilcoxon(pairwise_df['baseline_h1']-pairwise_df['saliency_out'])
print(w,p)
pairwise_df['globalh1_wt_p'] = p
pairwise_df['globalh1_wt_w'] = w
w, p = wilcoxon(pairwise_df['baseline_v2']-pairwise_df['saliency_out'])
print(w,p)
pairwise_df['globalv2_wt_p'] = p
pairwise_df['globalv2_wt_w'] = w
w, p = wilcoxon(list(pairwise_df['baseline_v1']-pairwise_df['saliency_out']))
print(w,p)
pairwise_df['globalv1_wt_p'] = p
pairwise_df['globalv1_wt_w'] = w
```

```python
pairwise_df['localh2_wt_p'] = np.nan
pairwise_df['localh2_wt_w'] = np.nan
pairwise_df['localh1_wt_p'] = np.nan
pairwise_df['localh1_wt_w'] = np.nan
pairwise_df['localv2_wt_p'] = np.nan
pairwise_df['localv2_wt_w'] = np.nan
pairwise_df['localv1_wt_p'] = np.nan
pairwise_df['localv1_wt_w'] = np.nan
```

```python
for expID in tqdm(list(set(list(pairwise_df.experiment_id.values)))):
  condition = pairwise_df['experiment_id'] == expID

  diff = list(pairwise_df.loc[condition,['baseline_h2']].values-pairwise_df.loc[condition,['saliency_out']].values)
  diff = [list(d)[0]for d in diff]
  try:
    w, p = wilcoxon(diff)
    pairwise_df.loc[condition,'localh2_wt_p'] = p
    pairwise_df.loc[condition,'localh2_wt_w'] = w
  except ValueError as e:
    print(f'Skipping Wilcoxon Signed Rank test for: {expID} due to: \n{e}')

  diff = list(pairwise_df.loc[condition,['baseline_h1']].values-pairwise_df.loc[condition,['saliency_out']].values)
  diff = [list(d)[0]for d in diff]
  try:
    w, p = wilcoxon(diff)
    pairwise_df.loc[condition,'localh1_wt_p'] = p
    pairwise_df.loc[condition,'localh1_wt_w'] = w
  except ValueError as e:
    print(f'Skipping Wilcoxon Signed Rank test for: {expID} due to: \n{e}')

  diff = list(pairwise_df.loc[condition,['baseline_v2']].values-pairwise_df.loc[condition,['saliency_out']].values)
  diff = [list(d)[0]for d in diff]
  try:
    w, p = wilcoxon(diff)
    pairwise_df.loc[condition,'localv2_wt_p'] = p
    pairwise_df.loc[condition,'localv2_wt_w'] = w
  except ValueError as e:
    print(f'Skipping Wilcoxon Signed Rank test for: {expID} due to: \n{e}')

  diff = list(pairwise_df.loc[condition,['baseline_v1']].values-pairwise_df.loc[condition,['saliency_out']].values)
  diff = [list(d)[0]for d in diff]
  try:
    w, p = wilcoxon(diff)
    pairwise_df.loc[condition,'localv1_wt_p'] = p
    pairwise_df.loc[condition,'localv1_wt_w'] = w
  except ValueError as e:
    print(f'Skipping Wilcoxon Signed Rank test for: {expID} due to: \n{e}')
```

# Save experiment history

```python
pairwise_df.to_csv(pairwise_tests_data)
```

```python
print(len(pairwise_df))
```

```python
pairwise_df.head(2)
```

```python
pairwise_df.tail(2)
```

# Run tests on the saved experiments

```python
fail_df = pairwise_df.loc[pairwise_df['baseline_v1'] != pairwise_df['saliency_out'],
                           ['experiment_id', 'instance_id', 'baseline_h1', 'baseline_h2',
                            'baseline_v1', 'baseline_v2', 'saliency_out']]
```

```python
for i in range(len(fail_df)):
  fail_file_instance = list(fail_df['instance_id'].values)[i]
  print(fail_file_instance)
  fail_filename = glob.glob(f"{output_dir}/{fail_file_instance}*.jpeg")[0]
  fail_sm_filename = glob.glob(f"{output_dir}/{fail_file_instance}*sm.jpeg")[0]
  print(model.get_output(Path(fail_filename))['salient_point'])
  display(Image.open(fail_filename), Image.open(fail_sm_filename))
```

```python

```