```python
import re
from tqdm import tqdm
import pickle
from functools import reduce
import mafan
from mafan import text
import itertools
import sys
import os
bos = " <bos> "
eos = " <eos> "
```

# Tokenizer Functions

## Sentence Tokenizer

```python
def zng(paragraph):
    for sent in re.findall(u'[^!?。\.\!\?]+[!?。\.\!\?]?', paragraph, flags=re.U):
        yield sent
```

## Simplified Chinese Tokenizer

Below is the code for simplified to traditional mapping dictionary.

We have a large dictionary *conversions.txt* that includes words, characters, common phrases, locations and idioms. Each entry contains the traditional chinese word and simplified chinese word.

```python
infile = open("conversions.txt", "r+", encoding="utf-8")

s2t_dict = dict()

for line in infile:
    line = line.rstrip()
    arr = line.split()
    trad = arr[0]
    sim = arr[1]
    if sim not in s2t_dict:
        s2t_dict[sim] = [trad]
    else:
        s2t_dict[sim].append(trad)
s2t_dict['-'] = ['-']
```

Tokeniser is used for identifying dictionary words and phrases in the input sentence. We always prefer longer phrases because it gives more meaning and less translation mappings. Hence we use Byte Pair Encoding (BPE) for identifying words, while BPE candidates are constrained by the defined list of vocabs in the dictionary. Since the longest phrase in the dictionary has 8 characters we start with 8-character phrases and do it backwards.

```python
def tokenizer(sentence, n = 8):
    '''
    This function tokenizes input sentences according to the dicitionary.
    Input: a sentence or paragraph
    Output: a list of tokens from the input in order according to the original paragraph; a list of non-chinese characters from the original text.
    '''
    text, charList = prepare(sentence)
    token_list = []
    input_text = text
    for k in range(n, 0, -1):
        candidates = [input_text[i:i + k] for i in range(len(input_text) - k + 1)]
        for candidate in candidates:
            if candidate in s2t_dict:
                token_list.append(candidate)
                input_text = re.sub(candidate, '', input_text)
    final = sequencer(token_list, text)
    return final, charList
```

```python
def output_list(sentence_list, char_list):
    count = 0
    original = [] # sentence we want to output

    for word in sentence_list:
        if "-" in word:
            original.append(list(char_list[count]))
            count += 1
        else:
            original.append(word)
    return original
```

```python
def output(sentence, char_list):
    count = 0
    original = "" # sentence we want to output

    for char in list(sentence):
        if char == "-":
            original += char_list[count] # append character if non-chinese
            count += 1
        else:
            original += char # append chinese
    return original
```

```python
def prepare(sentence):
    new = "" # input to your tokenizer
    char_list = [] # punct / english to be omitted

    for char in list(sentence):
        if text.identify(char) is mafan.NEITHER:
            new += "-" # sub - with non-chinese chars
            char_list.append(char)
        else:
            new += char

    return new, char_list
```

```python
def sequencer(tokens, example):

    flags = [1] * len(example)
    sequence = []
    for token in tokens:
        for match in re.finditer(token, example):
            location = (token, match.span()[0], match.span()[1])
            valid = reduce(lambda x,y:x*y, flags[location[1]:location[2]])
            if valid:
                sequence.append(location)
                for i in range(location[1], location[2]):
                    flags[i] = 0
            else:
                continue
    sequence.sort(key=lambda x: x[1])
    result = [x[0] for x in sequence]
    return result
```

## Corpus Preparation

First, we need to prepare our corpus.
1. We will add paddings (sentinels) to our sentences.
2. Take one sentence at a time.
3. Change non-chinese words to FW to avoid data explosion.
4. Slice the n-grams and add them to dictionary

```python
def add_stuff(order):
    '''
    This function divides the corpus into n-grams and stores them in dictionary.
    Input: order of n-gram (like 2 for bi-gram)
    Output: none
    '''
    infile = open("hk-zh.txt", "r+") # this contains our corpus
    start_padding = bos * order # add padding
    end_padding = eos * order

    for line in tqdm(infile, total=1314726):
        line = line.rstrip()
        sentences = list(zng(line)) # tokenize sentence by sentence
        for sentence in sentences:
            candidate = start_padding + sentence + end_padding # form sentence
            word_list = candidate.split()
            word_list_tokens = []
            for word in word_list:
                if not(bool(re.match('^[a-zA-Z0-9]+$', word))):
                    word_list_tokens.append(word) # add if not chinese
                else:
                    word_list_tokens.append("FW") # turn non-chinese (except punc) to FW
            word_list = word_list_tokens
            ordered = [word_list[i:i + order] for i in range(1, len(word_list) - order)] # extract n-grams through slicing
            # for each ngram, convert to tuple and add to dictionary
            for ngram in ordered:
                ngram = tuple(ngram)
                if ngram not in corpus:
                    corpus[ngram] = 1
                else:
                    corpus[ngram] += 1
```

Let's say you want to extract till trigrams.

We want to do 3 iterations, for trigram, bi-gram and then unigram. Each iteration takes 2 minutes. This is only time-consuming part of this code. Once you prep the dictionary, you don't need to do this again.

```python
corpus = dict()
start_order = 3
for i in range(start_order, 0, -1):
    add_stuff(i)
```

Once you made the dictionary, dump it into a pickle.

```python
import pickle
with open('corpus.pkl', 'wb') as handle:
    pickle.dump(corpus, handle)
```

Here's a way to load a pickle so you don't need to process data everytime.

```python
with open('corpus.pkl', 'rb') as fp:
    corpus = pickle.load(fp)
```

# Making Candidate Lists

1. Tokenize the input.
2. Check the mappings of each input.
3. Add all possible mappings to candidate list.

```python
def convert(sentence):
    '''
    Returns list of possible mappings.
    Input: Simplified chinese sentence
    Output: List of lists. Each list has a set of possible traditional chinese tokens
    '''
    tokens, char_list = tokenizer(sentence)
    candidate_list = []
    for token in tokens:
        candidate_list.append(s2t_dict[token])
    candidate_list = output_list(candidate_list, char_list)
    return(candidate_list)
```

# Maximum log-likelihood calculations

Compute the log likelihood of a sentence with different \\(\alpha\\) penalties for unigram, bigram and trigrams.

```python
num_tokens = 4526000 # total number of tokens in corpus

def prob(word_list, alpha_0 = 0.25, alpha_1 = 0.5, alpha_2 = 1.0):
    '''
    Computes the log likelihood probability.
    Input: A sequence of words in form of list
    Output: Log probabilties
    '''
    word_list = tuple(word_list) # change word list to tuple
    if word_list in corpus:
        # word found in dictionary
        numerator = corpus[word_list] # get the frequency of that word list
        denominator = num_tokens # let denominator be num tokens
        # cutoff the last word and check whether it's in corpus
        if len(word_list[:-1]) > 1 and word_list[:-1] in corpus:
            denom_list = word_list[:-1]
            denominator = corpus[denom_list]
        if len(word_list[:-1]) == 1 and word_list[:-1] in corpus:
            return alpha_0 * log(numerator / denominator) # log of prob*alpha
        elif len(word_list[:-1]) == 2 and word_list[:-1] in corpus:
            return alpha_1 * log(numerator / denominator)
        elif len(word_list[:-1]) == 3 and word_list[:-1] in corpus:
            return alpha_2 * log(numerator / denominator)
        else:
            return log(numerator/denominator)
    else:
        word_list = list(word_list) # convert it back to list
        k = len(word_list) - 1 # backoff, reduce n gram length
        if k > 0:
            # recursive function, divide the sequence into smaller n and find probs
            probs = [prob(word_list[i:i + k]) for i in range(len(word_list) - k + 1)]
            return sum(probs)
        else:
            # we found an unseen word
            if not(bool(re.match('^[a-zA-Z0-9]+$', word_list[0]))):
                return log(1 / num_tokens) # return a small probability
            else:
                return prob(["FW"]) # we encountered a non-chinese word
```

# Language Model

Generative n-gram language model that estimates the conditional probability of a word given its history in the n-gram. It's calculated by backing off through progressively shorter history models.

Stupid Backoff does not generate normalized probabilities. The main difference is that we don’t apply any discounting and instead directly use the relative frequencies (S is used instead of P to emphasize that these are not probabilities but scores).

\\(S(w^i|w^{i−1}_{i−k+1}) =
\begin{cases}
    \frac{f(w^{i}_{i−k+1})}{f(w^{i-1}_{i−k+1})} & \text{if } f(w^{i}_{i−k+1})> 0\\
    \\\alpha S(w^i|w^{i−1}_{i−k+2}),              & \text{otherwise}
\end{cases}
\\)

Where \\(\alpha\\) is the backoff factor.

Stupid Backoff is inexpensive to calculate in a distributed environment while approaching the quality
of Kneser-Ney smoothing for large amounts of data.

```python
from math import log
def backoff(sentence, order, alpha_0 = 0.25, alpha_1 = 0.5, alpha_2 = 1.0):
    '''
    Calcuates log likelihood using backoff language model
    Input: Sentence and order of the n-gram
    Output: Log prob of that sentence
    '''
    score = 0
    sentences = list(zng(sentence)) # sentence tokenizer
    for sentence in sentences:
        start_padding = bos * order # beginning padding
        end_padding = eos * order # ending padding
        candidate = start_padding + sentence + end_padding # add paddings
        word_list = candidate.split()
        word_list_tokens = []
        for word in word_list:
            # append only non-chinese words
            if not(bool(re.match('^[a-zA-Z0-9]+$', word))):
                word_list_tokens.append(word)
            else:
                word_list_tokens.append("FW")
        word_list = word_list_tokens
        ordered = [word_list[i:i + order] for i in range(1, len(word_list) - order)] # shingle into n-grams
        probs = [prob(x, alpha_0, alpha_1, alpha_2) for x in ordered] # calculate probabilities
        score += sum(probs) # final answer
    return score
```

# Translator

Take simplified sentence as input, generate candidate list for sentence.
For words with many to one mappings add the candidate to a temporary sentence, calculate perplexity and choose the option which gives the lowest perplexity.

Call function to add back spaces at the end and output the final sentence.

```python
def translate(sentence, alpha_0 = 0.25, alpha_1 = 0.5, alpha_2 = 1.0):
    '''
    Translate a given sentence to traditional
    Input: Simplified Sentence
    Output: Traditional Sentence
    '''
    candidates = convert(sentence) # get the candidate lists
    final_sent = ""
    for words in candidates:
        if len(words) > 1:
            # many to one mappings
            score = -50000.0 # start with extreme negative value
            likely = ""
            for candidate in words:
                temp = final_sent
                temp = temp + " "  + candidate # add a candidate to temp sentence
                current_score = backoff(temp, 3, alpha_0, alpha_1, alpha_2) # check perplexity
                if current_score > score:
                    # if performing good, include that
                    score = current_score
                    likely = candidate
            final_sent = final_sent + " " + likely
        else:
            final_sent = final_sent + " " + words[0]
    final_sent = final_sent.replace(" ", "")
    final_sent = add_back_spaces(sentence, final_sent) #call function to add the spaces back and output translation
    return final_sent
```

Add spaces back by enumerating through the original and the appended list.

```python
def add_back_spaces(original, current):
    current_list = list(current)
    original_list = list(original)
    count = 1
    for index, char in enumerate(original_list):
        if char == " ":
            current_list[index - count] += " "
            count += 1
    current = "".join(current_list)
    return current
```

Test sentence for translate.

```python
sentence = "早在23岁，伍兹就参与了世界上首个核反应堆Chicago Pile-1的建设，她是导师费米领导的项目团队中最年轻的一员。此外，伍兹在建立和使用实验所需的盖革计数器上起到关键作用。反应堆成功运转并达到自持状态时，她也是唯一在场的女性。曼哈顿计划中，她与费米合作；同时，她曾与第一任丈夫约翰·马歇尔（John Marshall）一同解决了汉福德区钚生产厂氙中毒的问题，并负责监督钚生产反应炉的建造和运行。"
a = translate(sentence)
print(a)
```

# Joint Probability Based Tokenizer

Greedy tokenizer would generally work for most of the cases, however, it could lead to an undesirable segmentation, due to the preference towards longer chunks.
We propose a joint consideration for sub-word segmentation by considering both source and target sentences.

A translator needs a source sentence $\mathbf{S}$ consisting of segmentations where $\mathbf{S} = s_0 s_1 \dots s_n$ and a target sentence $\mathbf{T}$ consisting of segmentations where $\mathbf{T} = t_0 t_1 \dots t_m$.

We want to find optimal arrangement of $\mathbf{S}$ which is $\mathbf{S}^*$ and optimal arrangement of $\mathbf{T}$ which is $\mathbf{T}^*$. Mathematically:
\begin{align}
\label{eq1}
    \mathbf{S}^*, \mathbf{T}^* = \underset{{s_i \in \mathbf{S}, t_j \in \mathbf{T}}}{\operatorname{argmax}} P(\mathbf{S}, \mathbf{T})
\end{align}
where $P(\mathbf{S}, \mathbf{T})$ is the joint probability of sequences.

We assume that the prior probabilities, which are $P(\mathbf{S})$ and $P(\mathbf{T})$, are language model based probabilities.

```python
def dp_tokenizer(sentence):
    s = sentence
    global orig_len
    orig_len = len(s)
    return segment(s)
```

```python
model = kenlm.Model("sim_train.klm")
```

```python
def memo(f):
    "Memoize function f, whose args must all be hashable."
    cache = {}
    def fmemo(*args):
        if args not in cache:
            cache[args] = f(*args)
        return cache[args]
    fmemo.cache = cache
    return fmemo
```

```python
def splits(text, start=0, L=20):
    "Return a list of all (first, rest) pairs; start <= len(first) <= L."
    return [(text[:i], text[i:])
            for i in range(start, min(len(text), L)+1)]
```

We chose Viterbi for segmentation the given sentence. The scoring function is obtained from the constructed language models.

Optimal segmentation depends on the following:

language model score of source sentence of a candidate segment.

language model score of target sentence of a candidate segment.

item mapping conversions from source segment to target segment

```python
@memo
def segment(text):
    "Return a list of words that is the most probable segmentation of text."
    if not text:
        return []
    else:
        candidates = ([first] + segment(rest)
                      for (first, rest) in splits(text, 1))
        return max(candidates, key=Pwords)
```

To avoid OOVs as output segmentations, we imposed a penalty on OOV outputs, which is given by: $\alpha \times \frac{\texttt{len(segment)}}{\texttt{len(sentence)}}$.

```python
penalty_constant = 15.0
```

```python
def Pwords(words):
    "Probability of words, assuming each word is independent of others."
    sentence = " ".join(words)
    score = 0
    words_ = ['<s>'] + sentence.split() + ['</s>']
    for i, (prob, length, oov) in enumerate(model.full_scores(sentence)):
        if oov:
            penalty = len(words_[i+1]) / orig_len
            score += penalty_constant * prob * penalty
        else:
            score += prob
    return score
```

```python
import string
alphanumerics = 'a-zA-Z0-9'
known_stops = u'。。…！？'
known_punctuation = u'／（）、，。：「」…。『』！？《》“”；’ ‘【】·〔〕'
eng_punct = string.punctuation
avoid = re.compile("([%s%s%s%s]+)" % (alphanumerics, known_stops, known_punctuation, eng_punct))
```

# Tokenize Sentence
Tokenize sentence and output the tokens.

```python
def tokenize_sentence(sentence):
    split_words = re.split(avoid, sentence)
    split_words_values = [(i, bool(re.search(avoid, i))) for i in split_words]
    answer = []
    for (word, value) in split_words_values:
        segmented_text = []
        if value == False:
            orig_len = len(word)
            segmented_text = dp_tokenizer(word)
        else:
            segmented_text = list(word)
        for segs in segmented_text:
            answer.append(segs)
    return answer
```

```python
sentence = "姚松炎、周庭势被「DQ」? 泛民质疑，政府再取消参选人资格涉政治筛选，要求律政司司长郑若骅解释法律理据。 有报道指，据全国人大常委会就《基本法》第一百零四条进行的释法，代表泛民参选立法会港岛及九龙西补选的香港众志周庭和被「DQ」前议员姚松炎，势被取消参选资格。律政司表示，法律政策专员黄惠冲将于稍后时间与泛民议员会面，确实时间待定。 民主派议员前晚在律政中心外静坐要求与律政司司长郑若骅会面不果后，昨在立法会召开记者招待会，要求郑就撤销参选人资格的理据，及其给予选举主任的法律意见作出详细交代。公民党议员郭荣铿批评，郑不向公众交代的做法是「冇承担，冇责任」的表现，不能只把责任交托予公务员。"
a = tokenize_sentence(sentence)
```

# Evaluation

Evaluating the accuracy on 100 lines on sample test data.

```python
sim_filename = "simplified100.txt"
tra_filename = "traditional100.txt"
```

```python
checklist = []
for key in s2t_dict:
    if len(s2t_dict[key]) > 1:
        for t in s2t_dict[key]:
            checklist.append(t)
```

The translated characters are matched with the original traditional corpus during evaluation. The mismatch characters include wrongly characters and variant characters. Variant characters are characters that are homophones and synonyms. In some cases, simplified characters can have multiple traditional variant characters mapped to them, which gives the same meaning and context. Thus a mismatch in this case does not necessarily means a incorrect conversion. However this is not a common case and does not affect the evaluation result substantially.

```python
def eval(sim_filename, tra_filename, alpha_0 = 0.25, alpha_1 = 0.5, alpha_2 = 1.0):

    total = 0
    correct = 0
    wrong = 0
    micro_total = 0
    micro_correct = 0

    sim_file = open(sim_filename, "r+", encoding="utf-8")
    tra_file = open(tra_filename, "r+", encoding="utf-8")
    tra_lines = tra_file.readlines()
    line_count = 0

    for line in sim_file:

        line = line.rstrip()

        line = translate(line, alpha_0 , alpha_1 , alpha_2)
        tra_line = tra_lines[line_count].rstrip()

        if len(line) == len(tra_line):
            char_count = 0
            for c in line:
                total = total + 1
                if c == tra_line[char_count]:
                    correct = correct + 1
                else:
                    # print(c + tra_line[char_count])
                    wrong = wrong + 1

                if tra_line[char_count] in checklist:
                    micro_total += 1
                    if c == tra_line[char_count]:
                        micro_correct = micro_correct + 1

                char_count = char_count + 1

        line_count += 1
    results = []
    results.append(('Total', (total)))
    results.append(('Correct' , (correct)))
    results.append(('Wrong' , (wrong)))
    results.append(('Percentage' , (correct/total*100)))
    results.append(('Micro Total' , (micro_total)))
    results.append(('Micro Correct' , (micro_correct)))
    results.append(('Micro Percentage' , (micro_correct/micro_total*100)))

    return results
```

Overall Accuracy is defined as (no. of correctly converted characters) / (no. of converted characters). We also calculate the Micro-average accuracy to evaluate the performance for one-to-many character conversions only.

```python
print(eval(sim_filename, tra_filename, 0.7, 0.5, 1))
```

# Hyper-Parameter Tuning

Testing the evaluation function on the test set with multiple hyperparameter values in order to determine the optimal values for the hyperparameter.

```python
outfile = open('alpha_tuning.txt', 'w+')
max = [99.5, 0.25, 0.5, 1]
for a_0 in range(100, 10, -10):
    for a_1 in range(100, 10, -10):
        outfile.write(str(eval(sim_filename, tra_filename, alpha_0 = float(a_0)/100.0, alpha_1 = float(a_1)/100.0, alpha_2 = 1.0)) + "a0: " + str(a_0/100.0) + " " + "a1: " + str(a_1/100.0) + '\n')
```

Part 1, Disambiguation and Conversion.ipynb


# Data Augmentation Using Generative Adversarial Networks (GANs)

Notebook to perform data augmentation using Generative Adversarial Networks (GANs). This notebook must be run after running the notebook `prepare_dataset_and_create_project_structure.ipynb` that should be available in the same directory.

---
## Prerequisites

1. Ensure that the Cityscapes dataset is downloaded and placed in the directory named **dataset**. More information about the dataset can be found in the notebook `prepare_dataset_and_create_project_structure.ipynb` that should be available in the same directory.
2. Run all the cells in the notebook `prepare_dataset_and_create_project_structure.ipynb` to understand the dataset, process the dataset, create the project structure required to ensure correct results from the project.

### 1. Check Python version

It is crucial to ensure that the notebook runs on the correct version of Python to guarantee proper functionality.

```python
import platform
assert (platform.python_version_tuple()[:2] >= ('3','7')), "[ERROR] The notebooks are tested on Python 3.7 and higher. Please updated your Python to evaluate the code"
```

### 2. Check Notebook server has access to all required resources

```python
from pathlib import Path

dataset_folder = Path("dataset")
dataset_folder = Path.joinpath(Path.cwd(), dataset_folder)

if not dataset_folder.exists():
    raise FileNotFoundError("[ERROR] Add `{}` folder in the current directory (`{}`)".format(dataset_folder.name, Path.cwd()))
```

```python
dataset_preparation_notebook = Path("prepare_dataset_and_create_project_structure.ipynb")
dataset_preparation_notebook = Path.joinpath(Path.cwd(), dataset_preparation_notebook)

if not dataset_preparation_notebook.exists():
    raise FileNotFoundError("[ERROR] The notebook `{}` is unavailable in the current directory (`{}`). Please download and run the notebook `{}` before this notebook to ensure proper results.".format(dataset_preparation_notebook.name, Path.cwd(), dataset_preparation_notebook.name))
```

```python
test_dataset = Path.joinpath(dataset_folder, "test_dataset")
test_dataset_A = Path.joinpath(test_dataset, "A")
test_dataset_B = Path.joinpath(test_dataset, "B")

test_dataset_overall = [test_dataset, test_dataset_A, test_dataset_B]

for dataset in test_dataset_overall:
    if not dataset.exists():
        raise FileNotFoundError("[ERROR] The folder `{}` is unavailable. Please run the notebook `prepare_dataset_and_create_project_structure.ipynb` available in the current directory (`{}`) before running this notebook.".format(dataset.name, Path.cwd()))
```

```python
training_dataset = Path.joinpath(dataset_folder, "training_dataset")
training_dataset_A = Path.joinpath(training_dataset, "A")
training_dataset_B = Path.joinpath(training_dataset, "B")

training_dataset_overall = [training_dataset, training_dataset_A, training_dataset_B]

for dataset in training_dataset_overall:
    if not dataset.exists():
        raise FileNotFoundError("[ERROR] The folder `{}` is unavailable. Please run the notebook `prepare_dataset_and_create_project_structure.ipynb` available in the current directory (`{}`) before running this notebook.".format(dataset.name, Path.cwd()))
```

```python
validatation_dataset = Path.joinpath(dataset_folder, "validatation_dataset")
validatation_dataset_A = Path.joinpath(validatation_dataset, "A")
validatation_dataset_B = Path.joinpath(validatation_dataset, "B")

validatation_dataset_overall = [validatation_dataset, validatation_dataset_A, validatation_dataset_B]

for dataset in validatation_dataset_overall:
    if not dataset.exists():
        raise FileNotFoundError("[ERROR] The folder `{}` is unavailable. Please run the notebook `prepare_dataset_and_create_project_structure.ipynb` available in the current directory (`{}`) before running this notebook.".format(dataset.name, Path.cwd()))
```

---
## Introduction

One of the biggest bottlenecks in creating generalized deep learning models is a scarcity of high-quality data. The collection of high-quality data and its conversion is expensive. Most of the data collection methods are labor-intensive and error-prone, requiring considerable editing afterward to clean the data. Since large amounts of data are needed to achieve generalized deep learning models, standard data augmentation methods are routinely used to increase the dataset's generalizability. Data augmentation methods are also used when the datasets are imbalanced, improving the model's overall performance.

Generative Adversarial Networks, popularly known as GANs, are a novel method for data augmentation. The generation of artificial training data can not only be instrumental in situations such as imbalanced data sets, but it can also be useful when the original dataset contains sensitive information. In such cases, it is then desirable to avoid using the original data as much as possible (For example, Medical data).

This report proposes a GAN architecture based on a [paper](https://arxiv.org/abs/1611.07004) from UC Berkeley to perform data augmentation using the popular image-to-image translation method. Generative Adversarial Networks trained on these methods learn the mapping from an input image to an output image and learn a loss function to train this mapping. Therefore, this approach makes it possible to apply the same generic approach to problems that traditionally require very different loss formulations. In this particular report, we demonstrate that this approach can effectively synthesize photos from label maps. To evaluate the performance of the proposed GAN architecture, we utilize a standard dataset named Cityscapes. The Cityscapes Dataset focuses on semantic understanding of urban street scenes. The dataset contains 5000 images with detailed annotations and 20000 images with coarse annotations apart from the original images.

---
## Background Theory

With the advancements in deep learning, the most striking successes have involved discriminative models, usually those that map a high-dimensional, rich sensory input to a class label. These striking successes have primarily been based on the backpropagation and dropout algorithms, using piecewise linear units. However, deep generative models have had less impact due to the challenges of approximating many probabilistic computations that occur due to the usage of piecewise linear units in the generative context.

Generative Adversarial Networks, popularly known as GANs, is a machine learning framework class that sidesteps these difficulties by pitting the generative model against an adversary. In other words, a GAN is a machine learning framework where two neural networks compete against each other in a zero-sum game (i.e., one network's gain is the other network's loss). The two networks in a GAN can be considered as a generator and a discriminator. The generator learns to create images that look real, while the discriminator learns to tell real images apart from fakes. Competition in this game drives both networks to improve their models until the counterfeits are indistinguishable from the genuine images. An overview of the Generative Adversarial Network is represented below:

Image Courtesy: Google Developers (Link: https://developers.google.com/machine-learning/gan/gan_structure)

One of the most significant characteristics of GANs is the lack of loss function. GANs learn the loss function to classify if the output image is real or fake while simultaneously training a generative model to minimize this loss. This property of GANs allows it to learn a loss that adapts to the data, making them a perfect solution to a multitude of tasks that traditionally would require very different kinds of loss functions such as image-to-image translation. The image-to-image translation is the task of translating one possible representation of a scene into another, given sufficient training data. Traditionally, each image-to-image translation task has been tackled with separate, special-purpose machinery, although the setting is always the same: predict pixels from pixels. The biggest problem with this approach is the need to formulate specialized loss functions that drive the neural network to do what we want – e.g., output sharp, realistic images. Fortunately, the need for specialized loss functions is precisely eliminated with the use of GANs.

In this report, we explore GANs in the conditional setting. Just as GANs learn a generative data model, conditional GANs (cGANs) learn a conditional generative model. In other words, just as GANs are generative models that learn a mapping from random noise vector $z$ to output image $y$, $G:\,z\,\rightarrow\,y$, the conditional GANs (cGANs) learn a mapping from observed image $x$ and random noise vector $z$, to output image $y$,$G:\,\{x,\, z\}\,\rightarrow\,y$. This ability to map from an observed image to an output image makes cGANs fitting for image-to-image translation tasks, where we condition on an input image and generate a corresponding output image.

Finally, to understand the objective function of the conditional GANs (i.e., cGANS), let us first examine the objective function of a traditional GAN. The equation below refers to the objective function of a traditional GAN.

$$
\begin{align}
\mathbb{L}_{GAN}(G,D)\,=\,&\mathbb{E}_{y}[logD(y)]\,+\, \\
&\mathbb{E}_{x\,,\,z}[log(1\,-\,D(G(x\,,\,z)))]
\end{align}
$$

It can be observed from the Equation above that Generator $G$ tries to minimize the objective function against an adversarial $D$ that tries to maximize it. Therefore, the optimal value of G can be represented as the equation below.

$$
\begin{align}
G^{*}\,=\,arg\,min_{G}\,max_{D}\,\mathbb{L}_{GAN}(G\,,\,D)
\end{align}
$$

Based on the Equation above, the objective function of a conditional GAN can be represented as Equation below.

$$
\begin{align}
\mathbb{L}_{cGAN}(G,D)\,=\,&\mathbb{E}_{x\,,\,y}[logD(x\,,\,y)]\,+\, \\
&\mathbb{E}_{x\,,\,z}[log(1\,-\,D(x\,,\,G(x\,,\,z)))]
\end{align}
$$

Similar to the equation above, Generator $G$ tries to minimize the objective function against an adversarial $D$ that tries to maximize it. By training both $G$ and $D$ on the objective function $\mathbb{L}_{cGAN}(G\,,\,D)$ simultaneously, the optimal values for both the parameters can be calculated.

---
## Proposed Solution

### 0. Imports

```python
import os
import glob
import time
import numpy as np
import scipy as sp
from imageio import imread
import matplotlib.pyplot as plt
from keras.optimizers import Adam
from keras.models import Sequential, Model
from keras.layers.advanced_activations import LeakyReLU
from keras.layers.convolutional import UpSampling2D, Conv2D
from keras.layers import BatchNormalization, Activation, ZeroPadding2D
from keras.layers import Input, Dense, Reshape, Flatten, Dropout, Concatenate
```

### 1. Load Data

As mentioned in the previous sections, the Cityscapes dataset was used to translate photos from label maps. The Cityscapes Dataset is perfect for our project as the dataset focuses on semantic understanding of urban street scenes. The dataset contains 5000 images with detailed annotations and 20000 images with coarse annotations apart from the original images. Some sample images from the dataset are presented in below:

Image Courtesy: Cityscapes Datatset (Link: https://www.cityscapes-dataset.com/)

```python
def draw_randomized_data(sample_size, dataset_type="training"):
    """
    Function to draw randomized data samples from dataset. This function would be required while displaying training output.

    Parameters
    ----------
    sample_size  : Number of data samples to be loaded (type=int)
    dataset_type : Type of dataset to draw randomized data samples from (type=str)

    Returns
    -------
    random_ground_truth_data  : Randomized "Ground Truth Data" drawn from the dataset "dataset_type" (type=list)
    random_input_data         : Randomized "Input Data" drawn from the dataset "dataset_type" (type=list)
    """

    # Calculate Dataset Paths
    if(dataset_type == "training"):
        ground_truth_dataset_expr = str(training_dataset_A) + "\\**\\*.jpg"
        input_dataset_expr = str(training_dataset_B) + "\\**\\*.jpg"
    else:
        ground_truth_dataset_expr = str(validatation_dataset_A) + "\\**\\*.jpg"
        input_dataset_expr = str(validatation_dataset_B) + "\\**\\*.jpg"

    ground_truth_dataset_paths = glob.glob(ground_truth_dataset_expr, recursive=True)
    ground_truth_dataset_paths = sorted(ground_truth_dataset_paths)

    input_dataset_paths = glob.glob(input_dataset_expr, recursive=True)
    input_dataset_paths = sorted(input_dataset_paths)

    # Check if Datasets Are Modified After Preprocessing
    assert (len(ground_truth_dataset_paths) == len(input_dataset_paths)), ("[ERROR] Datasets Modified After Preprocessing! Please reload the dataset from the beginning!")

    # Draw Random Samples from the dataset
    random_data_samples = np.random.choice(range(len(ground_truth_dataset_paths)), sample_size) # Pick any random data from the dataset

    random_ground_truth_data = []
    random_input_data = []

    # Process Randomly Chosen Samples
    for random_sample in random_data_samples:

        ground_truth_data_path = ground_truth_dataset_paths[random_sample]
        input_data_path = input_dataset_paths[random_sample]

        ground_truth_data = imread(ground_truth_data_path)
        input_data = imread(input_data_path)

        # Perform Basic Transformations to Avoid Overfitting
        if((dataset_type != "training") and (np.random.random() > 0.5)):
            ground_truth_data = np.fliplr(ground_truth_data)
            input_data = np.fliplr(input_data)

        random_ground_truth_data.append(ground_truth_data)
        random_input_data.append(input_data)

    # Normalizing the image between [-1, 1] since tanh() activation functions are used
    random_ground_truth_data = np.array(random_ground_truth_data)/127.5 - 1
    random_input_data = np.array(random_input_data)/127.5 - 1

    return random_ground_truth_data, random_input_data
```

```python
def load_batches(batch_size, dataset_type="training"):
    """
    Function to load batches. This function would be required while training the neural network.

    Parameters
    ----------
    batch_size   : Size of the batches to be loaded (type=int)
    dataset_type : Type of dataset to load batches from (type=str)

    Yeilds (Returns a Generator and Not an Iterator [IMPORTANT!])
    ------
    ground_truth_data_batch  : Batch of "Ground Truth Data" loaded from the dataset "dataset_type" (type=list)
    input_data_batch         : Batch of "Input Data" loaded from the dataset "dataset_type" (type=list)
    """

    # Calculate Dataset Paths
    if(dataset_type == "training"):
        ground_truth_dataset_expr = str(training_dataset_A) + "\\**\\*.jpg"
        input_dataset_expr = str(training_dataset_B) + "\\**\\*.jpg"
    else:
        ground_truth_dataset_expr = str(validatation_dataset_A) + "\\**\\*.jpg"
        input_dataset_expr = str(validatation_dataset_B) + "\\**\\*.jpg"

    ground_truth_dataset_paths = glob.glob(ground_truth_dataset_expr, recursive=True)
    ground_truth_dataset_paths = sorted(ground_truth_dataset_paths)

    input_dataset_paths = glob.glob(input_dataset_expr, recursive=True)
    input_dataset_paths = sorted(input_dataset_paths)

    # Check if Datasets Are Modified After Preprocessing
    assert (len(ground_truth_dataset_paths) == len(input_dataset_paths)), ("[ERROR] Datasets Modified After Preprocessing! Please reload the dataset from the beginning!")

    # Load Batches
    for batch_iterator in range(int(len(ground_truth_dataset_paths)/batch_size) - 1):

        ground_truth_dataset_paths_batch = ground_truth_dataset_paths[(batch_iterator)*batch_size:(batch_iterator+1)*batch_size]
        input_dataset_paths_batch = input_dataset_paths[(batch_iterator)*batch_size:(batch_iterator+1)*batch_size]

        ground_truth_data_batch = []
        input_data_batch = []

        for (ground_truth_data_path_iterator, input_data_path_iterator) in zip(ground_truth_dataset_paths_batch, input_dataset_paths_batch):

            ground_truth_data = imread(ground_truth_data_path_iterator)
            input_data = imread(input_data_path_iterator)

            # Perform Basic Transformations to Avoid Overfitting
            if((dataset_type != "training") and (np.random.random() > 0.5)):
                ground_truth_data = np.fliplr(ground_truth_data)
                input_data = np.fliplr(input_data)

            ground_truth_data_batch.append(ground_truth_data)
            input_data_batch.append(input_data)

        # Normalizing the image between [-1, 1] since tanh() activation functions are used
        ground_truth_data_batch = np.array(ground_truth_data_batch)/127.5 - 1
        input_data_batch = np.array(input_data_batch)/127.5 - 1

        # Return a Generator Object that Calculates the Batches "On-the-fly"
        yield ground_truth_data_batch, input_data_batch
```

### 2. Define cGAN Architecture

**The Generator**

A defining feature of image-to-image translation problems is that they map a high-resolution input grid to a high-resolution output grid. Furthermore, for the problems we consider, although the input and output differ in surface appearance, both are renderings of the same underlying structure. Since the input structure is roughly aligned with the structure in the output, an encoder-decoder network is used as a generator layer.  In an encoder-decoder network, the input is passed through a series of layers that progressively downsample. This process is continued until a bottleneck layer, at which point the process is reversed. Such a network requires that all information flow through all the layers, including the bottleneck. For many image translation problems, there is a great deal of low-level information shared between the input and output, and it would be desirable to shuttle this information directly across the net. Therefore, to allow the Generator network to map the representation of the input image and the output image, the U-Net architecture was chosen based on the suggestion from UC Berkley paper. The generator architecture chosen is represented in figure below.

```python
def build_generator():
    """
    Closure Function to Build A Generator for cGAN.

    Parameters
    ----------
    None

    Returns
    -------
    Model   : Generator Model for cGAN (type=tf.keras.models.Model)
    """
    def conv2d(input_layer, filters, filter_shape=4, batch_normalization=True):
        """
        Nested Function to create a 2D-Convolution Layer

        Parameters
        ----------
        input_layer         : Input Layer to the 2D-Convolution Layer
        filters             : Number of filters in the convolution
        filter_shape        : Size of the 2D convolution window
        batch_normalization : Flag to set Batch Normalization

        Returns
        -------
        d    : 2D-Convolution Layer
        """
        d = Conv2D(filters, kernel_size=filter_shape, strides=2, padding='same')(input_layer)
        d = LeakyReLU(alpha=0.2)(d)
        if(batch_normalization == True):
            d = BatchNormalization(momentum=0.8)(d)

        return d

    def deconv2d(input_layer, skip_input, filters, filter_shape=4, dropout_rate=0):
        """
        Nested Function to create a 2D-Deconvolution Layer

        Parameters
        ----------
        input_layer   : Input Layer to the 2D-Convolution Layer
        skip_input    : Input Layer to be Skipped (i.e., Inputs of this layer are concatenated)
        filters       : Number of filters in the convolution
        filter_shape  : Size of the 2D convolution window
        dropout_rate  : Dropout Rate

        Returns
        -------
        u    : 2D-Deconvolution Layer
        """
        u = UpSampling2D(size=2)(input_layer)
        u = Conv2D(filters, kernel_size=filter_shape, strides=1, padding='same', activation='relu')(u)
        if dropout_rate:
            u = Dropout(dropout_rate)(u)
        u = BatchNormalization(momentum=0.8)(u)
        u = Concatenate()([u, skip_input]) #skip connection

        return u

    d0 = Input(shape=image_shape)

    # Downsampling
    d1 = conv2d(d0, gf, batch_normalization=False)
    d2 = conv2d(d1, gf*2)
    d3 = conv2d(d2, gf*4)
    d4 = conv2d(d3, gf*8)
    d5 = conv2d(d4, gf*8)
    d6 = conv2d(d5, gf*8)
    d7 = conv2d(d6, gf*8)

    # Upsampling
    u1 = deconv2d(d7, d6, gf*8)
    u2 = deconv2d(u1, d5, gf*8)
    u3 = deconv2d(u2, d4, gf*8)
    u4 = deconv2d(u3, d3, gf*4)
    u5 = deconv2d(u4, d2, gf*2)
    u6 = deconv2d(u5, d1, gf)

    u7 = UpSampling2D(size=2)(u6)
    generated_image = Conv2D(channels, kernel_size=4, strides=1, padding='same', activation='tanh')(u7)

    return Model(d0, generated_image)
```

**The Discriminator**

For the discriminator, we decided to use a convolutional *PatchGAN* classifier, which penalizes the Generator based on the scale of image patches, the same as in the UC Berkley paper. Therefore, the discriminator tries to classify if each patch in an image is real or fake. We run this discriminator convolutionally across the image, averaging all responses to provide the ultimate output. One of the most significant advantages of this approach is that the patch can be much smaller than the full size of the image and still produce high-quality results. Having a smaller discriminator like PatchGAN is advantageous as the network has fewer parameters, runs faster, and can be applied to arbitrarily large images.

```python
def build_discriminator():
    """
    Closure Function to Build A Discriminator for cGAN.

    Parameters
    ----------
    None

    Returns
    -------
    Model   : Discriminator Model for cGAN (type=tf.keras.models.Model)
    """
    def d_layer(input_layer, filters, filter_shape=4, batch_normalization=True):
        """
        Nested Function to create a single layer discriminator

        Parameters
        ----------
        input_layer         : Input Layer to the 2D-Convolution Layer
        filters             : Number of filters in the convolution
        filter_shape        : Size of the 2D convolution window
        batch_normalization : Flag to set Batch Normalization

        Returns
        -------
        d    : Discriminator layer
        """
        d = Conv2D(filters, kernel_size=filter_shape, strides=2, padding='same')(input_layer)
        d = LeakyReLU(alpha=0.2)(d)
        if batch_normalization:
            d = BatchNormalization(momentum=0.8)(d)

        return d

    ground_truth_image = Input(shape=image_shape)
    input_image = Input(shape=image_shape)

    # Concatenate image and conditioning image by channels to produce input
    combined_images = Concatenate(axis=-1)([ground_truth_image, input_image])

    d1 = d_layer(combined_images, df, batch_normalization=False)
    d2 = d_layer(d1, df*2)
    d3 = d_layer(d2, df*4)
    d4 = d_layer(d3, df*8)

    validity = Conv2D(1, kernel_size=4, strides=1, padding='same')(d4)

    return Model([ground_truth_image, input_image], validity)
```

### 3. Training

To optimize our networks, we follow the standard approach from the UC Berkley paper. We use minibatch SGD and apply the Adam solver, with a learning rate of 0.0002.

```python
# Input shape
image_rows = 128
image_columns = 128
channels = 3
image_shape = (image_rows, image_columns, channels)

# Calculate output shape of D (PatchGAN)
patch = int(image_rows / 2**4)
discriminator_patch = (patch, patch, 1)

# Number of filters in the first layer of G and D
gf = 64
df = 64

optimizer = Adam(0.0002, 0.5)

# Build and compile the discriminator
discriminator = build_discriminator()
discriminator.compile(loss='mse', optimizer=optimizer, metrics=['accuracy'])

# Build the generator
generator = build_generator()

# Input images and their conditioning images
ground_truth_image = Input(shape=image_shape)
input_image = Input(shape=image_shape)

# By conditioning on B generate a fake version of A
generated_ground_truth_image = generator(input_image)

# For the combined model we will only train the generator
discriminator.trainable = False

# Discriminators determines validity of translated images / condition pairs
valid = discriminator([generated_ground_truth_image, input_image])

gan = Model(inputs=[ground_truth_image, input_image], outputs=[valid, generated_ground_truth_image])
gan.compile(loss=['mse', 'mae'], loss_weights=[1, 100], optimizer=optimizer)
```

```python
# Function to display output

def display_output(output_data_type, number_of_outputs=3):

    real_ground_truth_images, input_images = draw_randomized_data(number_of_outputs, dataset_type=output_data_type)

    generated_ground_truth_image = generator.predict(input_images)
    overall_generated_images = np.concatenate(input_images, generated_ground_truth_image, real_ground_truth_images)

    # Rescale Image
    overall_generated_images = 0.5 * overall_generated_images + 0.5

    titles = ["Input", "Output", "Ground Truth"]
    f, axarr = plt.subplots(3, number_of_outputs, figsize=(20,30))

    for row_iterator in range(3):
        for column_iterator in range(number_of_outputs):
            axarr[row_iterator, column_iterator].imshow(overall_generated_images[row_iterator + column_iterator])
            axarr[row_iterator, column_iterator].set_title(titles[row_iterator])
            axarr[row_iterator, column_iterator].axis("off")

    plt.show()
    plt.close()
```

```python
# Function to perform training
def train_neural_network(epochs, batch_size, display_interval=10):

    start_time = time.time()
    print("cGAN Training started at {}".format(time.asctime(time.localtime(start_time))))

    # Calculate Batch Shape and Target Values
    batch_shape = (batch_size,) + discriminator_patch    # Joining Tuples to create a batch shape of dimensions: (batch, height, width, channels)
    target_real_images = np.ones(batch_shape)
    target_fake_images = np.zeros(batch_shape)

    # Initialize Generaor and Discriminator Losses
    generator_loss_overall = []
    discriminator_loss_overall = []

    for current_epoch in range(epochs):

        for batch_iterator, (ground_truth_images, input_images) in enumerate(load_batches(batch_size)):

            # Generate Fake Ground Truth Images
            generated_ground_truth_image = generator.predict(input_images)

            # Train Discriminator
            discriminator_loss_real_ground_truth = discriminator.train_on_batch([ground_truth_images, input_images], target_real_images)
            discriminator_loss_fake_ground_truth = discriminator.train_on_batch([generated_ground_truth_image, input_images], target_fake_images)

            print(discriminator_loss_fake_ground_truth)
            print(type(discriminator_loss_fake_ground_truth))
            print(discriminator_loss_fake_ground_truth.shape)

            discriminator_loss_current = 0.5*np.add(discriminator_loss_real_ground_truth, discriminator_loss_fake_ground_truth)

            print(discriminator_loss_current)
            print(type(discriminator_loss_current))
            print(discriminator_loss_current.shape)

            # Train Generator
            generator_loss_current = gan.train_on_batch([ground_truth_images, input_images], [target_real_images, ground_truth_images])

            print(generator_loss_current)
            print(type(generator_loss_current))
            print(generator_loss_current.shape)

            # Calculate Elapsed Time
            current_time = time.time()
            elapsed_time = current_time - start_time

        # Append Losses
        generator_loss_overall.append(generator_loss_current)
        discriminator_loss_overall.append(discriminator_loss_current)

        # Display Results
        print(f"Epoch: {current_epoch + 1}/{epochs}, Generator Loss: {generator_loss_current:.2f}, Discriminator Loss: {discriminator_loss_current:.2f}, Elapsed Time: {elapsed_time:.2f}")

        # Display Images
        if(current_epoch % display_interval == 0):
            display_output(output_data_type="training")
```

```python
train_neural_network(epochs=50, batch_size=32, display_interval=1)
```

---
## Results

```python
display_output(output_data_type="training")
```

```python
display_output(output_data_type="validation")
```

---
## Conclusions

**Conclusion**

Based on the outputs from different epochs in figures above, it can be observed how the translation of the images improves over time—starting with a very foggy version of the grounded truth image, the model produces a sharper and more precise image. The final version shows a much-improved image. However, the synthetic image is still a somewhat blurry version of the ground truth image.  Although the results were not as realistic as what we desired, it shows the real potential of the cGAN in producing content that can increase/diversify the existing dataset.

One of the biggest challenges that we faced while training the network was the vanishing gradients problem. Due to the complicated networks used, the discriminator got too successful multiple times, causing the generator gradient to vanish. Another challenge that we faced was the implementation of the network. GANs are known for being notoriously difficult to implement and train, and even the slightest imbalance between the parameters of the Generator and Discriminator can cause the model to fail.

Lastly, evaluating the quality of synthesized images is an open and challenging problem. Traditional metrics such as per-pixel mean-squared error do not assess the image's quality, and therefore do not measure the very structure that structured losses aim to capture. In other words, there is no standard metric similar to a confusion metric for classifiers to evaluate the performance of the network. Therefore, to holistically evaluate our results' visual quality, we considered the plausibility to a human observer as the ultimate goal.

**Future Work**

This project's original plan was to use datasets relating to the health care industry, such as MRI scans or X-ray images. Unfortunately, this was not possible because of the lack of quality data. Nevertheless, the method proposed can be considered a good starting point to further research into the potential of data augmentation. The method proposed in this report could prove particularly prove useful in solving imbalances in medical datasets, ultimately leading to better prediction models. A comparison between a classifier's performance trained on real data and artificial data can be the next logical step.

```python

```