# Homework and bake-off: word-level entailment with neural networks

```python
__author__ = "Christopher Potts"
__version__ = "CS224u, Stanford, Spring 2020"
```

## Contents

1. [Overview](#Overview)
1. [Set-up](#Set-up)
1. [Data](#Data)
  1. [Edge disjoint](#Edge-disjoint)
  1. [Word disjoint](#Word-disjoint)
1. [Baseline](#Baseline)
  1. [Representing words: vector_func](#Representing-words:-vector_func)
  1. [Combining words into inputs: vector_combo_func](#Combining-words-into-inputs:-vector_combo_func)
  1. [Classifier model](#Classifier-model)
  1. [Baseline results](#Baseline-results)
1. [Homework questions](#Homework-questions)
  1. [Hypothesis-only baseline [2 points]](#Hypothesis-only-baseline-[2-points])
  1. [Alternatives to concatenation [2 points]](#Alternatives-to-concatenation-[2-points])
  1. [A deeper network [2 points]](#A-deeper-network-[2-points])
  1. [Your original system [3 points]](#Your-original-system-[3-points])
1. [Bake-off [1 point]](#Bake-off-[1-point])

## Overview

The general problem is word-level natural language inference.

Training examples are pairs of words $(w_{L}, w_{R}), y$ with $y = 1$ if $w_{L}$ entails $w_{R}$, otherwise $0$.

The homework questions below ask you to define baseline models for this and develop your own system for entry in the bake-off, which will take place on a held-out test-set distributed at the start of the bake-off. (Thus, all the data you have available for development is available for training your final system before the bake-off begins.)

## Set-up

See [the first notebook in this unit](nli_01_task_and_data.ipynb) for set-up instructions.

```python
from collections import defaultdict
import json
import numpy as np
import os
import pandas as pd
from torch_shallow_neural_classifier import TorchShallowNeuralClassifier
import nli
import utils
```

```python
DATA_HOME = 'data'

NLIDATA_HOME = os.path.join(DATA_HOME, 'nlidata')

wordentail_filename = os.path.join(
    NLIDATA_HOME, 'nli_wordentail_bakeoff_data.json')

GLOVE_HOME = os.path.join(DATA_HOME, 'glove.6B')
```

## Data

I've processed the data into two different train/test splits, in an effort to put some pressure on our models to actually learn these semantic relations, as opposed to exploiting regularities in the sample.

* `edge_disjoint`: The `train` and `dev` __edge__ sets are disjoint, but many __words__ appear in both `train` and `dev`.
* `word_disjoint`: The `train` and `dev` __vocabularies are disjoint__, and thus the edges are disjoint as well.

These are very different problems. For `word_disjoint`, there is real pressure on the model to learn abstract relationships, as opposed to memorizing properties of individual words.

```python
with open(wordentail_filename) as f:
    wordentail_data = json.load(f)
```

The outer keys are the  splits plus a list giving the vocabulary for the entire dataset:

```python
wordentail_data.keys()
```

    dict_keys(['edge_disjoint', 'vocab', 'word_disjoint'])

### Edge disjoint

```python
wordentail_data['edge_disjoint'].keys()
```

    dict_keys(['dev', 'train'])

This is what the split looks like; all three have this same format:

```python
wordentail_data['edge_disjoint']['dev'][: 5]
```

    [[['sweater', 'stroke'], 0],
     [['constipation', 'hypovolemia'], 0],
     [['disease', 'inflammation'], 0],
     [['herring', 'animal'], 1],
     [['cauliflower', 'outlook'], 0]]

Let's test to make sure no edges are shared between `train` and `dev`:

```python
nli.get_edge_overlap_size(wordentail_data, 'edge_disjoint')
```

    0

As we expect, a *lot* of vocabulary items are shared between `train` and `dev`:

```python
nli.get_vocab_overlap_size(wordentail_data, 'edge_disjoint')
```

    2916

This is a large percentage of the entire vocab:

```python
len(wordentail_data['vocab'])
```

    8470

Here's the distribution of labels in the `train` set. It's highly imbalanced, which will pose a challenge for learning. (I'll go ahead and reveal that the `dev` set is similarly distributed.)

```python
def label_distribution(split):
    return pd.DataFrame(wordentail_data[split]['train'])[1].value_counts()
```

```python
label_distribution('edge_disjoint')
```

    0    14650
    1     2745
    Name: 1, dtype: int64

### Word disjoint

```python
wordentail_data['word_disjoint'].keys()
```

    dict_keys(['dev', 'train'])

In the `word_disjoint` split, no __words__ are shared between `train` and `dev`:

```python
nli.get_vocab_overlap_size(wordentail_data, 'word_disjoint')
```

    0

Because no words are shared between `train` and `dev`, no edges are either:

```python
nli.get_edge_overlap_size(wordentail_data, 'word_disjoint')
```

    0

The label distribution is similar to that of `edge_disjoint`, though the overall number of examples is a bit smaller:

```python
label_distribution('word_disjoint')
```

    0    7199
    1    1349
    Name: 1, dtype: int64

## Baseline

Even in deep learning, __feature representation is vital and requires care!__ For our task, feature representation has two parts: representing the individual words and combining those representations into a single network input.

### Representing words: vector_func

Let's consider two baseline word representations methods:

1. Random vectors (as returned by `utils.randvec`).
1. 50-dimensional GloVe representations.

```python
def randvec(w, n=50, lower=-1.0, upper=1.0):
    """Returns a random vector of length `n`. `w` is ignored."""
    return utils.randvec(n=n, lower=lower, upper=upper)
```

```python
# Any of the files in glove.6B will work here:

glove_dim = 50

glove_src = os.path.join(GLOVE_HOME, 'glove.6B.{}d.txt'.format(glove_dim))

# Creates a dict mapping strings (words) to GloVe vectors:
GLOVE = utils.glove2dict(glove_src)

def glove_vec(w):
    """Return `w`'s GloVe representation if available, else return
    a random vector."""
    return GLOVE.get(w, randvec(w, n=glove_dim))
```

### Combining words into inputs: vector_combo_func

Here we decide how to combine the two word vectors into a single representation. In more detail, where `u` is a vector representation of the left word and `v` is a vector representation of the right word, we need a function `vector_combo_func` such that `vector_combo_func(u, v)` returns a new input vector `z` of dimension `m`. A simple example is concatenation:

```python
def vec_concatenate(u, v):
    """Concatenate np.array instances `u` and `v` into a new np.array"""
    return np.concatenate((u, v))
```

`vector_combo_func` could instead be vector average, vector difference, etc. (even combinations of those) – there's lots of space for experimentation here; [homework question 2](#Alternatives-to-concatenation-[1-point]) below pushes you to do some exploration.

### Classifier model

For a baseline model, I chose `TorchShallowNeuralClassifier`:

```python
net = TorchShallowNeuralClassifier(hidden_dim=50, max_iter=100)
```

### Baseline results

The following puts the above pieces together, using `vector_func=glove_vec`, since `vector_func=randvec` seems so hopelessly misguided for `word_disjoint`!

```python
word_disjoint_experiment = nli.wordentail_experiment(
    train_data=wordentail_data['word_disjoint']['train'],
    assess_data=wordentail_data['word_disjoint']['dev'],
    model=net,
    vector_func=glove_vec,
    vector_combo_func=vec_concatenate)

print("macro-f1: {0}".format(word_disjoint_experiment['macro-F1']))
```

    Finished epoch 100 of 100; error is 0.02750065876170993

                  precision    recall  f1-score   support

               0      0.919     0.939     0.929      1910
               1      0.406     0.335     0.367       239

        accuracy                          0.872      2149
       macro avg      0.662     0.637     0.648      2149
    weighted avg      0.862     0.872     0.866      2149

    macro-f1: 0.6477534575895934

## Homework questions

Please embed your homework responses in this notebook, and do not delete any cells from the notebook. (You are free to add as many cells as you like as part of your responses.)

### Hypothesis-only baseline [2 points]

During our discussion of SNLI and MultiNLI, we noted that a number of research teams have shown that hypothesis-only baselines for NLI tasks can be remarkably robust. This question asks you to explore briefly how this baseline effects the 'edge_disjoint' and 'word_disjoint' versions of our task.

For this problem, submit two functions:

1. A `vector_combo_func` function called `hypothesis_only` that simply throws away the premise, using the unmodified hypothesis (second) vector as its representation of the example.

1. A function called `run_hypothesis_only_evaluation` that does the following:
    1. Loops over the two conditions 'word_disjoint' and 'edge_disjoint' and the two `vector_combo_func` values `vec_concatenate` and `hypothesis_only`, calling `nli.wordentail_experiment` to train on the conditions 'train' portion and assess on its 'dev' portion, with `glove_vec` as the `vector_func`. So that the results are consistent, use an `sklearn.linear_model.LogisticRegression` with default parameters as the model.
    1. Returns a `dict` mapping `(condition_name, function_name)` pairs to the 'macro-F1' score for that pair, as returned by the call to `nli.wordentail_experiment`. (Tip: you can get the `str` name of your function `hypothesis_only` with `hypothesis_only.__name__`.)

The test functions `test_hypothesis_only` and `test_run_hypothesis_only_evaluation` will help ensure that your functions have the desired logic.

```python
##### YOUR CODE HERE
def hypothesis_only(u, v):
    """Just return the hypothesis part"""
    return v

def run_hypothesis_only_evaluation():
    ##### YOUR CODE HERE

    from sklearn.linear_model import LogisticRegression

    eval_results = {}
    net = LogisticRegression()
    for condition_name in ['edge_disjoint', 'word_disjoint']:
        for vec_combo_func in [vec_concatenate, hypothesis_only]:
            result = nli.wordentail_experiment(
                train_data=wordentail_data[condition_name]['train'],
                assess_data=wordentail_data[condition_name]['dev'],
                model=net,
                vector_func=glove_vec,
                vector_combo_func=vec_combo_func)

            print("macro-f1: {0}".format(result['macro-F1']))
            eval_results[(condition_name, vec_combo_func.__name__)] = result['macro-F1']

    return eval_results
```

```python
def test_hypothesis_only(hypothesis_only):
    v = hypothesis_only(1, 2)
    assert v == 2
```

```python
test_hypothesis_only(hypothesis_only)
```

```python
def test_run_hypothesis_only_evaluation(run_hypothesis_only_evaluation):
    results = run_hypothesis_only_evaluation()
    assert ('word_disjoint', 'vec_concatenate') in results, \
        "The return value of `run_hypothesis_only_evaluation` does not have the intended kind of keys"
    assert isinstance(results[('word_disjoint', 'vec_concatenate')], float), \
        "The values of the `run_hypothesis_only_evaluation` result should be floats"
```

```python
test_run_hypothesis_only_evaluation(run_hypothesis_only_evaluation)
```

                  precision    recall  f1-score   support

               0      0.875     0.969     0.920      7376
               1      0.574     0.230     0.328      1321

        accuracy                          0.857      8697
       macro avg      0.725     0.600     0.624      8697
    weighted avg      0.830     0.857     0.830      8697

    macro-f1: 0.6242497026339122
                  precision    recall  f1-score   support

               0      0.872     0.975     0.920      7376
               1      0.584     0.199     0.297      1321

        accuracy                          0.857      8697
       macro avg      0.728     0.587     0.609      8697
    weighted avg      0.828     0.857     0.826      8697

    macro-f1: 0.6086585700699786
                  precision    recall  f1-score   support

               0      0.902     0.979     0.939      1910
               1      0.474     0.151     0.229       239

        accuracy                          0.887      2149
       macro avg      0.688     0.565     0.584      2149
    weighted avg      0.854     0.887     0.860      2149

    macro-f1: 0.5837810695455686
                  precision    recall  f1-score   support

               0      0.893     0.989     0.939      1910
               1      0.382     0.054     0.095       239

        accuracy                          0.885      2149
       macro avg      0.638     0.522     0.517      2149
    weighted avg      0.836     0.885     0.845      2149

    macro-f1: 0.5169358178053831

### Alternatives to concatenation [2 points]

We've so far just used vector concatenation to represent the premise and hypothesis words. This question asks you to explore two simple alternative:

1. Write a function `vec_diff` that, for a given pair of vector inputs `u` and `v`, returns the element-wise difference between `u` and `v`.

1. Write a function `vec_max` that, for a given pair of vector inputs `u` and `v`, returns the element-wise max values between `u` and `v`.

You needn't include your uses of `nli.wordentail_experiment` with these functions, but we assume you'll be curious to see how they do!

```python
def vec_diff(u, v):
    ##### YOUR CODE HERE
    return u - v

def vec_max(u, v):
    ##### YOUR CODE HERE
    return np.maximum(u, v)

```

```python
def test_vec_diff(vec_diff):
    u = np.array([10.2, 8.1])
    v = np.array([1.2, -7.1])
    result = vec_diff(u, v)
    expected = np.array([9.0, 15.2])
    assert np.array_equal(result, expected), \
        "Expected {}; got {}".format(expected, result)
```

```python
test_vec_diff(vec_diff)
```

```python
def test_vec_max(vec_max):
    u = np.array([1.2,  8.1])
    v = np.array([10.2, -7.1])
    result = vec_max(u, v)
    expected = np.array([10.2, 8.1])
    assert np.array_equal(result, expected), \
        "Expected {}; got {}".format(expected, result)
```

```python
test_vec_max(vec_max)
```

```python
if 'IS_GRADESCOPE_ENV' not in os.environ:

    net = TorchShallowNeuralClassifier(hidden_dim=50, max_iter=100)
    print(net)

    for vec_combo_func in [vec_diff, vec_max, vec_concatenate, hypothesis_only]:
        result = nli.wordentail_experiment(
            train_data=wordentail_data['word_disjoint']['train'],
            assess_data=wordentail_data['word_disjoint']['dev'],
            model=net,
            vector_func=glove_vec,
            vector_combo_func=vec_combo_func)

        print("macro-f1: {0}".format(result['macro-F1']))
```

    TorchShallowNeuralClassifier(
    	hidden_dim=50,
    	hidden_activation=Tanh(),
    	batch_size=1028,
    	max_iter=100,
    	eta=0.01,
    	optimizer=<class 'torch.optim.adam.Adam'>,
    	l2_strength=0)

    Finished epoch 100 of 100; error is 0.28842502646148205

                  precision    recall  f1-score   support

               0      0.909     0.842     0.874      1910
               1      0.205     0.326     0.252       239

        accuracy                          0.785      2149
       macro avg      0.557     0.584     0.563      2149
    weighted avg      0.831     0.785     0.805      2149

    macro-f1: 0.5630849852522789

    Finished epoch 100 of 100; error is 1.056394025683403

                  precision    recall  f1-score   support

               0      0.910     0.890     0.900      1910
               1      0.252     0.297     0.273       239

        accuracy                          0.824      2149
       macro avg      0.581     0.593     0.586      2149
    weighted avg      0.837     0.824     0.830      2149

    macro-f1: 0.586104297300003

    Finished epoch 100 of 100; error is 0.02869832795113325

                  precision    recall  f1-score   support

               0      0.927     0.924     0.925      1910
               1      0.409     0.423     0.416       239

        accuracy                          0.868      2149
       macro avg      0.668     0.673     0.671      2149
    weighted avg      0.870     0.868     0.869      2149

    macro-f1: 0.6705681430526949

    Finished epoch 100 of 100; error is 1.5673212707042694

                  precision    recall  f1-score   support

               0      0.903     0.919     0.911      1910
               1      0.248     0.213     0.229       239

        accuracy                          0.840      2149
       macro avg      0.575     0.566     0.570      2149
    weighted avg      0.830     0.840     0.835      2149

    macro-f1: 0.5700959707451074

### A deeper network [2 points]

It is very easy to subclass `TorchShallowNeuralClassifier` if all you want to do is change the network graph: all you have to do is write a new `define_graph`. If your graph has new arguments that the user might want to set, then you should also redefine `__init__` so that these values are accepted and set as attributes.

For this question, please subclass `TorchShallowNeuralClassifier` so that it defines the following graph:

$$\begin{align}
h_{1} &= xW_{1} + b_{1} \\
r_{1} &= \textbf{Bernoulli}(1 - \textbf{dropout\_prob}, n) \\
d_{1} &= r_1 * h_{1} \\
h_{2} &= f(d_{1}) \\
h_{3} &= h_{2}W_{2} + b_{2}
\end{align}$$

Here, $r_{1}$ and $d_{1}$ define a dropout layer: $r_{1}$ is a random binary vector of dimension $n$, where the probability of a value being $1$ is given by $1 - \textbf{dropout_prob}$. $r_{1}$ is multiplied element-wise by our first hidden representation, thereby zeroing out some of the values. The result is fed to the user's activation function $f$, and the result of that is fed through another linear layer to produce $h_{3}$. (Inside `TorchShallowNeuralClassifier`, $h_{3}$ is the basis for a softmax classifier, so no activation function is applied to it.)

For your implementation, please use `nn.Sequential`, `nn.Linear`, and `nn.Dropout` to define the required layers.

For comparison, using this notation, `TorchShallowNeuralClassifier` defines the following graph:

$$\begin{align}
h_{1} &= xW_{1} + b_{1} \\
h_{2} &= f(h_{1}) \\
h_{3} &= h_{2}W_{2} + b_{2}
\end{align}$$

The following code starts this sub-class for you, so that you can concentrate on `define_graph`. Be sure to make use of `self.dropout_prob`

For this problem, submit just your completed  `TorchDeepNeuralClassifier`. You needn't evaluate it, though we assume you will be keen to do that!

You can use `test_TorchDeepNeuralClassifier` to ensure that your network has the intended structure.

```python
import torch.nn as nn

class TorchDeepNeuralClassifier(TorchShallowNeuralClassifier):
    def __init__(self, dropout_prob=0.7, **kwargs):
        self.dropout_prob = dropout_prob
        super().__init__(**kwargs)

    def define_graph(self):
        """Complete this method!

        Returns
        -------
        an `nn.Module` instance, which can be a free-standing class you
        write yourself, as in `torch_rnn_classifier`, or the outpiut of
        `nn.Sequential`, as in `torch_shallow_neural_classifier`.

        """
        ##### YOUR CODE HERE
        return nn.Sequential(
            nn.Linear(self.input_dim, self.hidden_dim),
            nn.Dropout(p=self.dropout_prob),
            self.hidden_activation,
            nn.Linear(self.hidden_dim, self.n_classes_))

##### YOUR CODE HERE
if 'IS_GRADESCOPE_ENV' not in os.environ:
    net = TorchDeepNeuralClassifier()
    for vec_combo_func in [vec_diff, vec_max, vec_concatenate, hypothesis_only]:
        result = nli.wordentail_experiment(
            train_data=wordentail_data['word_disjoint']['train'],
            assess_data=wordentail_data['word_disjoint']['dev'],
            model=net,
            vector_func=glove_vec,
            vector_combo_func=vec_combo_func)

        print("macro-f1: {0}".format(result['macro-F1']))
```

    Finished epoch 100 of 100; error is 3.103418618440628

                  precision    recall  f1-score   support

               0      0.919     0.919     0.919      1910
               1      0.351     0.351     0.351       239

        accuracy                          0.856      2149
       macro avg      0.635     0.635     0.635      2149
    weighted avg      0.856     0.856     0.856      2149

    macro-f1: 0.6351563013428553

    Finished epoch 100 of 100; error is 3.0264479219913483

                  precision    recall  f1-score   support

               0      0.904     0.982     0.941      1910
               1      0.534     0.163     0.250       239

        accuracy                          0.891      2149
       macro avg      0.719     0.573     0.596      2149
    weighted avg      0.863     0.891     0.864      2149

    macro-f1: 0.5956472654290015

    Finished epoch 100 of 100; error is 2.4502106606960297

                  precision    recall  f1-score   support

               0      0.905     0.987     0.944      1910
               1      0.621     0.172     0.269       239

        accuracy                          0.896      2149
       macro avg      0.763     0.579     0.607      2149
    weighted avg      0.873     0.896     0.869      2149

    macro-f1: 0.6065023627413547

    Finished epoch 100 of 100; error is 2.7340420484542847

                  precision    recall  f1-score   support

               0      0.895     0.989     0.940      1910
               1      0.447     0.071     0.123       239

        accuracy                          0.887      2149
       macro avg      0.671     0.530     0.531      2149
    weighted avg      0.845     0.887     0.849      2149

    macro-f1: 0.5311554770666994

```python
def test_TorchDeepNeuralClassifier(TorchDeepNeuralClassifier):
    dropout_prob = 0.55
    assert hasattr(TorchDeepNeuralClassifier(), "dropout_prob"), \
        "TorchDeepNeuralClassifier must have an attribute `dropout_prob`."
    try:
        inst = TorchDeepNeuralClassifier(dropout_prob=dropout_prob)
    except TypeError:
        raise TypeError("TorchDeepNeuralClassifier must allow the user "
                        "to set `dropout_prob` on initialization")
    inst.input_dim = 10
    inst.n_classes_ = 5
    graph = inst.define_graph()
    assert len(graph) == 4, \
        "The graph should have 4 layers; yours has {}".format(len(graph))
    expected = {
        0: 'Linear',
        1: 'Dropout',
        2: 'Tanh',
        3: 'Linear'}
    for i, label in expected.items():
        name = graph[i].__class__.__name__
        assert label in name, \
            "The {} layer of the graph should be a {} layer; yours is {}".format(i, label, name)
    assert graph[1].p == dropout_prob, \
        "The user's value for `dropout_prob` should be the value of `p` for the Dropout layer."
```

```python
test_TorchDeepNeuralClassifier(TorchDeepNeuralClassifier)
```

### Your original system [3 points]

This is a simple dataset, but our focus on the 'word_disjoint' condition ensures that it's a challenging one, and there are lots of modeling strategies one might adopt.

You are free to do whatever you like. We require only that your system differ in some way from those defined in the preceding questions. They don't have to be completely different, though. For example, you might want to stick with the model but represent examples differently, or the reverse.

Keep in mind that, for the bake-off evaluation, the 'edge_disjoint' portions of the data are off limits. You can, though, train on the combination of the 'word_disjoint' 'train' and 'dev' portions. You are free to use different pretrained word vectors and the like. Please do not introduce additional entailment datasets into your training data, though.

Please embed your code in this notebook so that we can rerun it.

In the cell below, please provide a brief technical description of your original system, so that the teaching team can gain an understanding of what it does. This will help us to understand your code and analyze all the submissions to identify patterns and strategies.

```python
# Enter your system description in this cell.

# Tried out different systems:
# system_0 : Original system.
#            Uses bidirectional RNN classifier (0.68)
#            Choosing this as original system.

# system_1 : Variations of vector_combo_func (0.70)
# system_2 : Retrofit GLOVE using WordNet (0.64)
# system_3 : Data augmentation of 'entails' class using WordNet to avoid imbalance during training. (0.48)

# My peak score was: 0.68

if 'IS_GRADESCOPE_ENV' not in os.environ:
    from nltk.corpus import wordnet as wn
    from retrofitting import Retrofitter
    from torch_rnn_classifier import TorchRNNClassifier

    def get_wordnet_edges():
        edges = defaultdict(set)
        for ss in wn.all_synsets():
            lem_names = {lem.name() for lem in ss.lemmas()}
            for lem in lem_names:
                edges[lem] |= lem_names
        return edges

    wn_edges = get_wordnet_edges()

    # Idea: Bidirectional RNN classifier
    def system_0_original():

        # Data------------
        with open(wordentail_filename) as f:
            wordentail_data = json.load(f)

        print("Distribution of labels : \n{0}".format(pd.DataFrame(wordentail_data['word_disjoint']['train'])[1].value_counts()))

        # Model-----------
        X_glove = pd.DataFrame(GLOVE)
        X_glove['$UNK'] = 0
        X_glove = X_glove.T

        vocab = list(X_glove.index)
        embedding = X_glove.values
        net = TorchRNNClassifier(vocab=vocab, embedding=embedding, bidirectional=True)

        # Exp-------------
        result = nli.wordentail_experiment(
            train_data=wordentail_data['word_disjoint']['train'],
            assess_data=wordentail_data['word_disjoint']['dev'],
            model=net,
            vector_func=lambda x: np.array([x]),
            vector_combo_func=vec_concatenate)

        return result['macro-F1']

    #############################################################################
    # Idea: Variations of vector_combo_func.
    def system_1():

        # Data------------
        with open(wordentail_filename) as f:
            wordentail_data = json.load(f)

        print("Distribution of labels : \n{0}".format(pd.DataFrame(wordentail_data['word_disjoint']['train'])[1].value_counts()))

        def vec_merge(u, v):
            """Merge different feature reps including array diff, max, avg etc."""
            return np.concatenate((u, v, vec_diff(u, v), vec_max(u,v)))

        # Model-----------
        net = TorchShallowNeuralClassifier(hidden_dim=50, max_iter=100)
        print(net)

        # Exp-------------
        result = nli.wordentail_experiment(
            train_data=wordentail_data['word_disjoint']['train'],
            assess_data=wordentail_data['word_disjoint']['dev'],
            model=net,
            vector_func=glove_vec,
            vector_combo_func=vec_merge)

        return result['macro-F1']

    #######################################################################
    # Idea: Retrofit GLOVE using WordNet
    def system_2():

        # Data------------
        with open(wordentail_filename) as f:
            wordentail_data = json.load(f)

        X_glove = pd.DataFrame(GLOVE).T
        print(X_glove.shape)

        def convert_edges_to_indices(edges, Q):
            lookup = dict(zip(Q.index, range(Q.shape[0])))
            index_edges = defaultdict(set)
            for start, finish_nodes in edges.items():
                s = lookup.get(start)
                if s:
                    f = {lookup[n] for n in finish_nodes if n in lookup}
                    if f:
                        index_edges[s] = f
            return index_edges

        wn_index_edges = convert_edges_to_indices(wn_edges, X_glove)
        wn_retro = Retrofitter(verbose=True)
        X_retro = wn_retro.fit(X_glove, wn_index_edges)
        print(X_retro.shape)

        def retro_vec(w):
            """Return `w`'s Retrofitted representation if available, else return
            a random vector."""
            return X_retro.loc[w].values if w in X_retro.index else randvec(w, n=glove_dim)

        # Model-----------
        net = TorchShallowNeuralClassifier(hidden_dim=50, max_iter=100)
        print(net)

        # Exp-------------
        result = nli.wordentail_experiment(
            train_data=wordentail_data['word_disjoint']['train'],
            assess_data=wordentail_data['word_disjoint']['dev'],
            model=net,
            vector_func=retro_vec,
            vector_combo_func=vec_concatenate)

        return result['macro-F1']

    ################################################################
    # Idea: Data augmentation of 'entails' class using wordnet
    def system_3():

        # Data------------
        with open(wordentail_filename) as f:
            wordentail_data = json.load(f)

        x_train = wordentail_data['word_disjoint']['train']
        print("Existing distribution of labels : \n{0}".format(pd.DataFrame(x_train)[1].value_counts()))

        # get wordnet edges
        def get_wordnet_edges():
            edges = defaultdict(set)
            for ss in wn.all_synsets():
                lem_names = {lem.name() for lem in ss.lemmas()}
                for lem in lem_names:
                    edges[lem] |= lem_names
            return edges

        wn_edges = get_wordnet_edges()

        # data augmentation of positive entailments.
        positive_entailments = []
        for premise_hypothesis, label in x_train:
            if label == 1:
                positive_entailments.append(premise_hypothesis)

        print("Current count of positives: {0}".format(len(positive_entailments)))

        positive_entailments_ex = []
        for premise_hypothesis in positive_entailments:
            premise = premise_hypothesis[0]
            hypothesis = premise_hypothesis[1]

            for wn_premise in wn_edges[premise]:
                if premise == wn_premise:
                    continue
                for wn_hypothesis in wn_edges[hypothesis]:
                    if wn_hypothesis == hypothesis:
                        continue

                    positive_entailments_ex.append([wn_premise, wn_hypothesis])

        print("New count of positives to add: {0}".format(len(positive_entailments_ex)))
        x_train.extend([[item, 1] for item in positive_entailments_ex])

        print("New distribution of labels : \n{0}".format(pd.DataFrame(wordentail_data['word_disjoint']['train'])[1].value_counts()))

        # Model-----------
        net = TorchShallowNeuralClassifier(hidden_dim=50, max_iter=100)

        # Exp-------------
        result = nli.wordentail_experiment(
            train_data=wordentail_data['word_disjoint']['train'],
            assess_data=wordentail_data['word_disjoint']['dev'],
            model=net,
            vector_func=glove_vec,
            vector_combo_func=vec_concatenate)

        return result['macro-F1']

    ###################################################################
    print("System 0 (Original) Score:{0}".format(system_0_original()))
    print("="*100)

#     print("System 1 Score:{0}".format(system_1()))
#     print("="*100)

#     print("System 2 Score:{0}".format(system_2()))
#     print("="*100)

#     print("System 3 Score:{0}".format(system_3()))
#     print("="*100)
    ####################################################################

# Please do not remove this comment.

```

    Distribution of labels :
    0    7199
    1    1349
    Name: 1, dtype: int64

    Finished epoch 100 of 100; error is 0.02302360360044986

                  precision    recall  f1-score   support

               0      0.932     0.910     0.921      1910
               1      0.396     0.469     0.429       239

        accuracy                          0.861      2149
       macro avg      0.664     0.690     0.675      2149
    weighted avg      0.872     0.861     0.866      2149

    System 0 (Original) Score:0.6750996412104682
    ====================================================================================================

## Bake-off [1 point]

The goal of the bake-off is to achieve the highest macro-average F1 score on __word_disjoint__, on a test set that we will make available at the start of the bake-off. The announcement will go out on the discussion forum. To enter, you'll be asked to run `nli.bake_off_evaluation` on the output of your chosen `nli.wordentail_experiment` run.

The cells below this one constitute your bake-off entry.

The rules described in the [Your original system](#Your-original-system-[3-points]) homework question are also in effect for the bake-off.

Systems that enter will receive the additional homework point, and systems that achieve the top score will receive an additional 0.5 points. We will test the top-performing systems ourselves, and only systems for which we can reproduce the reported results will win the extra 0.5 points.

Late entries will be accepted, but they cannot earn the extra 0.5 points. Similarly, you cannot win the bake-off unless your homework is submitted on time.

The announcement will include the details on where to submit your entry.

```python
# Enter your bake-off assessment code into this cell.
# Please do not remove this comment.
##### YOUR CODE HERE

```

```python
# On an otherwise blank line in this cell, please enter
# your macro-avg f1 value as reported by the code above.
# Please enter only a number between 0 and 1 inclusive.
# Please do not remove this comment.

##### YOUR CODE HERE

```


 _Por: V. Robles B._

# Descenso por Gradiente: una breve introducción - Parte 2

En este cuaderno se presenta un ejemplo sobre cómo aplicar el método de descenso por gradiente en el caso de que tengamos una función con dos incógnitas, es decir, cuando se $f$ depende de dos variables: $f(x,y)$.

Recuerde que en diversos textos también se representa esta función de la siguiente forma: $f(x_{1}, x_{2})$.

La forma de realizar el cálculo es la misma, sin embargo, ahora en lugar de tener una línea, tendremos un plano por el cual nos moveremos en búsqueda de minimizar la función dada.

## Ejemplo de aplicación:

En el siguiente ejemplo vamos a ver cómo aplicar el algoritmo de descenso por gradiente para minimizar esta función que tiene dos incógnitas, asumiendo que el punto de partida es $x^{(0)}=(2,3)$:

$f(x,y)=x^{3}+x y+y^{3}+3.3$

Para ello, procedemos a realizar una gráfica de la función a través de la librería de utilidades de **matplotlib** (_mpl_toolkits.mplot3d_) a fin de tener una idea más clara de cómo se representa dicha función:

```python
import matplotlib.pyplot as pp
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter
import numpy as np

%matplotlib inline

def fxy(x,y):
    return (np.power(x,3.)+np.dot(x,y)+np.power(y,3.)+3.3)

fig=pp.figure()
#fig.set_size_inches(17.,10.)
axes=fig.gca(projection='3d')

x=np.arange(-3.,3.,0.23)
y=np.arange(-3.,3.,0.23)

x, y = np.meshgrid(x,y) # creamos la malla
z=fxy(x,y)

# Dibujamos la superficie
surface=axes.plot_surface(x,y,z,cmap=cm.coolwarm,linewidth=0,antialiased=False)

axes.set_zlim(43.,123.)
axes.zaxis.set_major_locator(LinearLocator(10))
axes.zaxis.set_major_formatter(FormatStrFormatter('%.02f'))

fig.colorbar(surface, shrink=0.5, aspect=5)
pp.show()

#print(x,"\n",y,"\n")
#print(z)

#pp.plot(x,y)
#pp.grid(True)
#pp.show()
```

### Resolución usando el método de Descenso por Gradiente [3]:

Dado que no conocemos el valor óptimo de $x$, hacemos $k\leftarrow 0$, como nos indica el algoritmo. A continuación procedemos a calcular los siguientes elementos:

* La derivada **parcial** de la función original $f(x,y)=x^{3}+x y+y^{3}+3.3$.
* Evaluamos la función original en el punto $x^{(0)}=(2,3)$.

```python
from sympy import Function, Symbol, diff, solve

x=Symbol('x')
y=Symbol('y')
f=Function('f')(x)

fx=x**3+x*y+y**3+3.3

fpx=diff(fx,x)
fpy=diff(fx,y)
print("[",fpx,",",fpy,"]")

print("Valor en el punto (x^(0),y^(0)) => (2,3) = ",fx.subs(x,2.).subs(y,3.).evalf())
```

    [ 3*x**2 + y , x + 3*y**2 ]
    Valor en el punto (x^(0),y^(0)) => (2,3) =  44.3000000000000

Lo que nos da como resultado los siguientes valores:

$\nabla f(x,y) =  \binom{\frac{\partial f(x,y)}{\partial x}}{\frac{\partial f(x,y)}{\partial y}}= \binom{3x^{2}+y}{x+3y^{2}} $

$\nabla f\left((x^{(0)},y^{(0)}) = (2,3)\right) = 44.3$

1. Ahora buscaremos encontrar el siguiente punto con coordenadas $\left(x^{(k+1)}, y^{(k+1)}\right)$, para ello debemos calcular:

$\left(x^{(1)},y^{(1)}\right)=(x^{(0)},y^{(0)})-t^{*}_{0} \nabla f(x^{(0)},y^{(0)})$

2. Para encontrar el valor de $t^{*}_{0}$, debemos hallar el mínimo de la función $\theta(t) = f(x^{(0)},y^{(0)})-t \nabla f(x^{(0),y^{(0)}}$. Para realizar este paso, buscamos en el punto estacionario trabajando con la derivada de la función a minimizar. En este punto, usaremos una notación vectorial:

  - Para esto, calculamos el valor de $t^{*}_{k}$ empleando el punto $x^{(0)}$, y la función de $t$ y en base a la derivada de la función a minimizar:

$\theta^{'}(t)=-\nabla f\left( (x^{(0)},y^{(0)} )- t \nabla f(x^{(0)},y^{(0)}) \right) \nabla f(x^{(0)},y^{(0)})$

$=-\nabla f\left( (2,3)- t \nabla f(2,3)  \right) \nabla f(x^{(0)},y^{(0)})$

   Para hallar $\nabla f(x^{(0)},y^{(0)})$, simplemente sustituimos el punto $(2,3)$ en ambas partes de la función original que se derivó con respecto a $x$ y con respecto a $y$:

$\nabla f(x^{(0)},y^{(0)}) = \binom{3x^{2}+y}{x+3y^{2}}$

$\nabla f(2,3) = \binom{3\cdot 2^{2}+3}{2+3\cdot 3^{2}}$

$\nabla f(2,3) = (15, 29)$

   Con ello, ahora volvemos a la función $\theta^{'}(t)$ y reemplazamos el valor calculado:

$=-\nabla f\left( (2,3)- t (15,29)  \right) (15,29)$

$=-\nabla f\left( (2-15t),(3-29t) \right) (15,29)$

   Ahora, evaluamos la función que derivamos con los nuevos valores $x=(2-15t), y=(3-29t)$:

$=-\left( 3\cdot (2-15t)^{2}+(3-29t), (2-15t)+3\cdot (3-29t)^{2} \right) (15,29)$

$=-\lbrace{ 15\left( 3\cdot (2-15t)^{2}+(3-29t)\right)+29\left( (2-15t)+3\cdot (3-29t)^{2} \right) \rbrace}$

   Factoramos con ayuda de **sympy** y buscamos las raíces:

```python
t=Symbol('t')
ft=Function('ft')(t)

fnabla=-(15.*(3.*(2.-15.*t)**2+(3.-29.*t))+29.*((2.-15.*t)+3*(3.-29.*t)**2))

print(fnabla.expand())

fnablap=fnabla.diff(t)

print("Derivada para buscar raices: ",fnablap)
print("Raices: [",solve(fnablap),"]")
```

    -83292.0*t**2 + 18708.0*t - 1066.0
    Derivada para buscar raices:  -166584.0*t + 18708.0
    Raices: [ [0.112303702636508] ]

   Con ello, nuestra ecuación en función de $t$ queda como sigue:

$=-83292t^{2} + 18708t - 1066$

   Y la raíz que encontramos luego de derivarla (ya que nos salen valores imaginarios) es $t_{0}=0.1123$.

3. Dado lo anterior, el siguiente punto $(x^{(1)},y^{(1)})$ será:

$(x^{(1)},y^{(1)})=(x^{(0)},y^{(0)})-t^{*}_{0} \nabla f(x^{(0)},y^{(0)})$

$(x^{(1)},y^{(1)})=(2,3)-0.1123\cdot (15, 29)$

$(x^{(1)},y^{(1)})=(0.3155, -0.25669)$

Si ahora evaluamos la función original con el nuevo punto $(x^{(1)},y^{(1)})=(0.3155, -0.25669)$, obtenemos obtendremos lo siguiente:

```python
print("Valor en el punto (x^(1),y^(1)) => (0.3155,-0.25669) = ",fx.subs(x,0.3155).subs(y,-0.25669).evalf())
```

    Valor en el punto (x^(1),y^(1)) => (0.3155,-0.25669) =  3.23350601238169

 Como se aprecia, tenemos un valor menor, es decir, se ha minimizado la función: $f(x,y)=x^{3}+x y+y^{3}+3.3$.
 Deberemos seguir repitiendo los pasos, hasta que el error obtenido sea menor al planteado en $\epsilon$.

## Boletín de prácticas 2
### Objetivos:
* Familiarizarse con los principales aspectos y etapas del método de descenso por gradiente para funciones que dependen de dos variables $f(x,y)$.
* Conocer cómo aplicar el proceso de descenso por gradiente de forma automatizada con soporte de **sympy** y **Jupyter Notebook*.

### Enunciado:

1. Seleccionar una función matemática $f(x,y)$ para realizar el proceso de minimización. Ejecutar al menos **3 pasos** del método de descenso por gradiente.
2. Diseñar y desarrollar un cuaderno en Jupyter Notebook donde se realicen todos los pasos correspondientes al proceso de minimización de la función a través del método de descenso por gradiente.
3. El cuaderno deberá incluir los siguientes puntos:
    1. Gráfica de la función y los puntos que se obtienen a medida que se ejecutan los pasos de cálculo (hasta $k=3$).
    2. Aplicación de las funciones de derivación y evaluación de forma similar a la que se ha detallado en el presente cuaderno.
    3. Incluir un acápite sobre las funciones cóncavas y los puntos estacionarios (incluir gráficos).
    4. Emplear las funcionalidades que proveen los paquetes **matplotlib** y **sympy**.

### Criterios de evaluación:
Los criterios o rúbrica de evaluación del Boletín de Prácticas 2 son los siguientes:
1. Adecuada complejidad de la función seleccionada para el proceso de minimización (procurar no usar funciones cóncavas).
2. Mejorar la gráfica en 3D presentada en este cuaderno.
3. Correcta explicación y detalle de cada paso ejecutado con el método de descenso por gradiente.
4. Verificación de la solución encontrada.
5. Elementos extra: incluye recta (flecha) con la dirección del gradiente, animaciones, etc.
6. Ejecutar los pasos indicados en el segundo video-tutorial.

### Prerrequisitos:
A fin de poder realizar esta práctica, deberá contar con los siguientes prerrequisitos:
1. Haber leído de forma completa el presente cuaderno.
2. Tener instalados los siguientes paquetes en su computador:
    1. Python 2.7+ (de preferencia 3.6+)
    2. [Sympy](http://www.sympy.org)
    3. [matplotlib](https://matplotlib.org/)

### Detalles de la entrega:
El cuaderno de Jupyter Notebook deberá ser cargado en el AVAC con todos los recursos necesarios (imágenes, enlaces, etc.) en un archivo comprimido, dentro de la tarea **Boletín de Prácticas 2: Descenso por gradiente f(x,y)**. Los detalles de la fecha y hora de entrega se encuentran especificados en el AVAC.

### Algunos enlaces de utilidad:
A continuación se presentan algunos enlaces que pueden resultar de utilidad:
* Wiki con algunas funciones básicas de la librería [sympy](https://github.com/sympy/sympy/wiki/Quick-examples)
* Un tutorial interactivo con ejemplos de técnicas de [optimización](http://www.benfrederickson.com/numerical-optimization/).

### Vídeos complementarios:
A continuación se presenta un listado de vídeos de interés para reforzar los conceptos vistos en clase:
* Vídeo tutorial introductorio al método de descenso por gradiente

 [](https://youtu.be/A6FiCDoz8_4)

* Vídeo tutorial - Práctico con Jupytper Notebook sobre descenso por gradiente

 [](https://www.youtube.com/watch?v=-_A_AAxqzCg)

## Referencias
1. [Nedrich Matt, An Introduction to Gradient Descent and Linear Regression, Atomic Object, 2014.](https://spin.atomicobject.com/2014/06/24/gradient-descent-linear-regression/)
2. [Renjie Liao, CSC 411 Tutorial: Optimization for Machine Learning, Torondo University, 2016.](https://www.cs.toronto.edu/~urtasun/courses/CSC411_Fall16/tutorial2.pdf)
3. [Singer Yaron, Advanced Optimization: Lecture 9, Harvard University, 2016](https://people.seas.harvard.edu/~yaron/AM221-S16/lecture_notes/AM221_lecture9.pdf)


# Задание №1

## Теория

Этот ноутбук содержит вспомогательный материал для выполнения первого задания, в котором надо предсказать целевую переменную по двум признакам с помощью линейной регрессии. Хочу сразу предупредить, что здесь собрана вводная информация, причем в некоторых аспектах я намеренно привел упрощенное или не до конца полное описание. В дальнейшем мы отдельно размерем все эти аспекты.

Для начала, вспомним, что такое линейная регрессия и как выглядит алгоритм подбора оптимальных параметров предсказания.

Назовем множество имеющихся примеров **обучающей выборкой** или **тренировочными данными**. Обучающая выборка — это множество пар $D = \{(\vec{x^{(i)}},y^{(i)})\}_{i=1,2,...,n}$, где $n$ - размер выборки. В каждой паре через $y \in R$ обозначена целевая переменная (или ответ), который соответствует вектору признаков $\vec{x} \in R^{m\times1}$. Необходимо научиться предсказывать значение $y$ по $\vec{x}$. Обозначим предсказанное значение через $\hat{y}$. В линейной регрессии предсказание делается с помощью следующей формулы:

\begin{equation}
\hat{y} = h(\vec{x},\vec{\theta}) = \vec{x}^T\vec{\theta} = \langle \vec{x}, \vec{\theta} \rangle = \sum_{j=1}^{m} x_j\theta_j,
\end{equation}

где через $h(\vec{x},\vec{\theta})$ обозначена функция предсказания, а $\vec{\theta} \in R^{m\times1}$ — вектор параметров линейной регрессии. **Обучение** модели на основе линейной регрессии заключается в подборе $\vec{\theta}$ так, чтобы предсказанные значения $\hat{y}^{(i)}$ были как можно ближе к целевым переменным $y^{(i)}$ из обучающей выборки. Термин "ближе" значит, что есть способ измерения похожести $\hat{y}^{(i)}$ и $y^{(i)}$. В рассматриваемой задаче в качестве критерия похожести можно воспользоваться квадратичным отклонением:

\begin{equation}
e^{(i)} = \left( y^{(i)} - \hat{y}^{(i)} \right)^2.
\end{equation}

Значение $e^{(i)}$, или ошибка предсказания, показывает насколько сильно предсказанное значение целевой переменной отличается от требуемого значения для $i$-го примера обучающей выборки. Средняя ошибка предсказания, или **эмпирический риск**, на обучающей выборке определяется как

\begin{equation}
E(D, \vec{\theta}) = \frac{1}{n} \sum_{i=1}^{n} e^{(i)} = \frac{1}{n} \sum_{i=1}^{n} \left( y^{(i)} - h(\vec{x^{(i)}},\vec{\theta}) \right)^2 = \frac{1}{n}\sum_{i=1}^{n} \left(y^{(i)} - \sum_{j=1}^{m} x^{(i)}_j\theta_j \right)^2.
\end{equation}

Таким образом, задача обучения формулируется как "найти такой вектор параметров линейной регрессии, при котором минимальна средняя ошибка предсказания". Для поиска оптимального $\vec{\theta}$ можно воспользоваться методом градиентного спуска.

Рассмотрим идею этого метода. "Градиент" функции в точке показывает направление роста функции в этой точке. Фактически, градиент -- это обобщение понятия производной на многомерный случай. Для рассмотренной нами функции эмпирического риска:

\begin{equation}
\nabla E(D, \vec{\theta}) \big\vert_{\vec{\theta}=\vec{t}} =
\left(
\begin{aligned}
& \frac{\partial E(D, \vec{\theta})}{\partial \theta_0} \big\vert_{\vec{\theta}=\vec{t}} \\
& \frac{\partial E(D, \vec{\theta})}{\partial \theta_1} \big\vert_{\vec{\theta}=\vec{t}} \\
& ... \\
& \frac{\partial E(D, \vec{\theta})}{\partial \theta_m} \big\vert_{\vec{\theta}=\vec{t}} \\
\end{aligned}
\right).
\end{equation}

Нас интересует минимизация эмпирического риска. Это значит, что можно выбрать некоторое, например случайное, значение вектора $\vec{\theta}$, посчитать в нем значение градиента, а потом изменить $\vec{\theta}$ так, чтобы значение фунцкии средней ошибки уменьшилось, т.е. изменить $\vec{\theta}$ в направлении, обратном направлению градиента. Далее повторить эту процедуру для измененного $\vec{\theta}$. Если функция эмпирического риска является выпуклой и унимодальной (имеет один оптимум), то подобный итеративный пересчет через достаточное число итераций (спусков) приведет к значению параметров $\vec{\theta}$, в которых достигается минимум средней ошибки предсказания.

Опишем теперь алгоритм поиска оптимальных параметров линейной регрессии с помощью градиентного спуска.

**Инициализация:**
- задать $\alpha$ - скорость градиентного спуска;
- задать начальные значения переменных: $\vec{\theta}^{(0)}$ = random, k = 0.

**Повторять, пока не выполнятся условия остановки**

\begin{equation}
\vec{\theta}^{(k+1)} = \vec{\theta}^{(k)} - \alpha E(D, \vec{\theta}) \big\vert_{\vec{\theta}=\vec{\theta^{(k)}}} \\
k = k + 1
\end{equation}

Условия остановки:
- достигнуто максимальное число итераций градиентного спуска;
- значения параметров $\theta$ почти не меняются от итерации к итерации;
- значение градиента близко у нулю.

## Практика

Перейдем теперь к практической части, где нужно реализовать линейную регрессию. Далее приведен "скелет" программы, в которую Вам необходимо встроить недостающие фрагменты. Места для Вашег кода помечены "TODO".

Программа состоит из следующих блоков.

1. Подключение всех необходимых библиотек.
2. Чтение и вывод тренировочных данных.
3. Прототипы функций, которые Вам надо заполнить.
4. Цикл обучения.

### Подключение библиотек.

В python есть много готовых библиотек для работы с данными. Наши тренировочные данные хранятся в csv файлах и для их парсинга и отображения можно воспользоваться удобной библиотекой pandas.

Удобнее всего работать с данными, представленными в формате матриц и векторов. Для работы с ними в python есть библиотека numpy.

Отображение графиков можно сделать с помощью matplotlib.

Удобный вывод прогресса - tqdm.

```python
# Step 1. Import all the necessary packages
import sys
import tqdm
import pandas as pd

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm

import numpy as np
```

### Чтение и вывод тренировочных данных.

Для чтения данных воспользуемся методом read_csv из pandas. Эта функция возвращает специальную структуру данных DataFrame, которая содержит много методов для вывода данных и расчета различных статистик по ним. Так как данные разбиты на 2 файла, перед выводом на экран первых десяти сэмплов, сконкатенируем их в один DataFrame.

Затем преобразуем данные из DataFrame в матрицы numpy.

```python
# Step 2. Parse and visualize data
# parse train data: read CSV files with train features (train_x) and train targets (train_y)
x_train = pd.read_csv("train_x.csv", header=None)
y_train = pd.read_csv("train_y.csv", header=None)

# show first 10 samples
pd.concat([x_train, y_train], axis=1).head(10)
```

      0
      1
      0

      0
      23.98
      6.459
      11.8

      1
      21.52
      6.193
      11.0

      2
      7.74
      6.750
      23.7

      3
      4.81
      7.249
      35.4

      4
      18.06
      5.454
      15.2

      5
      5.90
      6.487
      24.4

      6
      2.94
      6.998
      33.4

      7
      6.36
      7.163
      31.6

      8
      17.44
      6.749
      13.4

      9
      4.56
      6.975
      34.9

```python
# convert pandas dataframe to numpy arrays and matrices and diplay the dimensions of train dataset
x_train = x_train.to_numpy()
y_train = y_train.to_numpy()

print("Shape of train features:", x_train.shape)
print("Shape of train targets:", y_train.shape)
```

    Shape of train features: (354, 2)
    Shape of train targets: (354, 1)

### Прототипы функций.

* predict_fn - предсказание с помощью линейной регрессии;
* loss_fn - расчет среднего значения ошибки предсказания;
* gradient_fn - расчет градиента в точке.

Вам необходимо реализовать predict_fn и gradient_fn.

```python
# Step 3. Prototypes.

# In this demo we will use linear regression to predict targets from features.
# In linear regression model with parameters thetas
# the prediction y is calculated from features x using linear combination of x and thetas.
# For example, for the case of 2 features:
# y = theta_0 * x_o + theta_1 * x_1

# Let's define some helper functions

def predict_fn(x, thetas):
    '''
    Predict target from features x using parameters thetas and linear regression

    param x: input features, shape NxM, N - number of samples to predict, M - number of features
    param thetas: vector of linear regression parameters, shape Mx1
    return y_hat: predicted scalar value for each input samples, shape Nx1
    '''
    # TODO: calculate y_hat using linear regression
    y_hat = np.zeros((x.shape[0], 1))
    return y_hat

def loss_fn(x_train, y_train, thetas):
    '''
    Calculate average loss value for train dataset (x_train, y_train).

    param x_train: input features, shape NxM, N - number of samples to predict, M - number of features
    param y_train: input tagrets, shape Nx1
    param thetas: vector of linear regression parameters, shape Mx1
    return loss: predicted scalar value for each input samples, shape Mx1
    '''
    y_predicted = predict_fn(x_train, thetas)
    loss = np.mean(np.power(y_train - y_predicted, 2))
    return loss

def gradient_fn(x_train, y_train, thetas):
    '''
    Calculate gradient value for linear regression.

    param x_train: input features, shape NxM, N - number of samples to predict, M - number of features
    param y_train: input tagrets, shape Nx1
    param thetas: vector of linear regression parameters, shape Mx1
    return g: predicted scalar value for each input samples, shape Mx1
    '''
    # TODO: calculate vector gradient
    g = np.zeros_like(thetas)
    return g
```

### Цикл обучения

Ниже дана упрощенная реализация градиентного спуска для поиска оптимальных параметров линейной регрессии. Вам необходимо подобрать значения скорости (alpha), максимальное число итераций (MAX_ITER) и условия остановки.

```python
# Step 4. Gradient descent.

# now let's find optimal parameters using gradient descent
MAX_ITER = 100000
thetas = np.random.randn(2, 1)
alpha = 1e-3

progress = tqdm.tqdm(range(MAX_ITER), "Training", file=sys.stdout)
loss_val = loss_fn(x_train, y_train, thetas)
progress.set_postfix(loss_val=loss_val)

for iter in progress:
    gradient = gradient_fn(x_train, y_train, thetas)
    thetas = thetas - alpha*gradient

    # TODO: add stop conditions
    # if stop_condition is True:
    #     progress.close()
    #     loss_val = loss_fn(x_train, y_train, thetas)
    #     print("Stop condition detected")
    #     print("Final loss:", loss_val)
    #     break

    if iter % 100 == 0:
        loss_val = loss_fn(x_train, y_train, thetas)
        progress.set_postfix(loss_val=f"{loss_val:8.4f}", thetas=f"{thetas[0][0]:5.4f} {thetas[1][0]:5.4f}")

progress.close()
```

    Training: 100%|██████████| 100000/100000 [00:01<00:00, 68118.77it/s, loss_val=588.6201, thetas=-1.5970 1.5719]

Выведем несколько предсказаний для примера.

```python
for i in range(10):
    y_hat = predict_fn(x_train[i], thetas)
    print("Target: ", y_train[i][0], ", predicted:", y_hat[0][0])
```

    Target:  11.8 , predicted: 0.0
    Target:  11.0 , predicted: 0.0
    Target:  23.7 , predicted: 0.0
    Target:  35.4 , predicted: 0.0
    Target:  15.2 , predicted: 0.0
    Target:  24.4 , predicted: 0.0
    Target:  33.4 , predicted: 0.0
    Target:  31.6 , predicted: 0.0
    Target:  13.4 , predicted: 0.0
    Target:  34.9 , predicted: 0.0

Скорее всего, в результате получится довольно большое значение ошибки и предсказания будут не очень точными. Как поднять точность, мы рассмотрим далее :)


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 3 Derivatives

Chapter 3 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

By the end of this session, you should be able to:
- Remember basic derivative rules for polynomials, adding, multiplying, dividing...
- Remember the chain rule
- Understand the basics of partial derivatives

### Basic Rules

Before we get into any rules for derivatives, let's remember what the [derivative](https://en.wikipedia.org/wiki/Derivative) means **conceptually**.

Consider a generic function and the slope between two points:
(Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

That's fairly easy to describe:

$$ slope = \frac{\Delta f}{\Delta x} = \frac{f(x+\Delta x) - f(x)}{\Delta x} $$

The question that [Newton and Leibnitz](https://en.wikipedia.org/wiki/Calculus) used when establishing calculus - what happens as $\Delta x \rightarrow 0$?

Naturally, we can evaluate that ourselves using Python. Let's evaluate the derivative of $e^x$ near $x = 1$

```python
import math

# evaluate the derivative of exp(x = 1.0)
#  by finite difference
x = 1.0
for step in [ 2, 4, 8, 10, 25, 50, 100, 250]:
    delta_x = 1 / step
    delta_f = math.exp(x + delta_x) - math.exp(x)
    print(round(delta_x, 3), round(delta_f / delta_x, 3))

print('limit', math.e)
```

So as we decrease the $\Delta x$ size, we slowly appproach a limit. Feel free to change the step sizes further (e.g., 0.01, 0.001, etc.) to convince yourself it will eventually reach

$$\frac{d}{dx}[e^x] = e^x$$

One key point is that the derivative represents the instantaneous slope at a particular point - a tangent to the curve. Secondly, even if we do not *know* the formula for a derivative of a particular function, it's possible to use a numerical derivative using *finite differences* and a small $\Delta x$ to approximate one.

In our example above, we've used the "forward" method (i.e., $x + \Delta x$ to approximate the derivative. There are three general methods for numeric approximate derivatives:

***Forward Difference***

a.k.a "take a step forward in $\Delta x$

$$\frac{df}{dx} \approx \frac{f(x+\Delta x) - f(x)}{\Delta x} $$

***Backwards Difference***

a.k.a. "take a step backwards in $\Delta x$

$$\frac{df}{dx} \approx \frac{f(x) - f(x-\Delta x)}{\Delta x} $$

***Central Difference***

Use *both* a step forward and a step backwards:

$$\frac{df}{dx} \approx \frac{f(x+\Delta x) - f(x-\Delta x)}{2\Delta x} $$

Let's test these quickly with Python:

```python
x = 1.0
delta_x = 0.05

# forward difference for derivative of e^x
delta_f = math.exp(x + delta_x) - math.exp(x)
print('forward difference:', delta_f / delta_x)

# backwards difference
delta_f = math.exp(x) - math.exp(x - delta_x)
print('backwards difference:', delta_f / delta_x)

# central difference
delta_f = math.exp(x + delta_x) - math.exp(x - delta_x)
print('central difference:', delta_f / (2.0*delta_x)) # remember the 2*delta_x

print('exact:', math.exp(x))
```

Often the central difference method is a more accurate approximation if the function does not oscillate significantly in the range considered. The error in the approximation decreases with $\Delta x^2$ for central difference, vs. $\Delta x$ for forward and backwards finite difference methods.

### Notation

Calculus was derived independently by both Newton and Leibnitz - who used different notation. As a result, there are a few ways to indicate the "take a derivative" operation.

- $\frac{df}{dx}$ - Leibnitz
- $\dot f$ - Newton (a.k.a. 'f-dot')
- $f'$ - Lagrange (a.k.a. 'f-prime')
- $D_x f(x)$ - Euler

Similar notation exists for second derivatives, third derivatives, etc.:

- $\frac{d^2f}{dx^2}$ - Leibnitz
- $\ddot{f}, \dddot{f}$ - Newton
- $f''$ - Lagrange

Depending on needs, I tend to mix Leibnitz and Lagrange notation, but you may see the "dot" notation elsewhere.

### Taking derivatives

You should be able to do any of the following:

- Constants

$$\frac{d}{dx} c = 0$$

- Linear functions

$$\frac{d}{dx} ax + b = a$$

- Polynomials

$$\frac{d}{dx} x^3 + x^2 = 3x^2 + 2x$$

- Sum / Difference

$$\frac{d}{dx} [f(x) + g(x)] = \frac{d}{dx}[f(x)] + \frac{d}{dx}[g(x)]$$

- Sine

$$\frac{d}{dx} \sin ax = a \cos ax$$

- Cosine

$$\frac{d}{dx} \cos ax = -a \sin ax$$

- Exponential

$$\frac{d}{dx} e^x = e^x$$

- Log

$$\frac{d}{dx} \log x = \frac{1}{x} $$

- Powers ax

$$\frac{d}{dx} a^x = a^x \ln a$$

- Chain Rule

$$\frac{d}{dx} f(g(x)) = \frac{df}{dg}\frac{dg}{dx}$$

>For example:

$$\frac{d}{dx} e^{ax^2} = 2axe^{ax^2}$$

- Product Rule

$$
\frac{d}{d x}[f(x) g(x)]=f(x) \frac{d g}{d x}+g(x) \frac{d f}{d x}
$$

- Quotient Rule

$$
d\left(\frac{u}{v}\right)=\frac{v d u-u d v}{v^{2}}
$$

>For example:

$$
f(x)=\frac{\sin x}{\cos x} \quad \begin{array}{ll}u=\sin x & d u=\cos x \\ v=\cos x & d v=-\sin x\end{array}
$$

$$\frac{df}{dx} = \frac{\cos^2 x + \sin^2 x}{\cos^2 x} = \frac{1}{\cos^2 x} = \sec^2 x
$$

Note that the "quotient rule" can also be considered as a combination of the product rule and chain rule that are easier to remember:

$$
\begin{aligned} \frac{d}{d x}\left[\frac{f(x)}{g(x)}\right] &=\frac{1}{g(x)} \frac{d f}{d x}+f(x) \frac{d}{d x}\left[\frac{1}{g(x)}\right] \\ &=\frac{1}{g^{2}(x)} g(x) \frac{d f}{d x}+f(x)\left[-\frac{1}{g^{2}(x)} \frac{d g}{d x}\right] \\ &=\frac{1}{g^{2}(x)}\left[g(x) \frac{d f}{d x}-f(x) \frac{d g}{d x}\right] \end{aligned}
$$

```python
# We're using a new module, sympy to help us with symbolic math
import sys
!{sys.executable} -m pip install sympy --upgrade --user

# the init_session function creates some variables but also
# tweaks the output to display nice formulas
from sympy import init_session
init_session()
```

```python
# we can take derivatives using the "diff" function
diff(cos(2*x**2), x)
```

```python
# chain rule
diff(exp(x**2), x)
```

```python
# quotient rule
diff(sin(x)/cos(x), x)
```

```python
diff(x**x, x)
```

### Second derivatives, etc.

You should remember that second derivatives are just repeated application of the rules:

$$\frac{d^2}{dx^2}[x^2] = \frac{d}{dx}[2x] = 2$$

```python
# we can use Sympy to take second derivatives too - just add the order after the variable
diff(x**2, x, 2)
```

```python
# this next line would be the third-order derivative
diff(x**2, x, 3)
```

### Partial Derivatives

Remember that a first derivative is simply the limit of an infinitesimal change in $x$. If we have functions of more than one variable, we can define a partial derivative similarly:

$$
\left(\frac{\partial f}{\partial x}\right)_{y}=\lim _{\Delta x \rightarrow 0} \frac{f(x+\Delta x, y)-f(x, y)}{\Delta x}
$$

We could define a similar derivative with respect to $y$: (Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/))

For second derivatives, there are a few possibilities:

$$
f_{x x}=\frac{\partial^{2} f}{\partial x^{2}} \quad f_{y y}=\frac{\partial^{2} f}{\partial y^{2}} \quad f_{x y}=\frac{\partial^{2} f}{\partial x \partial y}=\frac{\partial}{\partial x} f_{y} \quad f_{y x}=\frac{\partial^{2} f}{\partial y \partial x}=\frac{\partial}{\partial y} f_{x}
$$

The key insight is that when taking a derivative with respect to $x$, all other variables (e.g. $y$) are constants.

For example:
$$
\left(\frac{\partial f}{\partial x}\right)_{y}=\frac{\partial}{\partial x}\left(3 x y^{2}+x+2 y^{2}\right)=\frac{\partial}{\partial x} 3 x y^{2}+\frac{\partial}{\partial x} x+\frac{\partial}{\partial x} 2 y^{2}
= 3y^2 + 1
$$

```python
# Sympy already knows how to do partial derivatives
f = 3*x*y**2 + x + 2*y**2
diff(f, x)
```

What about if we want to take a partial derivative with respect to $y$? Now $x$ is constant:

$$
\begin{aligned}\left(\frac{\partial f}{\partial y}\right)_{x} &=\frac{\partial}{\partial y}\left(3 x y^{2}+x+2 y^{2}\right)=\frac{\partial}{\partial y} 3 x y^{2}+\frac{\partial}{\partial y} x+\frac{\partial}{\partial y} 2 y^{2} \\ &=3 x \frac{\partial}{\partial y} y^{2}+x \frac{\partial}{\partial y} 1+2 \frac{\partial}{\partial y} y^{2}=6 x y+4 y \end{aligned}
$$

```python
# what about if I want the partia derivative with respect to y?
diff(f, y)
```

 Notice that $\frac{\partial}{\partial x}$ is usually not equal to $\frac{\partial}{\partial y}$

 We can also use Sympy to help us take second-order partial derivatives:

```python
diff(f, y, y)
```

```python
# take the derivative with respect to x, then with respect to y
# e.g., f_xy
diff(f, x, y)
```

```python
# take the derivative with respect to y, then to x
# e.g. f_yx
diff(f, y, x)
```

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000


```python
%%HTML
<!-- Mejorar visualización en proyector -->
<style>
.rendered_html {font-size: 1.2em; line-height: 150%;}
div.prompt {min-width: 0ex; }
.container {width:95% !important;}
</style>
```

<!-- Mejorar visualización en proyector -->
<style>
.rendered_html {font-size: 1.2em; line-height: 150%;}
div.prompt {min-width: 0ex; }
.container {width:95% !important;}
</style>

```python
%matplotlib notebook
import numpy as np
import scipy.fftpack
import matplotlib.pyplot as plt
from matplotlib import animation, patches
from IPython.display import display, Audio, HTML
plt.rcParams['image.cmap'] = 'gray'
from ipywidgets import interact, FloatSlider, IntSlider, SelectionSlider, Layout
from functools import partial
slider_layout = Layout(width='600px', height='20px')
slider_style = {'description_width': 'initial'}
FloatSlider_nice = partial(FloatSlider, style=slider_style, layout=slider_layout, continuous_update=False)
IntSlider_nice = partial(IntSlider, style=slider_style, layout=slider_layout, continuous_update=False)
SelectionSlider_nice = partial(SelectionSlider, style=slider_style, layout=slider_layout, continuous_update=False)

def color2bw(img):
    return np.dot(img, [0.299, 0.587, 0.114])
```

***
### Universidad Austral de Chile
## Computación de alto rendimiento
# Fast Fourier Transform

### Cristobal Navarro y Pablo Huijse

***

# Serie trigonométrica

Podemos descomponer una señal periódica $P=1/f$ en una suma de sinusoides

$$
\begin{align}
s(t)
&= \sum_{k=0}^\infty A_k \cos(2\pi k f t + \phi_k) \nonumber \\
&= \sum_{k=0}^\infty a_k \cos(2\pi k f t) + b_k \sin(2\pi k f t), \nonumber
\end{align}
$$

donde

$$
a_k =  f \int_{-\frac{1}{2f}}^{\frac{1}{2f}} s(t) \cos(2\pi k f t) \,dt \quad b_k = f \int_{-\frac{1}{2f}}^{\frac{1}{2f}} s(t) \sin(2\pi k f t) \,dt
$$

```python
plt.close('all'); fig, ax = plt.subplots(figsize=(6, 4))
f = 1.51518; t = np.linspace(-2/f, 2/f, num=5000);
line = ax.plot(t, np.zeros_like(t))
ax.set_xticks([-2/f, -1/f, 0, 1/f, 2/f]);
ax.set_xticklabels([r"$-2/f_0$", r"$-1/f_0$", "0", r"$1/f_0$", r"$2/f_0$"]);
a, b = np.random.randn(101), np.random.randn(101)
def update(K):
    y = np.zeros_like(t)
    for k in range(1, K+1):
        #y += (1/k)*np.sin(2.0*np.pi*k*f*t)
        #y += (2*np.sin(np.pi*k/2)/(np.pi*k))*np.cos(2.0*np.pi*k*f*t)
        y += a[k]*np.cos(2.0*np.pi*k*f*t)/k**2 + b[k]*np.sin(2.0*np.pi*k*f*t)/k**2
    line[0].set_ydata(y); ax.set_ylim([np.amin(y)*1.1, np.amax(y)*1.1])
interact(update, K=SelectionSlider_nice(options=[1, 2, 3, 4, 5, 10, 20, 30, 50, 100]));
```

Visualización interactiva de la FS: https://bl.ocks.org/jinroh/7524988

Serie de Fourier interesante: https://www.wolframalpha.com/input/?i=longcat+curve

# Transformada de Fourier
***
- El concepto de frecuencia puede aplicarse también a señales no-periódicas
- **Joseph Fourier:** Una señal no-periódica puede ser vista como una señal periódica **con un período infinito**
- El único requisito es que ahora las frecuencias son un continuo, con un espaciado infinitesimal

Directa:
$$
S(\omega) = \mathbb{FT}[s(t)] =  \int_{-\infty}^{\infty} s(t) e^{-j\omega t } dt,
$$

***

Inversa:
$$
s(t) =  \mathbb{FT}^{-1}[S(\omega)] = \frac{1}{2\pi} \int_{-\infty}^{\infty} S(\omega) e^{j \omega t } d\omega,
$$

***

# Discrete Fourier Transform (DFT)

***

- Nos interesa trabajar con señales digitales que están muestreadas en el tiempo
- Asumimos que la señal fue observada en un ventana de tiempo de ancho $T$ [s]
- Sea un sistema muestreador con frecuencia de muestreo $F_s$ [Hz] tal que
$$
s(t) = \sum_{n=0}^{N-1} s[n] \delta(t - n/F_s),
$$
del cual hemos tomado $N = T F_s$ muestras de $s(t)$

- Reemplazando en la transformada de Fourier
$$
\begin{align}
S(\omega) &= \int s(t) e^{-j\omega t} dt \nonumber \\
&= \int \sum_{n=0}^{N-1} s[n] \delta(t - n/F_s) e^{-j\omega t} dt \nonumber \\
&=  \sum_{n=0}^{N-1} s[n] \int \delta(t - n/F_s) e^{-j\omega t} dt \nonumber \\
&=  \sum_{n=0}^{N-1} s[n] e^{-j\omega n/F_s} \nonumber
\end{align}
$$
- Definiendo entonces $\omega = 2 \pi f = 2 \pi k \Delta f$ donde $\Delta f = \frac{1}{T} = \frac{F_s}{N}$ y reemplazando
$$
S[k] =  \sum_{n=0}^{N-1} s[n] e^{-j \frac{2 \pi}{N} k n},
$$
donde $k = [0, 1, \ldots N-1]$, ¿A qué frecuencias corresponden estos índices?

***

## DFT como producto matricial

Sea $\{s_n\}_{n=0,\ldots,N-1}$ y definiendo

$$
W_N = e^{-j \frac{2\pi}{N}} = \cos \left(\frac{2\pi}{N}\right) - j \sin \left(\frac{2\pi}{N}\right)
$$

podemos expresar la transformada de Fourier discreta como

$$
S[k] =  \sum_{n=0}^{N-1} s[n] W_N^{kn}, \quad k = [0, 1, \ldots N-1],
$$

que también puede ser expresado matricialmente como

$$
\begin{align}
\begin{pmatrix}
S[0] \\
S[1] \\
S[2] \\
\vdots \\
S[N-1] \\
\end{pmatrix} &=
\begin{pmatrix}
1 & 1 & 1 & \cdots & 1 \\
1 & W_N & W_N^2 & \cdots & W_N^{N-1} \\
1 & W_N^2 & W_N^4 & \cdots & W_N^{N-2} \\
\vdots & \dots & \dots & \ddots &  \vdots \\
1 & W_N^{N-1} & W_N^{N-2} & \cdots & W_N \\
\end{pmatrix}
\begin{pmatrix}
s[0] \\
s[1] \\
s[2] \\
\vdots \\
s[N-1] \\
\end{pmatrix} \nonumber  \\
S &= \Omega s,
\end{align}
$$

Notemos que:
- Por definición $W_N^{kn} = \left(e^{-j \frac{2\pi}{N}}\right)^{kn} = e^{-j \frac{2\pi}{N}kn}$
- Por periodicidad $W_N^{2(N-1)} = W_N^{2(N-1) - N}  = W_N^{N-2}$
- También se tiene simetría hermítica: $W_N^{k(-n)} = W_N^{-kn} = (W_N^{kn})^*$
- $\Omega$ es una matriz cuadrada y simétrica

***
La DFT tiene complejidad cuadrática: $N^2$ multiplicaciones y $N$ sumas
***

```python
F_s = 100
t = np.linspace(0, 2, 2*F_s, endpoint=False)
x = np.sin(2*np.pi*t*5) + 0.5*np.cos(2*np.pi*t*12 + np.pi/4)

fig, ax = plt.subplots(figsize=(7, 3), tight_layout=True)
ax.plot(t, x)
ax.set_xlabel('Tiempo [s]'); ax.set_ylabel('Amplitud');

def DFT(x):
    N = len(x)
    W_N = np.exp(-1j*2*np.pi/N)
    n = np.arange(N)
    Omega = W_N**(n*n.reshape(1,-1).T)
    return np.dot(Omega, x)

fig, ax = plt.subplots(figsize=(7, 3), tight_layout=True)
ax.plot(np.linspace(0, F_s, num=len(t)), np.abs(DFT(x)))
ax.set_xlabel('Frecuencia [Hz]');
display(np.angle(DFT(x))[np.abs(DFT(x)) > 10]*180/np.pi);
```

***

# Transformada Rápida de Fourier (FFT)

***

- La computación de la DFT tiene complejidad $\mathcal{O}(N^2)$
- Existe una aproximación numérica con complejidad $\mathcal{O}(N\log N)$: la Fast Fourier Transform (FFT).

El algoritmo *radix-2* obtiene una FFT recursiva que explota las simetrías en la DFT

$$
\begin{align}
S[k] &=  \sum_{n=0}^{N-1} s[n] W_N^{kn} \nonumber \\
&= \sum_{n=0}^{N/2-1} s[2n] W_N^{k 2n} + \sum_{n=0}^{N/2-1} s[2n+1] W_N^{k(2n+1)} \nonumber \\
&= \sum_{n=0}^{N/2-1} s[2n] W_{N/2}^{kn} + W_N^{k} \sum_{n=0}^{N/2-1} s[2n+1] W_{N/2}^{kn} \nonumber \\
&= S_E[k] + W_N^{k} S_O[k] ~~ \forall k \in [0,N/2]  \nonumber
\end{align}
$$

Notar que se calculan dos "medias" DFT

Por periodicidad de la DFT tenemos que
$$
\begin{align}
S_E[k + N/2] &=  \sum_{n=0}^{N/2-1} s[2n] W_{N/2}^{(k+N/2)n} \nonumber \\
&=  \sum_{n=0}^{N/2-1} s[2n] W_{N/2}^{kn} \exp \left(-j2\pi n \right) = S_E[k], \nonumber
\end{align}
$$

e igualmente

$$
S_O[k + N/2] = S_O[k],
$$

juntando ambos tenemos que
$$
\begin{align}
S[k + N/2] &=  S_E[k + N/2] + W_{N}^{(k+N/2)} S_O[k + N/2] \nonumber  \\
&=  S_E[k] + W_{N}^{k} \exp \left(-j\pi\right) S_O[k] \nonumber \\
&=  S_E[k] - W_{N}^{k} S_O[k] \nonumber
\end{align}
$$

es decir

$$
\begin{align}
S[k] &=  S_E[k] + W_{N}^{k} S_O[k] \nonumber \\
S[k + N/2] &=  S_E[k] - W_{N}^{k} S_O[k] \quad \forall k \in [0,N/2]  \nonumber
\end{align}
$$

- La DFT de $k$ y $k+N/2$ difieren en un signo
- Se han explotado las simetrías de la DFT para reducir el costo computacional

Diagramas de mariposa: http://www.themobilestudio.net/the-fourier-transform-part-14

```python
t = np.linspace(0, 10, num=20)
x = np.random.randn(20)
plt.figure(figsize=(7, 4))
plt.plot(t, x, c='k', alpha=0.5)
plt.scatter(t[::2], x[::2], marker='x', zorder=100)
plt.scatter(t[1::2], x[1::2], marker='o', zorder=100);
```

```python
np.set_printoptions(precision=3)
display(scipy.fftpack.fft(x),
        DFT(x))
```

```python
S = DFT(x) # N*N multiplicaciones
Se = DFT(x[0::2]) # N/2*N/2
So = DFT(x[1::2]) # N/2*N/2
# Se y So = N*N/2 multiplicaciones
display(S[:10],
        Se + np.exp(-1j*2*np.pi*np.arange(10)/len(x))*So,
        S[10:],
        Se - np.exp(-1j*2*np.pi*np.arange(10)/len(x))*So)
```

# Trabajo práctico

Genere una señal y calcule su transformada de Fourier escribiendo una rutina en C++

Optimize su rutina usando el algoritmo FFT

Paralelice el algoritmo FFT usando OpenMP y CUDA

```python
times, dt = np.linspace(0, 10, num=2048, retstep=True)
freqs = 0.01 + 20*np.random.rand(5)
phases = np.pi*np.random.randn(5)
signal = np.zeros(shape=(len(times),))
for freq, phase in zip(freqs, phases):
    signal += np.cos(2.0*np.pi*times*freq + phase)

fig, ax = plt.subplots()
ax.plot(times, signal)
np.savetxt('signal.dat', signal)
```

```python
fig, ax = plt.subplots(2, figsize=(7, 4))
SIGNAL = scipy.fftpack.fft(signal, n=len(times))
SIGNAL_plot = scipy.fftpack.fftshift(SIGNAL)
freqs_plot = scipy.fftpack.fftshift(scipy.fftpack.fftfreq(n=len(times), d=dt))
ax[0].plot(freqs_plot, np.abs(SIGNAL_plot))
ax[1].plot(freqs_plot, np.angle(SIGNAL_plot));
```

```python
#%%bash
!make
%timeit -n1 !./prog 0
%timeit -n1 !./prog 1
#!./prog 1
!head spectrum.dat
!make clean
```

```python
SIGNAL_c = np.loadtxt("spectrum.dat")
SIGNAL_c_complex = SIGNAL_c[:, 0] + 1j*SIGNAL_c[:, 1]
np.allclose(SIGNAL, SIGNAL_c_complex, rtol=1e-9)
```

```python
fig, ax = plt.subplots(2, figsize=(7, 4), tight_layout=True)
ax[0].plot(freqs_plot, np.abs(SIGNAL_plot), alpha=0.5)
SIGNAL_c_plot = scipy.fftpack.fftshift(SIGNAL_c_complex)
ax[0].plot(freqs_plot, np.abs(SIGNAL_c_plot), alpha=0.5)
ax[1].plot(freqs_plot, np.abs(SIGNAL_plot) - np.abs(SIGNAL_c_plot));
```

# Apéndices

## Transformada de Fourier bidimensional

La DFT se puede aplicar a funciones multi-dimensionales. En el caso discreto de una señal bidimensional $g[n_1, n_2]$ con índices $n_1 \in [0, N_1-1]$ y $n_2 \in [0, N_2-1]$ tenemos

$$
G[k_1, k_2] = \sum_{n_1=0}^{N_1-1} \sum_{n_2=0}^{N_2-1} g[n_1, n_2] \exp \left ( -j2\pi  \left[\frac{n_1 k_1}{N_1} + \frac{n_2 k_2}{N_2} \right] \right)
$$
y su inversa

$$
g[n_1, n_2] = \frac{1}{N_1 N_2}\sum_{k_1=0}^{N_1-1} \sum_{k_2=0}^{N_2-1} G[k_1, k_2] \exp \left ( j2\pi  \left[\frac{n_1 k_1}{N_1} + \frac{n_2 k_2}{N_2} \right] \right)
$$

Notemos que

\begin{align}
G[k_1, k_2] &= \sum_{n_1=0}^{N_1-1} \left(\sum_{n_2=0}^{N_2-1} g[n_1, n_2] \exp \left (-j2\pi \frac{n_2 k_2}{N_2}\right) \right) \exp \left (-j2\pi \frac{n_1 k_1}{N_1}\right) \\
&= \sum_{n_1=0}^{N_1-1} \gamma_{n_1}[n_2] \exp \left (-j2\pi \frac{n_1 k_1}{N_1}\right),
\end{align}

*i.e.* se descompone como dos DFT de una dimensión. En cada paso podemos usar la FFT

```python
x = np.arange(0, 32, step=1)
X, Y = np.meshgrid(x, x)
fig, ax = plt.subplots(9, 9, figsize=(11, 11), tight_layout=False)
for n in range(9):
    for m in range(9):
        ax[n, m].matshow(np.cos(2.0*np.pi*X*m/len(x) + 2.0*np.pi*Y*n/len(x)),
                         cmap=plt.cm.RdBu_r, vmin=-1, vmax=1)
        ax[n, m].axis('off')
```

```python
img_doge = color2bw(plt.imread('doge.jpg'))

plt.figure(figsize=(8, 5))
plt.imshow(img_doge)
plt.colorbar(orientation='horizontal');
```

```python
from scipy import fftpack
fig, ax = plt.subplots(1, 2, figsize=(10, 5), tight_layout=True)
S_img = fftpack.fft2(img_doge)
im = ax[0].imshow(fftpack.fftshift(np.log(1.+np.abs(S_img))))
fig.colorbar(im, ax=ax[0], orientation='horizontal')
im = ax[1].imshow(fftpack.fftshift(np.angle(S_img)))  # arctan(imag/real)
fig.colorbar(im, ax=ax[1], orientation='horizontal');
```

```python
plt.close('all'); fig, ax = plt.subplots(1, 2, figsize=(6, 3), tight_layout=True);
cy, cx = img_doge.shape[0]/2, img_doge.shape[1]/2
x = np.arange(0, img_doge.shape[1]); y = np.arange(0, img_doge.shape[0]);
X, Y = np.meshgrid(x, y)

def update(sigma1=1, sigma2=1):
    for ax_ in ax:
        ax_.cla()
        ax_.axis('off')
    mask1 = np.exp(-(((X-cx)/sigma1)**2 + ((Y-cy)/sigma1)**2))
    mask2 = np.exp(-(((X-cx)/sigma2)**2 + ((Y-cy)/sigma2)**2))
    im = ax[0].imshow(fftpack.fftshift(np.log(1.0+np.abs(S_img)))*(mask1-mask2))
    im = ax[1].imshow(np.real(fftpack.ifft2(fftpack.ifftshift(fftpack.fftshift(S_img)*(mask1-mask2)))))
interact(update, sigma1=FloatSlider_nice(min=1, max=200.0, value=200, description="$\sigma_1$"),
        sigma2=FloatSlider_nice(min=1, max=200.0, value=1, description="$\sigma_2$"));
```

```python

```


```python
# Third Party
import matplotlib.pyplot as plt
import astropy.stats.histogram as histogram
import astropy.units as u
import numpy as np
# Package
import stellar
```

# Initialize tracer profile
## plummer sphere:
Params:

- a: Plummer radius,

mass OR density:

- mass: total stellar mass $\mathrm{M}_{\odot}$

- density: scale density: $\mathrm{M}_{\odot}\,\mathrm{kpc}^{-3}$

```python
# Plummer parameters:
a    = .25 * u.kpc     # Plummer radius
mass = 1e5 * u.solMass # Mass of stellar systems #! Doesn't matter - set it to number of tracers you want to draw for easy comparison

#* Initialize stellar distribution
tracer = stellar.plummer(a=a, mass=mass)
```

```python
# for viewing convenience
tracer
```

$\displaystyle \rho(r)=\frac{3M_{0}}{4\pi{}a^3}\left(1+\frac{r^2}{a^2}\right)^{-\frac{5}{2}}$

    M = 1.00e+05 solMass
    a = 2.50e-01 kpc

# Initialize dark matter profile
## Herquist-Zhao:

```python
thetaNFW = {'rho_s':2e7 *u.solMass/u.kpc**3, # scale radius
            'r_s'  : 2*u.kpc,                # scale density
            'a'    : 1,                      # inner-slope
            'b'    : 1,                      # "width" of transition
            'c'    : 3                       # outer-slope
        }
dm = stellar.HerquistZhao(**thetaNFW)
dm
```

$\displaystyle \rho(r)=\frac{\rho_s}{\left(\frac{r}{r_s}\right)^{a}\left(1+\left(\frac{r}{r_s}\right)^{b}\right)^{\frac{c-a}{b}}}$

    rho_s = 2.00e+07 solMass / kpc3
    r_s = 2.00e+00 kpc
    a = 1
    b = 1
    c = 3

```python
dracoLike = stellar.System(dark_matter=dm,tracer=tracer,beta=0,pm=False)
```

```python
priors={'a'     : 1 ,
        'lnrho' : 1 ,
        'lnr'   : 1 ,
        'beta'  : 1 ,
        'b'     : 1 ,
        'c'     : 1 }
```

```python
R_observed = np.logspace(-2,0,20)*u.kpc
sigma,cov = dracoLike.Covariance(R_observed,dv=2*u.km/u.s,priors=priors)
# print(cov)
```

```python
sigma
```

      $\sigma_a$
      $\sigma_{\ln\rho_s}$
      $\sigma_{\ln{r_s}}$
      $\sigma_{\beta}$
      $\sigma_{b}$
      $\sigma_{c}$

      0
      0.489994
      0.928371
      0.814538
      0.590341
      0.876547
      0.986246

```python
cov
```

    array([[ 0.24009446, -0.21353256, -0.1917102 , -0.19280684, -0.0314963 ,
            -0.0178725 ],
           [-0.21353256,  0.86187313, -0.20226944,  0.05263578, -0.14226295,
             0.03778146],
           [-0.1917102 , -0.20226944,  0.66347261,  0.11240649, -0.26497164,
             0.0814366 ],
           [-0.19280684,  0.05263578,  0.11240649,  0.34850255,  0.142388  ,
            -0.03352441],
           [-0.0314963 , -0.14226295, -0.26497164,  0.142388  ,  0.76833513,
             0.07493248],
           [-0.0178725 ,  0.03778146,  0.0814366 , -0.03352441,  0.07493248,
             0.97268112]])

```python

```


Quantum Error Correction

```
%load_ext sympyprinting
```

```
from sympy import sqrt, symbols, Rational
from sympy import expand, Eq, Symbol, simplify, exp, sin
from sympy.physics.quantum import *
from sympy.physics.quantum.qubit import *
from sympy.physics.quantum.gate import *
from sympy.physics.quantum.grover import *
from sympy.physics.quantum.qft import QFT, IQFT, Fourier
from sympy.physics.quantum.circuitplot import circuit_plot
```

5 qubit code

```
M0 = Z(1)*X(2)*X(3)*Z(4); M0

```

```
M1 = Z(2)*X(3)*X(4)*Z(0); M1

```

```
M2 = Z(3)*X(4)*X(0)*Z(1); M2

```

```
M3 = Z(4)*X(0)*X(1)*Z(2); M3

```

These operators should mutually commute.

```
gate_simp(Commutator(M0,M1).doit())

```

And square to the identity.

```
for o in [M0,M1,M2,M3]:
    display(gate_simp(o*o))

```

Codewords

```
zero = Rational(1,4)*(1+M0)*(1+M1)*(1+M2)*(1+M3)*IntQubit(0, 5); zero

```

```
qapply(4*zero)

```

```
one = Rational(1,4)*(1+M0)*(1+M1)*(1+M2)*(1+M3)*IntQubit(2**5-1, 5); one

```

```
qapply(4*one)

```

The encoding circuit

```
encoding_circuit = H(3)*H(4)*CNOT(2,0)*CNOT(3,0)*CNOT(4,0)*H(1)*H(4)*\
                   CNOT(2,1)*CNOT(4,1)*H(2)*CNOT(3,2)*CNOT(4,2)*H(3)*\
                   H(4)*CNOT(4, 3)*Z(4)*H(4)*Z(4)

```

```
circuit_plot(encoding_circuit, nqubits=5, scale=0.5)
```

```
represent(4*encoding_circuit, nqubits=5)
```

$$\left(\begin{smallmatrix}1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0\\0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1\\0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1\\1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0\\0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1\\-1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0\\1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0\\0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1\\0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1\\-1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0\\-1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0\\0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1\\1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0\\0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1\\0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1\\-1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0\\0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1\\1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0\\-1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0\\0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1\\-1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0\\0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1\\0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1\\-1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0\\1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & -1 & 0 & -1 & 0 & -1 & 0 & -1 & 0\\0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 0 & -1 & 0 & -1 & 0 & -1 & 0 & -1\\0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1\\-1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & 1 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0\\0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1\\-1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & -1 & -1 & 0 & -1 & 0 & 1 & 0 & 1 & 0\\-1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0\\0 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & -1 & 0 & 1 & 0 & 1 & 0 & -1 & 0 & 1 & 0 & -1 & 0 & -1 & 0 & 1 & 0 & 0 & -1 & 0 & 1 & 0 & 1 & 0 & -1\end{smallmatrix}\right)$$


```python
# import packages
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
from sympy import *
from scipy.optimize import fsolve
%matplotlib inline
```

```python
# set up variables
length = 25 #(nm)
dx = length/12 #(nm)
dt = 0.016667*332/12 #(s)
```

```python
# set up constant for follwing calculation
a = 5.6737/2*10**(-6)
b = 4.295*10**(-5)
c = 6.3129*10**(-5)
d = 1.76386*10**(-6)
e = 1.5863*10**(-8)
f = 6.29973*10**(-10)
```

```python
# import two-phase data
df = pd.read_excel('D.xlsx', sheet_name='4')
A = df['SymbolA']
B = df['SymbolB']
C = df['Li Fraction'] # the concentration in two-phase region
```

```python
C[0]
```

    0.028169014084507043

```python
def myFunctionA(AA):

    for i in range(0,12):
        A[i] = AA[i]

    F = np.empty((12))

    F[0] = A[0] - C[0]
    F[11] = A[11] - 0

    for i in range(1,11):

        F[i] = (C[i]-2*A[i]) * ( a*(C[i]-A[i])**2 + b*(C[i]-A[i]) - a*(C[i]-A[i])*C[i] - b*C[i] + c*C[i]**3 - d*C[i]**2 + e*C[i] - f) - (A[i+1]-A[i])**2/(A[i+1]-A[i-1])*dx/dt - (C[i+1]-A[i+1]-C[i]+A[i])**2/(C[i+1]-A[i+1]-1.99*C[i]+1.99*A[i]+C[i-1]-A[i-1])*dx/dt

    return F

AAGuess = np.linspace(0.02, 0, 12)
AA = fsolve(myFunctionA,AAGuess)
print(AA)
```

    C:\Users\user\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      after removing the cwd from sys.path.

    [2.81585098e-02 1.08349088e-01 1.93442715e-01 4.11667645e-01
     4.78610705e-01 1.45106143e+00 1.63165168e+00 1.72117269e+00
     1.80121426e+00 2.46385126e+00 2.54386105e+00 4.38472723e-06]

```python
def myFunction1(BB):

    for i in range(0,12):
        B[i] = BB[i]

    F = np.empty((12))

    F[0] = B[0] - 0
    F[11] = B[11] - C[11]

    for i in range(1,11):

        F[i] = (2*B[i]-C[i]) * (a*B[i]**2 + b*B[i] - a*B[i]*C[i] - b*C[i] + c*C[i]**3 - d*C[i]**2 + e*C[i] - f) - ( (C[i+1]-2*B[i+1]-C[i]+2*B[i])**2 / (C[i+1]-B[i+1]-C[i-1]+B[i-1]) *delta_x / delta_t - (B[i+1]-B[i])**2 / (B[i+1]-1.99*B[i]+B[i-1])* delta_x / delta_t )

    return F

BBGuess = np.linspace(0, 1, 12)
BB = fsolve(myFunction1,BBGuess)
print(BB[0],BB[6],BB[11])
```

    C:\Users\user\Anaconda3\lib\site-packages\ipykernel_launcher.py:4: SettingWithCopyWarning:
    A value is trying to be set on a copy of a slice from a DataFrame

    See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
      after removing the cwd from sys.path.

    2.888845185697965e-15 0.2652138559880587 0.807088320080344

    C:\Users\user\Anaconda3\lib\site-packages\scipy\optimize\minpack.py:163: RuntimeWarning: The iteration is not making good progress, as measured by the
      improvement from the last five Jacobian evaluations.
      warnings.warn(msg, RuntimeWarning)

```python
BB
```

    array([2.88884519e-15, 3.18000542e-02, 6.52265596e-02, 1.01818872e-01,
           1.45239962e-01, 2.00002137e-01, 2.65213856e-01, 3.34357707e-01,
           4.03071788e-01, 5.22602103e-01, 7.76763935e-01, 8.07088320e-01])

```python
plt.plot(BB, 'ro')
plt.xlabel('location')
plt.ylabel('concentration')
plt.title('C_Beta')
plt.show()
```

```python
AA = C-BB
AA
```

    0     0.028169
    1     0.066791
    2     0.103788
    3     0.137618
    4     0.164619
    5     0.180280
    6     0.185490
    7     0.186769
    8     0.188478
    9     0.139370
    10   -0.044370
    11    0.000423
    Name: Li Fraction, dtype: float64

```python
plt.plot(AA, 'ro')
plt.xlabel('location')
plt.ylabel('concentration')
plt.title('C_Alpha')
plt.show()
```

```python
def myFunction2(AA):

    for i in range(0,12):
        B[i] = AA[i]

    F = np.empty((12))

    F[0] = B[0] - C[0]
    F[11] = B[11] - 0

    for i in range(1,11):

        F[i] = (C[i]-2*B[i]) * (a*B[i]**2 + b*B[i] - a*B[i]*C[i] - b*C[i] + c*C[i]**3 - d*C[i]**2 + e*C[i] - f) - ( (C[i+1]-2*B[i+1]-C[i]+2*B[i])**2 / (C[i+1]-B[i+1]-C[i-1]+B[i-1]) *delta_x / delta_t - (B[i+1]-B[i])**2 / (B[i+1]-1.99*B[i]+B[i-1])* delta_x / delta_t )

    return F

BBGuess = np.linspace(0, 1, 12)
BB = fsolve(myFunction2,BBGuess)
print(BB[0],BB[6],BB[11])
```


We now extend SEIR model to SEAI5R:

\begin{align}
\dot{S_{i}}	&=-\lambda_{i}(t)S_{i}+\sigma_{i}, \\
\dot{E}_{i}	&=\lambda_{i}(t)S_{i}-\gamma_{E}E_{i},\\
\dot{A}_{i}	&=\gamma_{E}E_{i}-\gamma_{A}A_{i},\\
\dot{I}_{i}^{a}	&=\alpha\gamma_{A}A_{i}-\gamma_{I^{a}}I_{i}^{a},\\
\dot{I}_{i}^{s}	&=\bar{\alpha}\gamma_{A}A_{i}-\gamma_{I^{s}}I_{i}^{s},\\
\dot{I}_{i}^{h}	&=h_{i}\gamma_{I^{s}}I_{i}^{s}-\gamma_{I^{h}}I_{i}^{h},\\
\dot{I}_{i}^{c}	&=c_{i}\gamma_{I^{h}}I_{i}^{h}-\gamma_{I^{c}}I_{i}^{c},\\
\dot{I}_{i}^{m}	&=m_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{R}_{i} &=\gamma_{I^{a}}I_{i}^{a}+\bar{h}_{i}\gamma_{I^{s}}I_{i}^{s}+\bar{c}_{i}\gamma_{I^{h}}I_{i}^{h}+\bar{m}_{i}\gamma_{I^{c}}I_{i}^{c},\\
\dot{N}_{i}	&=\sigma_{i}-I_{i}^{m}.
\end{align}

$\lambda_{i}(t)=\beta\sum_{j=1}^{M}\left(C_{ij}^{a}\frac{A_{j}}{N_{j}}+C_{ij}^{a}\frac{I_{j}^{a}}{N_{j}}+C_{ij}^{s}\frac{I_{j}^{s}}{N_{j}}+C_{ij}^{h}\frac{I_{j}^{h}}{N_{j}}\right)$.

Here
* $I^{a}$ : asymptomatic infectives
* $I^{s}$ : symptomatic infectives
* $I^{h}$ : hospitalized infectives
* $I^{c}$ : ICU cases
* $I^{m}$ : mortality
* ${h}_{i}=1-\bar h_{i}$ is the fraction of syptomatics who are hospitalized
* $c_{i}=1-\bar{c}_{i}$, is the fraction of hospitalizeds, who are in ICU
* $m_{i}=1-\bar{m}_{i}$, is the fraction of ICU which leads to mortality
* $C_{ij}^{s}=f^{s}C_{ij}^{a}\equiv f^{s}C_{ij}$
* $C_{ij}^{h}=f^{h}C_{ij}^{a}\equiv f^{h}C_{ij}$.

S ---> E

E ---> A

A ---> Ia, Is

Ia ---> R

Is ---> Ih, R

Ih ---> Ic, R

Ic ---> Im, R

```python
%%capture
## compile PyRoss for this notebook
import os
owd = os.getcwd()
os.chdir('../../')
%run setup.py install
os.chdir(owd)
```

```python
%matplotlib inline
import numpy as np
import pyross
import pandas as pd
import matplotlib.pyplot as plt
```

```python
M=16  # number of age groups

# load age structure data
my_data = np.genfromtxt('../data/age_structures/UK.csv', delimiter=',', skip_header=1)
aM, aF = my_data[:, 1], my_data[:, 2]

# set age groups
Ni=aM+aF;   Ni=Ni[0:M];  N=np.sum(Ni)
```

```python
# contact matrices
CH, CW, CS, CO = pyross.contactMatrix.UK()

## matrix of total contacts
C=CH+CW+CS+CO

fig,aCF =  plt.subplots(2,2);
aCF[0][0].pcolor(CH, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[0][1].pcolor(CW, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][0].pcolor(CS, cmap=plt.cm.get_cmap('GnBu', 10));
aCF[1][1].pcolor(CO, cmap=plt.cm.get_cmap('GnBu', 10));
```

## Fraction of asymptomatic infectives is constant

```python
beta  = 0.036692            # infection rate

gE    = 1/5
gA    = 1/3
gIa   = 1./7                # recovery rate of asymptomatic infectives
gIs   = 1./7                # recovery rate of symptomatic infectives
alpha = 0.3                 # fraction of asymptomatic infectives
fsa   = 0.2                 # the self-isolation parameter
fh    = 0
gIh   = 1/14
gIc   = 1/14

sa    = 100*np.ones(M)      #  rate of additional/removal of population by birth etc
sa[0]     = 1500            # birth
sa[12:16] = -300            # mortality

hh   = 0.1*np.ones(M)       # fraction which goes from Is to hospital
cc   = 0.05*np.ones(M)      # fraction which goes from  hospital to ICU
mm   = 0.4*np.ones(M)       # mortality from IC

# initial conditions
Is_0 = np.zeros((M));  #Is_0[6:13]=8;  Is_0[2:6]=4; Is_0[13:16]=4

Ia_0 = 1000*np.ones((M));
R_0  = np.zeros((M))
E_0  = np.zeros((M))
A_0  = np.zeros((M))
Ih_0  = np.zeros((M))
Ic_0  = np.zeros((M))
Im_0  = np.zeros((M))

S_0  = Ni - (E_0 + A_0 + Ia_0 + Is_0 + Ih_0 + Ic_0 +Im_0 + R_0)

# matrix for linearised dynamics
L0 = np.zeros((M, M))
L  = np.zeros((2*M, 2*M))

for i in range(M):
    for j in range(M):
        L0[i,j]=C[i,j]*Ni[i]/Ni[j]

L[0:M, 0:M]     =    alpha*beta/gIs*L0
L[0:M, M:2*M]   = fsa*alpha*beta/gIs*L0
L[M:2*M, 0:M]   =    ((1-alpha)*beta/gIs)*L0
L[M:2*M, M:2*M] = fsa*((1-alpha)*beta/gIs)*L0

r0 = np.max(np.linalg.eigvals(L))
print("The basic reproductive ratio for these parameters is", r0)
```

    The basic reproductive ratio for these parameters is (1.319908017634234+0j)

```python
# duration of simulation and data file
Tf=200;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE, 'gA':gA,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEAI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC  = np.zeros((Nf))
for i in range(1*M):
        IC += data['X'][:,3*M+i]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC, '-', lw=4, color='#A60628', label='forecast', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```

## Fraction of asymptomatic infectives is age-dependent

```python
alpha = 0.5*np.ones(M)      #  rate of additional/removal of population by birth etc
alpha[6:M] = 0.1              # everyone is symtomatic in large age-group

Ia_0 = 1000*np.ones((M));
```

```python

# duration of simulation and data file
Tf=200;  Nf=2000;

# intantiate model
parameters = {'alpha':alpha,'beta':beta, 'gIa':gIa,'gIs':gIs,
              'gIh':gIh,'gIc':gIc,  'gE':gE, 'gA':gA,
              'fsa':fsa, 'fh':fh,
              'sa':sa, 'hh':hh, 'cc':cc, 'mm':mm}
model = pyross.deterministic.SEAI5R(parameters, M, Ni)

# the contact structure is independent of time
def contactMatrix(t):
    return C
# run model
data=model.simulate(S_0, E_0, A_0, Ia_0, Is_0, Ih_0, Ic_0, Im_0, contactMatrix, Tf, Nf)

t = data['t']; IC1  =  data['X'][:,3*M+1];
IC2  =  data['X'][:,3*M+8]

fig = plt.figure(num=None, figsize=(10, 8), dpi=80, facecolor='w', edgecolor='k')
plt.rcParams.update({'font.size': 22})

plt.plot(t, IC1, '-', lw=4, color='#A60628', label='M=1', alpha=0.8)
plt.plot(t, IC2, '-', lw=4, color='green', label='M=16', alpha=0.8)

plt.legend(fontsize=26, loc='upper left'); plt.grid()
plt.autoscale(enable=True, axis='x', tight=True)
plt.ylabel('Asymtomatic Infected individuals'); #plt.xlim(0, 40); plt.ylim(0, 4);
#plt.savefig('/Users/rsingh/Desktop/2a.png', format='png', dpi=212)
IC[0]
```