# DeepSets
## Permutation Invariant Neural Networks

This notebook is focused on DeepSets, a neural network architecture that takes into input sets rather than fixed dimensional vectors, and whose output should be invariant to permutation of this set. You will learn what is DeepSets, how it works, and implement yourself a DeepSets network that learns to compute the sum of any set of MNIST Digits, based on PyTorch.

This notebook is based on mainly two article : [Zaheer et al., 2017](https://arxiv.org/abs/1703.06114) which first introduced and applied the concept in 2017, and the response of [Wagstaff et al., 2019](https://arxiv.org/abs/1901.09006) on the limitations of this method of representing functions on sets. Both paper can be found in the /biblio repository. The code is inspired by [this Github repository](https://github.com/yassersouri/pytorch-deep-sets), by Yasser Souri.

#### First let's import a few modules while you read the intro. You might need to install tensorboardX, use pip ! (pip install tensorboardX)

```python
import matplotlib.pyplot as plt
import numpy as np
import torchvision
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch import FloatTensor
from torch.utils.data.dataset import Dataset
from torchvision.datasets import MNIST
from torchvision.transforms import Compose, ToTensor, Normalize
from torch import optim
from torch.autograd import Variable

from tensorboardX import SummaryWriter

from typing import Tuple
from typing import Union
from tqdm import tqdm

RANDOM_SEED = 0
DATA_ROOT = './datasets/'
NetIO = Union[FloatTensor, Variable]
MNIST_TRANSFORM = Compose([ToTensor(), Normalize((0.1307,), (0.3081,))])
```

## 0 - Introduction

Most successful deep learning approaches make use of the *structure* in their inputs: CNNs are the state-of-the-art for images, RNNs and temporal convolutions for sequences, etc. The success of convolutional networks boils down to exploiting a key invariance property: translation invariance.

But images and sequences are far from the only data we want to build neural networks for. Often our inputs are sets: sequences of items, where the ordering of items caries no information for the task in hand. In such a situation, the invariance property we can exploit is **permutation invariance**. Such problems are frequent, for instance parsing a scene composed of a set of objects ([Eslami et al., 2016](https://arxiv.org/abs/1603.08575)), making predictions from a set of points forming a 3D point cloud ([Qi et al., 2017](https://arxiv.org/abs/1612.00593)) or training a set of agents in reinforcement learning ([Sunehag et al., 2017](https://www.sciencedirect.com/science/article/pii/S240589631932035X)). In the original paper, the authors [Zaheer et al., 2017](https://arxiv.org/abs/1703.06114) applied it in cosmology by improving red-shift estimation and anomaly detection in celebrities faces databases ! The model showed both qualitative and quantitative results in every case.

**Note** : Another feature discussed in the original paper is *permutation equivariance* . However, since it is a relatively different concept from *invariance*, and do not apply in the same way, nor to the same goal, we will discuss it in the Appendix.

## I - Permutation Invariance

Intuitively, permutation invariances states that the output of the model must not change if the inputs are reordered. For instance, this is what a permutation invariant function with three inputs would look like : $ f(a,b,c) = f(b,a,c) = f(b,c,a) = \cdots $. More mathematically speaking, the exact definition is the following:

A function $f(\mathbf{x})$ is permutation invariant if $f(\{x_1, ... , x_M\}) = f(\{x_{\pi(1)}, ... , x_{\pi(M)}\})$ for any permutation $\pi$.

Having this in my mind, how do we define permutation-invariant neural networks? Remember, we need to define a model that does not rely on the order of its inputs.

The main idea is to make use of some operation $P$ that is already permutation invariant. We map each of our inputs separately to another space, and apply to them our $P$ to obtain a representation of the set as a whole. $P$ destroys the ordering information, leaving the overall model permutation invariant.

In particular, DeepSets does this by setting $P$ to be summation on the latent space $Z$. Other operations are used as well, e.g. elementwise max. We call the case where the sum is used sum-decomposition via the latent space :

    A function $f$ is sum-decomposable if there are functions $\rho$ and $\phi$ such that : \[f(X) = \rho(\sum_{x \in X} \phi(x))\]
    In this case, we say that $(\rho,\phi)$ is a sum-decomposition of $f$. The codomain of $\phi$ (hence, the domain of $\rho$) is called the latent space $Z$.

We can now introduce the main backbone theorem for our work:

    We denote by $\mathcal{P}(\mathfrak{X}) $ the power set of $\mathfrak{X}$, i.e the set of the sets of $\mathfrak{X}$.
    Let $f : \mathcal{P}(\mathfrak{X}) \rightarrow \mathbb{R}$ where $\mathfrak{X}$ is $\textbf{countable}$ (i.e, its cardinality is smaller or equal to the number of elements in $\mathbb{N}$. This includes both finite and countably infinite sets; e.g. $\mathbb{N}$, $\mathbb{Q}$, and subsets thereof). Then $f$ is permutation-invariant if and only if it is sum-decomposable via the latent space $\mathbb{R}$.

Proof; you may try to do it as an exercice! (click to expand)

Since $\mathfrak{X}$ is countable, each $x \in \mathfrak{X}$ can be mapped to a unique element in $\mathbb{N}$ by a function $c(x) : \mathfrak{X} \rightarrow \mathbb{N}$. Let $\Phi(X) = \sum_{􏰄x \in X} \phi(x)$. If we can choose $\phi$ so that $\Phi$ is injective, then we can set $\rho = f ◦ \Phi^{−1}$, giving \begin{equation} f = \rho ◦ \Phi = \rho(\sum_{x \in X} \phi(x)) \quad \text{i.e, $f$ is sum-decomposable via $\mathbb{R}$.}\end{equation}
Now, consider $\phi(x) = 4^{-c(x)}$. Under this mapping, each $X \subset \mathfrak{X}$ corresponds to a unique real number expressed in base 4. Therefore $\Phi$ is injective, hence the theorem.

Great ! We now have a fully-functional theorem that states that if we can sum-decompose our network to $\mathbb{R}$, then it is permutation-invariant. The high-level description of the full architecture is now reasonably straightforward - transform your inputs into some latent space, destroy the ordering information in the latent space by applying the sum, and then transform from the latent space to the final output. This is illustrated in the following figure:

Replacing $\phi$ and $\rho$ by universal approximators leaves matters unchanged, since, in particular, $\phi$ and $\rho$ can be used to approximate arbitrary polynomials. Therefore, we can develop our own DeepSet network by learning these approximator!

## II - Implementation : MNIST digits sum
In this applied section, we want to build a network than learns to compute **any sum** of **any set** of MNIST Digits using DeepSets.
Why not use a classic Neural Network and train it to add its inputs? Let's look at the following picture.

Well this would certainly work for a fixed number N of digits. But remember, we want to be able to compute the sum of **any number M of digits** ! We would need to retrain the network on this new number of inputs, which isn't ideal... Would a recurrent network work then?

It could work. However, we have **no guaranty** that the sum of digits will remain the same if you switch two or more digits. Which is why we will use DeepSets, which has this nice property !

First of, let's define a class that instantiates a set of MNIST and returns its correct sum based on the labels of the MNIST digits.

```python
# This class instantiates a set of MNIST and returns its sum based on the label (targets) of the MNIST digits
# min_len : minimum length of the sets generated
# max_len : maximum length of the sets generated
# dataset_len : length of the dataset, ie. how many sets of MNIST digits there is

class MNISTSummation(Dataset):
    def __init__(self, min_len: int, max_len: int, dataset_len: int, train: bool = True, transform: Compose = None):
        self.min_len = min_len
        self.max_len = max_len
        self.dataset_len = dataset_len
        self.train = train
        self.transform = transform

        self.mnist = MNIST(DATA_ROOT, train=self.train, transform=self.transform, download=True)

        mnist_len = self.mnist.__len__()
        mnist_items_range = np.arange(0, mnist_len)

        items_len_range = np.arange(self.min_len, self.max_len + 1)
        items_len = np.random.choice(items_len_range, size=self.dataset_len, replace=True)
        self.mnist_items = []
        for i in range(self.dataset_len):
            self.mnist_items.append(np.random.choice(mnist_items_range, size=items_len[i], replace=True))

    def __len__(self) -> int:
        return self.dataset_len

    def __getitem__(self, item: int) -> Tuple[FloatTensor, FloatTensor]:
        mnist_items = self.mnist_items[item]

        the_sum = 0
        images = []
        for mi in mnist_items:
            img, target = self.mnist.__getitem__(mi)
            the_sum += target
            images.append(img)

        return torch.stack(images, dim=0), torch.FloatTensor([the_sum])
```

Let's test it for a batch of 5 sets, each having a length between 2 and 10. This is so we can visualize sets of digits of different length.

```python
dataset_length = 5

MNISTSum = MNISTSummation(min_len=2, max_len=10, dataset_len=dataset_length, train=True, transform=MNIST_TRANSFORM)

for i in range(dataset_length):
    img = MNISTSum.__getitem__(i)
    print("-------------------------------------------------------")
    print("Correct sum of the MNIST digits  : ",img[1].item())
    plt.imshow(torchvision.utils.make_grid(img[0], nrow=10).permute(1, 2, 0), cmap='gray')
    plt.axis('off')
    plt.show()

```

It works! Great. Now let's define the $\phi$ and $\rho$ functions of our DeepSets model.

```python
class SmallMNISTCNNPhi(nn.Module):
    def __init__(self):
        super().__init__()
        self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
        self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
        self.conv2_drop = nn.Dropout2d()
        self.fc1 = nn.Linear(320, 50)
        self.fc1_drop = nn.Dropout2d()
        self.fc2 = nn.Linear(50, 10)

    def forward(self, x: NetIO) -> NetIO:
        x = F.relu(F.max_pool2d(self.conv1(x), 2))
        x = self.conv2_drop(self.conv2(x))
        x = F.relu(F.max_pool2d(x, 2))
        x = x.view(-1, 320)
        x = F.relu(self.fc1(x))
        x = self.fc1_drop(x)
        x = F.relu(self.fc2(x))
        return x

class SmallRho(nn.Module):
    def __init__(self, input_size: int, output_size: int = 1):
        super().__init__()

        self.input_size = input_size
        self.output_size = output_size

        self.fc1 = nn.Linear(self.input_size, 10)
        self.fc2 = nn.Linear(10, self.output_size)

    def forward(self, x: NetIO) -> NetIO:
        x = F.relu(self.fc1(x))
        x = self.fc2(x)
        return x
```

We can now define our own DeepSets ! Your turn to work. Based on the figure in the first part, and the explanations, complete the DeepSets model.

```python
### Complete the code below!
# If you get stuck, uncomment the line in the next cell to load a correction (then you can execute the code).

class InvariantModel(nn.Module):
    def __init__(self, phi: nn.Module, rho: nn.Module):
        super().__init__()
        self.phi = phi
        self.rho = rho

    def forward(self, x: NetIO) -> NetIO:

        # Complete here !

        return out
```

```python
# True solution : uncomment and execute the line below
#%load solution/solution.py
```

Good job ! Now we can define our final model that will learn to sum up the digits.

```python
# DeepSets Model that sums up the digits. For better performance, use a dataset_len >10000 for training and testing
# This allows the model to train better.

class SumOfDigits(object):
    def __init__(self, lr=1e-3, wd=5e-3): #learning rate, weight decay
        self.lr = lr
        self.wd = wd

        self.train_db = MNISTSummation(min_len=2, max_len=30, dataset_len=2000, train=True, transform=MNIST_TRANSFORM)
        self.test_db = MNISTSummation(min_len=2, max_len=30, dataset_len=2000, train=False, transform=MNIST_TRANSFORM)

        self.the_phi = SmallMNISTCNNPhi()
        self.the_rho = SmallRho(input_size=10, output_size=1)

        self.model = InvariantModel(phi=self.the_phi, rho=self.the_rho)
        if torch.cuda.is_available():
            self.model.cuda()

        self.optimizer = optim.Adam(self.model.parameters(), lr=self.lr, weight_decay=self.wd)

        self.summary_writer = SummaryWriter(
            log_dir='./logs/exp-lr:%1.5f-wd:%1.5f/' % (self.lr, self.wd))

    # Train a complete epoch
    def train_1_epoch(self, epoch_num: int = 0):
        self.model.train()
        for i in tqdm(range(len(self.train_db))):
            loss = self.train_1_item(i)
            self.summary_writer.add_scalar('train_loss', loss, i + len(self.train_db) * epoch_num)

    # Train a single MNIST digit set
    def train_1_item(self, item_number: int) -> float:
        x, target = self.train_db.__getitem__(item_number)
        if torch.cuda.is_available():
            x, target = x.cuda(), target.cuda()

        x, target = Variable(x), Variable(target)

        self.optimizer.zero_grad()
        pred = self.model.forward(x)
        the_loss = F.mse_loss(pred, target)

        the_loss.backward()
        self.optimizer.step()

        the_loss_tensor = the_loss.data
        if torch.cuda.is_available():
            the_loss_tensor = the_loss_tensor.cpu()

        the_loss_numpy = the_loss_tensor.numpy().flatten()
        the_loss_float = float(the_loss_numpy[0])

        return the_loss_float

    # Evaluate the quality of prediction
    def evaluate(self):

        self.model.eval()

        totals = [0] * 51
        corrects = [0] * 51

        for i in tqdm(range(len(self.test_db))):

            x, target = self.test_db.__getitem__(i)

            item_size = x.shape[0]

            if torch.cuda.is_available():
                x = x.cuda()

            pred = self.model.forward(Variable(x)).data

            if torch.cuda.is_available():
                pred = pred.cpu().numpy().flatten()

            pred = int(round(float(pred[0])))
            target = int(round(float(target.numpy()[0])))

            totals[item_size] += 1

            if pred == target:
                corrects[item_size] += 1

        totals = np.array(totals)
        corrects = np.array(corrects)

        print("Totals: ",totals)
        print("Corrects: ",corrects)

        print(corrects / totals)

    # Simple function for you to test and visualize the model performance
    def test_after_training(self, x_test):
        pred_test = self.model.forward(Variable(x_test)).data
        pred_test = int(round(float(pred_test[0])))
        return pred_test
```

Now let's train our model.

For better results, I recommend changing the dataset_len to 10000 (**at least**) in the previous class SumOfDigits, and train our model on 30-35 iterations below.

However **be careful, this might be REALLY heavy for your computer**. You might want to launch it on Google Colab or any other external GPU.

```python
the_experiment = SumOfDigits(lr=1e-3) #our model

for i in range(10):
    the_experiment.train_1_epoch(i)
    the_experiment.evaluate()
```

      0%|          | 0/2000 [00:00<?, ?it/s]/Users/hugo/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:42: UserWarning: Using a target size (torch.Size([1])) that is different to the input size (torch.Size([1, 1])). This will likely lead to incorrect results due to broadcasting. Please ensure they have the same size.
    100%|██████████| 2000/2000 [00:19<00:00, 103.07it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 240.02it/s]
    /Users/hugo/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:92: RuntimeWarning: invalid value encountered in true_divide
      1%|          | 13/2000 [00:00<00:16, 117.73it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [0 0 6 7 2 3 2 5 3 6 2 2 5 2 1 3 2 5 2 0 0 3 2 0 2 3 1 4 2 3 3 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0]
    [       nan        nan 0.09375    0.11290323 0.04       0.05357143
     0.02777778 0.06097561 0.03947368 0.08571429 0.03076923 0.03846154
     0.0862069  0.02298851 0.01265823 0.03947368 0.03225806 0.07042254
     0.03278689 0.         0.         0.0483871  0.02531646 0.
     0.02597403 0.03846154 0.01470588 0.05797101 0.025      0.04477612
     0.03947368        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:20<00:00, 98.77it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 244.57it/s]
      1%|          | 11/2000 [00:00<00:18, 109.85it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [0 0 5 5 6 4 1 5 3 6 5 0 0 5 3 3 3 4 0 1 1 0 2 3 1 0 3 2 0 1 3 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0]
    [       nan        nan 0.078125   0.08064516 0.12       0.07142857
     0.01388889 0.06097561 0.03947368 0.08571429 0.07692308 0.
     0.         0.05747126 0.03797468 0.03947368 0.0483871  0.05633803
     0.         0.01265823 0.01785714 0.         0.02531646 0.04545455
     0.01298701 0.         0.04411765 0.02898551 0.         0.01492537
     0.03947368        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:19<00:00, 104.71it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 241.85it/s]
      0%|          | 8/2000 [00:00<00:25, 79.14it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [0 0 9 9 7 6 4 5 6 2 2 2 3 3 3 7 1 5 1 2 3 5 1 2 7 4 3 0 5 2 2 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0]
    [       nan        nan 0.140625   0.14516129 0.14       0.10714286
     0.05555556 0.06097561 0.07894737 0.02857143 0.03076923 0.03846154
     0.05172414 0.03448276 0.03797468 0.09210526 0.01612903 0.07042254
     0.01639344 0.02531646 0.05357143 0.08064516 0.01265823 0.03030303
     0.09090909 0.05128205 0.04411765 0.         0.0625     0.02985075
     0.02631579        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:20<00:00, 99.32it/s]
    100%|██████████| 2000/2000 [00:09<00:00, 220.70it/s]
      0%|          | 10/2000 [00:00<00:20, 95.62it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [0 0 6 9 7 2 6 9 5 5 2 4 5 9 7 3 6 2 4 3 3 4 6 1 1 7 0 5 4 4 5 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0]
    [       nan        nan 0.09375    0.14516129 0.14       0.03571429
     0.08333333 0.1097561  0.06578947 0.07142857 0.03076923 0.07692308
     0.0862069  0.10344828 0.08860759 0.03947368 0.09677419 0.02816901
     0.06557377 0.03797468 0.05357143 0.06451613 0.07594937 0.01515152
     0.01298701 0.08974359 0.         0.07246377 0.05       0.05970149
     0.06578947        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:18<00:00, 106.80it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 247.40it/s]
      1%|          | 12/2000 [00:00<00:17, 115.97it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [ 0  0 15  7  4  4  2  6  7  8  8  2  5  5  6  2  2  3  3  4  2  2  3  2
      1  4  3  3  1  2  4  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    [       nan        nan 0.234375   0.11290323 0.08       0.07142857
     0.02777778 0.07317073 0.09210526 0.11428571 0.12307692 0.03846154
     0.0862069  0.05747126 0.07594937 0.02631579 0.03225806 0.04225352
     0.04918033 0.05063291 0.03571429 0.03225806 0.03797468 0.03030303
     0.01298701 0.05128205 0.04411765 0.04347826 0.0125     0.02985075
     0.05263158        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:20<00:00, 98.83it/s]
    100%|██████████| 2000/2000 [00:09<00:00, 212.04it/s]
      1%|          | 12/2000 [00:00<00:16, 118.83it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [ 0  0 10 10  8  5  5  4  5  2  1  2  4  6  5  6  3  4  6  2  1  2  1  4
      5  2  5  2  3  1  3  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    [       nan        nan 0.15625    0.16129032 0.16       0.08928571
     0.06944444 0.04878049 0.06578947 0.02857143 0.01538462 0.03846154
     0.06896552 0.06896552 0.06329114 0.07894737 0.0483871  0.05633803
     0.09836066 0.02531646 0.01785714 0.03225806 0.01265823 0.06060606
     0.06493506 0.02564103 0.07352941 0.02898551 0.0375     0.01492537
     0.03947368        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:19<00:00, 100.30it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 227.15it/s]
      0%|          | 10/2000 [00:00<00:19, 99.95it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [ 0  0 15 10  8  4  6  6  2  6  2  3  8  0  4  6  3  3  7  3  3  2  3  3
      2  2  2  1  1  2  5  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    [       nan        nan 0.234375   0.16129032 0.16       0.07142857
     0.08333333 0.07317073 0.02631579 0.08571429 0.03076923 0.05769231
     0.13793103 0.         0.05063291 0.07894737 0.0483871  0.04225352
     0.1147541  0.03797468 0.05357143 0.03225806 0.03797468 0.04545455
     0.02597403 0.02564103 0.02941176 0.01449275 0.0125     0.02985075
     0.06578947        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:19<00:00, 102.83it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 222.35it/s]
      1%|          | 13/2000 [00:00<00:16, 119.90it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [ 0  0 12 10  7  4  6  4  4  5  3  4  2  0  3  1  1  2  0  0  2  0  0  2
      1  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    [       nan        nan 0.1875     0.16129032 0.14       0.07142857
     0.08333333 0.04878049 0.05263158 0.07142857 0.04615385 0.07692308
     0.03448276 0.         0.03797468 0.01315789 0.01612903 0.02816901
     0.         0.         0.03571429 0.         0.         0.03030303
     0.01298701 0.01282051 0.         0.         0.         0.
     0.                nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:19<00:00, 103.07it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 236.10it/s]
      0%|          | 8/2000 [00:00<00:25, 78.76it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [ 0  0 11  4  5  3  5  3 12  9  4  5  3 11  6  5  5  9  3  5  5  4  5  6
      6  4  8  4  2  6  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    [       nan        nan 0.171875   0.06451613 0.1        0.05357143
     0.06944444 0.03658537 0.15789474 0.12857143 0.06153846 0.09615385
     0.05172414 0.12643678 0.07594937 0.06578947 0.08064516 0.12676056
     0.04918033 0.06329114 0.08928571 0.06451613 0.06329114 0.09090909
     0.07792208 0.05128205 0.11764706 0.05797101 0.025      0.08955224
     0.01315789        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

    100%|██████████| 2000/2000 [00:19<00:00, 104.61it/s]
    100%|██████████| 2000/2000 [00:08<00:00, 243.39it/s]

    Totals:  [ 0  0 64 62 50 56 72 82 76 70 65 52 58 87 79 76 62 71 61 79 56 62 79 66
     77 78 68 69 80 67 76  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    Corrects:  [ 0  0 16 14 10  3 10  4  3  2  2  4  2  1  3  2  2  2  2  0  0  1  0  0
      3  0  0  0  1  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0  0
      0  0  0]
    [       nan        nan 0.25       0.22580645 0.2        0.05357143
     0.13888889 0.04878049 0.03947368 0.02857143 0.03076923 0.07692308
     0.03448276 0.01149425 0.03797468 0.02631579 0.03225806 0.02816901
     0.03278689 0.         0.         0.01612903 0.         0.
     0.03896104 0.         0.         0.         0.0125     0.
     0.                nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan        nan        nan        nan
            nan        nan        nan]

Finally, let's visualize it ourselves! We will generate a simple dataset, print it, and pass it to our model to assess and visualize its performance.

```python
dataset_length = 1

MNISTSum = MNISTSummation(min_len=2, max_len=5, dataset_len=dataset_length, train=True, transform=MNIST_TRANSFORM)

x, target = MNISTSum.__getitem__(0)

print("Real value of the sum of digits (target) : ",target.item())
plt.imshow(torchvision.utils.make_grid(x, nrow=10).permute(1, 2, 0), cmap='gray')
plt.axis('off')
plt.show()

pred = the_experiment.test_after_training(x)
print("Computed value by the newtork (prediction) : ", pred)
```

Be wary that this value may not be correct and you may need to re-run the previous cell multiple times to obtain a "good" result... It really depends on how much we trained the model.

But at last we have (almost) successfully built up our MNIST Digits summer. The results are 'usually' good and prove the utility of DeepSets. But is there a catch ? If you followed closely the theoritical part, you may already have noticed it... What about the case where the domain of our function $f$ is **continous** ? There appears the limitations of this method that we'll discuss in our final section.

## III - Limitations of DeepSets and the representation of functions on sets
Recall that we defined our backbone theorem as the following :

    We denote by $\mathcal{P}(\mathfrak{X}) $ the power set of $\mathfrak{X}$, i.e the set of the sets of $\mathfrak{X}$.
    Let $f : \mathcal{P}(\mathfrak{X}) \rightarrow \mathbb{R}$ where $\mathfrak{X}$ is $\textbf{countable}$ (i.e, its cardinality is smaller or equal to the number of elements in $\mathbb{N}$. This includes both finite and countably infinite sets; e.g. $\mathbb{N}$, $\mathbb{Q}$, and subsets thereof). Then $f$ is permutation-invariant if and only if it is sum-decomposable via the latent space $\mathbb{R}$.

But what if the domain of our network (where the inputs "belong") is not countable? Said, $\mathbb{R}$? In fact, we do have a second theorem, albeit slightly less strong.

    Uncountable Case :
    Let $M \in \mathbb{N}$, and let $f : [0, 1]^M \rightarrow \mathbb{R}$ be a continuous function. Then $f$ is
    permutation-invariant if and only if it is continuously sum-decomposable via $\mathbb{R}^{M+1}$
.

The proof is more complicated, we will not include it here (you can find it in the first paper refered). What this second theorem states is really interesting as it places a limit on the set size for a fixed finite-dimensional latent space. In particular, it shows that with a latent space of $M$ dimension is sufficient for representing all continuous permutation-invariant functions on sets of size $\leq M$. If you want to feed the model larger sets, there's no guarantee that it can represent your target function. This means that **if we want to use continuous mappings, the dimension of the latent space must be at least the maximum set size** . The details of this implications are nicely discussed in the answering paper of [Wagstaff et al., 2019](https://arxiv.org/abs/1901.09006). I encourage you to read it, especially the sections 3 and 4, as it discusses both the limitations of this method and the importance of continuity in neural networks !

## Conclusion

As a conclusion, we showed in this notebook a new take on neural networks, providing a great framework for building invariant models on sets. This method however suffers from limitations when working with more complex domains such as $\mathbb{R}$, and require attention when tuning the model.

As further reading, I can recommend these articles :
* [Lee et al., 2018](https://arxiv.org/abs/1810.00825), proposing an attention-based architecture to implement set functions, Set Transformer.
* [Bloem-Reddy and Teh, 2019](https://arxiv.org/abs/1901.06082) for a more theoretical take on invariances in neural networks.
* [Vartak et al., 2017](https://papers.nips.cc/paper/7266-a-meta-learning-perspective-on-cold-start-recommendations-for-items) developed a recommender system for Twitter using DeepSets. User's preferences are summarized by the set of tweets they recently engaged with on the platform. This set is processed by a DeepSets architecture, and the output of this set function is then fed into another neural network that scores new tweets the user might find interesting.
* [Maron et al., 2018](https://openreview.net/forum?id=Syx72jC9tm) for invariant and equivariant Graph Networks.

The implications for this topic are endless and will certainly develop even more in the future !

## Appendix : permutation equivariance

Permutation equivariance is another concept developed in the original paper by [Zaheer et al., 2017](https://arxiv.org/abs/1703.06114). Think of it as independance from permutation : if we take a function $f$ and a permutation $\pi$, $f$ is permutation equivariant if \begin{equation} f(\{x_1, ... , x_M\}) = \{f_{\pi(1)}(x), ... , f_{\pi(M)}(x)\} \end{equation}

This different concept is discussed in the original paper, albeit slightly less developed and applied. Up to you to implement it in Pytorch !

```python

```


# Exercises

We have four exercises in this section:
1. Rewrite the CART method to the Gini index.
2. Use pydot do draw the tree for C4.5 example.
3. Implement the minimum number of objects pruning method.
4. Plot OC1 tree, but instead of elements id, print the feature id it was split by.

## Exercise 1: Rewrite the CART method to use Gini index as shown in the lecture

Use Gini index can be calculated with the following equation:
\begin{equation}
I_{G}(X)=1-\sum_{i=1}^{m}p^{2}_{i},
\end{equation}
and
\begin{equation}
I_{G}(\text{feature})=\sum_{i=1}^{n}p_{i}*I_{G}(X_{i}).
\end{equation}

You need to fill the ``calculate_gini`` function and change the ``build`` function a bit.

```python
import math
import numpy as np
import pydot
import copy

class BinaryLeaf:

    def __init__(self, elements, labels, ids):
        self.L = None
        self.R = None
        self.elements = elements
        self.split_feature = None
        self.labels = labels
        self.completed = False
        self.ids = ids

    def set_R(self, Rleaf):
        self.R = Rleaf

    def set_L(self, Lleaf):
        self.L = Lleaf

    def set_elements(self, elements):
        self.elements = elements

    def get_elements(self):
        return self.elements

    def set_p(self, threshold):
        self.p = threshold

    def get_L(self):
        return self.L

    def get_R(self):
        return self.R

    def set_completed(self):
        self.completed = True

    def is_completed(self):
        return self.completed

    def get_labels(self):
        return self.labels

    def set_split(self, feature):
        self.split_feature = feature

    def get_split(self):
        return self.split_feature

    def set_ids(self, ids):
        self.ids = ids

    def get_ids(self):
        return self.ids

labels = [1, 1, -1, 1, -1, -1, 1, 1, 1, 1, 1, 1, -1, -1, -1]
data_set = [[1, 1, 2, 2], [2, 1, 2, 2], [1, 1, 1, 2], [1, 2, 1, 2], [2, 3, 2, 2],
            [2, 2, 1, 2], [3, 2, 2, 1], [1, 3, 2, 2], [3, 3, 2, 1], [2, 3, 1, 2],
            [3, 1, 1, 1], [1, 2, 1, 1], [2, 3, 1, 1], [2, 1, 1, 2], [2, 2, 1, 1]]

labels_count = len(np.unique(labels))

ids = list(range(len(data_set)))
root = BinaryLeaf(data_set, labels, ids)
current_node = root

## helper methods:

def get_unique_labels(labels):
    return np.unique(np.array(labels)).tolist()

def get_unique_values(elements):
    features_number = len(elements[0])
    unique = []
    for i in range(features_number):
        features_list = []
        for j in range(len(elements)):
            features_list.append(elements[j][i])
        unique.append(np.unique(np.array(features_list)))
    return unique

def is_leaf_completed(node):
    if node.is_completed():
        if node.get_L() != None and not node.get_L().is_completed():
            return node.get_L()
        elif node.get_R() != None and not node.get_R().is_completed():
            return node.get_R()
        elif node.get_L() == None and node.get_R() == None:
            return None
        elif node.get_L().is_completed() or node.get_R().is_completed():
            new_node = is_leaf_completed(node.get_L())
            if new_node == None:
                return is_leaf_completed(node.get_R())
            else:
                return new_node
        else:
            return None
    return node

def find_leaf_not_completed(root):
    return is_leaf_completed(root)

## CART methods:

def get_split_candidates(unique_values):
    split_list = []
    for i in range(len(unique_values)):
        current_list = []
        temp_list = copy.deepcopy(unique_values)
        current_list.append(temp_list[i])
        del temp_list[i]
        current_list.append(temp_list)
        split_list.append(current_list)
    return split_list

def get_number_of_labels_for_value(elements, column_id, label):
    count = 0
    if not isinstance(elements, list):
        elements_list = [elements]
    else:
        elements_list = elements

    column_elements = get_node_elements_column(column_id)

    for i in range(len(elements_list)):
        for j in range(len(column_elements)):
            if column_elements[j] == elements_list[i]:
                if current_node.labels[j] == label:
                    count = count + 1
    return count

def get_node_elements_column(column_id):
    return np.array(current_node.elements)[..., column_id].tolist()

def count_number_of_elements(elements, column_id):
    count = 0
    if isinstance(elements, list):
        column_elements = get_node_elements_column(column_id)
        for i in range(len(elements)):
            count = count + column_elements.count(elements[i])
    else:
        count = count + get_node_elements_column(column_id).count(elements)
    return count

# fill:
def calculate_gini(elements, column_id):
    t_l = count_number_of_elements(elements[0], column_id)
    t_r = count_number_of_elements(elements[1], column_id)
    elements_count = len(current_node.elements)
    p_l = t_l * 1.0 / elements_count * 1.0
    p_r = t_r * 1.0 / elements_count * 1.0

    sum_p_l = 1
    sum_p_r = 1
    labels = get_unique_labels(current_node.labels)
    for i in range(labels_count):
        p_class_t_l = (get_number_of_labels_for_value(elements[0], column_id, labels[i]) * 1.0) / (
                count_number_of_elements(elements[0], column_id) * 1.0)
        p_class_t_r = (get_number_of_labels_for_value(elements[1], column_id, labels[i]) * 1.0) / (
                count_number_of_elements(elements[1], column_id) * 1.0)
        sum_p_l = sum_p_l - p_class_t_l**2
        sum_p_r = sum_p_l - p_class_t_r**2
    return p_l*sum_p_l + p_r*sum_p_r

def check_completed(labels, elements):
    ratio = len(get_unique_labels(labels))
    if ratio == 1:
        return True
    elements = sorted(elements)
    duplicated = [elements[i] for i in range(len(elements)) if i == 0 or elements[i] != elements[i - 1]]
    if len(duplicated) == 1:
        return True
    return False

def split_node(current_node, value, split_id, split_history):
    left_leaf = []
    left_leaf_labels = []
    left_leaf_ids = []
    right_leaf = []
    right_leaf_labels = []
    right_leaf_ids = []
    for i in range(len(current_node.elements)):
        if current_node.elements[i][split_id] == value:
            left_leaf.append(current_node.elements[i])
            left_leaf_labels.append(current_node.labels[i])
            left_leaf_ids.append(current_node.ids[i])
        else:
            right_leaf.append(current_node.elements[i])
            right_leaf_labels.append(current_node.labels[i])
            right_leaf_ids.append(current_node.ids[i])
    if len(right_leaf_labels) == 0 or len(left_leaf_labels) == 0:
        current_node.set_completed()
        return current_node, split_history
    split_history.append([str(current_node.ids), str(left_leaf_ids)])
    split_history.append([str(current_node.ids), str(right_leaf_ids)])
    current_node.set_L(BinaryLeaf(left_leaf, left_leaf_labels, left_leaf_ids))
    current_node.set_R(BinaryLeaf(right_leaf, right_leaf_labels, right_leaf_ids))
    current_node.set_split(split_id)
    current_node.set_completed()
    if check_completed(left_leaf_labels, left_leaf):
        current_node.L.set_completed()
    if check_completed(right_leaf_labels, right_leaf):
        current_node.R.set_completed()
    return current_node, split_history

def get_current_node():
    return find_leaf_not_completed()

def build(root_node):
    current_node = root_node
    stop_criterion = False
    split_history = []
    while stop_criterion == False:
        unique_values = get_unique_values(current_node.get_elements())
        max_unique_id = 0
        max_split_id = 0
        max_value = 0
        for i in range(len(unique_values)):  # co gdy len(unique) == 1 ?
            if len(unique_values[i]) == 1:
                continue
            split_candidates = get_split_candidates(unique_values[i].tolist())
            for j in range(len(split_candidates)):
                # change here:
                current_value = calculate_gini(split_candidates[j], i)
                ###
                if max_value < current_value:
                    max_unique_id = i
                    max_split_id = j
                    max_value = current_value
        current_node, split_history = split_node(current_node, unique_values[max_unique_id][max_split_id], max_unique_id, split_history)
        new_node = find_leaf_not_completed(root_node)
        if new_node != None:
            current_node = new_node
        else:
            stop_criterion = True
    return root_node, split_history

def plot_tree(split_history):
    tree = pydot.Dot(graph_type='graph')
    for split in split_history:
        new_edge = pydot.Edge(split[0], split[1], fillcolor = 'red')
        tree.add_edge(new_edge)
    tree.write('cart_tree.png', format='png')

cart_tree, split_history = build(current_node)
plot_tree(split_history)
```

## Exercise 2: Use pydot do draw the tree for C4.5 example

Fill the code below.

```python
%store -r split_history_c45
```

```python
def plot_tree(split_history):
    tree = pydot.Dot(graph_type='graph')
    for split in split_history:
        new_edge = pydot.Edge(split[0], split[1], fillcolor = 'red')
        tree.add_edge(new_edge)
    tree.write('c45_tree.png', format='png')

plot_tree(split_history_c45)

from IPython.display import Image
Image(filename='c45_tree.png')
```

## Exercise 3: Implement the minimum number of objects pruning method

The MNO method checkes the accuracy at each split and prune the node if the number of objects in a leaf is below a given value N. Load the CART method first as below.

```python
import math
import numpy as np
import copy
from random import *

N = 2

labels = [1,1,-1,1,-1,-1,1,1,1,1,1,1,-1,-1,-1]
data_set = [[1,1,2,2],[2,1,2,2],[1,1,1,2],[1,2,1,2],[2,3,2,2],
                [2,2,1,2],[3,2,2,1],[1,3,2,2],[3,3,2,1],[2,3,1,2],
                [3,1,1,1],[1,2,1,1],[2,3,1,1],[2,1,1,2],[2,2,1,1]]

class BinaryLeaf:

    def __init__(self, elements, labels, ids):
        self.L = None
        self.R = None
        self.elements = elements
        self.split_feature = None
        self.split_value = None
        self.labels = labels
        self.completed = False
        self.ids = ids
        self.validated = False

    def set_R(self, Rleaf):
        self.R = Rleaf

    def set_L(self, Lleaf):
        self.L = Lleaf

    def set_elements(self, elements):
        self.elements = elements

    def get_elements(self):
        return self.elements

    def set_p(self, threshold):
        self.p = threshold

    def get_L(self):
        return self.L

    def get_R(self):
        return self.R

    def set_completed(self):
        self.completed = True

    def is_completed(self):
        return self.completed

    def get_labels(self):
        return self.labels

    def set_split(self, feature):
        self.split_feature = feature

    def get_split(self):
        return self.split_feature

    def set_split_value(self, value):
        self.split_value = value

    def get_split_value(self):
        return self.split_value

    def set_validated(self):
        self.validated = True

    def is_validated(self):
        return self.validated

    def set_ids(self, ids):
        self.ids = ids

    def get_ids(self):
        return self.ids

labels_count = len(np.unique(labels))

ids = list(range(len(data_set)))
root = BinaryLeaf(data_set, labels, ids)
current_node = root

def get_unique_labels(labels):
    return np.unique(np.array(labels)).tolist()

def get_unique_values(elements):
    features_number = len(elements[0])
    unique = []
    for i in range(features_number):
        features_list = []
        for j in range(len(elements)):
            features_list.append(elements[j][i])
        unique.append(np.unique(np.array(features_list)))
    return unique

def is_leaf_completed(node):
    if node.is_completed():
        if node.get_L() != None and not node.get_L().is_completed():
            return node.get_L()
        elif node.get_R() != None and not node.get_R().is_completed():
            return node.get_R()
        elif node.get_L() == None and node.get_R() == None:
            return None
        elif node.get_L().is_completed() or node.get_R().is_completed():
            new_node = is_leaf_completed(node.get_L())
            if new_node == None:
                return is_leaf_completed(node.get_R())
            else:
                return new_node
        else:
            return None
    return node

def find_leaf_not_completed(root):
    return is_leaf_completed(root)

def get_split_candidates(unique_values):
    split_list = []
    for i in range(len(unique_values)):
        current_list = []
        temp_list = copy.deepcopy(unique_values)
        current_list.append(temp_list[i])
        del temp_list[i]
        current_list.append(temp_list)
        split_list.append(current_list)
    return split_list

def get_number_of_labels_for_value(elements, column_id, label):
    count = 0
    if not isinstance(elements, list):
        elements_list = [elements]
    else:
        elements_list = elements

    column_elements = get_node_elements_column(column_id)

    for i in range(len(elements_list)):
        for j in range(len(column_elements)):
            if column_elements[j] == elements_list[i]:
                if current_node.labels[j] == label:
                    count = count + 1
    return count

def get_node_elements_column(column_id):
    return np.array(current_node.elements)[..., column_id].tolist()

def count_number_of_elements(elements, column_id):
    count = 0
    if isinstance(elements, list):
        column_elements = get_node_elements_column(column_id)
        for i in range(len(elements)):
            count = count + column_elements.count(elements[i])
    else:
        count = count + get_node_elements_column(column_id).count(elements)
    return count

def calculate_omega(elements, column_id):
    t_l = count_number_of_elements(elements[0], column_id)
    t_r = count_number_of_elements(elements[1], column_id)
    p_l = t_l * 1.0 / len(current_node.elements) * 1.0
    p_r = t_r * 1.0 / len(current_node.elements) * 1.0

    sum_p = 0
    labels = get_unique_labels(current_node.labels)
    for i in range(labels_count):
        p_class_t_l = (get_number_of_labels_for_value(elements[0], column_id, labels[i]) * 1.0) / (
                count_number_of_elements(elements[0], column_id) * 1.0)
        p_class_t_r = (get_number_of_labels_for_value(elements[1], column_id, labels[i]) * 1.0) / (
                count_number_of_elements(elements[1], column_id) * 1.0)
        sum_p = sum_p + math.fabs(p_class_t_l - p_class_t_r)
    return 2.0 * p_l * p_r * sum_p

def check_completed(labels, elements):
    ratio = len(get_unique_labels(labels))
    if ratio == 1:
        return True
    elements = sorted(elements)
    duplicated = [elements[i] for i in range(len(elements)) if i == 0 or elements[i] != elements[i - 1]]
    if len(duplicated) == 1:
        return True
    return False

def build(root_node):
    current_node = root_node
    stop_criterion = False
    split_history = []
    while stop_criterion == False:
        unique_values = get_unique_values(current_node.get_elements())
        max_unique_id = 0
        max_split_id = 0
        max_value = 0
        for i in range(len(unique_values)):
            if len(unique_values[i]) == 1:
                continue
            split_candidates = get_split_candidates(unique_values[i].tolist())
            for j in range(len(split_candidates)):
                current_value = calculate_omega(split_candidates[j], i)
                if max_value < current_value:
                    max_unique_id = i
                    max_split_id = j
                    max_value = current_value
        current_node, split_history = split_node(current_node, unique_values[max_unique_id][max_split_id],
                                                 max_unique_id, split_history)
        new_node = find_leaf_not_completed(root_node)
        if new_node != None:
            current_node = new_node
        else:
            stop_criterion = True
    return root_node, split_history
```

Fill the gap below:

```python
def split_node(current_node, value, split_id, split_history):
    left_leaf = []
    left_leaf_labels = []
    left_leaf_ids = []
    right_leaf = []
    right_leaf_labels = []
    right_leaf_ids = []
    for i in range(len(current_node.elements)):
        if current_node.elements[i][split_id] == value:
            left_leaf.append(current_node.elements[i])
            left_leaf_labels.append(current_node.labels[i])
            left_leaf_ids.append(current_node.ids[i])
        else:
            right_leaf.append(current_node.elements[i])
            right_leaf_labels.append(current_node.labels[i])
            right_leaf_ids.append(current_node.ids[i])
    if len(right_leaf_labels) == 0 or len(left_leaf_labels) == 0:
        current_node.set_completed()
        return current_node, split_history
    split_history.append([str(current_node.ids), str(left_leaf_ids)])
    split_history.append([str(current_node.ids), str(right_leaf_ids)])

    if len(left_leaf_ids) < N or len(right_leaf_ids) < N:
        current_node.set_completed()
    else:
        current_node.set_L(BinaryLeaf(left_leaf, left_leaf_labels, left_leaf_ids))
        current_node.set_R(BinaryLeaf(right_leaf, right_leaf_labels, right_leaf_ids))
        current_node.set_split(split_id)
        current_node.set_split_value(value)
        current_node.set_completed()
        if check_completed(left_leaf_labels, left_leaf):
            current_node.L.set_completed()
        if check_completed(right_leaf_labels, right_leaf):
            current_node.R.set_completed()
    ####
    return current_node, split_history
```

Test:

```python
cart_tree, split_history_cart = build(current_node)

def plot_tree(split_history):
    tree = pydot.Dot(graph_type='graph')
    for split in split_history:
        new_edge = pydot.Edge(split[0], split[1], fillcolor = 'red')
        tree.add_edge(new_edge)
    tree.write('cart2_tree.png', format='png')

plot_tree(split_history_cart)
```

## Exercise 4: Plot OC1 tree, but instead of elements id, print the feature id it was split by

To make the task done, you need to change the functions:
- ``build_level``,
- update the ``BinaryLeaf`` in two places to add the setters/getters and the feature and feature value split data.

```python
class BinaryLeaf:

    def __init__(self, elements, labels, ids):
        self.L = None
        self.R = None
        self.p = None
        self.elements = elements
        # add here features and value
        self.feature = None
        self.feature_values = None
        self.labels = labels
        self.completed = False
        self.ids = ids

    def set_R(self, Rleaf):
        self.R = Rleaf

    def set_L(self, Lleaf):
        self.L = Lleaf

    def set_elements(self, elements):
        self.elements = elements

    def get_elements(self):
        return self.elements

    def set_p(self, threshold):
        self.p = threshold

    def get_L(self):
        return self.L

    def get_R(self):
        return self.R

    def set_completed(self):
        self.completed = True

    def is_completed(self):
        return self.completed

    def get_labels(self):
        return self.labels

    # fill the methods here
    def get_feature(self):
        return self.feature

    def set_feature(self, feature):
        self.feature = feature

    def get_feature_values(self):
        return self.feature_values

    def set_feature_values(self, feature_values):
        self.feature_values = feature_values

    def set_ids(self, ids):
        self.ids = ids

    def get_ids(self):
        return self.ids

labels = [1, 1, -1, 1, -1, -1, 1, 1, 1, 1, 1, 1, -1, -1, -1]
data_set = [[1, 1, 2, 2], [2, 1, 2, 2], [1, 1, 1, 2], [1, 2, 1, 2], [2, 3, 2, 2],
            [2, 2, 1, 2], [3, 2, 2, 1], [1, 3, 2, 2], [3, 3, 2, 1], [2, 3, 1, 2],
            [3, 1, 1, 1], [1, 2, 1, 1], [2, 3, 1, 1], [2, 1, 1, 2], [2, 2, 1, 1]]
features = ['Location','Category','Gender','Product review']

labels_count = len(np.unique(labels))

ids = list(range(len(data_set)))
root = BinaryLeaf(data_set, labels, ids)
current_node = root
R = 10

## helper methods:

def compute_v(element, scv):
    return np.sum(np.multiply(element, scv[:-1])) + scv[-1]

def compare_two_leafs(leaf1, leaf2):
    labels1 = leaf1.labels
    labels2 = leaf2.labels
    if len(labels1) == len(labels2):
        for i in range(len(labels1)):
            if labels1[i] != labels2[i]:
                return False
        return True
    return False

def is_leaf_completed(node):
    if node.is_completed():
        if node.get_L() != None and not node.get_L().is_completed():
            return node.get_L()
        elif node.get_R() != None and not node.get_R().is_completed():
            return node.get_R()
        elif node.get_L() == None and node.get_R() == None:
            return None
        elif node.get_L().is_completed() or node.get_R().is_completed():
            new_node = is_leaf_completed(node.get_L())
            if new_node == None:
                return is_leaf_completed(node.get_R())
            else:
                return new_node
        else:
            return None
    return node

## oc1

def calculate_gini(labels):
    unique_labels = np.unique(labels)
    gini = 0
    for label in unique_labels:
        found = np.where(labels == label)
        gini = gini + len(found)/len(labels)
    return np.square(gini)

def get_all_possible_splits_by_gini(leaf):
    leaf_elements = leaf.elements
    labels = leaf.labels
    ginis = []
    for i in range(len(leaf_elements[0])):
        feature_ginis = []
        feature_column = np.array(leaf_elements)[:, i]
        for feature in feature_column:
            distinguish = feature_column <= feature
            left_labels  = np.array(labels)[distinguish]
            right_labels = np.array(labels)[~distinguish]
            gini = 1 - calculate_gini(left_labels) - calculate_gini(right_labels)
            feature_ginis.append([feature,gini])
        ginis.append(min(feature_ginis))
    return ginis

def find_current_level_data(root):
    return is_leaf_completed(root)

def divide_data_hiperplane(leaf,scv):
    below = []
    above = []
    below_labels = []
    above_labels = []
    below_ids = []
    above_ids = []
    for i in range(len(leaf.elements)):
        v = compute_v(leaf.elements[i],scv) > 0
        if v:
            above.append(leaf.elements[i])
            above_labels.append(leaf.labels[i])
            above_ids.append(leaf.ids[i])
        else:
            below.append(leaf.elements[i])
            below_labels.append(leaf.labels[i])
            below_ids.append(leaf.ids[i])
    return np.array(below), np.array(above), np.array(below_labels), np.array(above_labels), below_ids, above_ids

def get_coefficiency(splits):
    splits = np.array(splits)
    scv = np.zeros(len(splits)+1)
    min_split_index = np.argmin(splits[:,1])
    scv[min_split_index] = 1
    scv[-1] = -splits[min_split_index][1]
    return scv

def compute_u(element, scv, feature):
    return (scv[feature] * element[feature] - compute_v(element, scv)) / element[feature]

def sort_u(element):
    return np.sort(element)

def perturb(leaf, scv, feature, old_gini):
    u=[]
    for element in leaf.elements:
        u.append(compute_u(element, scv, feature))
    splits = sort_u(np.array(u))
    am = []
    for split in splits:
        new_scv = scv
        new_scv[feature] = split
        below, above, below_label, above_label, below_ids, above_ids = divide_data_hiperplane(leaf, scv)
        gini = 1 - (calculate_gini(below_label) - calculate_gini(above_label))
        am.append([new_scv, gini])
    am = np.array(am)
    best_split_index = np.argmin(am[:,1])
    if am[best_split_index][1] < old_gini:
        return am[best_split_index][1], am[best_split_index][0]
    elif am[best_split_index][1] == old_gini:
        if random() < 0.3:
            return am[best_split_index][1], am[best_split_index][0]
    return old_gini, scv

def build_level(root, split_history):
    leaf = find_current_level_data(root)
    if leaf == None:
        return
    splits = get_all_possible_splits_by_gini(leaf)
    split_coefficiency_vector = get_coefficiency(splits)
    below,above, below_label, above_label, below_ids, above_ids = divide_data_hiperplane(leaf,split_coefficiency_vector)
    gini = 1 - calculate_gini(below_label) - calculate_gini(above_label)
    for c in range(R):
        feature = randint(0,len(leaf.elements[0])-1)
        gini, split_coefficiency_vector = perturb(leaf, split_coefficiency_vector, feature, gini)
        below, above, below_label, above_label, below_ids, above_ids = divide_data_hiperplane(leaf,split_coefficiency_vector)
    left_leaf = BinaryLeaf(below, below_label, below_ids)
    right_leaf = BinaryLeaf(above, above_label, above_ids)
    leaf.set_feature(feature)
    split_history.append([str(leaf.feature), str(leaf.ids), str(left_leaf.ids)])
    split_history.append([str(leaf.feature), str(leaf.ids), str(right_leaf.ids)])
    leaf.set_completed()
    # change here
    ###
    if len(np.unique(below_label)) == 1:
        left_leaf.set_completed()
    if len(np.unique(above_label)) == 1:
        right_leaf.set_completed()
    if compare_two_leafs(leaf, left_leaf) or compare_two_leafs(leaf,right_leaf):
        leaf.set_completed()
    else:
        leaf.set_R(right_leaf)
        leaf.set_L(left_leaf)
    build_level(root, split_history)
    return root, split_history

def build(root):
    split_history = []
    return build_level(root, split_history)

def plot_tree(split_history):
    tree = pydot.Dot(graph_type='graph')
    for split in split_history:
        feature_2 = [split2[0] for split2 in split_history if split2[1] == split[2]]
        new_edge = pydot.Edge(split[0] + ": " + split[1], (feature_2[0] + ": " if len(feature_2) != 0 else "") + split[2], fillcolor = 'red')
        tree.add_edge(new_edge)
    tree.write('oc1_tree.png', format='png')

oc1_tree, split_history = build(root)
plot_tree(split_history)
```

    /usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:186: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray

```python

```


# Asymptotic solutions in long-times

Projectile motion in a non-homogenous potential field with drag is described by the equation

$$y_{\tau \tau} + \beta \epsilon y_{\tau} + \frac{1}{(1 + \epsilon y)^2} = 0,$$

with $y(0) = \epsilon$ and $y_{\tau}(0)=1$, and where $\epsilon \ll 1$ is expected.

```python
import sympy as sym
from sympy import init_printing
init_printing(order='rev-lex')
```

```python
y, eps, a, b, tau, t, beta, gamma = sym.symbols('y, epsilon, a, b, tau, t, beta, gamma')
y0 = sym.Function('y0')(t)
y1 = sym.Function('y1')(t)
y2 = sym.Function('y2')(t)
y3 = sym.Function('y3')(t)
y4 = sym.Function('y4')(t)
```

```python
y = sym.Eq(y0 + eps*y1 + eps**2*y2 + eps**3*y3 + eps**4*y4) # naive expansion

class f(sym.Function):

    @classmethod
    def eval(cls, y):
        return y.lhs.diff(t,t) + beta*eps*y.lhs.diff(t)**2 + 1/(1 + eps*y.lhs)**2
        #return y.lhs.diff(tau, tau) + eps/y.lhs**2
y
```

```python
the_series = sym.series(f(y), eps, x0=0, n=5)
by_order = sym.collect(the_series, eps, evaluate=False)
the_series
```

### $\mathcal{O} \left( 1 \right) \mbox{Solution}$

```python
sym.Eq(by_order[1].removeO())
```

```python
eqn = sym.Eq(by_order[1].removeO())    #1 + y0(tau).diff(tau, tau))
soln0 = sym.dsolve(eqn, y0)
constants = sym.solve([soln0.rhs.subs(t,0) - 0, \
                      soln0.rhs.diff(t).subs(t,0) - 1])
C1, C2 = sym.symbols('C1 C2')
soln0 = soln0.subs(constants)
soln0
```

### $\mathcal{O} \left( \epsilon \right) \mbox{Solution}$

```python
by_order[eps]
```

```python
try:
    eqn = sym.Eq(by_order[eps].replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps])

soln1 = sym.dsolve(eqn, y1)
constants = sym.solve([soln1.rhs.subs(t,0) - 0, \
                      soln1.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln1 = soln1.subs(constants)
soln1

```

### $\mathcal{O} \left( \epsilon^2 \right) \mbox{Solution}$

```python
by_order[eps**2]
```

```python
try:
    eqn = sym.Eq(by_order[eps**2].replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**2].replace(y1, soln1.rhs))

soln2 = sym.dsolve(eqn, y2)
constants = sym.solve([soln2.rhs.subs(t,0) - 0, \
                      soln2.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln2 = soln2.subs(constants)
sym.factor(soln2)
```

### $\mathcal{O} \left( \epsilon^3 \right) \mbox{Solution}$

```python
by_order[eps**3]
```

```python
try:
    eqn = sym.Eq(by_order[eps**3].replace(y2, soln2.rhs).replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**3].replace(y2, soln2.rhs))

soln3 = sym.dsolve(eqn, y3)
constants = sym.solve([soln3.rhs.subs(t,0) - 0, \
                      soln3.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln3 = soln3.subs(constants)
sym.factor(soln3)
```

### $\mathcal{O} \left( \epsilon^4 \right) \mbox{Solution}$

```python
by_order[eps**4]
```

```python
try:
    eqn = sym.Eq(by_order[eps**4].replace(y3, soln3.rhs).replace(
        y2, soln2.rhs).replace(y1, soln1.rhs).replace(y0, soln0.rhs))
except NameError:
    eqn = sym.Eq(by_order[eps**4].replace(y3, soln3.rhs))

soln4 = sym.dsolve(eqn, y4)
constants = sym.solve([soln4.rhs.subs(t,0) - 0, \
                      soln4.rhs.diff(t,1).subs(t,0) - 0])
C1, C2 = sym.symbols('C1 C2')
soln4 = soln4.subs(constants)
sym.factor(soln4)
```

### $\mbox{Composite Solution}$

```python
y_comp = sym.symbols('y_{comp}', cls=sym.Function)
try:
    y_comp = sym.Eq(y_comp, soln0.rhs + eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)
except NameError:
    y_comp = sym.Eq(y_comp, eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)

#print(sym.latex(y_comp))
#print(str(y_comp.rhs.subs(beta,6.5E-4)))
print(sym.latex(y_comp.rhs.subs(beta,6.5E-4).subs(t, 1) * 1.37))
#sym.latex(y_comp.rhs.subs(beta, 6.5E-4))
y_comp.rhs
```

### $\mbox{The Trajectory}$

```python
def savefig(filename, pics):
    if pics == True:
        plt.savefig('../doc/figures/{}.pgf'.format(filename), bbox_inches='tight', dpi=400)
    else:
        pass
pics = True
```

```python
import matplotlib.pyplot as plt
import matplotlib
import numpy as np
import scipy as sp

%config InlineBackend.figure_format = 'retina'
plt.rc('text', usetex=True)
plt.rc('font', family='serif')
plt.rcParams['figure.dpi'] = 300
matplotlib.rcParams.update(
{   'text.color': 'k',
    'xtick.color': 'k',
    'ytick.color': 'k',
    'axes.labelcolor': 'k'
})
plt.rc('font', size=14)
eps_val = [.1, .5, 1.][::-1]
linestyle = ['rs--', 'bo-', 'cv-.', 'k+:', 'm']
tt = sp.arange(0,1,0.001)
bet = 6.5E-4
plt.figure(figsize=(6, 4))#, dpi=100)
for keys, vals in enumerate(eps_val):
    y_compP = sym.lambdify(t, y_comp.rhs.subs(eps, vals).subs(beta, bet), 'numpy')
    plt.plot(tt, y_compP(tt), linestyle[keys],label=r'$\phi \mathbf{E}\mbox{u}$'+ ' = {}'.format(vals).rstrip('0').rstrip('.')
             , markevery=100)

plt.ylim(ymin=0., ymax=0.7)
plt.xlim(xmax=1)
plt.ylabel(r'$y^*$')
plt.xlabel(r'$t^*$')
leg = plt.legend(title = r'$\mathbf{D}\mbox{g}$' +' = {:1.0E}'.format(bet)[:-4] + r'$\times 10^{-4}$')
leg.get_frame().set_linewidth(0.0)
savefig('long_times',pics)
plt.show();
```

## Time aloft

```python
y2 = sym.symbols('y2', cls=sym.Function)
y2 = sym.Function('y2')(t)
try:
    y2 = sym.Eq(y2, soln0.rhs + eps*soln1.rhs + eps**2*soln2.rhs
                + eps**3*soln3.rhs + eps**4*soln4.rhs) # + eps**2*soln2.rhs)
except NameError:
    y2 = sym.Eq(y2, eps*soln1.rhs + eps**2*soln2.rhs + eps**3*soln3.rhs + eps**4*soln4.rhs)
y2.rhs
#y2.diff(t)
```

```python
tau0, tau1, tau2, tau3 = sym.symbols('tau0 tau1 tau2 tau3')
tau = sym.Eq(tau0 + eps*tau1 + eps**2*tau2 + eps**3*tau3)
y3 = y2.rhs.subs(t, tau.lhs).series(eps)
col = sym.collect(y3, eps, evaluate=False)
```

### $\mathcal{O} \left( 1 \right) \mbox{Solution}$

```python
#tau0 = 2
sym.Eq(col[1].removeO())
```

Two roots, lets look at $\tau_0 = 2$.

### $\mathcal{O} \left( \epsilon \right) \mbox{Solution}$

```python
order_eps = col[eps].subs(tau0, 2)
order_eps
```

```python
soln_eps = sym.solve(order_eps, tau1)
```

### $\mathcal{O} \left( \epsilon^2 \right) \mbox{Solution}$

```python
order_eps2 = col[eps**2].subs(tau0, 2).subs(tau1, soln_eps[0])
order_eps2
```

```python
soln_eps2 = sym.solve(order_eps2, tau2)
```

### $\mathcal{O} \left( \epsilon^3 \right) \mbox{Solution}$

```python
order_eps3 = col[eps**3].subs(tau0, 2).subs(tau1, soln_eps[0]).subs(tau2, soln_eps2[0])
order_eps3
soln_eps3 = sym.solve(order_eps3, tau3)
```

### Composite solution

```python
0.2*8*np.pi
2*1.45
```

```python
tau0, tau1, tau2, tau3 = sym.symbols('tau0 tau1 tau2 tau3')
tau = sym.Eq(tau0 + eps*tau1 + eps**2*tau2 + eps**3*tau3)
tau = tau.subs(tau0, 2).subs(tau1, soln_eps[0]).subs(tau2, soln_eps2[0]).subs(tau3, soln_eps3[0])
print(tau.subs(beta, 6E-4).lhs*1.45)
print(str(tau.subs(beta, 6E-4).lhs))
tau_soln = sym.lambdify(eps, tau.subs(beta, 6E-4).lhs*1.45, 'numpy')
print(tau_soln(0.67))
tau
```

```python
a=y_comp.rhs.subs(t, tau.lhs.subs(eps, gamma)).subs(beta,6.5E-4)
#(sym.collect(a, eps, evaluate=False)[1])
b = sym.collect(a, eps, evaluate=False)[1]
sym.simplify(b)
```

```python
ttt = np.arange(0.01, 1,0.001)
betas = [bet]
linestyle = ['k','k--', 'bo-', 'cv-.', 'k+:', 'm']
plt.figure(figsize=(6, 4))#, dpi=100)
for keys, vals in enumerate(betas):
    taun = tau.subs(beta, vals)
    tau_soln = sym.lambdify(eps, taun.lhs, 'numpy')
    label=['{:1.0E}'.format(vals)[:-4] + r'$\times 10^{-4}$']
    plt.semilogx(ttt, tau_soln(ttt), linestyle[keys],
                 label=r'$\mathbf{D}\mbox{g}$ = '+ label[keys], markevery=100)
plt.xlabel(r'$\phi \mathbf{E}\mbox{u}$')
plt.ylabel(r'$t_f$')
plt.legend()
savefig('drag', pics)
plt.show();
```

## Drag vs. inertia

```python
t, V = sym.symbols('t V', real=True, positive=True)
u = sym.symbols('u', cls=sym.Function)
eq = sym.diff(u(t), t) + u(t)**2
sol = sym.dsolve(eq, u(t))
print(sol.rhs)
u = sol.rhs
```

    1/(C1 + t)

```python
C1, C2 = sym.symbols('C1 C2')
eq = sym.Eq(u.subs(t, 0), 1)
sol = sym.solve(eq, C1)[0]
print(sol)
print(u.subs(C1, sol))
```

    1
    1/(t + 1)

```python
tt = np.arange(0.01,1.,0.001)
plt.figure(figsize=(6, 4), dpi=100)
u_soln = sym.lambdify(t, u.subs(C1, sol).subs(V,1.), 'numpy')
plt.semilogy(tt, u_soln(tt), 'k')
plt.show();
```

```python

```


>###  Ejemplo 3  **Tarea**
>El tiempo en el cual un movimiento browniano se mantiene sobre su punto máximo en el intervalo [0,1] tiene una distribución
>$$F(x)=\frac{2}{\pi}\sin^{-1}(\sqrt x),\quad 0\leq x\leq 1$$
Genere muestres aleatorias que distribuyan según la función dada usando el método de la transformada inversa y grafique en una gráfica el historias 100 muestras generadas y comparela con el función F(x) dada, esto con el fín de validar que el procedimiento fue realizado de manera correcta

```python
import numpy as np
import scipy.stats as ss
from scipy.stats import rayleigh
import statsmodels as st
import matplotlib.pyplot as plt
from sympy import Derivative, diff, simplify
import matplotlib.pyplot as plt
```

```python
N=10**6
def d_fun(N):
    U= np.random(N)
    return np.sin(np.pi*U/2)**2
f= lambda x: 1/(np.pi*np.sqrt(x)*np.sqrt(1-x))
#dominio 0<=x<=1
x=np.arange(0.1,.99,.01)
plt.plot(x,f(x))

```

```python
d = d_fun(N)
plt.hist(d_fun, bins= 100, density=True)
plt.hist(d_fun)
plt.show()
```

### Ejemplo 4
Distribución de Rayleigh
$$F(x)=1-e^{-2x(x-b)},\quad x\geq b $$

```python
#derivar F(x) para obtener f(x)
import numpy as np
import scipy.stats as stats
from scipy.stats import rayleigh
import statsmodels as st
import matplotlib.pyplot as plt
from sympy import Derivative, diff, simplify
import matplotlib.pyplot as plt
#lambda x: np.diff(1-np.exp*(-2*x*(x-b)))
```

```python
def D_ray(x,b):
    return np.diff(1-np.exp*(-2*x*(x-b)))
r = rayleigh.rvs(size=1000)
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.legend(loc='best', frameon=False)
plt.show()

```

```python
values = plt.hist(np.random.rayleigh(3, 100000), bins=200, density=True)
```

### Ejemplo binomial:
La distribución binomial modela el número de éxitos de n ensayos independientes donde hay una probabilidad p de éxito en cada ensayo.

Generar una variable aletoria binomial con parámetros $n=10$ y $p=0.7$. Recordar que
$$X\sim binomial(n,p) \longrightarrow p_i=P(X=i)=\frac{n!}{i!(n-i)!}p^i(1-p)^{n-i},\quad i=0,1,\cdots,n$$
> ## Tarea: Demostrar la validez de la siguiente ecuación
>$$p_{i+1}=\frac{n-i}{i+1}\frac{p}{1-p} p_i \longrightarrow \text{Hablar de las ventajas que sea recursiva}$$

```python
#generarU, verificar si cumple X=0
#Comparar la aumulada hasta n
```

```python
n=
p=np.zeros(N)
N=
P[0]=(1-p)**n

----
for i in range (N):
    p.append(pi+1#toda la expresion)((n-i)/(i+1))*(p/(1-p)*p[-1])

#crear la binomial
# Función que calcula la probabilidad acumulada
def P_acum_Binomial(n,p):
    c  = p/(1-p)
    pr = [(1-p)**n]
    add = pr.append
    for i in range(n+1):
        add((c*(n-i)/(i+1))*pr[-1])
    return np.cumsum(pr)
----
#quitar for
#se crea un vector y se llena
n=
p=np.zeros(N)
N=
P[0]=(1-p)**n
def P_acum_Binomial(n,p): #sera global para usarla en cualqioer momento, para recursividad crear otra func dentro de la global
             p=np.zeros(N)
             P[0]=(1-p)**n
             def llenar_vector(i): #contador de 0 a 1
             global p
             p[i+1]=((c*(n-i)/(i+1))*p[i])
             #compresion de listas, llama al vector de ceros
             [llenar_vector(i) for i in range(N-1)]
             return np.cumsum(p)
```

> ## Tarea
Seguir un procedimiento similar al mostrado cuando se generó una distribución binomial, pero en esta caso genere un código que genere variables aletorias Poisson cuya función de distribución de probabilidad esta dada por:
>$$P(k,\lambda)=\frac{e^{-\lambda}(\lambda)^k}{k!}$$

> Demuestre matemáticamente que
> $$P(k+1)=\frac{\lambda}{k+1}P(k)$$
> y a partir de esta relación genere variables aletorias que distribuyen poisson.

Enlace: https://es.wikipedia.org/wiki/Distribuci%C3%B3n_de_Poisson

```python
def P_acum_poisson(l,k):
    c  = l/(l+1)
    pr = [(k+1)**n]
    add = pr.append
    for i in range(n+1):
        add((c*(n-i)/(i+1))*pr[-1])
    return np.cumsum(pr)
```

```python
# Función que calcula la probabilidad acumulada optimizada
def P_acum_poisson_o(l,k):
    global Pr
    Pr = np.zeros(n)
    Pr[0] = (1-p)**n
    def pr(i):
        global Pr
        c  =l/(l+1)
        Pr[i+1]=(c*(n-i)/(i+1))*Pr[i]

    # Lleno el vector Pr usando compresión de listas
    [pr(i) for i in range(n-1)]
    return np.cumsum(Pr)
```

```python
# def D_binomial_intermedia(n,p,N):
l = 10; k' = 0.7; N = 10**4

p_acum = P_acum_poisson_o(l,k)
u = np.random.rand(N)

# Gráfica de resultados
plt.figure(figsize=[13,4])
plt.subplot(121)
d_poisson = list(map(lambda j:sum(1 for i in p_acum if i<u[j]),range(N)))
plt.hist(d_poisson,100,density=True,label='Usando función creada')
plt.legend()

plt.subplot(122)
d_poi_numpy = np.random.binomial(n,p,N)
plt.hist(d_poi_numpy,100,density=True,label='Usando Numpy')
plt.legend()
np.histogram
plt.show()
```

```python
# Graficando Poisson
mu = 3.6 # parametro de forma
poisson = stats.poisson(mu) # Distribución
x = np.arange(poisson.ppf(0.01),
poisson.ppf(0.99))
fmp = poisson.pmf(x) # Función de Masa de Probabilidad
plt.plot(x, fmp)
plt.vlines(x, 0, fmp, colors='b', lw=5, alpha=0.5)
plt.title('Distribucion Poisson')
plt.ylabel('probabilidad')
plt.xlabel('valores')
plt.show()
# histograma
aleatorios = poisson.rvs(1000) # genera aleatorios
cuenta, cajas, ignorar = plt.hist(aleatorios, 20)
plt.ylabel('frequencia')
plt.xlabel('valores')
plt.title('Histograma Poisson')
plt.show()
```

```python

```


# Block 1: Linear programming
# Python Notebook
### Alfred Galichon (NYU)
## `math+econ+code' masterclass on matching models, optimal transport and applications
© 2018-2019 by Alfred Galichon. Support from NSF grant DMS-1716489 is acknowledged. James Nesbit contributed.

## Linear programming: duality

### Learning objectives

* Linear programming duality

* Economic interpretation of the dual

* Numerical computation

### References

* [OTME], App. B

* Stigler (1945), The cost of subsistence. *Journal of Farm Economics*.

* Dantzig (1990), The diet problem. *Interface*.

* Complements:

    * Gale (1960), *The theory of linear economic models*.

    * Vohra (2011), *Mechanism Design: A Linear Programming Approach*.

### The diet problem

During World War II, engineers in US Army were wondering how to feed their personnel at minimal cost, leading to what is now called the **optimal diet problem**.

* Nutritionists have identified a number of vital nutrients (calories, protein, calcium, iron, etc.) that matter for a person's health, and have determined the minimum daily intake of each nutrient

* For each basic food (pasta, butter, bread, etc), nutritionists have characterized the intake in each of the various nutrients

* Each food has a unit cost, and the problem is to find the optimal diet = combination of foods that meet the minimal intake in each of the nutrients and achieves minimal cost

The problem was taken on by G. Stigler, who published a paper about it in 1945, giving a first heuristic solution, exhibiting a diet that costs $\$39.93$ per year in $1939$ dollars. Later (in $1947$) it was one of the first
application of G.B. Dantzig's method (the simplex algorithm), which provided the exact solution ($\$39.67$). It then took $120$ man-day to perform this operation. However today the computer will perform it for us in a
fraction of second.

However, don't try this diet at home! Dantzig did so and almost died from it...

## Motivation

### A look at the Data

Our dataset was directly taken from Stigler's article. It is a .csv file called `StiglerData1939.txt':

```python
import gurobipy as grb
import scipy.sparse as sp
import pandas as pd
import numpy as np
import os

filename = 'data_mec_optim/lp_stigler-diet/StiglerData1939.txt'
thepath = os.path.join(os.getcwd(),'..')
thedata = pd.read_csv(os.path.join(thepath ,filename), sep='\t')
thedata = thedata.dropna(how = 'all')
thedata
```

      Commodity
      Unit
      Price Aug.15 1939(cents)
      Edible Weight per $1.00 (grams)
      Calories (1000)
      Protein(grams)
      Calcium(grams)
      Iron(mg.)
      Vitamin A(1000 I.U)
      Thiamine(mg.)
      Riboflavin(mg.)
      Niacin(mg.)
      Asorbic Acid (mg.)

      0
      1. Wheat Flour (Enriched)
      10 lb.
      36.0
      12600.0
      44.7
      1411.0
      2.0
      365.0
      NaN
      55.4
      33.3
      441.0
      NaN

      1
      2. Macaroni
      1 lb.
      14.1
      3217.0
      11.6
      418.0
      0.7
      54.0
      NaN
      3.2
      1.9
      68.0
      NaN

      2
      3. Wheat Cereal (Enriched)
      28 oz.
      24.2
      3280.0
      11.8
      377.0
      14.4
      175.0
      NaN
      14.4
      8.8
      114.0
      NaN

      3
      4. Corn Flakes
      8 oz.
      7.1
      3194.0
      11.4
      252.0
      0.1
      56.0
      NaN
      13.5
      2.3
      68.0
      NaN

      4
      5. Corn Meal
      1 lb.
      4.6
      9861.0
      36.0
      897.0
      1.7
      99.0
      30.9
      17.4
      7.9
      106.0
      NaN

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      73
      74. Sugar
      10 lb.
      51.2
      8773.0
      34.9
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN

      74
      75. Corn Sirup
      24 oz.
      13.7
      4966.0
      14.7
      NaN
      0.5
      74.0
      NaN
      NaN
      NaN
      5.0
      NaN

      75
      76. Molasses
      18 oz.
      13.6
      3752.0
      9.0
      NaN
      10.3
      244.0
      NaN
      1.9
      7.5
      146.0
      NaN

      76
      77. Strawberry Preserves
      1 lb.
      20.5
      2213.0
      6.4
      11.0
      0.4
      7.0
      0.2
      0.2
      0.4
      3.0
      NaN

      77
      Daily allowance
      NaN
      NaN
      NaN
      3.0
      70.0
      0.8
      12.0
      5.0
      1.8
      2.7
      18.0
      75.0

78 rows × 13 columns

Our dataset has the nutritional content of $77$ commodities, and in the final row, the daily minimum requirement of each of these nutrients.

### The Diet problem

Problem setup:

* Assume there are nutrients $i\in\left\{  1,...,m\right\}  $ (calories, protein, calcium, iron, etc.) that matter for a person's health, in such way that the minimum daily intake of nutrient $i$ should be $d_{i}$.

* Nutrients do not come as standalone elements, but are combined into various foods. Each unit of food $j\in\left\{  1,...,n\right\}$ yields a quantity $N_{ij}$ of nutrient $i\in\left\{1,...,m\right\}$. The dollar cost of food $j$ is $c_{j}$.

The problem is to find the diet that achieves the minimal intake of each nutrient at a cheapest price. If $q\in\mathbb{R}^{n}$ is a vector such that $q_{j}\geq0$ is the quantity of food $i$ purchased, the quantity of nutrient $i$ ingested is $\sum_{j=1}^{n}N_{ij}q_{j}$, and the cost of the diet is $\sum_{j=1}^{n}q_{j}c_{j}$. The optimal diet is therefore given by

\begin{align*}
\min_{q\geq0}  &  ~c^{\top}q\\
s.t.~  &  Nq\geq d.\nonumber
\end{align*}

Before we tackle this problem, let's look into the linear programming problem in standard form.

## A Crash Course on Linear Programming

### Linear programming in standard form

Let $c\in\mathbb{R}^{n}$, $d\in\mathbb{R}^{m}$, $A$ be a $m\times n$ matrix, and consider the following problem

\begin{align}
V_{P} = \max_{x\in\mathbb{R}_{+}^{n}}  & \, c^{\top} x \\
s.t.~Ax  &  \leq d
\end{align}

This problem is a *linear programming problem*, as the objective function, namely $x\rightarrow c^{\top}x$ is linear, and as the constraint, namely $x\in\mathbb{R}_{+}^{n}$ and $Ax=d$ are also linear (or more accurately, affine). This problem is called the *primal program*, for reasons to be explained soon. The set of $x$'s that satisfy the constraint are called *feasible solutions*; the set of solutions of the primal problem are called *optimal solutions*.

### Remarks

* The previous diet problem can be reformulate into this problem - why?

* A problem does not necessarly have a feasible solution (e.g. if $A=0$ and $d\neq0$), in which case (by convention) $V_{P}=-\infty$.

* The whole space may be solution (e.g. if $A=0$ and $d=0$), in which case $V_{P}=+\infty$.

### Duality

There is a powerful tool called duality which provides much insight into the analysis of the primal problem. The idea is to rewrite the problem as

\begin{align*}
V_{P}=\max_{x\in\mathbb{R}_{+}^{n}}\left\{  c^{\top}x+L_{P}\left(
d-Ax\right)  \right\}
\end{align*}

where $L_{P}\left(z\right)$ is a penalty function whose value is zero if the constraint is met, that is if $z=0$, and $-\infty$ if it is not, namely if $z\neq0$. The simplest choice of such penalty function is given by $L_{P}\left(  z\right)  =\min_{y\in\mathbb{R}^{m}}\left\{  z^{\top}y\right\}$. One has

\begin{align*}
V_{P}=\max_{x\in\mathbb{R}_{+}^{n}}\min_{y\in\mathbb{R}^{m}}\left\{c^{\top}x+\left(  d-Ax\right)  ^{\top}y\right\}  .
\end{align*}

However, the minimax inequality $\max_{x}\min_{y}\leq\min_{y}\max_{x}$ always holds, thus

\begin{align*}
V_{P}  &  \leq\min_{y\in\mathbb{R}^{m}}\max_{x\in\mathbb{R}_{+}^{n}}\left\{
c^{\top}x+\left(  d-Ax\right)  ^{\top}y\right\}  =\min_{y\in\mathbb{R}^{m}
}\max_{x\in\mathbb{R}_{+}^{n}}\left\{  x^{\top}\left(  c-A^{\top}y\right)
+d^{\top}y\right\} \\
&  \leq\min_{y\in\mathbb{R}^{m}}\left\{  d^{\top}y+L_{D}\left(  c-A^{\top
}y\right)  \right\}  =:V_{D}
\end{align*}

where $L_{D}\left(z\right) = \max_{x\in\mathbb{R}_{+}^{n}}\left\{x^{\top}z\right\}$ is equal to $0$ if $z\in\mathbb{R}_{-}^{n}$, and to $+\infty$ if not. Therefore, the value $V_{D}$ is expressed by the *dual program*

\begin{align}
V_{D}=\min_{y\in\mathbb{R}^{m}}  & \,  d^{\top}y, \\
s.t.~A^{\top}y  &  \geq c
\end{align}

and the weak duality inequality $V_{P}\leq V_{D}$ holds. It turns out that as
soon as either the primal or dual program has an optimal solution, then both
programs have an optimal solution and the values of the two programs coincide,
so the weak duality becomes an equality $V_{P}=V_{D}$ called strong duality.
Further, if $x^{\ast}\in\mathbb{R}_{+}^{n}$ is an optimal primal solution, and
$y^{\ast}\in\mathbb{R}^{m}$ is an optimal dual solution, then complementary
slackness holds, that is $x_{i}^{\ast}>0$ implies $\left(  A^{\top}y^{\ast
}\right)  _{i}=c_{i}$.

### Duality theorem
We summarize these results into the following statement.

---
**Theorem.** In the setting described above:

1. The weak duality inequality holds:

\begin{align}
V_{P}\leq V_{D}.
\end{align}

2. As soon as the primal or the dual program have an optimal solution, then both programs have an optimal solution, and strong duality holds:

\begin{align}
V_{P}=V_{D}.
\end{align}

3. If $x^{\ast}\in\mathbb{R}_{+}^{n}$ is an optimal primal solution, and $y^{\ast}\in\mathbb{R}^{m}$ is an optimal dual solution, then complementary slackness holds:

\begin{align}
x_{i}^{\ast}>0\text{  implies  }\left(  A^{\top}y^{\ast}\right)  _{i}=c_{i}.
\end{align}

---

## The diet problem (revisited)

Recall the optimal diet problem

\begin{align*}
\min_{q\geq0}  &  \, c^{\top}q\\
s.t.~  &  Nq\geq d.
\end{align*}

which has minimax formulation $\min_{q\geq0}\max_{\pi\geq0}c^{\top}q+d^{\top}\pi-q^{\top}N^{\top}\pi$, so the dual is

\begin{align*}
\max_{\pi\geq0}  & \, d^{\top}\pi\\
s.t.~  &  N^{\top}\pi\leq c
\end{align*}

Interpretation: imagine that there is a new firm called Nutrient Shoppe, who sells raw nutrients. Let $\pi_{i}$ be the price of nutrient $i$. The cost of the diet is $d^{\top}\pi$. Consumer purchase raw nutrients and can generate
"synthetic foods". The cost of the synthetic version of food $j$ is $\sum_{i=1}^{m}N_{ij}\pi_{i}=\left(N^{\intercal}\pi\right)_{j}$. The constraint thus means that each "synthetic food" is more affordable than
its natural counterpart.

The duality means that it is possible to price the nutrients so that the
synthetic foods are cheaper than the natural ones, in such a way that the
price of the synthetic diet equals the price of the natural diet.

Complementary slackness yields:

* $q_{j}>0$ implies $\left(  N^{\intercal}\pi\right)  _{j}=c_{j}$; that
is, if natural food $j$ is actually purchased, then the prices of its
synthetic and natural versions coincide

* $\pi_{i}>0$ implies $\left(  Nq\right)  _{i}=d_{i}$; that is, if
nutrient $i$ has a positive price, then the natural diet has the
"just right" amount.

### Solving the diet problem

To solve the primal problem we need to construct the objects $c$, $N$ and $d$. $c$ is simply a vector of ones, the size of the number of commodities. $N$ is a matrix of amount of nutrients in each commodity. $d$ is the required daily allowance of each nutrient.

```python
commodity = (thedata['Commodity'].values)[:-1]
intake = thedata.iloc[:-1, 4:].fillna(0).transpose().values
allowance = thedata.iloc[-1, 4:].fillna(0).transpose()
```

Set up the model:Set up the model: prior to version 9.0, Gurobi in Python was different from R, in the sense that we could not input a matrix of constraints: note that Gurobi in Python is different from R, in the sense that we cannot a matrix of constraints

```python
m = grb.Model('optimalDiet')
meal = m.addVars(commodity, name='meal')
m.setObjective(meal.sum(), grb.GRB.MINIMIZE)
m.addConstrs((grb.quicksum(meal[k] * intake[i, j] for j, k in enumerate(commodity)) >= allowance[i]
              for i in range(intake.shape[0])), name='c')
```

    Using license file C:\Users\jmcgn\gurobi.lic
    Academic license - for non-commercial use only

    {0: <gurobi.Constr *Awaiting Model Update*>,
     1: <gurobi.Constr *Awaiting Model Update*>,
     2: <gurobi.Constr *Awaiting Model Update*>,
     3: <gurobi.Constr *Awaiting Model Update*>,
     4: <gurobi.Constr *Awaiting Model Update*>,
     5: <gurobi.Constr *Awaiting Model Update*>,
     6: <gurobi.Constr *Awaiting Model Update*>,
     7: <gurobi.Constr *Awaiting Model Update*>,
     8: <gurobi.Constr *Awaiting Model Update*>}

```python
m.optimize()
if m.status == grb.GRB.Status.OPTIMAL:
    total = 0
    solution = m.getAttr('x', meal)
    pi = m.getAttr('pi')
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 9 rows, 77 columns and 570 nonzeros
    Model fingerprint: 0x7ae6b743
    Coefficient statistics:
      Matrix range     [1e-01, 5e+03]
      Objective range  [1e+00, 1e+00]
      Bounds range     [0e+00, 0e+00]
      RHS range        [8e-01, 8e+01]
    Presolve removed 0 rows and 47 columns
    Presolve time: 0.01s
    Presolved: 9 rows, 30 columns, 240 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    0.0000000e+00   1.384688e+01   0.000000e+00      0s
           5    1.0866228e-01   0.000000e+00   0.000000e+00      0s

    Solved in 5 iterations and 0.01 seconds
    Optimal objective  1.086622782e-01

```python
print('***Optimal solution***')
for food in commodity:
    if solution[food] > 0:
        print(food, solution[food] * 365)
        total += solution[food] * 365
print('Total cost (optimal) =', total)
```

    ***Optimal solution***
    1. Wheat Flour (Enriched) 10.774457511918214
    30. Liver (Beef) 0.6907834111074215
    46. Cabbage 4.0932688648428766
    52. Spinach  1.8277960703546998
    69. Navy Beans Dried 22.27542568724304
    Total cost (optimal) = 39.66173154546625

Since version 9.0, Gurobi now allows passing a matrix of constraints from Python:

```python
m2 = grb.Model('optimalDietMatrix')
x = m2.addMVar(shape=commodity.shape, name="x")
m2.setObjective(x.sum(), grb.GRB.MINIMIZE)
m2.addConstr(sp.csr_matrix(intake) @ x >= np.array(allowance), name="c")
m2.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 9 rows, 77 columns and 570 nonzeros
    Model fingerprint: 0x7ae6b743
    Coefficient statistics:
      Matrix range     [1e-01, 5e+03]
      Objective range  [1e+00, 1e+00]
      Bounds range     [0e+00, 0e+00]
      RHS range        [8e-01, 8e+01]
    Presolve removed 0 rows and 47 columns
    Presolve time: 0.01s
    Presolved: 9 rows, 30 columns, 240 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    0.0000000e+00   1.384688e+01   0.000000e+00      0s
           5    1.0866228e-01   0.000000e+00   0.000000e+00      0s

    Solved in 5 iterations and 0.02 seconds
    Optimal objective  1.086622782e-01

As promised, we achieve the minimum cost bundle at $\$39.67$ per year in $1939$ dollars. If we compare this to Stigler's solutions which was

|Food| Annual Quantities| Annual Cost|
| ---------- | ------------------ | ------------ |
| Wheat Flour | 	370 lb.|   \$13.33 |
| Evaporated Milk | 	57 cans |	  \$3.84 |
|Cabbage| 	111 lb. 	  |\$4.11|
|Spinach| 	23 lb. 	  |\$1.85|
|Dried Navy Beans| 	285 lb. |	\$16.80|
|Total Annual Cost| 	  	| \$39.93 |