# Memory Aware Synapses
[Paper](https://arxiv.org/abs/1711.09601)
[Code](https://github.com/rahafaljundi/MAS-Memory-Aware-Synapses)
[Code-PyTorch](https://github.com/wannabeOG/MAS-PyTorch)

## Main Contributions:

* Proposes a regularization-based method (parameters are regularized based on their importance) that uses the sensititivity of the network's output function to the change in parameters as a measure of parameter importance.
* Parameter importance can be computed in an unsupervised manner without the need of labels (parameter importance is not dependent on the loss)
* Specific groups or tasks can be prioritized for not forgetting. This way parameters can be freed up from remembering unimportant things.
* Link with Hebbian Learning
* Global MAS - uses $L_2^2$ norm of output, Local MAS - uses layer wise outputs for gradient computation.
* Better performance than SI, EWC, LwF, ELLB, IMM.
* Among the competing methods, SI and EWC perform the best.
* Can be used with a pretrained model (importance calculation can be independent from initial training phase)
* Points out short-comings of SI (check related work for hand-written comments on disadvantages of SI and advantages of MAS over SI)

## How is parameter importance calculated?

A trained model learns an approximation $F$ of the true function $\bar{F}$.

Need to measure how sensitive the function $F$ is to changes in the network parameters. For a given datapoint $x_k$, the output is $F(x_k, \theta)$.

A small perturbation $\delta=\{\delta_{ij}\}$ in the parameters $\theta=\{\theta_{ij}\}$ results in a change in $F$, that can be approximated by:
\begin{equation}
 F(x_k; \theta+\delta) -F(x_k; \theta) \approx \sum_{i,j} \cfrac{\partial{(F(x_k;\theta))}}{\partial \theta_{ij}}\delta_{ij} = \sum_{i,j} g_{ij}(x_k)\delta_{ij}
\end{equation}

The importance of a parameter $\theta_{ij}$ is measured by the magnitude of the gradient $g_{ij}$. The importance is summed over all datapoints to get the final importance measure.

\begin{equation}
    \Omega_{ij} = \cfrac{1}{N} \sum_{k=1}^{N} \vert\vert g_{ij}(x_k) \vert\vert
\end{equation}

In most cases, $F$ is multi-dimensional. Thus multiple backward passes need to be performed to compute the gradient w.r.t. each dimension of $F$. To avoid this, the squared-$L_2$ norm of $F$ is used to compute the gradient. In this way only 1 backward pass is necessary. Thus $g_{ij}(x_k) = \cfrac{\partial{[L_2^2(F(x_k;\theta))]}}{\partial \theta_{ij}}$. It is verified empirically in the paper that using the $L_2$ norm has the same performance.

The loss function for learning the $n^{\text{th}}$ task is given by

\begin{equation}
    \mathcal{L}(\theta) = \mathcal{L_n}(\theta) + \lambda \sum_{i,j} \Omega_{ij}(\theta_{ij} - \theta^*_{ij})^2
\end{equation}

where $\lambda$ is the regularization strength, $\theta^*_{ij}$ is the parameters learned till task $n-1$ and $\mathcal{L_n}(\theta)$ is the task specific loss for task $n$    . The importance term $\Omega$ is accumulated (summed, or averaged?) with the previous $\Omega$.

## Specifics
* 3 kinds of evaluations:
    - 2 task sequences using 3 different **image classification** datasets, comparing 6 methods
    - Longer sequence of 8 tasks using 8 different classification datasets, comparing 7 methods
    - 4 task sequence using different groups of a **fact-learning** dataset (6DS), comparing 4 methods (including joint training and finetuning)
* Network types:
    - For image classification experiments: AlexNet (PyTorch model has 61.1 million parameters). Multi-headed architecture is used (one head per task).
    - For fact learning task, a VGG-16 network (PyTorch model has 138.4 million parameters) is used in which all parameters are shared (not multi-headed for tasks). The network has 3 heads corresponding to the 3 outputs that are needed for all tasks.
* Global MAS is used for experiments.
* Importance weights are estimated using training data.
* $\lambda=1$ is used.

```python
import numpy as np
import random
from sklearn.metrics import accuracy_score
from tqdm.notebook import trange, tqdm
import matplotlib.pyplot as plt
%matplotlib inline

import torch
import torch.nn as nn
from torch.nn import Sequential
import torch.optim as optim
import torch.nn.functional as F
from torch.utils.data import DataLoader

from continuum import Rotations, Permutations
from continuum.datasets import MNIST

import time
```

```python
def check_cuda():
    """
    Checks if GPU is available.
    """
    cuda_available = torch.cuda.is_available()
    device = torch.device('cuda' if cuda_available else 'cpu')
    return cuda_available, device

def set_seed(seed=1000):
    """
    Sets the seed for reproducability
    """
    torch.manual_seed(seed)
    random.seed(seed)
    np.random.seed(seed)

    cuda_available, _ = check_cuda()
    if cuda_available:
        torch.backends.cudnn.benchmark = False
        torch.use_deterministic_algorithms(True)
        torch.backends.cudnn.deterministic = True
        torch.cuda.manual_seed_all(seed)
```

```python
def get_cl_metrics(arr):
    """
    Given an array of validation accuracies (each current task along the rows,
    and accuracy of the tasks in the columns), this function computes the
    CL metrics according to the following paper:
    Don't forget, there is more than forgetting: new metrics for Continual Learning,
    Natalia Díaz-Rodríguez et al. [https://arxiv.org/pdf/1810.13166.pdf]

    We are using our own function instead of the Logger from continvvm because of
    a lack of sufficient documentation, and to illustrate how the CL metrics can
    be computed.

    The following metrics are computed:
        - Accuracy,
        - Backward Transfer,
        - BWT+,
        - REM,
        - FWT

    Args:
        arr (Numpy array): Array of validation accuracies

    Returns:
        cl_metrics (dict): Dictionary of CL metrics acc, bwt, bwt_plus, rem, fwt
    """

    n = arr.shape[0]

    # Accuracy considers the average accuracy by considering the diagonal
    # elements as well as all elements below it
    # This is equivalent to computing the sum of the lower traingular matrix
    # and dividing that sum by N(N+1)/2
    acc = np.sum(np.tril(arr))/(n*(n+1)/2.0)

    # Backward transfer (BWT)
    bwt = 0.0
    for i in range(1, n):
        for j in range(0, i):
            bwt += (arr[i,j] - arr[j,j])
    bwt /= (n*(n-1)/2.0)

    rem = 1.0 - np.abs(np.min([bwt, 0.0]))
    bwt_plus = np.max([bwt, 0.0])

    # Forward Transfer:  FWT measures the influence that learning a task has
    # on the performance of future tasks
    # This is the average accuracy for the train-test accuracy entries
    # above the principal diagonal, excluding it
    # This is equivalent to the sum of the upper tril minus the trace
    fwt = (np.sum(np.triu(arr)) - np.trace(arr))/(n*(n-1)/2.0)

    cl_metrics = {'acc': acc, 'bwt': bwt, 'bwt_plus': bwt_plus, 'rem': rem, 'fwt': fwt}
    return cl_metrics
```

```python
def get_avg_acc(arr):
    """
    Computes the average accuracies over validation tasks

    Args:
        arr (Numpy array): Array of validation accuracies (each current task
                           along the rows,and accuracy of the tasks in the columns)

    Returns:
        avg_acc_curr_past (Numpy array):
        avg_acc_all (Numpy array):
    """

    (nb_tasks, _) = arr.shape

    avg_acc_curr_past = np.zeros((nb_tasks,))
    avg_acc_all = np.zeros((nb_tasks,))

    for curr_task in range(nb_tasks):
        avg_acc_curr_past[curr_task] = np.mean(arr[curr_task,:curr_task+1])
        avg_acc_all[curr_task] = np.mean(arr[curr_task,:])

    return avg_acc_curr_past, avg_acc_all
```

```python
class MemoryAwareSynapsesLearner(nn.Module):

    def __init__(self, mas_lambda, lr, model=None):
        """
        Initialize the MemoryAwareSynapsesLearner object.
        Creates the network, sets the MAS hyperparameters and
        initializes the buffer objects required for updates.

        Args:
            mas_lambda (float): Regularization strength parameters
            lr (float): Learning rate for optimizer
            model (torch.nn.module): The network model

        """
        super().__init__()

        # Set the network
        if model is not None:
            self.model = model
        else:
            self.model = Sequential(nn.Linear(784, 2000),
                         nn.ReLU(),
                         nn.Linear(2000,2000),
                         nn.ReLU(),
                         nn.Linear(2000,10))

        # Set the MAS hyperparameters
        self.mas_lambda = mas_lambda

        # Optimizer hyperparameters
        self.lr = lr

        # Setting buffers for regularization parameters
        for n, p in self.model.named_parameters():
            if p.requires_grad:
                n = n.replace('.', '__')
                self.register_buffer(f'{n}_theta_prev', p.data.detach().clone().zero_())
                self.register_buffer(f'{n}_omega_curr', p.data.detach().clone().zero_())
                self.register_buffer(f'{n}_omega_prev', p.data.detach().clone().zero_())

        # Set the optimizer
        self.reset_optimizer()

    def reset_optimizer(self):
        """
        Resets the optimizer
        """
        self.optimizer = optim.Adam(self.model.parameters(), self.lr)

    def update_omega(self, dataloader, device='cpu', timeit=False):
        """
        Updates the per-parameter importance once training is complete for a task.

        Here, we compute the per-sample gradient by iterating over a dataloader
        with a bach size of 1, but this is painfully slow.
        More efficient implements are possible. For example,
        see: [BackPACK](https://github.com/f-dangel/backpack). However,
        further tests are needed for more complex architectures.
        """
        start = time.time()

        # Find the number of data points
        datalen = len(dataloader.dataset)

        # Find the number of batches
        num_batches = len(dataloader)

        # We need per sample gradients
        # Make sure that the batch size is 1
        # that is the number of batches should equal the number of data points
        assert num_batches == datalen, 'Batch-size must be 1 for per-sample gradients'

        # Set the model to eval
        self.model = self.model.to(device)
        self.model.eval()

        for x, _, _ in tqdm(dataloader):

            # Need to convert to cuda
            # How does this work in PL?
            x = x.to(device)

            # Reset all gradients of the model
            self.model.zero_grad()

            # Forward pass to compute the output
            x = torch.flatten(x, 1)
            output = self.model(x)

            # L2 norm of the output
            l2_norm = torch.norm(output, p=2, dim=1)

            # Square the L2 norm and sum
            sum_sq_l2_norm = torch.sum(l2_norm**2)

            # Compute the gradients (backward pass)
            sum_sq_l2_norm.backward()

            # Loop over all parameters
            for n, p in self.model.named_parameters():
                if p.requires_grad:
                    n = n.replace('.', '__')

                    # Retrieve current omega for this parameter
                    try:
                        omega_curr = getattr(self, f'{n}_omega_curr')
                    except AttributeError:
                        omega_curr = p.detach().clone().zero_()

                    # Retrieve the absolute current gradient for this parameter
                    p_abs_grad = torch.abs(p.grad.detach().clone())

                    # Update the current omega for this parameter
                    # The gradient is normalized by the number of datapoints
                    omega_curr = torch.add(omega_curr, p_abs_grad, alpha=1.0/datalen)

                    # Update the current omega in the buffer
                    self.register_buffer(f'{n}_omega_curr', omega_curr)

        end = time.time()
        if timeit:
            print(f'Total time for omega update: {end-start}')

    def merge_omega(self):
        """
        Once omega for the current task has been computed, merge this
        with the omega computed for the previous tasks
        """

        # Loop over all parameters
        for n, p in self.model.named_parameters():
            if p.requires_grad:
                n = n.replace('.', '__')

                # Retrieve previous omega for this parameter
                try:
                    omega_prev = getattr(self, f'{n}_omega_prev')
                except AttributeError:
                    omega_prev = p.detach().clone().zero_()

                # Retrieve current omega for this parameter
                omega_curr = getattr(self, f'{n}_omega_curr')

                # Update the current omega
                omega_curr = torch.add(omega_prev, omega_curr)
                self.register_buffer(f'{n}_omega_prev', omega_curr)

    def update_theta_prev(self):
        """
        Once training is over for the current task, the current
        parameter values need to be saved
        """
        for n, p in self.model.named_parameters():
            if p.requires_grad:
                n = n.replace('.', '__')
                self.register_buffer(f'{n}_theta_prev', p.data.detach().clone())

    def regularization_loss(self):
        """
        Computes the regularization loss in MAS using the per-parameter importance and
        the current and previous parameter values.
        """
        try:
            losses = []
            for n, p in self.model.named_parameters():
                if p.requires_grad:
                    # Retrieve previous parameter values and the corresponding omega
                    n = n.replace('.', '__')
                    prev_values = getattr(self, f'{n}_theta_prev')
                    omega_prev = getattr(self, f'{n}_omega_prev')
                    # Calculate MAS's surrogate loss, sum over all parameters
                    losses.append((omega_prev * (p-prev_values)**2).sum())
            return sum(losses)
        except AttributeError:
            # Regularization loss is 0 if there is no stored omega yet
            return torch.tensor(0.)

    def train_step(self, x, y, t):
        """
        Single batch of training

        Args:
            x (torch tensor): Batch of inputs
            y (torch tensor): Batch of labels
            t (torch tensor): Batch of task ids
        """

        # Zero the parameter gradients
        self.optimizer.zero_grad()

        # Perform a forward pass and compute the cross entropy loss
        x = torch.flatten(x, 1)
        y_hat = self.model(x)
        loss = F.cross_entropy(input=y_hat, target=y, reduction='mean')

        # Compute the MAS regularization loss
        regularization_loss = self.regularization_loss()
        loss += self.mas_lambda * regularization_loss

        # Backprop and gradient step
        loss.backward()
        self.optimizer.step()

        # Compute the training accuracy
        pred = torch.argmax(y_hat, dim=-1)
        acc = accuracy_score(y.detach().cpu().numpy(), pred.detach().cpu().numpy(), normalize=True)

        # Return the training accuracy
        return acc

    def val_step(self, x, y, t):
        """
        Single batch of validation

        Args:
            x (torch tensor): Batch of inputs
            y (torch tensor): Batch of labels
            t (torch tensor): Batch of task ids
        """

        with torch.no_grad():

            # Forward pass
            x = torch.flatten(x, 1)
            y_hat = self.model(x)

            # Compute the validation accuracy
            pred = torch.argmax(y_hat, dim=-1)
            acc = accuracy_score(y.detach().cpu().numpy(), pred.detach().cpu().numpy(), normalize=True)

        return acc
```

```python
def train_val(model,
              list_degrees,
              train_dataset,
              val_dataset,
              num_epochs,
              batch_size,
              seeds,
              mnist_type='Rotated',
              num_tasks=None,
              reg=True,
              device='cpu'):
    """
    Complete training and validation loop over all tasks.

    Once training is complete for a task, validation is performed using the validation
    datasets of all past, current and future tasks. This is needed for computing the
    CL metrics.

    Args:
        model (MemoryAwareSynapsesLearner): The learner class
        list_degrees (list): List of angles in degrees for the RotatedMNIST tasks
        train_dataset (MNIST): Training MNIST Dataset object
        val_dataset (MNIST): Validation MNIST Dataset object
        num_epochs (int): Number of training epochs
        seeds (int): List of seeds for PMNIST
        mnist_type (str): Rotated or Permuted
        batch_size (int): Batch size for training
        device (str): Device to use for training ('cpu' or 'cuda')

    Returns:
        val_acc_arr (Numpy array): Array of validation accuracies (each current task
                                   along the rows,and accuracy of the tasks in the columns)
    """
    nb_tasks = len(list_degrees) if num_tasks is None else num_tasks

    # Array to store validation accuracies (each current task along the rows,
    # and accuracy of the tasks in the columns)
    val_acc_arr = np.zeros((nb_tasks, nb_tasks))

    if mnist_type == 'Rotated':
        train_scenario = Rotations(cl_dataset=train_dataset,
                                   nb_tasks=nb_tasks,
                                   list_degrees=list_degrees)

        val_scenario = Rotations(cl_dataset=val_dataset,
                                 nb_tasks=nb_tasks,
                                 list_degrees=list_degrees)

    elif mnist_type == 'Permuted':
        train_scenario = Permutations(cl_dataset=train_dataset,
                                      nb_tasks=nb_tasks,
                                      seed=seeds,
                                      shared_label_space=True)

        val_scenario = Permutations(cl_dataset=val_dataset,
                                    nb_tasks=nb_tasks,
                                    seed=seeds,
                                    shared_label_space=True)

    for task_id, train_taskset in enumerate(train_scenario):

        print(f'Starting task_id: {task_id}')

        train_loader = DataLoader(train_taskset, batch_size=batch_size, shuffle=True)

        for epoch in range(num_epochs):

            print(f'Starting epoch: {epoch}')

            model.train()

            train_accuracies = list()

            for x, y, t in tqdm(train_loader):
                train_accuracies.append(model.train_step(x.to(device), y.to(device), t.to(device)))

            train_acc = np.mean(train_accuracies)

            print(f'task_id: {task_id}, epoch: {epoch}, train_accuracy: {train_acc}')

        if reg:
            # Update omega on ending a task
            # We need to pass a dataloader with a batch size of 1
            data_loader_omega = DataLoader(val_scenario[task_id], batch_size=1, shuffle=False)
            model.update_omega(data_loader_omega, device=device)

            # Save the parameters
            model.update_theta_prev()

            # Merge with previous omega if there were previous tasks
            if task_id > 0:
                model.merge_omega()

        # At the end of the epoch validate on all tasks (past, current, and future tasks)
        model.eval()
        for val_task_id, val_taskset in enumerate(val_scenario):
            val_loader = DataLoader(val_taskset, batch_size=1, shuffle=False)
            val_accuracies = list()

            for x, y, t in val_loader:
                val_accuracies.append(model.val_step(x.to(device), y.to(device), t.to(device)))

            val_acc = np.mean(val_accuracies)

            val_acc_arr[task_id][val_task_id] = val_acc

            print(f'Current task: {task_id}, Validating on task: {val_task_id}, Accuracy: {val_acc}')

    return val_acc_arr
```

## Rotated MNIST (5 tasks)

```python
# Setting hyperparameters
NUM_EPOCHS = 1  # instead of 20 (in the paper)
MAS_LAMBDA = 1.0
LEARNING_RATE = 0.001
BATCH_SIZE = 256
```

```python
# Check if CUDA is available
cuda_available, device = check_cuda()
print(f'cuda_available: {cuda_available}, device: {device}')
```

    cuda_available: False, device: cpu

```python
# Create the Dataset objects
list_degrees = [0, 45, 90, 135, 180]
train_dataset = MNIST(data_path="dataset", download=True, train=True)
val_dataset = MNIST(data_path="dataset", download=True, train=False)
```

```python
set_seed()

# Create the learner object
mas_model = MemoryAwareSynapsesLearner(mas_lambda=MAS_LAMBDA, lr=LEARNING_RATE)
mas_model = mas_model.to(device)
```

```python
# Train using MAS
val_acc_arr = train_val(model=mas_model,
                        list_degrees=list_degrees,
                        train_dataset=train_dataset,
                        val_dataset=val_dataset,
                        num_epochs=NUM_EPOCHS,
                        batch_size=BATCH_SIZE)
```

    Starting task_id: 0
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 0, train_accuracy: 0.9314937943262412

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 0, Validating on task: 0, Accuracy: 0.9729
    Current task: 0, Validating on task: 1, Accuracy: 0.3472
    Current task: 0, Validating on task: 2, Accuracy: 0.1639
    Current task: 0, Validating on task: 3, Accuracy: 0.1865
    Current task: 0, Validating on task: 4, Accuracy: 0.2796
    Starting task_id: 1
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 0, train_accuracy: 0.9427526595744681

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 1, Validating on task: 0, Accuracy: 0.709
    Current task: 1, Validating on task: 1, Accuracy: 0.9735
    Current task: 1, Validating on task: 2, Accuracy: 0.3916
    Current task: 1, Validating on task: 3, Accuracy: 0.142
    Current task: 1, Validating on task: 4, Accuracy: 0.2616
    Starting task_id: 2
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 0, train_accuracy: 0.9028257978723404

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 2, Validating on task: 0, Accuracy: 0.3755
    Current task: 2, Validating on task: 1, Accuracy: 0.8003
    Current task: 2, Validating on task: 2, Accuracy: 0.9523
    Current task: 2, Validating on task: 3, Accuracy: 0.3657
    Current task: 2, Validating on task: 4, Accuracy: 0.1527
    Starting task_id: 3
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 0, train_accuracy: 0.85823359929078

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 3, Validating on task: 0, Accuracy: 0.3839
    Current task: 3, Validating on task: 1, Accuracy: 0.682
    Current task: 3, Validating on task: 2, Accuracy: 0.8351
    Current task: 3, Validating on task: 3, Accuracy: 0.9215
    Current task: 3, Validating on task: 4, Accuracy: 0.3584
    Starting task_id: 4
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 0, train_accuracy: 0.8510693705673759

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 4, Validating on task: 0, Accuracy: 0.3166
    Current task: 4, Validating on task: 1, Accuracy: 0.4687
    Current task: 4, Validating on task: 2, Accuracy: 0.4883
    Current task: 4, Validating on task: 3, Accuracy: 0.7157
    Current task: 4, Validating on task: 4, Accuracy: 0.9292

```python
# Compute the CL metrics
cl_metrics = get_cl_metrics(val_acc_arr)

# Compute the average accuracy (over other tasks) for each current task
avg_acc_curr_past, avg_acc_all = get_avg_acc(val_acc_arr)
```

```python
set_seed()

# Train another model without any continual learning
mas_model_no_cl = MemoryAwareSynapsesLearner(mas_lambda=0.0, lr=LEARNING_RATE)
mas_model_no_cl = mas_model_no_cl.to(device)
val_acc_arr_no_cl = train_val(model=mas_model_no_cl,
                              list_degrees=list_degrees,
                              train_dataset=train_dataset,
                              val_dataset=val_dataset,
                              num_epochs=NUM_EPOCHS,
                              batch_size=BATCH_SIZE,
                              reg=False,
                              device=device)
```

    Starting task_id: 0
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 0, train_accuracy: 0.9314937943262412
    Current task: 0, Validating on task: 0, Accuracy: 0.9729
    Current task: 0, Validating on task: 1, Accuracy: 0.3472
    Current task: 0, Validating on task: 2, Accuracy: 0.1639
    Current task: 0, Validating on task: 3, Accuracy: 0.1865
    Current task: 0, Validating on task: 4, Accuracy: 0.2796
    Starting task_id: 1
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 0, train_accuracy: 0.94182734929078
    Current task: 1, Validating on task: 0, Accuracy: 0.685
    Current task: 1, Validating on task: 1, Accuracy: 0.9653
    Current task: 1, Validating on task: 2, Accuracy: 0.3718
    Current task: 1, Validating on task: 3, Accuracy: 0.1271
    Current task: 1, Validating on task: 4, Accuracy: 0.2511
    Starting task_id: 2
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 0, train_accuracy: 0.9448581560283689
    Current task: 2, Validating on task: 0, Accuracy: 0.2556
    Current task: 2, Validating on task: 1, Accuracy: 0.6963
    Current task: 2, Validating on task: 2, Accuracy: 0.9718
    Current task: 2, Validating on task: 3, Accuracy: 0.4062
    Current task: 2, Validating on task: 4, Accuracy: 0.1608
    Starting task_id: 3
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 0, train_accuracy: 0.9452681737588652
    Current task: 3, Validating on task: 0, Accuracy: 0.2435
    Current task: 3, Validating on task: 1, Accuracy: 0.2795
    Current task: 3, Validating on task: 2, Accuracy: 0.7516
    Current task: 3, Validating on task: 3, Accuracy: 0.9744
    Current task: 3, Validating on task: 4, Accuracy: 0.4547
    Starting task_id: 4
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 0, train_accuracy: 0.9514738475177306
    Current task: 4, Validating on task: 0, Accuracy: 0.29
    Current task: 4, Validating on task: 1, Accuracy: 0.2847
    Current task: 4, Validating on task: 2, Accuracy: 0.3746
    Current task: 4, Validating on task: 3, Accuracy: 0.7678
    Current task: 4, Validating on task: 4, Accuracy: 0.9728

```python
# Compute the CL metrics
cl_metrics_no_cl = get_cl_metrics(val_acc_arr_no_cl)

# Compute the average accuracy (over other tasks) for each current task
avg_acc_curr_past_no_cl, avg_acc_all_no_cl = get_avg_acc(val_acc_arr_no_cl)
```

```python
# Compare the results

# Line plots for average accuracy

fig, ax = plt.subplots(1,2,figsize=(20,5))

ax[0].plot(avg_acc_curr_past, lw=2, color='blue', marker='o', label='MAS')
ax[0].plot(avg_acc_curr_past_no_cl, lw=2, color='red', marker='o', label='FT')

ax[1].plot(avg_acc_all, lw=2, color='blue', marker='o', label='MAS')
ax[1].plot(avg_acc_all_no_cl, lw=2, color='red', marker='o', label='FT')

ax[0].set_title("Average accuracy over current and past tasks")
ax[1].set_title("Average accuracy over current, past, and future tasks")
for a in ax:
    a.set_xlabel("Current task")
    a.set_ylabel("Accuracy")
    a.set_ylim([0.0,1.0])
    a.grid(True)
    a.legend()
```

```python
# Bar plots for CL metrics

fig, ax = plt.subplots(1,1,figsize=(10,5))
width = 0.2

ax.bar(np.arange(4)-width/2.0,
      (cl_metrics['acc'], cl_metrics['bwt_plus'], cl_metrics['rem'], cl_metrics['fwt']),
       width=width,
       label='MAS',
       color='blue')
ax.bar(np.arange(4)+width/2.0,
      (cl_metrics_no_cl['acc'], cl_metrics_no_cl['bwt_plus'], cl_metrics_no_cl['rem'], cl_metrics_no_cl['fwt']),
       width=width,
       label='FT',
       color='red')

ax.grid(True)
ax.set_xticks([0,1,2,3])
ax.set_xticklabels(['acc', 'bwt_plus', 'rem', 'fwt'])
ax.set_xlabel('CL Metrics')
ax.set_ylabel('Metric score')
ax.set_ylim([0.0, 1.0])
ax.legend()
_ = ax.set_title('CL Metrics')
```

## Permuted MNIST (5 tasks)

```python
# Setting hyperparameters
SEED = 1000
NUM_EPOCHS = 5
MAS_LAMBDA = 1.0
LEARNING_RATE = 0.001
BATCH_SIZE = 256
SEEDS = [100, 200, 300, 400]

set_seed(SEED)

# Check if CUDA is available
cuda_available, device = check_cuda()
print(f'cuda_available: {cuda_available}, device: {device}')

train_dataset = MNIST(data_path="dataset", download=True, train=True)
val_dataset = MNIST(data_path="dataset", download=True, train=False)

# Create the learner object
mas_model = MemoryAwareSynapsesLearner(mas_lambda=MAS_LAMBDA, lr=LEARNING_RATE)
mas_model = mas_model.to(device)

# Train using MAS
val_acc_arr = train_val(model=mas_model,
                        list_degrees=None,
                        train_dataset=train_dataset,
                        val_dataset=val_dataset,
                        num_epochs=NUM_EPOCHS,
                        batch_size=BATCH_SIZE,
                        seeds=SEEDS,
                        mnist_type='Permuted',
                        num_tasks=5)

# Compute the CL metrics
cl_metrics = get_cl_metrics(val_acc_arr)

# Compute the average accuracy (over other tasks) for each current task
avg_acc_curr_past, avg_acc_all = get_avg_acc(val_acc_arr)
```

    cuda_available: False, device: cpu
    Starting task_id: 0
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 0, train_accuracy: 0.9314937943262412
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 1, train_accuracy: 0.9759363918439715
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 2, train_accuracy: 0.9847129875886524
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 3, train_accuracy: 0.9895777925531914
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 4, train_accuracy: 0.9906471631205673

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 0, Validating on task: 0, Accuracy: 0.9771
    Current task: 0, Validating on task: 1, Accuracy: 0.0848
    Current task: 0, Validating on task: 2, Accuracy: 0.0624
    Current task: 0, Validating on task: 3, Accuracy: 0.0756
    Current task: 0, Validating on task: 4, Accuracy: 0.0756
    Starting task_id: 1
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 0, train_accuracy: 0.9358820921985816
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 1, train_accuracy: 0.9801252216312056
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 2, train_accuracy: 0.988031914893617
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 3, train_accuracy: 0.991373005319149
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 4, train_accuracy: 0.9945478723404255

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 1, Validating on task: 0, Accuracy: 0.8526
    Current task: 1, Validating on task: 1, Accuracy: 0.9831
    Current task: 1, Validating on task: 2, Accuracy: 0.1093
    Current task: 1, Validating on task: 3, Accuracy: 0.1092
    Current task: 1, Validating on task: 4, Accuracy: 0.0511
    Starting task_id: 2
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 0, train_accuracy: 0.9209275265957447
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 1, train_accuracy: 0.9698803191489361
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 2, train_accuracy: 0.9787123226950354
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 3, train_accuracy: 0.984624335106383
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 4, train_accuracy: 0.987605274822695

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 2, Validating on task: 0, Accuracy: 0.8428
    Current task: 2, Validating on task: 1, Accuracy: 0.9783
    Current task: 2, Validating on task: 2, Accuracy: 0.9743
    Current task: 2, Validating on task: 3, Accuracy: 0.0861
    Current task: 2, Validating on task: 4, Accuracy: 0.073
    Starting task_id: 3
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 0, train_accuracy: 0.9136081560283689
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 1, train_accuracy: 0.9655141843971631
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 2, train_accuracy: 0.9753823138297872
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 3, train_accuracy: 0.9805851063829787
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 4, train_accuracy: 0.9854388297872341

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 3, Validating on task: 0, Accuracy: 0.8485
    Current task: 3, Validating on task: 1, Accuracy: 0.9734
    Current task: 3, Validating on task: 2, Accuracy: 0.9641
    Current task: 3, Validating on task: 3, Accuracy: 0.9734
    Current task: 3, Validating on task: 4, Accuracy: 0.1035
    Starting task_id: 4
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 0, train_accuracy: 0.9064771719858157
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 1, train_accuracy: 0.9580064273049647
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 2, train_accuracy: 0.9687444592198582
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 3, train_accuracy: 0.9744071365248226
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 4, train_accuracy: 0.9796154698581561

      0%|          | 0/10000 [00:00<?, ?it/s]

    Current task: 4, Validating on task: 0, Accuracy: 0.8301
    Current task: 4, Validating on task: 1, Accuracy: 0.9684
    Current task: 4, Validating on task: 2, Accuracy: 0.9549
    Current task: 4, Validating on task: 3, Accuracy: 0.9649
    Current task: 4, Validating on task: 4, Accuracy: 0.9705

```python
set_seed(SEED)

# Train another model without any continual learning
mas_model_no_cl = MemoryAwareSynapsesLearner(mas_lambda=0.0, lr=LEARNING_RATE)
mas_model_no_cl = mas_model_no_cl.to(device)
val_acc_arr_no_cl = train_val(model=mas_model_no_cl,
                        list_degrees=None,
                        train_dataset=train_dataset,
                        val_dataset=val_dataset,
                        num_epochs=NUM_EPOCHS,
                        batch_size=BATCH_SIZE,
                        seeds=SEEDS,
                        mnist_type='Permuted',
                        num_tasks=5,
                        reg=False)

# Compute the CL metrics
cl_metrics_no_cl = get_cl_metrics(val_acc_arr_no_cl)

# Compute the average accuracy (over other tasks) for each current task
avg_acc_curr_past_no_cl, avg_acc_all_no_cl = get_avg_acc(val_acc_arr_no_cl)
```

    Starting task_id: 0
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 0, train_accuracy: 0.9314937943262412
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 1, train_accuracy: 0.9759363918439715
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 2, train_accuracy: 0.9847129875886524
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 3, train_accuracy: 0.9895777925531914
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 0, epoch: 4, train_accuracy: 0.9906471631205673
    Current task: 0, Validating on task: 0, Accuracy: 0.9771
    Current task: 0, Validating on task: 1, Accuracy: 0.0848
    Current task: 0, Validating on task: 2, Accuracy: 0.0624
    Current task: 0, Validating on task: 3, Accuracy: 0.0756
    Current task: 0, Validating on task: 4, Accuracy: 0.0756
    Starting task_id: 1
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 0, train_accuracy: 0.9379654255319149
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 1, train_accuracy: 0.9793882978723404
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 2, train_accuracy: 0.9881316489361702
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 3, train_accuracy: 0.9920434397163119
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 1, epoch: 4, train_accuracy: 0.994315159574468
    Current task: 1, Validating on task: 0, Accuracy: 0.8202
    Current task: 1, Validating on task: 1, Accuracy: 0.979
    Current task: 1, Validating on task: 2, Accuracy: 0.1092
    Current task: 1, Validating on task: 3, Accuracy: 0.0911
    Current task: 1, Validating on task: 4, Accuracy: 0.0655
    Starting task_id: 2
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 0, train_accuracy: 0.9315104166666668
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 1, train_accuracy: 0.9793384308510639
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 2, train_accuracy: 0.9882535460992907
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 3, train_accuracy: 0.992436835106383
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 2, epoch: 4, train_accuracy: 0.9954454787234043
    Current task: 2, Validating on task: 0, Accuracy: 0.7003
    Current task: 2, Validating on task: 1, Accuracy: 0.8944
    Current task: 2, Validating on task: 2, Accuracy: 0.9774
    Current task: 2, Validating on task: 3, Accuracy: 0.085
    Current task: 2, Validating on task: 4, Accuracy: 0.1173
    Starting task_id: 3
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 0, train_accuracy: 0.933200354609929
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 1, train_accuracy: 0.9801085992907801
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 2, train_accuracy: 0.9880097517730497
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 3, train_accuracy: 0.992891179078014
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 3, epoch: 4, train_accuracy: 0.9952626329787234
    Current task: 3, Validating on task: 0, Accuracy: 0.6218
    Current task: 3, Validating on task: 1, Accuracy: 0.8765
    Current task: 3, Validating on task: 2, Accuracy: 0.9299
    Current task: 3, Validating on task: 3, Accuracy: 0.9783
    Current task: 3, Validating on task: 4, Accuracy: 0.0867
    Starting task_id: 4
    Starting epoch: 0

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 0, train_accuracy: 0.9328845301418439
    Starting epoch: 1

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 1, train_accuracy: 0.9793162677304965
    Starting epoch: 2

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 2, train_accuracy: 0.9893062943262412
    Starting epoch: 3

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 3, train_accuracy: 0.9926640070921987
    Starting epoch: 4

      0%|          | 0/235 [00:00<?, ?it/s]

    task_id: 4, epoch: 4, train_accuracy: 0.9949800531914894
    Current task: 4, Validating on task: 0, Accuracy: 0.455
    Current task: 4, Validating on task: 1, Accuracy: 0.8381
    Current task: 4, Validating on task: 2, Accuracy: 0.9239
    Current task: 4, Validating on task: 3, Accuracy: 0.9334
    Current task: 4, Validating on task: 4, Accuracy: 0.9773

```python
# Compare the results

# Line plots for average accuracy

fig, ax = plt.subplots(1,2,figsize=(20,5))

ax[0].plot(avg_acc_curr_past, lw=2, color='blue', marker='o', label='MAS')
ax[0].plot(avg_acc_curr_past_no_cl, lw=2, color='red', marker='o', label='FT')

ax[1].plot(avg_acc_all, lw=2, color='blue', marker='o', label='MAS')
ax[1].plot(avg_acc_all_no_cl, lw=2, color='red', marker='o', label='FT')

ax[0].set_title("Average accuracy over current and past tasks")
ax[1].set_title("Average accuracy over current, past, and future tasks")
for a in ax:
    a.set_xlabel("Current task")
    a.set_ylabel("Accuracy")
    a.set_ylim([0.0,1.0])
    a.grid(True)
    a.legend()
```

```python
# Bar plots for CL metrics

fig, ax = plt.subplots(1,1,figsize=(10,5))
width = 0.2

ax.bar(np.arange(4)-width/2.0,
      (cl_metrics['acc'], cl_metrics['bwt_plus'], cl_metrics['rem'], cl_metrics['fwt']),
       width=width,
       label='MAS',
       color='blue')
ax.bar(np.arange(4)+width/2.0,
      (cl_metrics_no_cl['acc'], cl_metrics_no_cl['bwt_plus'], cl_metrics_no_cl['rem'], cl_metrics_no_cl['fwt']),
       width=width,
       label='FT',
       color='red')

ax.grid(True)
ax.set_xticks([0,1,2,3])
ax.set_xticklabels(['acc', 'bwt_plus', 'rem', 'fwt'])
ax.set_xlabel('CL Metrics')
ax.set_ylabel('Metric score')
ax.set_ylim([0.0, 1.0])
ax.legend()
_ = ax.set_title('CL Metrics')
```


# Announcements
- __Please familiarize yourself with the term projects, and sign up for your (preliminary) choice__ using [this form](https://forms.gle/ByLLpsthrpjCcxG89). _You may revise your choice, but I'd recommend settling on a choice well before Thanksgiving._
- Recommended reading on ODEs: [Lecture notes by Prof. Hjorth-Jensen (University of Oslo)](https://www.asc.ohio-state.edu/physics/ntg/6810/readings/hjorth-jensen_notes2013_08.pdf)
- Problem Set 5 will be posted on D2L on Oct 12, due Oct 20.
- Update to problem set due date: going forward, problem sets are due Tuesdays at 5 pm. I will stay on for 15 minutes after Monday's classes for last minute clarifications. _To make this work, familiarize yourself with the current problem set before office hours (Wed) or consultation hours (Fr) and ask your first round of questions_ **before Monday**.
- __Outlook__: high-dimensional liner and non-linear equations, then Boundary Value Problems and Partial Differential Equations.
- Conference for Undergraduate Women in Physics: online event in 2021, [applications accepted until 10/25](https://www.aps.org/programs/women/cuwip/)

[2020 Noble Prize in Physics](https://www.nobelprize.org/prizes/physics/2020/summary/) for theory and observation of black holes. Here's a [beautiful animation of stars orbiting the black hole in the center of our Milky Way](http://www.astro.ucla.edu/~ghezgroup/gc/animations.html) from the UCLA group.

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

# The gravitational constant G
G = 6.67428e-11
AU = (149.6e6 * 1000)     # 149.6 million km, in meters.
```

## Example: Planetary Motion in the Solar System
Today we will integrate the orbits of planets in the solar system interacting through gravity.

In general, the gravitational force on object $j$ with mass $m_j$ and position $\vec{r}_j$ through its gravitational interaction with objects $i$ with mass $m_i$ and position $\vec{x}_i$ is given by
$$
 \vec F_j = -\sum_{i\ne j}\frac{G m_j m_i}{|\vec x_i - \vec x_j|^2} \times \frac{\left(\vec x_i - \vec x_j\right)}{|\vec x_i - \vec x_j|}= m_j \vec a_j \tag{1}\label{FG}\,
$$
with the unit vector $\hat{\vec{x}} = \frac{\vec{x}}{|x|}$. _Note the direction of this force, gravity should pull two particles closer to each other! If that's not the case in your implementation, this is a good point to start..._

Here are approximate initial values for the inner four planets [(ref)](https://www.cs.princeton.edu/courses/archive/fall16/cos126/assignments/nbody.html)

Name |$p_x$ [m]|$p_y$ [m]|$v_x$ [m/s]| $v_y$ [m/s]| mass [kg]|
---- | ---- | ---- |---- | ----| ----|
Sun | 0.0000e+00 | 0.0000e+00 | 0.0000e+00 | 0.0000e+00 | 1.9890e+30
Mercury | 5.7900e+10 | 0.0000e+00 | 0.0000e+00 | 4.7900e+04 | 3.3020e+23
Venus | 1.0820e+11 | 0.0000e+00 | 0.0000e+00 | 3.5000e+04 | 4.8690e+24
Earth| 1.4960e+11 | 0.0000e+00 | 0.0000e+00 | 2.9800e+04 | 5.9740e+24
Mars | 2.2790e+11 | 0.0000e+00 | 0.0000e+00 | 2.4100e+04 | 6.4190e+23

We integrated the special case of a two-body problem in Lecture 16, in this case we only needed to keep track of the position and velocity of the separation vector. However, even in this simple case, passing different vectors around turned out to be cumbersome. To make keeping track of different data easier, we will use __Classes__ today.

_Classes_ are a way of grouping related bits of information together into a single unit (also known as an object), along with functions that can be called to manipulate that object (also known as methods). For example, if you want to track information about a solar system body, you might want to record its name, mass, position, etc., and be able to manipulate all of these as a single unit.

Before we start, it's important to understand the difference between a class and an object. A class is simply a description of what things should look like, what variables will be grouped together, and what functions can be called to manipulate those variables. Using this description, Python can then create many instances of the class (or objects), which can then be manipulated independently.

### Defining a Python Class

Let's start off by defining a simple class:

```python
class Foo:
    def __init__(self, val):
        self.val = val
    def printVal(self):
        print(self.val)
```

The class is called Foo and as usual, we use indentation to tell Python where the class definition starts and ends. In this example, the class definition consists of two function definitions (or methods), one called ``__init__`` and the other ``printVal``. There is also one member variable, not explicitly defined, but we'll explain below how it gets created. Using this definition, we can then create many instances of the class.

Both methods take a parameter called self. It doesn't have to be called self but this is the Python convention. Because we can create many instances of a class, when a class method is called, it needs to know which instance it is working with, and that's what Python will pass in via the self parameter.

``__init__`` is a special method that is called whenever Python creates a new instance of the class. In our example, it accepts one parameter (other than the mandatory self), called ``val``, and takes a copy of it in a member variable, also called ``val``.  To differentiate between the val variable that was passed in as a parameter and the ``val`` class member variable, we prefix the latter with ``self``. So, in the statement

``self.val = val``

``self.val`` refers to the ``val`` member variable belonging to the class instance the method is being called for, while ``val`` refers to the parameter that was passed into the method.

Let's take a look at a simple example:

```python
obj1 = Foo(1)
```

This creates an instance of our class . Python automatically calls the ``__init__`` method for us, passing in the value we specified (1).

Let's create another one:

```python
obj2 = Foo(2)
```

Exactly the same thing happens, except that 2 gets passed in to the ``__init__`` method.
We now have two independent objects, with different values for the ``val`` member variable.

Here's a class ``Body`` that we'll use for solar system bodies in today's Problem:

```python
class Body():
    """Class representing a gravitationally-acting body, attributes for vVerlet integration

    Attributes:
        mass : mass in kg
        vx, vy: x, y velocities in m/s
        px, py: x, y positions in m
        ax, ay: acceleration
        color: python color name (for plotting)
    """
    def __init__(self):
        self.name = 'Body'
        self.mass = None
        self.vx = self.vy = 0.0
        self.px = self.py = 0.0
        self.ax = self.ay = 0.0
    def pairwise_attraction(self, other):
        """(Body): (Fx, Fy)

        Returns the force exerted upon this body by the other body.
        """
        # Report an error if the other object is the same as this one.
        if self is other:
            raise ValueError("Attraction of object %r to itself requested"
                             % self.name)

        # Compute the distance of the other body.
        r = np.array([other.px-self.px,other.py-self.py])
        rr = np.sum(r*r)
        rhat = r/np.sqrt(rr)
        # Report an error if the distance is zero; otherwise we'll
        # get a ZeroDivisionError exception further down.
        if rr == 0:
            raise ValueError("Collision between objects %r and %r"
                             % (self.name, other.name))

        # Compute the force of attraction, Eq. 1
        # this is a numpy array consisting of [F_x,F_y]
        F = G * self.mass * other.mass / rr *rhat
        return F[0], F[1]
```

Let's create a few example instances already:

```python
    sun = Body()
    sun.name = 'Sun'
    sun.color = 'red'
    sun.mass = 1.9890e+30

    earth = Body()
    earth.name = 'Earth'
    earth.color = 'blue'
    earth.mass = 5.9740e+24
    earth.px = 1.4960e+11
    earth.vy = 2.9800e+04

    venus = Body()
    venus.name = 'Venus'
    venus.color = 'yellow'
    venus.mass = 4.8685 * 10**24
    venus.px = 1.0820e+11
    venus.vy = 3.5000e+04
```

```python
def set_acceleration(bodies):
    for body in bodies:
    # Add up all of the forces exerted on 'body'.
        total_fx = total_fy = 0.0
        for other in bodies:
            # Don't calculate the body's attraction to itself
            if body is other:
                continue
            fx, fy = body.pairwise_attraction(other)
            total_fx += fx
            total_fy += fy
        body.ax = total_fx/body.mass
        body.ay = total_fy/body.mass
```

```python
def copy_acceleration(bodies):
    """Return a dictionary with current acceleration values for all bodies
    Parameter:
    bodies: list of Body instances

    Returns:
    a_current: dictonary, with a_current[body] = (body.ax,body.ay)
    """
    a_current ={}
    for body in bodies:
        a_current[body] = (body.ax, body.ay)
    return a_current

```

We will again integrate orbits using the __velocity form__ of the Verlet algorithm (Lecture 15)
\begin{align}
x_{n+1} &= x_n + v_n\delta t + {1\over 2}a_n h^2 + \mathcal{O}(h^3) \tag{2a}\\
a_{n+1} &= a(x_{n+1}) \label{eq:Vv}\tag{2b}\\
v_{n+1} &= v_n + {1\over 2}\left(a_n + a_{n+1}\right)h + \mathcal{O}(h^2).\tag{2c}
\end{align}

 Implement the velocity Verlet step below, this time using the ``Body`` class.

__Before you start coding, discuss in your breakout room group in which order to proceed - do you implement (2a-2c) for one body at a time, or do you complete each step (e.g., (2a)) for all bodies before moving on to the next step?__

```python
def vVerlet_Step(bodies,h):

    #implement velocity Verlet step here

    #copy the current acceleration of all bodies into dictionary a_current
    #to access the x-acceleration of body: a_current[body][0]
    #to access the y-acceleration of body: a_current[body][1]
    a_current = copy_acceleration(bodies)

    #advance position (Equation 2a)

    #now update acceleration based on new position (Equation 2b)

    #now body.ax/y contains acceleration based on updated position
    #next, advance velocities (Equation 2c)

```

```python
def loop(bodies,years):
    """([Body])

    loops through the simulation, updating the
    positions of all the provided bodies.
    """
    h = 24*3600  # One day
    steps = years*365
    trajectories = np.zeros((len(bodies),steps,2))
    for n, body in enumerate(bodies):
        trajectories[n,0] = np.array([body.px,body.py])/AU

    #initialize acceleration
    set_acceleration(bodies)
    #acceleration base on current (initial) position in now in body.ax/y
    for day in range(1,steps):
        #take one Verlet step
        vVerlet_Step(bodies,h)

        #save current configuration for visualization
        for n, body in enumerate(bodies):
            trajectories[n,day] = np.array([body.px,body.py])/AU

    #visualize the trajectories
    for n, body in enumerate(bodies):
        print("%r shown in %r"%(body.name,body.color))
        plt.plot(trajectories[n,::10,0],trajectories[n,::10,1],color=body.color,marker = 'x',markersize = 2.5)
    plt.xlabel("x [AU]")
    plt.ylabel("y [AU]")
    plt.show()

def main(years = 1):

    bodies =[sun,earth,venus]
    loop(bodies,years)

```

```python
main()
```

 If there's time left, add the Mars and Mercury!

```python

```


.. _nb_custom:

## Custom Variable Type

In the following, we describe a custom variable problem. The variable is a string with a fixed length in our case.
However, we formulate the problem to be easily extended to have a variable length.
The objective function looks as follows:

\begin{align}
\begin{split}
\max f_1(x) & = & \, \# a \\[2mm]
\max f_2(x) & = & \, \# b
\end{split}
\end{align}

The first objective is the number of a's in the string and the second the number of b's.
For instance, for the variable "abdfgdgabb" the $f_1(x)=2$ and $f_2(x)=3$.

```python
import numpy as np
from pymoo.model.problem import Problem

class MyProblem(Problem):

    def __init__(self, n_characters=10):
        super().__init__(n_var=1, n_obj=2, n_constr=0, elementwise_evaluation=True)
        self.n_characters = n_characters
        self.ALPHABET = [c for c in string.ascii_lowercase]

    def _evaluate(self, x, out, *args, **kwargs):
        n_a, n_b = 0, 0
        for c in x[0]:
            if c == 'a':
                n_a += 1
            elif c == 'b':
                n_b += 1

        out["F"] = np.array([- n_a, - n_b], dtype=float)

```

The problem definition above defines a problem with just one variable. This variable can be considered a complex object, which is, in our case, a string. The same principle can be used to use other data structures such as trees or lists with variable lengths. Because both objectives have to be maximized, we are minimizing their negative values.

To solve the optimization problem, evolutionary operators sampling, crossover, mutation, and duplication, check needs to be implemented.
Each of the modules will be shown in the following.

### Sampling

Our sampling method just generates a random string, which is equivalent to choosing a random letter from the alphabet (only lower case).
Because of the implementation of having only one variable, we return a matrix with the shape (n,1).

```python
from pymoo.model.sampling import Sampling

class MySampling(Sampling):

    def _do(self, problem, n_samples, **kwargs):
        X = np.full((n_samples, 1), None, dtype=np.object)

        for i in range(n_samples):
            X[i, 0] = "".join([np.random.choice(problem.ALPHABET) for _ in range(problem.n_characters)])

        return X

```

### Crossover

The crossover operator combines parents to create offsprings. In our framework, the crossover operator retrieves the input already with predefined matings.
Our crossover randomly picks a character from the first or the second parent.

```python
from pymoo.model.crossover import Crossover

class MyCrossover(Crossover):
    def __init__(self):

        # define the crossover: number of parents and number of offsprings
        super().__init__(2, 2)

    def _do(self, problem, X, **kwargs):

        # The input of has the following shape (n_parents, n_matings, n_var)
        _, n_matings, n_var = X.shape

        # The output owith the shape (n_offsprings, n_matings, n_var)
        # Because there the number of parents and offsprings are equal it keeps the shape of X
        Y = np.full_like(X, None, dtype=np.object)

        # for each mating provided
        for k in range(n_matings):

            # get the first and the second parent
            a, b = X[0, k, 0], X[1, k, 0]

            # prepare the offsprings
            off_a = ["_"] * problem.n_characters
            off_b = ["_"] * problem.n_characters

            for i in range(problem.n_characters):
                if np.random.random() < 0.5:
                    off_a[i] = a[i]
                    off_b[i] = b[i]
                else:
                    off_a[i] = b[i]
                    off_b[i] = a[i]

            # join the character list and set the output
            Y[0, k, 0], Y[1, k, 0] = "".join(off_a), "".join(off_b)

        return Y
```

### Mutation

The mutation needs to be implemented for our string object as well. We either change the order of the string (40%), randomly pick a new character with a given probability (40%), or leave the string unmodified (20%).

```python
from pymoo.model.mutation import Mutation

class MyMutation(Mutation):
    def __init__(self):
        super().__init__()

    def _do(self, problem, X, **kwargs):

        # for each individual
        for i in range(len(X)):

            r = np.random.random()

            # with a probabilty of 40% - change the order of characters
            if r < 0.4:
                perm = np.random.permutation(problem.n_characters)
                X[i, 0] = "".join(np.array([e for e in X[i, 0]])[perm])

            # also with a probabilty of 40% - change a character randomly
            elif r < 0.8:
                prob = 1 / problem.n_characters
                mut = [c if np.random.random() > prob
                       else np.random.choice(problem.ALPHABET) for c in X[i, 0]]
                X[i, 0] = "".join(mut)

        return X
```

### Duplicates

Moreover, do not underestimate the importance of filtering out duplicates during evolution. Duplicate elimination helps to make sure a mating produces an offspring that does not already exist in the current population. If it does, another mating process is triggered until unique individuals do exist.

```python
from pymoo.model.duplicate import ElementwiseDuplicateElimination

class MyDuplicateElimination(ElementwiseDuplicateElimination):

    def is_equal(self, a, b):
        return a.X[0] == b.X[0]

```

### Optimize

Finally, we create an algorithm object with all the modules defined above.

```python
import string
import numpy as np

from pymoo.algorithms.nsga2 import NSGA2
from pymoo.optimize import minimize

algorithm = NSGA2(pop_size=11,
                  sampling=MySampling(),
                  crossover=MyCrossover(),
                  mutation=MyMutation(),
                  eliminate_duplicates=MyDuplicateElimination())

res = minimize(MyProblem(),
               algorithm,
               ('n_gen', 100),
               seed=1,
               verbose=False)

```

    /Users/blankjul/anaconda3/envs/dev/lib/python3.7/site-packages/ipykernel_launcher.py:6: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe.
    Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations

    /Users/blankjul/anaconda3/envs/dev/lib/python3.7/site-packages/ipykernel_launcher.py:16: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe.
    Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
      app.launch_new_instance()

```python
from pymoo.visualization.scatter import Scatter
Scatter().add(res.F).show()
```

```python
results = res.X[np.argsort(res.F[:, 0])]
count = [np.sum([e == "a" for e in r]) for r in results[:, 0]]
print(np.column_stack([results, count]))
```

    [['aaaaaaaaaa' 10]
     ['aaaaaabaaa' 9]
     ['aaabaabaaa' 8]
     ['baaaaaaaba' 8]
     ['bbaaaaaaba' 7]
     ['babbaaaaba' 6]
     ['bbaabababa' 5]
     ['babbabaabb' 4]
     ['bbabbabbba' 3]
     ['baabbbbbbb' 2]
     ['babbbbbbbb' 1]]