```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir( os.path.join('..', 'notebook_format') )
from formats import load_style
load_style(css_style = 'custom2.css')
```

```python
os.chdir(path)
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2

import sys
from math import ceil
from tqdm import trange
from subprocess import call
from itertools import islice
from sklearn.metrics import roc_auc_score
from sklearn.preprocessing import normalize
from sklearn.neighbors import NearestNeighbors
from scipy.sparse import csr_matrix, dok_matrix

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,matplotlib,scipy,sklearn,tqdm
```

    Ethen 2017-03-31 20:22:16

    CPython 3.5.2
    IPython 5.3.0

    numpy 1.12.1
    pandas 0.19.2
    matplotlib 2.0.0
    scipy 0.19.0
    sklearn 0.18
    tqdm 4.11.2

If you are new to the field recommendation system, please make sure you understand the basics of matrix factorizaion in this [other documentation](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/recsys/1_ALSWR.ipynb).

# Bayesian Personalized Ranking

Recall that when doing matrix factorization for implicit feedback data (users' clicks, view times), we start with a user-item matrix, $R$ where nonzero elements of the matrix are the users' interaction with the items. And what matrix factorization does is it decomposes a large matrix into products of matrices, namely, $R=U×V$.

Matrix factorization assumes that:

- Each user can be described by $d$ features. For example, feature 1 might be a referring to how much each user likes disney movies.
- Each item, movie in this case, can be described by an analagous set of $d$ features. To correspond to the above example, feature 1 for the movie might be a number that says how close the movie is to a disney movie.

With that notion in mind, we can denote our $d$ feature user into math by letting a user $u$ take the form of a $1 \times d$-dimensional vector $\textbf{x}_{u}$. Similarly, an item *i* can be represented by a $1 \times d$-dimensional vector $\textbf{y}_{i}$. And we would predict the interactions that the user $u$ will have for item $i$ is by doing a dot product of the two vectors

\begin{align}
\hat r_{ui} &= \textbf{x}_{u} \textbf{y}_{i}^{T} = \sum\limits_{d} x_{ud}y_{di}
\end{align}

Where $\hat r_{ui}$ represents our prediction for the true interactions $r_{ui}$. Next, we will choose a objective function to minimize the square of the difference between all interactions in our dataset ($S$) and our predictions. This produces a objective function of the form:

\begin{align}
L &= \sum\limits_{u,i \in S}( r_{ui} - \textbf{x}_{u} \textbf{y}_{i}^{T} )^{2}
\end{align}

This is all well and good, but a lot of times, what we wish to optimize for is not the difference between the true interaction and the predicted interaction, but instead is the ranking of the items. Meaning given a user, what is the top-N most likely item that the user prefers. And this is what **Bayesian Personalized Ranking (BPR)** tries to accomplish. The idea is centered around sampling positive (items user has interacted with) and negative (items user hasn't interacted with) items and running pairwise comparisons.

## Formulation

Suppose $U$ is the set of all users and $I$ is the set of all items, our goal is to provide user $u$ with a personalized ranking, deonted by $>_u$. As mentioned in the last section, the usual approach for item recommenders is to predict a personalized score $\hat r_{ui}$ for an item that reflects the user's preference for that item. Then the items are ranked by sorting them according to that score and the top-N is recommended to the user.

Here we'll use a different approach by using item pairs as training data and optimize for correctly ranking item pairs. From $S$, the whole dataset we try to reconstruct for each user parts of $>_u$. If the user has interacted with item $i$, i.e. $(u,i) \in S$, then we assume that the user prefers this item over all other non-observed items. E.g. in the figure below user $u_1$ has interacted with item $i_2$ but not item $i_1$, so we assume that this user prefers item $i_2$ over $i_1$: $i_2 >_u i_1$. We will denote this generally as $i >_u j$, where the $i$ stands for the positive item and $j$ is for the negative item. For items that the user have both interacted with, we cannot infer any preference. The same is true for two items that a user has not interacted yet (e.g. item $i_1$ and $i_4$ for user $u_1$).

Given these information, we can now get to the Bayesian part of this method. Let $\Theta$ be the parameter of the model that determines the personalized ranking. BPR's goal is to maximize the posterior probability:

\begin{align}
p(\Theta | i >_u j ) \propto p( i >_u j |\Theta) p(\Theta)
\end{align}

$p( i >_u j |\Theta)$ is the likelihood function, it captures the individual probability that a user really prefers item $i$ over item $j$. We compute this probability with the form:

\begin{align}
p( i >_u j |\Theta) = \sigma \big(\hat r_{uij}(\Theta) \big)
\end{align}

Where: $\sigma$ is the good old logistic sigmoid:

\begin{align}
\sigma(x) = \frac{1}{1 + e^{-x}}
\end{align}

And $r_{uij}(\Theta)$ captures the relationship between user $u$, item $i$ and item $j$, which can be further decomposed into:

\begin{align}
\hat r_{uij} = \hat r_{ui} - \hat r_{uj}
\end{align}

For convenience we skiped the argument $\Theta$ from $\hat r_{uij}$. The formula above is basically saying what is the difference between the predicted interaction between the positive item $i$ and the negative item $j$. Now because of this generic framework, we can apply any standard collaborative filtering (such as the matrix factorization) techniques that can predict the interaction between user and item. Keep in mind that although it may seem like we're using the same models as in other work, here we're optimizing against another criterion as we do not try to predict a single predictor $\hat r_{ui}$ but instead tries to classify the difference of two predictions $\hat r_{ui} - \hat r_{uj}$. For those that interested in diving deeper, there's a section in the original paper that showed that the BPR optimization criteria is actually optimizing AUC (Area Under the ROC curve).

So far, we have only discussed the likelihood function. In order to complete the Bayesian modeling approach of the personalized ranking task, we introduce a general prior density $p(\Theta)$ which is a normal distribution with zero mean and variance-covariance matrix $\Sigma (\Theta)$, to reduce the number of unknown hyperparameters, we set: $\Sigma (\Theta) = \lambda_{\Theta} I$

To sum it all up, the full form of the maximum posterior probability optimization (called BPR-Opt in the paper) can be specified as:

\begin{align}
BPR-Opt &\implies \prod_{u, i, j} p( i >_u j |\Theta) p(\Theta) \\
&\implies ln \big( \prod_{u, i, j} p( i >_u j |\Theta) p(\Theta) \big) \\
&\implies \sum_{u, i, j} ln  \sigma \big(\hat r_{ui} - \hat r_{uj} \big) + ln  p(\Theta) \\
&\implies \sum_{u, i, j} ln  \sigma \big(\hat r_{ui} - \hat r_{uj} \big)
- \lambda_{\Theta} \left\Vert \Theta \right\Vert ^{2} \\
&\implies \sum_{u, i, j} ln  \sigma \big(\hat r_{ui} - \hat r_{uj} \big)
- \frac{\lambda_{\Theta}}{2} \left\Vert x_u \right\Vert ^{2}
- \frac{\lambda_{\Theta}}{2} \left\Vert y_i \right\Vert ^{2}
- \frac{\lambda_{\Theta}}{2} \left\Vert y_j \right\Vert ^{2} \\
&\implies \sum_{u, i, j} ln  \sigma \big( x_u y_i^T - x_u y_j^T \big)
- \frac{\lambda_{\Theta}}{2} \left\Vert x_u \right\Vert ^{2}
- \frac{\lambda_{\Theta}}{2} \left\Vert y_i \right\Vert ^{2}
- \frac{\lambda_{\Theta}}{2} \left\Vert y_j \right\Vert ^{2} \\
&\implies \sum_{u, i, j} ln  \frac{1}{1 + e^{-(x_u y_i^T - x_u y_j^T) }}
- \frac{\lambda_{\Theta}}{2} \left\Vert x_u \right\Vert ^{2}
- \frac{\lambda_{\Theta}}{2} \left\Vert y_i \right\Vert ^{2}
- \frac{\lambda_{\Theta}}{2} \left\Vert y_j \right\Vert ^{2}
\end{align}

Where:

- We first take the natural log (it's a monotonic transformation, meaning taking the log does not affect the optimization process) to make the product a sum to make life easier on us
- As for the $p(\Theta)$ part, recall that because for each parameter we assume that it's a normal distribution with mean zero ($\mu = 0$), and unit variance ($\Sigma = I$, we ignore the $\lambda_{\Theta}$ for now), the formula for it is:

\begin{align}
N(x \mid \mu, \Sigma)
&\implies \frac{1}{(2\pi)^{d/2}\sqrt{|\Sigma|}} exp(-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)) \\
&\implies \frac{1}{(2\pi)^{d/2}} exp(-\frac{1}{2}\Theta^{T}\Theta)
\end{align}

In the formula above, the only thing that depends on $\Theta$ is the $exp(-\frac{1}{2}\Theta^{T}\Theta)$ part on the right, the rest is just a multiplicative constant that we don't need to worry about, thus if we take the natural log of that formula, then the exponential goes away and our $p(\Theta)$ can be written as $- \frac{1}{2} \left\Vert \Theta \right\Vert ^{2}$, and we simply multiply the $\lambda_{\Theta}$ back, which can be seen as the model specific regularization parameter.

Last but not least, in machine learning it's probably more common to try and minimize things, thus we simply flip all the signs of the maximization formula above, leaving us with:

\begin{align}
argmin_{x_u, y_i, y_j} \sum_{u, i, j} -ln \frac{1}{1 + e^{-(x_u y_i^T - x_u y_j^T) }}
+ \frac{\lambda_{\Theta}}{2} \left\Vert x_u \right\Vert ^{2}
+ \frac{\lambda_{\Theta}}{2} \left\Vert y_i \right\Vert ^{2}
+ \frac{\lambda_{\Theta}}{2} \left\Vert y_j \right\Vert ^{2}
\end{align}

## Optimization

In the last section we have derived an optimization criterion for personalized ranking. As the criterion is differentiable, gradient descent based algorithms are an obvious choice for maximization. But standard gradient descent is probably not the right choice for our problem given the size of all the possible combinations of the triplet $(u, i, j)$. To solve for this issue we use a stochastic gradient descent algorithm that chooses the triplets randomly (uniformly distributed and bootstrapped sampled).

To solve for the function using gradient descent, we derive the gradient for the three parameters $x_u$, $y_i$, $y_j$ separately. Just a minor hint when deriving the gradient, remember that the first part of the formula requires the chain rule:

\begin{align}
\dfrac{\partial}{\partial x} ln \sigma(x)
&\implies \dfrac{1}{\sigma(x)} \dfrac{\partial}{\partial x} \sigma(x) \\
&\implies \left( 1 + e^{-x} \right) \dfrac{\partial}{\partial x} \sigma(x) \\
&\implies \left( 1 + e^{-x} \right) \dfrac{\partial}{\partial x} \left[ \dfrac{1}{1 + e^{-x}} \right] \\
&\implies \left( 1 + e^{-x} \right) \dfrac{\partial}{\partial x} \left( 1 + \mathrm{e}^{-x} \right)^{-1} \\
&\implies \left( 1 + e^{-x} \right) \cdot -(1 + e^{-x})^{-2}(-e^{-x}) \\
&\implies \left( 1 + e^{-x} \right) \dfrac{e^{-x}}{\left(1 + e^{-x}\right)^2} \\
&\implies \dfrac{e^{-x}}{1 + e^{-x}}
\end{align}

---

\begin{align}
\dfrac{\partial}{\partial x_u}
&\implies \dfrac{e^{-(x_u y_i^T - x_u y_j^T)}}{1 + e^{-(x_u y_i^T - x_u y_j^T)}} \cdot (y_j - y_i) + \lambda x_u
\end{align}

\begin{align}
\dfrac{\partial}{\partial y_i}
&\implies \dfrac{e^{-(x_u y_i^T - x_u y_j^T)}}{1 + e^{-(x_u y_i^T - x_u y_j^T)}} \cdot -y_i + \lambda y_i
\end{align}

\begin{align}
\dfrac{\partial}{\partial y_j}
&\implies \dfrac{e^{-(x_u y_i^T - x_u y_j^T)}}{1 + e^{-(x_u y_i^T - x_u y_j^T)}} \cdot y_j + \lambda y_j
\end{align}

After deriving the gradient the update for each parameter using gradient descent is simply:

\begin{align}
\Theta & \Leftarrow \Theta - \alpha \dfrac{\partial}{\partial \Theta}
\end{align}

Where $\alpha$ is the learning rate.

# Implementation

We will again use the movielens data as an example.

```python
file_dir = 'ml-100k'
file_path = os.path.join(file_dir, 'u.data')
if not os.path.isdir(file_dir):
    call(['curl', '-O', 'http://files.grouplens.org/datasets/movielens/' + file_dir + '.zip'])
    call(['unzip', file_dir + '.zip'])

# we will not be using the timestamp column
names = ['user_id', 'item_id', 'rating', 'timestamp']
df = pd.read_csv(file_path, sep = '\t', names = names)
print('data dimension: \n', df.shape)
df.head()
```

    data dimension:
     (100000, 4)

      user_id
      item_id
      rating
      timestamp

      0
      196
      242
      3
      881250949

      1
      186
      302
      3
      891717742

      2
      22
      377
      1
      878887116

      3
      244
      51
      2
      880606923

      4
      166
      346
      1
      886397596

Because BPR assumes binary implicit feedback (meaing there's only positive and negative items), here we'll assume that an item is positive only if he/she gave the item a ratings above 3 (feel free to experiment and change the threshold). The next few code chunks, creates the sparse interaction matrix and split into train and test set.

```python
def create_matrix(data, users_col, items_col, ratings_col, threshold = None):
    """
    creates the sparse user-item interaction matrix,
    if the data is not in the format where the interaction only
    contains the positive items (indicated by 1), then use the
    threshold parameter to determine which items are considered positive

    Parameters
    ----------
    data : DataFrame
        implicit rating data

    users_col : str
        user column name

    items_col : str
        item column name

    ratings_col : str
        implicit rating column name

    threshold : int, default None
        threshold to determine whether the user-item pair is
        a positive feedback

    Returns
    -------
    ratings : scipy sparse csr_matrix [n_users, n_items]
        user/item ratings matrix

    data : DataFrame
        the implict rating data that retains only the positive feedback
        (if specified to do so)
    """
    if threshold is not None:
        data = data[data[ratings_col] >= threshold]
        data[ratings_col] = 1

    for col in (items_col, users_col, ratings_col):
        data[col] = data[col].astype('category')

    ratings = csr_matrix(( data[ratings_col],
                           (data[users_col].cat.codes, data[items_col].cat.codes) ))
    ratings.eliminate_zeros()
    return ratings, data
```

```python
items_col = 'item_id'
users_col = 'user_id'
ratings_col = 'rating'
threshold = 3
X, df = create_matrix(df, users_col, items_col, ratings_col, threshold)
X
```

    <943x1574 sparse matrix of type '<class 'numpy.int64'>'
    	with 82520 stored elements in Compressed Sparse Row format>

```python
def create_train_test(ratings, test_size = 0.2, seed = 1234):
    """
    split the user-item interactions matrix into train and test set
    by removing some of the interactions from every user and pretend
    that we never seen them

    Parameters
    ----------
    ratings : scipy sparse csr_matrix [n_users, n_items]
        The user-item interactions matrix

    test_size : float between 0.0 and 1.0, default 0.2
        Proportion of the user-item interactions for each user
        in the dataset to move to the test set; e.g. if set to 0.2
        and a user has 10 interactions, then 2 will be moved to the
        test set

    seed : int, default 1234
        Seed for reproducible random splitting the
        data into train/test set

    Returns
    -------
    train : scipy sparse csr_matrix [n_users, n_items]
        Training set

    test : scipy sparse csr_matrix [n_users, n_items]
        Test set
    """
    assert test_size < 1.0 and test_size > 0.0

    # Dictionary Of Keys based sparse matrix is more efficient
    # for constructing sparse matrices incrementally compared with csr_matrix
    train = ratings.copy().todok()
    test = dok_matrix(train.shape)

    # for all the users assign randomly chosen interactions
    # to the test and assign those interactions to zero in the training;
    # when computing the interactions to go into the test set,
    # remember to round up the numbers (e.g. a user has 4 ratings, if the
    # test_size is 0.2, then 0.8 ratings will go to test, thus we need to
    # round up to ensure the test set gets at least 1 rating)
    rstate = np.random.RandomState(seed)
    for u in range(ratings.shape[0]):
        split_index = ratings[u].indices
        n_splits = ceil(test_size * split_index.shape[0])
        test_index = rstate.choice(split_index, size = n_splits, replace = False)
        test[u, test_index] = ratings[u, test_index]
        train[u, test_index] = 0

    train, test = train.tocsr(), test.tocsr()
    return train, test
```

```python
X_train, X_test = create_train_test(X, test_size = 0.2, seed = 1234)
X_train
```

    <943x1574 sparse matrix of type '<class 'numpy.int64'>'
    	with 65641 stored elements in Compressed Sparse Row format>

```python
class BPR:
    """
    Bayesian Personalized Ranking (BPR) for implicit feedback data

    Parameters
    ----------
    learning_rate : float, default 0.01
        learning rate for gradient descent

    n_factors : int, default 20
        Number/dimension of user and item latent factors

    n_iters : int, default 15
        Number of iterations to train the algorithm

    n_batch_size : int, default 2000
        batch size for batch gradient descent, the original paper
        uses stochastic gradient descent (i.e., batch size of 1),
        but this can make the training unstable (very sensitive to
        learning rate)

    reg : int, default 0.01
        Regularization term for the user and item latent factors

    seed : int, default 1234
        Seed for the randomly initialized user, item latent factors

    verbose : boolean, default True
        Whether to print progress bar while training

    Attributes
    ----------
    user_factors : 2d numpy array [n_users, n_factors]
        User latent factors learnt

    item_factors : 2d numpy array [n_items, n_factors]
        Item latent factors learnt

    Reference
    ---------
    S. Rendle, C. Freudenthaler, Z. Gantner, L. Schmidt-Thieme
    Bayesian Personalized Ranking from Implicit Feedback
    - https://arxiv.org/pdf/1205.2618.pdf
    """
    def __init__(self, learning_rate = 0.01, n_factors = 15, n_iters = 10,
                 n_batch_size = 1000, reg = 0.01, seed = 1234, verbose = True):
        self.reg = reg
        self.seed = seed
        self.verbose = verbose
        self.n_iters = n_iters
        self.n_factors = n_factors
        self.n_batch_size = n_batch_size
        self.learning_rate = learning_rate

    def fit(self, ratings):
        """
        Parameters
        ----------
        ratings : scipy sparse csr_matrix [n_users, n_items]
            sparse matrix of user-item interactions
        """
        indptr = ratings.indptr
        indices = ratings.indices
        n_users, n_items = ratings.shape

        # ensure batch size makes sense
        batch_size = self.n_batch_size
        if ratings.nnz < batch_size:
            batch_size = ratings.nnz
            sys.stderr.write('WARNING: Batch size is greater than number of training interactions,'
                             'switching to a batch size of {}\n'.format(X.nnz))

        batch_iters = ratings.nnz // batch_size

        # initialize random weights
        rstate = np.random.RandomState(self.seed)
        self.user_factors = rstate.normal(size = (n_users, self.n_factors))
        self.item_factors = rstate.normal(size = (n_items, self.n_factors))

        # progress bar for training iteration if verbose is turned on
        loop = range(self.n_iters)
        if self.verbose:
            loop = trange(self.n_iters, desc = self.__class__.__name__)

        for _ in loop:
            for _ in range(batch_iters):
                sampled = self._sample(n_users, n_items, indices, indptr)
                sampled_users, sampled_pos_items, sampled_neg_items = sampled
                self._update(sampled_users, sampled_pos_items, sampled_neg_items)

        # to avoid re-computation at predict
        self._predicted = False
        return self

    def _sample(self, n_users, n_items, indices, indptr):
        """sample batches of random triplets u, i, j"""
        sampled_pos_items = np.zeros(self.n_batch_size, dtype = np.int)
        sampled_neg_items = np.zeros(self.n_batch_size, dtype = np.int)
        sampled_users = np.random.choice(n_users, size = self.n_batch_size)

        for idx, user in enumerate(sampled_users):
            pos_items = indices[ indptr[user]:indptr[user + 1] ]
            pos_item = np.random.choice(pos_items)
            neg_item = np.random.choice(n_items)
            while neg_item in pos_items:
                neg_item = np.random.choice(n_items)

            sampled_pos_items[idx] = pos_item
            sampled_neg_items[idx] = neg_item

        return sampled_users, sampled_pos_items, sampled_neg_items

    def _update(self, u, i, j):
        """
        update according to the bootstrapped user u,
        positive item i and negative item j
        """
        user_u = self.user_factors[u]
        item_i = self.item_factors[i]
        item_j = self.item_factors[j]

        # decompose the estimator, compute the difference between
        # the score of the positive items and negative items
        r_ui = np.diag(user_u.dot(item_i.T))
        r_uj = np.diag(user_u.dot(item_j.T))
        r_uij = r_ui - r_uj
        sigmoid = np.exp(-r_uij) / (1.0 + np.exp(-r_uij))

        # repeat the 1 dimension sigmoid n_factors times so
        # the dimension will match when doing the update
        sigmoid_tiled = np.tile(sigmoid, (self.n_factors, 1)).T

        # update using gradient descent
        grad_u = sigmoid_tiled * (item_j - item_i) + self.reg * user_u
        grad_i = sigmoid_tiled * -user_u + self.reg * item_i
        grad_j = sigmoid_tiled * user_u + self.reg * item_j
        self.user_factors[u] -= self.learning_rate * grad_u
        self.item_factors[i] -= self.learning_rate * grad_i
        self.item_factors[j] -= self.learning_rate * grad_j
        return self

    def predict(self):
        """
        Obtain the predicted ratings for every users and items
        by doing a dot product of the learnt user and item vectors.
        The result will be cached to avoid re-computing
        it every time we call predict, thus there will
        only be an overhead the first time we call it.
        Note, ideally you probably don't need to compute
        this as it returns a dense matrix and may take
        up huge amounts of memory for large datasets
        """
        if not self._predicted:
            self._get_prediction()
            self._predicted = True

        return self._pred

    def _get_prediction(self):
        """Predicted ratings (dot product of user and item vectors)"""
        self._pred = self.user_factors.dot(self.item_factors.T)
        return self

    def _predict_user(self, user):
        """
        returns the predicted ratings for the specified user,
        this is mainly used in computing evaluation metric
        """
        user_pred = self.user_factors[user].dot(self.item_factors.T)
        return user_pred

    def recommend(self, ratings, N = 5, user_ids = None):
        """
        Returns the top N ranked items for given user id,
        excluding the ones that the user already liked

        Parameters
        ----------
        ratings : scipy sparse csr_matrix [n_users, n_items]
            sparse matrix of user-item interactions

        N : int, default 5
            top-N similar items' N

        user_ids : 1d iterator, e.g. list or numpy array, default None
            users' id that we wish to find top-N recommended for,
            default None will find it for all users

        Returns
        -------
        recommendation : 2d numpy array [number of query users_ids, N]
            each row is the top-N ranked item for each query user
        """
        if user_ids is not None:
            recommendation = np.zeros((len(user_ids), N), dtype = np.int)
            for idx, user in enumerate(user_ids):
                top_n = self._recommend_user(ratings, user, N)
                recommendation[idx] = top_n
        else:
            n_users = ratings.shape[0]
            recommendation = np.zeros((n_users, N), dtype = np.int)
            for user in range(n_users):
                top_n = self._recommend_user(ratings, user, N)
                recommendation[user] = top_n

        return recommendation

    def _recommend_user(self, ratings, user, N):
        """the top-N ranked items for a given user"""
        scores = self._predict_user(user)

        # compute the top N items, removing the items that the user already liked
        # from the result and ensure that we don't get out of bounds error when
        # we ask for more recommendations than that are available
        liked = set(ratings[user].indices)
        count = N + len(liked)
        if count < scores.shape[0]:

            # when trying to obtain the top-N indices from the score,
            # using argpartition to retrieve the top-N indices in
            # unsorted order and then sort them will be faster than doing
            # straight up argort on the entire score
            # http://stackoverflow.com/questions/42184499/cannot-understand-numpy-argpartition-output
            ids = np.argpartition(scores, -count)[-count:]
            best_ids = np.argsort(scores[ids])[::-1]
            best = ids[best_ids]
        else:
            best = np.argsort(scores)[::-1]

        top_n = list( islice((rec for rec in best if rec not in liked), N) )
        return top_n

    def get_similar_items(self, N = 5, item_ids = None):
        """
        return the top N similar items for itemid, where
        cosine distance is used as the distance metric

        Parameters
        ----------
        N : int, default 5
            top-N similar items' N

        item_ids : 1d iterator, e.g. list or numpy array, default None
            the item ids that we wish to find the similar items
            of, the default None will compute the similar items
            for all the items

        Returns
        -------
        similar_items : 2d numpy array [number of query item_ids, N]
            each row is the top-N most similar item id for each
            query item id
        """
        # cosine distance is proportional to normalized euclidean distance,
        # thus we normalize the item vectors and use euclidean metric so
        # we can use the more efficient kd-tree for nearest neighbor search;
        # also the item will always to nearest to itself, so we add 1 to
        # get an additional nearest item and remove itself at the end
        normed_item_factors = normalize(self.item_factors)
        knn = NearestNeighbors(n_neighbors = N + 1, metric = 'euclidean')
        knn.fit(normed_item_factors)

        # returns a distance, index tuple,
        # we don't actually need the distance
        if item_ids is not None:
            normed_item_factors = normed_item_factors[item_ids]

        _, items = knn.kneighbors(normed_item_factors)
        similar_items = items[:, 1:]
        return similar_items
```

```python
bpr_params = {
    'reg': 0.01,
    'learning_rate': 0.05,
    'n_iters': 50,
    'n_factors': 20,
    'n_batch_size': 2000
}

bpr = BPR(**bpr_params)
bpr.fit(X_train)
```

    BPR: 100%|██████████| 50/50 [01:07<00:00,  1.43s/it]

    <__main__.BPR at 0x11a9e2160>

## Evaluation

In recommender systems, we are often interested in how well the method can rank a given set of items. And to do that we'll use AUC (Area Under ROC Curve as our evaluation metric. The best possible value that the AUC evaluation metric can take is 1, and any non-random ranking that makes sense would have an AUC > 0.5. An intuitive explanation of AUC is it specifies the probability that when we draw two examples at random, their predicted pairwise ranking is correct.

```python
def auc_score(model, ratings):
    """
    computes area under the ROC curve (AUC).
    The full name should probably be mean
    auc score as it is computing the auc
    for every user's prediction and actual
    interaction and taking the average for
    all users

    Parameters
    ----------
    model : BPR instance
        the trained BPR model

    ratings : scipy sparse csr_matrix [n_users, n_items]
        sparse matrix of user-item interactions

    Returns
    -------
    auc : float 0 ~ 1
        auc score of the model
    """
    auc = 0.0
    n_users, n_items = ratings.shape
    for user, row in enumerate(ratings):
        y_pred = model._predict_user(user)
        y_true = np.zeros(n_items)
        y_true[row.indices] = 1
        auc += roc_auc_score(y_true, y_pred)

    auc /= n_users
    return auc
```

```python
print(auc_score(bpr, X_train))
print(auc_score(bpr, X_test))
```

    0.952749796601
    0.883659780104

## Item Recommendations

Now that we have trained the model, we can get the most similar items by using the `get_similar_items` method, we can specify the number of most simliar items by specifying the `N` argument.

```python
bpr.get_similar_items(N = 5)
```

    array([[ 467, 1305,  272,  122,  737],
           [ 562,  160,  544, 1215,  383],
           [ 239,  815,  234,  146,  272],
           ...,
           [1335, 1477, 1247, 1257, 1243],
           [1433, 1573, 1259, 1490, 1336],
           [1472, 1572, 1433, 1545, 1491]])

On the other hand, we can also generate a top-N recommended item for each given user, by passing the sparse rating data and `N` to the `recommend` method.

```python
bpr.recommend(X_train, N = 5)
```

    array([[365, 431,  90,  41, 430],
           [669, 244, 269, 923,  92],
           [299, 867, 885, 324, 259],
           ...,
           [403, 150, 999,  92, 247],
           [133, 204, 186,  57, 352],
           [143, 544,  81, 572, 658]])

# Reference

- [Wiki: Area Under the ROC Curve](http://www.recsyswiki.com/wiki/Area_Under_the_ROC_Curve)
- [StackExchange: Derivative of sigmoid function](http://math.stackexchange.com/questions/78575/derivative-of-sigmoid-function-sigma-x-frac11e-x)
- [Blog: What you wanted to know about AUC](http://fastml.com/what-you-wanted-to-know-about-auc/)
- [Blog: Learning to Rank Sketchfab Models with LightFM](http://blog.ethanrosenthal.com/2016/11/07/implicit-mf-part-2/)
- [Blog (Chinese Mandarin): BPR [Bayesian Personalized Ranking]](http://liuzhiqiangruc.iteye.com/blog/2073526)
- [Github: An implementation of Bayesian Personalised Ranking in Theano](https://github.com/bbc/theano-bpr)
- [Paper: S. Rendle, C. Freudenthaler, Z. Gantner, L. Schmidt-Thieme Bayesian Personalized Ranking from Implicit Feedback](https://arxiv.org/pdf/1205.2618.pdf)


```python
!pip install onnx-pytorch
```

    Collecting onnx-pytorch
      Downloading onnx_pytorch-0.1.4-py3-none-any.whl (105 kB)
         |████████████████████████████████| 105 kB 5.1 MB/s
    Collecting sympy>=1.9
      Downloading sympy-1.9-py3-none-any.whl (6.2 MB)
         |████████████████████████████████| 6.2 MB 15.1 MB/s
    Requirement already satisfied: torchvision>=0.11.1 in /usr/local/lib/python3.7/dist-packages (from onnx-pytorch) (0.11.1+cu111)
    Requirement already satisfied: tqdm>=4.62.3 in /usr/local/lib/python3.7/dist-packages (from onnx-pytorch) (4.62.3)
    Requirement already satisfied: numpy>=1.19.5 in /usr/local/lib/python3.7/dist-packages (from onnx-pytorch) (1.19.5)
    Collecting onnx>=1.10.2
      Downloading onnx-1.10.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (12.7 MB)
         |████████████████████████████████| 12.7 MB 9.0 MB/s
    Requirement already satisfied: torch>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from onnx-pytorch) (1.10.0+cu111)
    Collecting onnxruntime>=1.9.0
      Downloading onnxruntime-1.10.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.9 MB)
         |████████████████████████████████| 4.9 MB 38.0 MB/s
    Collecting setuptools>=59.2.0
      Downloading setuptools-60.2.0-py3-none-any.whl (953 kB)
         |████████████████████████████████| 953 kB 45.6 MB/s
    Collecting PyYAML>=6.0
      Downloading PyYAML-6.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (596 kB)
         |████████████████████████████████| 596 kB 59.9 MB/s
    Collecting pytest>=6.2.5
      Downloading pytest-6.2.5-py3-none-any.whl (280 kB)
         |████████████████████████████████| 280 kB 56.3 MB/s
    Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from onnx>=1.10.2->onnx-pytorch) (1.15.0)
    Requirement already satisfied: typing-extensions>=3.6.2.1 in /usr/local/lib/python3.7/dist-packages (from onnx>=1.10.2->onnx-pytorch) (3.10.0.2)
    Requirement already satisfied: protobuf in /usr/local/lib/python3.7/dist-packages (from onnx>=1.10.2->onnx-pytorch) (3.17.3)
    Requirement already satisfied: flatbuffers in /usr/local/lib/python3.7/dist-packages (from onnxruntime>=1.9.0->onnx-pytorch) (2.0)
    Requirement already satisfied: attrs>=19.2.0 in /usr/local/lib/python3.7/dist-packages (from pytest>=6.2.5->onnx-pytorch) (21.2.0)
    Requirement already satisfied: importlib-metadata>=0.12 in /usr/local/lib/python3.7/dist-packages (from pytest>=6.2.5->onnx-pytorch) (4.8.2)
    Collecting pluggy<2.0,>=0.12
      Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
    Requirement already satisfied: iniconfig in /usr/local/lib/python3.7/dist-packages (from pytest>=6.2.5->onnx-pytorch) (1.1.1)
    Requirement already satisfied: py>=1.8.2 in /usr/local/lib/python3.7/dist-packages (from pytest>=6.2.5->onnx-pytorch) (1.11.0)
    Requirement already satisfied: toml in /usr/local/lib/python3.7/dist-packages (from pytest>=6.2.5->onnx-pytorch) (0.10.2)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from pytest>=6.2.5->onnx-pytorch) (21.3)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=0.12->pytest>=6.2.5->onnx-pytorch) (3.6.0)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy>=1.9->onnx-pytorch) (1.2.1)
    Requirement already satisfied: pillow!=8.3.0,>=5.3.0 in /usr/local/lib/python3.7/dist-packages (from torchvision>=0.11.1->onnx-pytorch) (7.1.2)
    Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->pytest>=6.2.5->onnx-pytorch) (3.0.6)
    Installing collected packages: pluggy, sympy, setuptools, PyYAML, pytest, onnxruntime, onnx, onnx-pytorch
      Attempting uninstall: pluggy
        Found existing installation: pluggy 0.7.1
        Uninstalling pluggy-0.7.1:
          Successfully uninstalled pluggy-0.7.1
      Attempting uninstall: sympy
        Found existing installation: sympy 1.7.1
        Uninstalling sympy-1.7.1:
          Successfully uninstalled sympy-1.7.1
      Attempting uninstall: setuptools
        Found existing installation: setuptools 57.4.0
        Uninstalling setuptools-57.4.0:
          Successfully uninstalled setuptools-57.4.0
      Attempting uninstall: PyYAML
        Found existing installation: PyYAML 3.13
        Uninstalling PyYAML-3.13:
          Successfully uninstalled PyYAML-3.13
      Attempting uninstall: pytest
        Found existing installation: pytest 3.6.4
        Uninstalling pytest-3.6.4:
          Successfully uninstalled pytest-3.6.4
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
    Successfully installed PyYAML-6.0 onnx-1.10.2 onnx-pytorch-0.1.4 onnxruntime-1.10.0 pluggy-1.0.0 pytest-6.2.5 setuptools-60.2.0 sympy-1.9

```python
from onnx_pytorch import code_gen
code_gen.gen("/content/Resnet34_3inputs_448x448_20200609 (3).onnx", "/content/new")
```

    WARNING:root:Cannot get default value for dilations of Conv.
    WARNING:root:Cannot get default value for kernel_shape of Conv.
    WARNING:root:Cannot get default value for pads of Conv.
    WARNING:root:Cannot get default value for strides of Conv.
    WARNING:root:Cannot get default value for dilations of MaxPool.
    WARNING:root:Cannot get default value for kernel_shape of MaxPool.
    WARNING:root:Cannot get default value for pads of MaxPool.
    WARNING:root:Cannot get default value for strides of MaxPool.
    WARNING:root:Cannot get default value for axis of Concat.
    WARNING:root:Cannot get default value for dilations of ConvTranspose.
    WARNING:root:Cannot get default value for kernel_shape of ConvTranspose.
    WARNING:root:Cannot get default value for output_padding of ConvTranspose.
    WARNING:root:Cannot get default value for output_shape of ConvTranspose.
    WARNING:root:Cannot get default value for pads of ConvTranspose.
    WARNING:root:Cannot get default value for strides of ConvTranspose.

    # Autogenerated by onnx-pytorch.

    import glob
    import os
    import math

    import numpy as np
    import torch
    import torch.nn as nn
    import torch.nn.functional as F
    import torchvision

    class Model(nn.Module):
      def __init__(self):
        super(Model, self).__init__()
        self._vars = nn.ParameterDict()
        self._regularizer_params = []
        for b in glob.glob(
            os.path.join(os.path.dirname(__file__), "variables", "*.npy")):
          v = torch.from_numpy(np.load(b))
          requires_grad = v.dtype.is_floating_point or v.dtype.is_complex
          self._vars[os.path.basename(b)[:-4]] = nn.Parameter(v, requires_grad=requires_grad)
        self.n_Conv_0 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [3, 3], 'kernel_size': (7, 7), 'stride': [2, 2], 'in_channels': 3, 'bias': False})
        self.n_Conv_0.weight.data = self._vars["conv1_weight"]
        self.n_BatchNormalization_0 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_0.weight.data = self._vars["bn1_weight"]
        self.n_BatchNormalization_0.bias.data = self._vars["bn1_bias"]
        self.n_BatchNormalization_0.running_mean.data = self._vars["bn1_running_mean"]
        self.n_BatchNormalization_0.running_var.data = self._vars["bn1_running_var"]
        self.n_MaxPool_0 = nn.MaxPool2d(**{'dilation': 1, 'kernel_size': [3, 3], 'ceil_mode': False, 'stride': [2, 2], 'return_indices': False, 'padding': [1, 1]})
        self.n_Conv_1 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [3, 3], 'kernel_size': (7, 7), 'stride': [2, 2], 'in_channels': 3, 'bias': False})
        self.n_Conv_1.weight.data = self._vars["conv1_1_weight"]
        self.n_BatchNormalization_1 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_1.weight.data = self._vars["bn1_1_weight"]
        self.n_BatchNormalization_1.bias.data = self._vars["bn1_1_bias"]
        self.n_BatchNormalization_1.running_mean.data = self._vars["bn1_1_running_mean"]
        self.n_BatchNormalization_1.running_var.data = self._vars["bn1_1_running_var"]
        self.n_MaxPool_1 = nn.MaxPool2d(**{'dilation': 1, 'kernel_size': [3, 3], 'ceil_mode': False, 'stride': [2, 2], 'return_indices': False, 'padding': [1, 1]})
        self.n_Conv_2 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [3, 3], 'kernel_size': (7, 7), 'stride': [2, 2], 'in_channels': 3, 'bias': False})
        self.n_Conv_2.weight.data = self._vars["conv1_2_weight"]
        self.n_BatchNormalization_2 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_2.weight.data = self._vars["bn1_2_weight"]
        self.n_BatchNormalization_2.bias.data = self._vars["bn1_2_bias"]
        self.n_BatchNormalization_2.running_mean.data = self._vars["bn1_2_running_mean"]
        self.n_BatchNormalization_2.running_var.data = self._vars["bn1_2_running_var"]
        self.n_MaxPool_2 = nn.MaxPool2d(**{'dilation': 1, 'kernel_size': [3, 3], 'ceil_mode': False, 'stride': [2, 2], 'return_indices': False, 'padding': [1, 1]})
        self.n_Conv_3 = nn.Conv2d(**{'groups': 128, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_3.weight.data = self._vars["marg_0_weight"]
        self.n_BatchNormalization_3 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_3.weight.data = self._vars["marg_1_weight"]
        self.n_BatchNormalization_3.bias.data = self._vars["marg_1_bias"]
        self.n_BatchNormalization_3.running_mean.data = self._vars["marg_1_running_mean"]
        self.n_BatchNormalization_3.running_var.data = self._vars["marg_1_running_var"]
        self.n_Conv_4 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_4.weight.data = self._vars["marg_3_weight"]
        self.n_BatchNormalization_4 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_4.weight.data = self._vars["marg_4_weight"]
        self.n_BatchNormalization_4.bias.data = self._vars["marg_4_bias"]
        self.n_BatchNormalization_4.running_mean.data = self._vars["marg_4_running_mean"]
        self.n_BatchNormalization_4.running_var.data = self._vars["marg_4_running_var"]
        self.n_Conv_5 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 64, 'bias': False})
        self.n_Conv_5.weight.data = self._vars["layer1_0_conv1_weight"]
        self.n_BatchNormalization_5 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_5.weight.data = self._vars["layer1_0_bn1_weight"]
        self.n_BatchNormalization_5.bias.data = self._vars["layer1_0_bn1_bias"]
        self.n_BatchNormalization_5.running_mean.data = self._vars["layer1_0_bn1_running_mean"]
        self.n_BatchNormalization_5.running_var.data = self._vars["layer1_0_bn1_running_var"]
        self.n_Conv_6 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 64, 'bias': False})
        self.n_Conv_6.weight.data = self._vars["layer1_0_conv2_weight"]
        self.n_BatchNormalization_6 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_6.weight.data = self._vars["layer1_0_bn2_weight"]
        self.n_BatchNormalization_6.bias.data = self._vars["layer1_0_bn2_bias"]
        self.n_BatchNormalization_6.running_mean.data = self._vars["layer1_0_bn2_running_mean"]
        self.n_BatchNormalization_6.running_var.data = self._vars["layer1_0_bn2_running_var"]
        self.n_Conv_7 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 64, 'bias': False})
        self.n_Conv_7.weight.data = self._vars["layer1_1_conv1_weight"]
        self.n_BatchNormalization_7 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_7.weight.data = self._vars["layer1_1_bn1_weight"]
        self.n_BatchNormalization_7.bias.data = self._vars["layer1_1_bn1_bias"]
        self.n_BatchNormalization_7.running_mean.data = self._vars["layer1_1_bn1_running_mean"]
        self.n_BatchNormalization_7.running_var.data = self._vars["layer1_1_bn1_running_var"]
        self.n_Conv_8 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 64, 'bias': False})
        self.n_Conv_8.weight.data = self._vars["layer1_1_conv2_weight"]
        self.n_BatchNormalization_8 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_8.weight.data = self._vars["layer1_1_bn2_weight"]
        self.n_BatchNormalization_8.bias.data = self._vars["layer1_1_bn2_bias"]
        self.n_BatchNormalization_8.running_mean.data = self._vars["layer1_1_bn2_running_mean"]
        self.n_BatchNormalization_8.running_var.data = self._vars["layer1_1_bn2_running_var"]
        self.n_Conv_9 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 64, 'bias': False})
        self.n_Conv_9.weight.data = self._vars["layer1_2_conv1_weight"]
        self.n_BatchNormalization_9 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_9.weight.data = self._vars["layer1_2_bn1_weight"]
        self.n_BatchNormalization_9.bias.data = self._vars["layer1_2_bn1_bias"]
        self.n_BatchNormalization_9.running_mean.data = self._vars["layer1_2_bn1_running_mean"]
        self.n_BatchNormalization_9.running_var.data = self._vars["layer1_2_bn1_running_var"]
        self.n_Conv_10 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 64, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 64, 'bias': False})
        self.n_Conv_10.weight.data = self._vars["layer1_2_conv2_weight"]
        self.n_BatchNormalization_10 = nn.BatchNorm2d(**{'num_features': 64, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_10.weight.data = self._vars["layer1_2_bn2_weight"]
        self.n_BatchNormalization_10.bias.data = self._vars["layer1_2_bn2_bias"]
        self.n_BatchNormalization_10.running_mean.data = self._vars["layer1_2_bn2_running_mean"]
        self.n_BatchNormalization_10.running_var.data = self._vars["layer1_2_bn2_running_var"]
        self.n_Conv_11 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [2, 2], 'in_channels': 64, 'bias': False})
        self.n_Conv_11.weight.data = self._vars["layer2_0_conv1_weight"]
        self.n_BatchNormalization_11 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_11.weight.data = self._vars["layer2_0_bn1_weight"]
        self.n_BatchNormalization_11.bias.data = self._vars["layer2_0_bn1_bias"]
        self.n_BatchNormalization_11.running_mean.data = self._vars["layer2_0_bn1_running_mean"]
        self.n_BatchNormalization_11.running_var.data = self._vars["layer2_0_bn1_running_var"]
        self.n_Conv_12 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_12.weight.data = self._vars["layer2_0_conv2_weight"]
        self.n_BatchNormalization_12 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_12.weight.data = self._vars["layer2_0_bn2_weight"]
        self.n_BatchNormalization_12.bias.data = self._vars["layer2_0_bn2_bias"]
        self.n_BatchNormalization_12.running_mean.data = self._vars["layer2_0_bn2_running_mean"]
        self.n_BatchNormalization_12.running_var.data = self._vars["layer2_0_bn2_running_var"]
        self.n_Conv_13 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [2, 2], 'in_channels': 64, 'bias': False})
        self.n_Conv_13.weight.data = self._vars["layer2_0_downsample_0_weight"]
        self.n_BatchNormalization_13 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_13.weight.data = self._vars["layer2_0_downsample_1_weight"]
        self.n_BatchNormalization_13.bias.data = self._vars["layer2_0_downsample_1_bias"]
        self.n_BatchNormalization_13.running_mean.data = self._vars["layer2_0_downsample_1_running_mean"]
        self.n_BatchNormalization_13.running_var.data = self._vars["layer2_0_downsample_1_running_var"]
        self.n_Conv_14 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_14.weight.data = self._vars["layer2_1_conv1_weight"]
        self.n_BatchNormalization_14 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_14.weight.data = self._vars["layer2_1_bn1_weight"]
        self.n_BatchNormalization_14.bias.data = self._vars["layer2_1_bn1_bias"]
        self.n_BatchNormalization_14.running_mean.data = self._vars["layer2_1_bn1_running_mean"]
        self.n_BatchNormalization_14.running_var.data = self._vars["layer2_1_bn1_running_var"]
        self.n_Conv_15 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_15.weight.data = self._vars["layer2_1_conv2_weight"]
        self.n_BatchNormalization_15 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_15.weight.data = self._vars["layer2_1_bn2_weight"]
        self.n_BatchNormalization_15.bias.data = self._vars["layer2_1_bn2_bias"]
        self.n_BatchNormalization_15.running_mean.data = self._vars["layer2_1_bn2_running_mean"]
        self.n_BatchNormalization_15.running_var.data = self._vars["layer2_1_bn2_running_var"]
        self.n_Conv_16 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_16.weight.data = self._vars["layer2_2_conv1_weight"]
        self.n_BatchNormalization_16 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_16.weight.data = self._vars["layer2_2_bn1_weight"]
        self.n_BatchNormalization_16.bias.data = self._vars["layer2_2_bn1_bias"]
        self.n_BatchNormalization_16.running_mean.data = self._vars["layer2_2_bn1_running_mean"]
        self.n_BatchNormalization_16.running_var.data = self._vars["layer2_2_bn1_running_var"]
        self.n_Conv_17 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_17.weight.data = self._vars["layer2_2_conv2_weight"]
        self.n_BatchNormalization_17 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_17.weight.data = self._vars["layer2_2_bn2_weight"]
        self.n_BatchNormalization_17.bias.data = self._vars["layer2_2_bn2_bias"]
        self.n_BatchNormalization_17.running_mean.data = self._vars["layer2_2_bn2_running_mean"]
        self.n_BatchNormalization_17.running_var.data = self._vars["layer2_2_bn2_running_var"]
        self.n_Conv_18 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_18.weight.data = self._vars["layer2_3_conv1_weight"]
        self.n_BatchNormalization_18 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_18.weight.data = self._vars["layer2_3_bn1_weight"]
        self.n_BatchNormalization_18.bias.data = self._vars["layer2_3_bn1_bias"]
        self.n_BatchNormalization_18.running_mean.data = self._vars["layer2_3_bn1_running_mean"]
        self.n_BatchNormalization_18.running_var.data = self._vars["layer2_3_bn1_running_var"]
        self.n_Conv_19 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 128, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 128, 'bias': False})
        self.n_Conv_19.weight.data = self._vars["layer2_3_conv2_weight"]
        self.n_BatchNormalization_19 = nn.BatchNorm2d(**{'num_features': 128, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_19.weight.data = self._vars["layer2_3_bn2_weight"]
        self.n_BatchNormalization_19.bias.data = self._vars["layer2_3_bn2_bias"]
        self.n_BatchNormalization_19.running_mean.data = self._vars["layer2_3_bn2_running_mean"]
        self.n_BatchNormalization_19.running_var.data = self._vars["layer2_3_bn2_running_var"]
        self.n_Conv_20 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [2, 2], 'in_channels': 128, 'bias': False})
        self.n_Conv_20.weight.data = self._vars["layer3_0_conv1_weight"]
        self.n_BatchNormalization_20 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_20.weight.data = self._vars["layer3_0_bn1_weight"]
        self.n_BatchNormalization_20.bias.data = self._vars["layer3_0_bn1_bias"]
        self.n_BatchNormalization_20.running_mean.data = self._vars["layer3_0_bn1_running_mean"]
        self.n_BatchNormalization_20.running_var.data = self._vars["layer3_0_bn1_running_var"]
        self.n_Conv_21 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_21.weight.data = self._vars["layer3_0_conv2_weight"]
        self.n_BatchNormalization_21 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_21.weight.data = self._vars["layer3_0_bn2_weight"]
        self.n_BatchNormalization_21.bias.data = self._vars["layer3_0_bn2_bias"]
        self.n_BatchNormalization_21.running_mean.data = self._vars["layer3_0_bn2_running_mean"]
        self.n_BatchNormalization_21.running_var.data = self._vars["layer3_0_bn2_running_var"]
        self.n_Conv_22 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [2, 2], 'in_channels': 128, 'bias': False})
        self.n_Conv_22.weight.data = self._vars["layer3_0_downsample_0_weight"]
        self.n_BatchNormalization_22 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_22.weight.data = self._vars["layer3_0_downsample_1_weight"]
        self.n_BatchNormalization_22.bias.data = self._vars["layer3_0_downsample_1_bias"]
        self.n_BatchNormalization_22.running_mean.data = self._vars["layer3_0_downsample_1_running_mean"]
        self.n_BatchNormalization_22.running_var.data = self._vars["layer3_0_downsample_1_running_var"]
        self.n_Conv_23 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_23.weight.data = self._vars["layer3_1_conv1_weight"]
        self.n_BatchNormalization_23 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_23.weight.data = self._vars["layer3_1_bn1_weight"]
        self.n_BatchNormalization_23.bias.data = self._vars["layer3_1_bn1_bias"]
        self.n_BatchNormalization_23.running_mean.data = self._vars["layer3_1_bn1_running_mean"]
        self.n_BatchNormalization_23.running_var.data = self._vars["layer3_1_bn1_running_var"]
        self.n_Conv_24 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_24.weight.data = self._vars["layer3_1_conv2_weight"]
        self.n_BatchNormalization_24 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_24.weight.data = self._vars["layer3_1_bn2_weight"]
        self.n_BatchNormalization_24.bias.data = self._vars["layer3_1_bn2_bias"]
        self.n_BatchNormalization_24.running_mean.data = self._vars["layer3_1_bn2_running_mean"]
        self.n_BatchNormalization_24.running_var.data = self._vars["layer3_1_bn2_running_var"]
        self.n_Conv_25 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_25.weight.data = self._vars["layer3_2_conv1_weight"]
        self.n_BatchNormalization_25 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_25.weight.data = self._vars["layer3_2_bn1_weight"]
        self.n_BatchNormalization_25.bias.data = self._vars["layer3_2_bn1_bias"]
        self.n_BatchNormalization_25.running_mean.data = self._vars["layer3_2_bn1_running_mean"]
        self.n_BatchNormalization_25.running_var.data = self._vars["layer3_2_bn1_running_var"]
        self.n_Conv_26 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_26.weight.data = self._vars["layer3_2_conv2_weight"]
        self.n_BatchNormalization_26 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_26.weight.data = self._vars["layer3_2_bn2_weight"]
        self.n_BatchNormalization_26.bias.data = self._vars["layer3_2_bn2_bias"]
        self.n_BatchNormalization_26.running_mean.data = self._vars["layer3_2_bn2_running_mean"]
        self.n_BatchNormalization_26.running_var.data = self._vars["layer3_2_bn2_running_var"]
        self.n_Conv_27 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_27.weight.data = self._vars["layer3_3_conv1_weight"]
        self.n_BatchNormalization_27 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_27.weight.data = self._vars["layer3_3_bn1_weight"]
        self.n_BatchNormalization_27.bias.data = self._vars["layer3_3_bn1_bias"]
        self.n_BatchNormalization_27.running_mean.data = self._vars["layer3_3_bn1_running_mean"]
        self.n_BatchNormalization_27.running_var.data = self._vars["layer3_3_bn1_running_var"]
        self.n_Conv_28 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_28.weight.data = self._vars["layer3_3_conv2_weight"]
        self.n_BatchNormalization_28 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_28.weight.data = self._vars["layer3_3_bn2_weight"]
        self.n_BatchNormalization_28.bias.data = self._vars["layer3_3_bn2_bias"]
        self.n_BatchNormalization_28.running_mean.data = self._vars["layer3_3_bn2_running_mean"]
        self.n_BatchNormalization_28.running_var.data = self._vars["layer3_3_bn2_running_var"]
        self.n_Conv_29 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_29.weight.data = self._vars["layer3_4_conv1_weight"]
        self.n_BatchNormalization_29 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_29.weight.data = self._vars["layer3_4_bn1_weight"]
        self.n_BatchNormalization_29.bias.data = self._vars["layer3_4_bn1_bias"]
        self.n_BatchNormalization_29.running_mean.data = self._vars["layer3_4_bn1_running_mean"]
        self.n_BatchNormalization_29.running_var.data = self._vars["layer3_4_bn1_running_var"]
        self.n_Conv_30 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_30.weight.data = self._vars["layer3_4_conv2_weight"]
        self.n_BatchNormalization_30 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_30.weight.data = self._vars["layer3_4_bn2_weight"]
        self.n_BatchNormalization_30.bias.data = self._vars["layer3_4_bn2_bias"]
        self.n_BatchNormalization_30.running_mean.data = self._vars["layer3_4_bn2_running_mean"]
        self.n_BatchNormalization_30.running_var.data = self._vars["layer3_4_bn2_running_var"]
        self.n_Conv_31 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_31.weight.data = self._vars["layer3_5_conv1_weight"]
        self.n_BatchNormalization_31 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_31.weight.data = self._vars["layer3_5_bn1_weight"]
        self.n_BatchNormalization_31.bias.data = self._vars["layer3_5_bn1_bias"]
        self.n_BatchNormalization_31.running_mean.data = self._vars["layer3_5_bn1_running_mean"]
        self.n_BatchNormalization_31.running_var.data = self._vars["layer3_5_bn1_running_var"]
        self.n_Conv_32 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 256, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 256, 'bias': False})
        self.n_Conv_32.weight.data = self._vars["layer3_5_conv2_weight"]
        self.n_BatchNormalization_32 = nn.BatchNorm2d(**{'num_features': 256, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_32.weight.data = self._vars["layer3_5_bn2_weight"]
        self.n_BatchNormalization_32.bias.data = self._vars["layer3_5_bn2_bias"]
        self.n_BatchNormalization_32.running_mean.data = self._vars["layer3_5_bn2_running_mean"]
        self.n_BatchNormalization_32.running_var.data = self._vars["layer3_5_bn2_running_var"]
        self.n_Conv_33 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [2, 2], 'in_channels': 256, 'bias': False})
        self.n_Conv_33.weight.data = self._vars["layer4_0_conv1_weight"]
        self.n_BatchNormalization_33 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_33.weight.data = self._vars["layer4_0_bn1_weight"]
        self.n_BatchNormalization_33.bias.data = self._vars["layer4_0_bn1_bias"]
        self.n_BatchNormalization_33.running_mean.data = self._vars["layer4_0_bn1_running_mean"]
        self.n_BatchNormalization_33.running_var.data = self._vars["layer4_0_bn1_running_var"]
        self.n_Conv_34 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_34.weight.data = self._vars["layer4_0_conv2_weight"]
        self.n_BatchNormalization_34 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_34.weight.data = self._vars["layer4_0_bn2_weight"]
        self.n_BatchNormalization_34.bias.data = self._vars["layer4_0_bn2_bias"]
        self.n_BatchNormalization_34.running_mean.data = self._vars["layer4_0_bn2_running_mean"]
        self.n_BatchNormalization_34.running_var.data = self._vars["layer4_0_bn2_running_var"]
        self.n_Conv_35 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [2, 2], 'in_channels': 256, 'bias': False})
        self.n_Conv_35.weight.data = self._vars["layer4_0_downsample_0_weight"]
        self.n_BatchNormalization_35 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_35.weight.data = self._vars["layer4_0_downsample_1_weight"]
        self.n_BatchNormalization_35.bias.data = self._vars["layer4_0_downsample_1_bias"]
        self.n_BatchNormalization_35.running_mean.data = self._vars["layer4_0_downsample_1_running_mean"]
        self.n_BatchNormalization_35.running_var.data = self._vars["layer4_0_downsample_1_running_var"]
        self.n_Conv_36 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_36.weight.data = self._vars["layer4_1_conv1_weight"]
        self.n_BatchNormalization_36 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_36.weight.data = self._vars["layer4_1_bn1_weight"]
        self.n_BatchNormalization_36.bias.data = self._vars["layer4_1_bn1_bias"]
        self.n_BatchNormalization_36.running_mean.data = self._vars["layer4_1_bn1_running_mean"]
        self.n_BatchNormalization_36.running_var.data = self._vars["layer4_1_bn1_running_var"]
        self.n_Conv_37 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_37.weight.data = self._vars["layer4_1_conv2_weight"]
        self.n_BatchNormalization_37 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_37.weight.data = self._vars["layer4_1_bn2_weight"]
        self.n_BatchNormalization_37.bias.data = self._vars["layer4_1_bn2_bias"]
        self.n_BatchNormalization_37.running_mean.data = self._vars["layer4_1_bn2_running_mean"]
        self.n_BatchNormalization_37.running_var.data = self._vars["layer4_1_bn2_running_var"]
        self.n_Conv_38 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_38.weight.data = self._vars["layer4_2_conv1_weight"]
        self.n_BatchNormalization_38 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_38.weight.data = self._vars["layer4_2_bn1_weight"]
        self.n_BatchNormalization_38.bias.data = self._vars["layer4_2_bn1_bias"]
        self.n_BatchNormalization_38.running_mean.data = self._vars["layer4_2_bn1_running_mean"]
        self.n_BatchNormalization_38.running_var.data = self._vars["layer4_2_bn1_running_var"]
        self.n_Conv_39 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_39.weight.data = self._vars["layer4_2_conv2_weight"]
        self.n_BatchNormalization_39 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_39.weight.data = self._vars["layer4_2_bn2_weight"]
        self.n_BatchNormalization_39.bias.data = self._vars["layer4_2_bn2_bias"]
        self.n_BatchNormalization_39.running_mean.data = self._vars["layer4_2_bn2_running_mean"]
        self.n_BatchNormalization_39.running_var.data = self._vars["layer4_2_bn2_running_var"]
        self.n_Conv_40 = nn.Conv2d(**{'groups': 512, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_40.weight.data = self._vars["model1_9_0_weight"]
        self.n_BatchNormalization_40 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_40.weight.data = self._vars["model1_9_1_weight"]
        self.n_BatchNormalization_40.bias.data = self._vars["model1_9_1_bias"]
        self.n_BatchNormalization_40.running_mean.data = self._vars["model1_9_1_running_mean"]
        self.n_BatchNormalization_40.running_var.data = self._vars["model1_9_1_running_var"]
        self.n_Conv_41 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_41.weight.data = self._vars["model1_9_3_weight"]
        self.n_BatchNormalization_41 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_41.weight.data = self._vars["model1_9_4_weight"]
        self.n_BatchNormalization_41.bias.data = self._vars["model1_9_4_bias"]
        self.n_BatchNormalization_41.running_mean.data = self._vars["model1_9_4_running_mean"]
        self.n_BatchNormalization_41.running_var.data = self._vars["model1_9_4_running_var"]
        self.n_ConvTranspose_0 = nn.ConvTranspose2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 512, 'padding': [0, 0], 'output_padding': [0, 0], 'kernel_size': (2, 2), 'stride': [2, 2], 'in_channels': 512, 'bias': False})
        self.n_ConvTranspose_0.weight.data = self._vars["model1_9_6_weight"]
        self.n_BatchNormalization_42 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_42.weight.data = self._vars["model1_9_7_weight"]
        self.n_BatchNormalization_42.bias.data = self._vars["model1_9_7_bias"]
        self.n_BatchNormalization_42.running_mean.data = self._vars["model1_9_7_running_mean"]
        self.n_BatchNormalization_42.running_var.data = self._vars["model1_9_7_running_var"]
        self.n_Conv_42 = nn.Conv2d(**{'groups': 512, 'dilation': [1, 1], 'out_channels': 512, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_42.weight.data = self._vars["model2_1_0_weight"]
        self.n_BatchNormalization_43 = nn.BatchNorm2d(**{'num_features': 512, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_43.weight.data = self._vars["model2_1_1_weight"]
        self.n_BatchNormalization_43.bias.data = self._vars["model2_1_1_bias"]
        self.n_BatchNormalization_43.running_mean.data = self._vars["model2_1_1_running_mean"]
        self.n_BatchNormalization_43.running_var.data = self._vars["model2_1_1_running_var"]
        self.n_Conv_43 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 512, 'bias': False})
        self.n_Conv_43.weight.data = self._vars["model2_1_3_weight"]
        self.n_BatchNormalization_44 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_44.weight.data = self._vars["model2_1_4_weight"]
        self.n_BatchNormalization_44.bias.data = self._vars["model2_1_4_bias"]
        self.n_BatchNormalization_44.running_mean.data = self._vars["model2_1_4_running_mean"]
        self.n_BatchNormalization_44.running_var.data = self._vars["model2_1_4_running_var"]
        self.n_Conv_44 = nn.Conv2d(**{'groups': 1024, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_44.weight.data = self._vars["heatmap_0_weight"]
        self.n_BatchNormalization_45 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_45.weight.data = self._vars["heatmap_1_weight"]
        self.n_BatchNormalization_45.bias.data = self._vars["heatmap_1_bias"]
        self.n_BatchNormalization_45.running_mean.data = self._vars["heatmap_1_running_mean"]
        self.n_BatchNormalization_45.running_var.data = self._vars["heatmap_1_running_var"]
        self.n_Conv_45 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 24, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_45.weight.data = self._vars["heatmap_3_weight"]
        self.n_BatchNormalization_46 = nn.BatchNorm2d(**{'num_features': 24, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_46.weight.data = self._vars["heatmap_4_weight"]
        self.n_BatchNormalization_46.bias.data = self._vars["heatmap_4_bias"]
        self.n_BatchNormalization_46.running_mean.data = self._vars["heatmap_4_running_mean"]
        self.n_BatchNormalization_46.running_var.data = self._vars["heatmap_4_running_var"]
        self.n_Conv_46 = nn.Conv2d(**{'groups': 1024, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_46.weight.data = self._vars["offset_0_weight"]
        self.n_BatchNormalization_47 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_47.weight.data = self._vars["offset_1_weight"]
        self.n_BatchNormalization_47.bias.data = self._vars["offset_1_bias"]
        self.n_BatchNormalization_47.running_mean.data = self._vars["offset_1_running_mean"]
        self.n_BatchNormalization_47.running_var.data = self._vars["offset_1_running_var"]
        self.n_Conv_47 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 48, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_47.weight.data = self._vars["offset_3_weight"]
        self.n_Conv_48 = nn.Conv2d(**{'groups': 1024, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_48.weight.data = self._vars["model2_2_0_weight"]
        self.n_BatchNormalization_48 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_48.weight.data = self._vars["model2_2_1_weight"]
        self.n_BatchNormalization_48.bias.data = self._vars["model2_2_1_bias"]
        self.n_BatchNormalization_48.running_mean.data = self._vars["model2_2_1_running_mean"]
        self.n_BatchNormalization_48.running_var.data = self._vars["model2_2_1_running_var"]
        self.n_Conv_49 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_49.weight.data = self._vars["model2_2_3_weight"]
        self.n_BatchNormalization_49 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_49.weight.data = self._vars["model2_2_4_weight"]
        self.n_BatchNormalization_49.bias.data = self._vars["model2_2_4_bias"]
        self.n_BatchNormalization_49.running_mean.data = self._vars["model2_2_4_running_mean"]
        self.n_BatchNormalization_49.running_var.data = self._vars["model2_2_4_running_var"]
        self.n_Conv_50 = nn.Conv2d(**{'groups': 1024, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_50.weight.data = self._vars["model3_0_0_weight"]
        self.n_BatchNormalization_50 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_50.weight.data = self._vars["model3_0_1_weight"]
        self.n_BatchNormalization_50.bias.data = self._vars["model3_0_1_bias"]
        self.n_BatchNormalization_50.running_mean.data = self._vars["model3_0_1_running_mean"]
        self.n_BatchNormalization_50.running_var.data = self._vars["model3_0_1_running_var"]
        self.n_Conv_51 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 672, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_51.weight.data = self._vars["model3_0_3_weight"]
        self.n_BatchNormalization_51 = nn.BatchNorm2d(**{'num_features': 672, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_51.weight.data = self._vars["heatmap3D_0_weight"]
        self.n_BatchNormalization_51.bias.data = self._vars["heatmap3D_0_bias"]
        self.n_BatchNormalization_51.running_mean.data = self._vars["heatmap3D_0_running_mean"]
        self.n_BatchNormalization_51.running_var.data = self._vars["heatmap3D_0_running_var"]
        self.n_Conv_52 = nn.Conv2d(**{'groups': 1024, 'dilation': [1, 1], 'out_channels': 1024, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_52.weight.data = self._vars["model3_2_0_weight"]
        self.n_BatchNormalization_52 = nn.BatchNorm2d(**{'num_features': 1024, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_52.weight.data = self._vars["model3_2_1_weight"]
        self.n_BatchNormalization_52.bias.data = self._vars["model3_2_1_bias"]
        self.n_BatchNormalization_52.running_mean.data = self._vars["model3_2_1_running_mean"]
        self.n_BatchNormalization_52.running_var.data = self._vars["model3_2_1_running_var"]
        self.n_Conv_53 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 672, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 1024, 'bias': False})
        self.n_Conv_53.weight.data = self._vars["model3_2_3_weight"]
        self.n_BatchNormalization_53 = nn.BatchNorm2d(**{'num_features': 672, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_53.weight.data = self._vars["model3_2_4_weight"]
        self.n_BatchNormalization_53.bias.data = self._vars["model3_2_4_bias"]
        self.n_BatchNormalization_53.running_mean.data = self._vars["model3_2_4_running_mean"]
        self.n_BatchNormalization_53.running_var.data = self._vars["model3_2_4_running_var"]
        self.n_BatchNormalization_54 = nn.BatchNorm2d(**{'num_features': 672, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_54.weight.data = self._vars["model3_1_0_weight"]
        self.n_BatchNormalization_54.bias.data = self._vars["model3_1_0_bias"]
        self.n_BatchNormalization_54.running_mean.data = self._vars["model3_1_0_running_mean"]
        self.n_BatchNormalization_54.running_var.data = self._vars["model3_1_0_running_var"]
        self.n_Conv_54 = nn.Conv2d(**{'groups': 1344, 'dilation': [1, 1], 'out_channels': 1344, 'padding': [1, 1], 'kernel_size': (3, 3), 'stride': [1, 1], 'in_channels': 1344, 'bias': False})
        self.n_Conv_54.weight.data = self._vars["offset3D_0_weight"]
        self.n_BatchNormalization_55 = nn.BatchNorm2d(**{'num_features': 1344, 'eps': 9.999999747378752e-06, 'momentum': 0.8999999761581421})
        self.n_BatchNormalization_55.weight.data = self._vars["offset3D_1_weight"]
        self.n_BatchNormalization_55.bias.data = self._vars["offset3D_1_bias"]
        self.n_BatchNormalization_55.running_mean.data = self._vars["offset3D_1_running_mean"]
        self.n_BatchNormalization_55.running_var.data = self._vars["offset3D_1_running_var"]
        self.n_Conv_55 = nn.Conv2d(**{'groups': 1, 'dilation': [1, 1], 'out_channels': 2016, 'padding': [0, 0], 'kernel_size': (1, 1), 'stride': [1, 1], 'in_channels': 1344, 'bias': False})
        self.n_Conv_55.weight.data = self._vars["offset3D_3_weight"]

      def forward(self, *inputs):
        input_1, input_4, input_7 = inputs
        t_340 = self.n_Conv_0(input_1)
        t_341 = self.n_BatchNormalization_0(t_340)
        t_342 = F.relu(t_341)
        t_343 = self.n_MaxPool_0(t_342)
        t_344 = self.n_Conv_1(input_4)
        t_345 = self.n_BatchNormalization_1(t_344)
        t_346 = F.relu(t_345)
        t_347 = self.n_MaxPool_1(t_346)
        t_348 = self.n_Conv_2(input_7)
        t_349 = self.n_BatchNormalization_2(t_348)
        t_350 = F.relu(t_349)
        t_351 = self.n_MaxPool_2(t_350)
        t_352 = torch.add(t_347, t_351)
        t_353 = torch.cat((t_352, t_343), **{'dim': 1})
        t_354 = self.n_Conv_3(t_353)
        t_355 = self.n_BatchNormalization_3(t_354)
        t_356 = F.relu(t_355)
        t_357 = self.n_Conv_4(t_356)
        t_358 = self.n_BatchNormalization_4(t_357)
        t_359 = F.relu(t_358)
        t_360 = self.n_Conv_5(t_343)
        t_361 = self.n_BatchNormalization_5(t_360)
        t_362 = F.relu(t_361)
        t_363 = self.n_Conv_6(t_362)
        t_364 = self.n_BatchNormalization_6(t_363)
        t_365 = torch.add(t_364, t_343)
        t_366 = F.relu(t_365)
        t_367 = self.n_Conv_7(t_366)
        t_368 = self.n_BatchNormalization_7(t_367)
        t_369 = F.relu(t_368)
        t_370 = self.n_Conv_8(t_369)
        t_371 = self.n_BatchNormalization_8(t_370)
        t_372 = torch.add(t_371, t_366)
        t_373 = F.relu(t_372)
        t_374 = self.n_Conv_9(t_373)
        t_375 = self.n_BatchNormalization_9(t_374)
        t_376 = F.relu(t_375)
        t_377 = self.n_Conv_10(t_376)
        t_378 = self.n_BatchNormalization_10(t_377)
        t_379 = torch.add(t_378, t_373)
        t_380 = F.relu(t_379)
        t_381 = torch.add(t_380, t_359)
        t_382 = self.n_Conv_11(t_381)
        t_383 = self.n_BatchNormalization_11(t_382)
        t_384 = F.relu(t_383)
        t_385 = self.n_Conv_12(t_384)
        t_386 = self.n_BatchNormalization_12(t_385)
        t_387 = self.n_Conv_13(t_381)
        t_388 = self.n_BatchNormalization_13(t_387)
        t_389 = torch.add(t_386, t_388)
        t_390 = F.relu(t_389)
        t_391 = self.n_Conv_14(t_390)
        t_392 = self.n_BatchNormalization_14(t_391)
        t_393 = F.relu(t_392)
        t_394 = self.n_Conv_15(t_393)
        t_395 = self.n_BatchNormalization_15(t_394)
        t_396 = torch.add(t_395, t_390)
        t_397 = F.relu(t_396)
        t_398 = self.n_Conv_16(t_397)
        t_399 = self.n_BatchNormalization_16(t_398)
        t_400 = F.relu(t_399)
        t_401 = self.n_Conv_17(t_400)
        t_402 = self.n_BatchNormalization_17(t_401)
        t_403 = torch.add(t_402, t_397)
        t_404 = F.relu(t_403)
        t_405 = self.n_Conv_18(t_404)
        t_406 = self.n_BatchNormalization_18(t_405)
        t_407 = F.relu(t_406)
        t_408 = self.n_Conv_19(t_407)
        t_409 = self.n_BatchNormalization_19(t_408)
        t_410 = torch.add(t_409, t_404)
        t_411 = F.relu(t_410)
        t_412 = self.n_Conv_20(t_411)
        t_413 = self.n_BatchNormalization_20(t_412)
        t_414 = F.relu(t_413)
        t_415 = self.n_Conv_21(t_414)
        t_416 = self.n_BatchNormalization_21(t_415)
        t_417 = self.n_Conv_22(t_411)
        t_418 = self.n_BatchNormalization_22(t_417)
        t_419 = torch.add(t_416, t_418)
        t_420 = F.relu(t_419)
        t_421 = self.n_Conv_23(t_420)
        t_422 = self.n_BatchNormalization_23(t_421)
        t_423 = F.relu(t_422)
        t_424 = self.n_Conv_24(t_423)
        t_425 = self.n_BatchNormalization_24(t_424)
        t_426 = torch.add(t_425, t_420)
        t_427 = F.relu(t_426)
        t_428 = self.n_Conv_25(t_427)
        t_429 = self.n_BatchNormalization_25(t_428)
        t_430 = F.relu(t_429)
        t_431 = self.n_Conv_26(t_430)
        t_432 = self.n_BatchNormalization_26(t_431)
        t_433 = torch.add(t_432, t_427)
        t_434 = F.relu(t_433)
        t_435 = self.n_Conv_27(t_434)
        t_436 = self.n_BatchNormalization_27(t_435)
        t_437 = F.relu(t_436)
        t_438 = self.n_Conv_28(t_437)
        t_439 = self.n_BatchNormalization_28(t_438)
        t_440 = torch.add(t_439, t_434)
        t_441 = F.relu(t_440)
        t_442 = self.n_Conv_29(t_441)
        t_443 = self.n_BatchNormalization_29(t_442)
        t_444 = F.relu(t_443)
        t_445 = self.n_Conv_30(t_444)
        t_446 = self.n_BatchNormalization_30(t_445)
        t_447 = torch.add(t_446, t_441)
        t_448 = F.relu(t_447)
        t_449 = self.n_Conv_31(t_448)
        t_450 = self.n_BatchNormalization_31(t_449)
        t_451 = F.relu(t_450)
        t_452 = self.n_Conv_32(t_451)
        t_453 = self.n_BatchNormalization_32(t_452)
        t_454 = torch.add(t_453, t_448)
        t_455 = F.relu(t_454)
        t_456 = self.n_Conv_33(t_455)
        t_457 = self.n_BatchNormalization_33(t_456)
        t_458 = F.relu(t_457)
        t_459 = self.n_Conv_34(t_458)
        t_460 = self.n_BatchNormalization_34(t_459)
        t_461 = self.n_Conv_35(t_455)
        t_462 = self.n_BatchNormalization_35(t_461)
        t_463 = torch.add(t_460, t_462)
        t_464 = F.relu(t_463)
        t_465 = self.n_Conv_36(t_464)
        t_466 = self.n_BatchNormalization_36(t_465)
        t_467 = F.relu(t_466)
        t_468 = self.n_Conv_37(t_467)
        t_469 = self.n_BatchNormalization_37(t_468)
        t_470 = torch.add(t_469, t_464)
        t_471 = F.relu(t_470)
        t_472 = self.n_Conv_38(t_471)
        t_473 = self.n_BatchNormalization_38(t_472)
        t_474 = F.relu(t_473)
        t_475 = self.n_Conv_39(t_474)
        t_476 = self.n_BatchNormalization_39(t_475)
        t_477 = torch.add(t_476, t_471)
        t_478 = F.relu(t_477)
        t_479 = self.n_Conv_40(t_478)
        t_480 = self.n_BatchNormalization_40(t_479)
        t_481 = F.relu(t_480)
        t_482 = self.n_Conv_41(t_481)
        t_483 = self.n_BatchNormalization_41(t_482)
        t_484 = F.relu(t_483)
        t_485 = self.n_ConvTranspose_0(t_484)
        t_486 = self.n_BatchNormalization_42(t_485)
        t_487 = F.relu(t_486)
        t_488 = self.n_Conv_42(t_487)
        t_489 = self.n_BatchNormalization_43(t_488)
        t_490 = F.relu(t_489)
        t_491 = self.n_Conv_43(t_490)
        t_492 = self.n_BatchNormalization_44(t_491)
        t_493 = F.relu(t_492)
        t_494 = self.n_Conv_44(t_493)
        t_495 = self.n_BatchNormalization_45(t_494)
        t_496 = F.relu(t_495)
        t_497 = self.n_Conv_45(t_496)
        t_498 = self.n_BatchNormalization_46(t_497)
        t_499 = F.relu(t_498)
        t_500 = self.n_Conv_46(t_493)
        t_501 = self.n_BatchNormalization_47(t_500)
        t_502 = F.relu(t_501)
        t_503 = self.n_Conv_47(t_502)
        t_504 = torch.tanh(t_503)
        t_505 = self.n_Conv_48(t_493)
        t_506 = self.n_BatchNormalization_48(t_505)
        t_507 = F.relu(t_506)
        t_508 = self.n_Conv_49(t_507)
        t_509 = self.n_BatchNormalization_49(t_508)
        t_510 = F.relu(t_509)
        t_511 = self.n_Conv_50(t_510)
        t_512 = self.n_BatchNormalization_50(t_511)
        t_513 = F.relu(t_512)
        t_514 = self.n_Conv_51(t_513)
        t_515 = self.n_BatchNormalization_51(t_514)
        t_516 = F.relu(t_515)
        t_517 = self.n_Conv_52(t_510)
        t_518 = self.n_BatchNormalization_52(t_517)
        t_519 = F.relu(t_518)
        t_520 = self.n_Conv_53(t_519)
        t_521 = self.n_BatchNormalization_53(t_520)
        t_522 = F.relu(t_521)
        t_523 = self.n_BatchNormalization_54(t_514)
        t_524 = F.relu(t_523)
        t_525 = torch.cat((t_522, t_524), **{'dim': 1})
        t_526 = self.n_Conv_54(t_525)
        t_527 = self.n_BatchNormalization_55(t_526)
        t_528 = F.relu(t_527)
        t_529 = self.n_Conv_55(t_528)
        t_530 = torch.tanh(t_529)
        return t_504, t_499, t_530, t_516

      def compatible_auto_pad(self, input, kernel_spatial_shape, nn_mod, auto_pad=None, **kwargs):
        input_spatial_shape = input.shape[2:]
        d = len(input_spatial_shape)
        strides = nn_mod.stride
        dilations = nn_mod.dilation
        output_spatial_shape = [math.ceil(float(l) / float(r)) for l, r in zip(input.shape[2:], strides)]
        pt_padding = [0] * 2 * d
        pad_shape = [0] * d
        for i in range(d):
          pad_shape[i] = (output_spatial_shape[i] - 1) * strides[i] + ((kernel_spatial_shape[i] - 1) * dilations[i] + 1) - input_spatial_shape[i]
          mean = pad_shape[i] // 2
          if auto_pad == b"SAME_UPPER":
            l, r = pad_shape[i] - mean, mean
          else:
            l, r = mean, pad_shape[i] - mean
          pt_padding.insert(0, r)
          pt_padding.insert(0, l)
        return F.pad(input, pt_padding)

    @torch.no_grad()
    def test_run_model(inputs=[torch.from_numpy(np.random.randn(*[1, 3, 448, 448]).astype(np.float32)), torch.from_numpy(np.random.randn(*[1, 3, 448, 448]).astype(np.float32)), torch.from_numpy(np.random.randn(*[1, 3, 448, 448]).astype(np.float32))]):
      model = Model()
      model.eval()
      rs = model(*inputs)
      print(rs)
      return rs