```python
 %matplotlib inline
```

Sequence-to-Sequence Modeling with nn.Transformer and TorchText
===============================================================

This is a tutorial on how to train a sequence-to-sequence model
that uses the
`nn.Transformer <https://pytorch.org/docs/master/nn.html?highlight=nn%20transformer#torch.nn.Transformer>`__ module.

PyTorch 1.2 release includes a standard transformer module based on the
paper `Attention is All You
Need <https://arxiv.org/pdf/1706.03762.pdf>`__. The transformer model
has been proved to be superior in quality for many sequence-to-sequence
problems while being more parallelizable. The ``nn.Transformer`` module
relies entirely on an attention mechanism (another module recently
implemented as `nn.MultiheadAttention <https://pytorch.org/docs/master/nn.html?highlight=multiheadattention#torch.nn.MultiheadAttention>`__) to draw global dependencies
between input and output. The ``nn.Transformer`` module is now highly
modularized such that a single component (like `nn.TransformerEncoder <https://pytorch.org/docs/master/nn.html?highlight=nn%20transformerencoder#torch.nn.TransformerEncoder>`__
in this tutorial) can be easily adapted/composed.

Define the model
----------------

In this tutorial, we train ``nn.TransformerEncoder`` model on a
language modeling task. The language modeling task is to assign a
probability for the likelihood of a given word (or a sequence of words)
to follow a sequence of words. A sequence of tokens are passed to the embedding
layer first, followed by a positional encoding layer to account for the order
of the word (see the next paragraph for more details). The
``nn.TransformerEncoder`` consists of multiple layers of
`nn.TransformerEncoderLayer <https://pytorch.org/docs/master/nn.html?highlight=transformerencoderlayer#torch.nn.TransformerEncoderLayer>`__. Along with the input sequence, a square
attention mask is required because the self-attention layers in
``nn.TransformerEncoder`` are only allowed to attend the earlier positions in
the sequence. For the language modeling task, any tokens on the future
positions should be masked. To have the actual words, the output
of ``nn.TransformerEncoder`` model is sent to the final Linear
layer, which is followed by a log-Softmax function.

```python
import math

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn import TransformerEncoder, TransformerEncoderLayer

class TransformerModel(nn.Module):

    def __init__(self, ntoken, ninp, nhead, nhid, nlayers, dropout=0.5):
        super(TransformerModel, self).__init__()
        self.model_type = 'Transformer'
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        encoder_layers = TransformerEncoderLayer(ninp, nhead, nhid, dropout)
        self.transformer_encoder = TransformerEncoder(encoder_layers, nlayers)
        self.encoder = nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.decoder = nn.Linear(ninp, ntoken)

        self.init_weights()

    def generate_square_subsequent_mask(self, sz):
        mask = (torch.triu(torch.ones(sz, sz)) == 1).transpose(0, 1)
        mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))
        return mask

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src, src_mask):
        src = self.encoder(src) * math.sqrt(self.ninp)
        # src = self.pos_encoder(src)
        output = self.transformer_encoder(src, src_mask)
        output = self.decoder(output)
        return output
```

``PositionalEncoding`` module injects some information about the
relative or absolute position of the tokens in the sequence. The
positional encodings have the same dimension as the embeddings so that
the two can be summed. Here, we use ``sine`` and ``cosine`` functions of
different frequencies.

```python
class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]

        return self.dropout(x)
```

Load and batch data
-------------------

This tutorial uses ``torchtext`` to generate Wikitext-2 dataset. The
vocab object is built based on the train dataset and is used to numericalize
tokens into tensors. Starting from sequential data, the ``batchify()``
function arranges the dataset into columns, trimming off any tokens remaining
after the data has been divided into batches of size ``batch_size``.
For instance, with the alphabet as the sequence (total length of 26)
and a batch size of 4, we would divide the alphabet into 4 sequences of
length 6:

\begin{align}\begin{bmatrix}
  \text{A} & \text{B} & \text{C} & \ldots & \text{X} & \text{Y} & \text{Z}
  \end{bmatrix}
  \Rightarrow
  \begin{bmatrix}
  \begin{bmatrix}\text{A} \\ \text{B} \\ \text{C} \\ \text{D} \\ \text{E} \\ \text{F}\end{bmatrix} &
  \begin{bmatrix}\text{G} \\ \text{H} \\ \text{I} \\ \text{J} \\ \text{K} \\ \text{L}\end{bmatrix} &
  \begin{bmatrix}\text{M} \\ \text{N} \\ \text{O} \\ \text{P} \\ \text{Q} \\ \text{R}\end{bmatrix} &
  \begin{bmatrix}\text{S} \\ \text{T} \\ \text{U} \\ \text{V} \\ \text{W} \\ \text{X}\end{bmatrix}
  \end{bmatrix}\end{align}

These columns are treated as independent by the model, which means that
the dependence of ``G`` and ``F`` can not be learned, but allows more
efficient batch processing.

```python
import io
import torch
from torchtext.datasets import WikiText2
from torchtext.data.utils import get_tokenizer
from collections import Counter
from torchtext.vocab import Vocab
import torchtext.vocab
import pandas as pd
from sklearn.model_selection import train_test_split
data = pd.read_csv("facies_vectors.csv")
device = torch.device("cpu")
data = data[data["Well Name"] == "SHRIMPLIN"].sort_values("Depth")
x = data["GR"]
depth = data["Depth"]
x_train, x_test, x_val = x[:260], x[260:460], x[460:]
#vocab = torch.from_numpy(x.drop_duplicates().to_numpy(dtype = 'int32'))
tokenizer = get_tokenizer('basic_english')
counter = Counter()
for line in x:
    line = str(int(line))
    counter.update(tokenizer(line))
#print(counter)
vocab = Vocab(counter)

def data_process(raw_text_iter):
  data = [torch.tensor([vocab[token] for token in tokenizer(str(item))],
                       dtype=torch.long) for item in raw_text_iter]
  return torch.cat(tuple(filter(lambda t: t.numel() > 0, data)))

#storage = Counter(x.drop_duplicates().to_numpy(dtype = 'int32').tolist())
#vocab =Vocab(storage)
train_data = data_process(x_train.to_numpy(dtype = 'int64'))
val_data = data_process(x_test.to_numpy(dtype = 'int64'))
test_data = data_process(x_test.to_numpy(dtype = 'int64'))

def batchify(data, bsz):
    # Divide the dataset into bsz parts.
    nbatch = data.size(0) // bsz
    # Trim off any extra elements that wouldn't cleanly fit (remainders).
    data = data.narrow(0, 0, nbatch * bsz)
    # Evenly divide the data across the bsz batxches.
    data = data.view(bsz, -1)
    return data.to(device)

batch_size = 10
eval_batch_size = 10
train_data = batchify(train_data, batch_size)
val_data = batchify(val_data, eval_batch_size)
test_data = batchify(test_data, eval_batch_size)
WikiText2(split='train')
```

    <torchtext.data.datasets_utils._RawTextIterableDataset at 0x7febdc6cf700>

```python
data
```

      Facies
      Formation
      Well Name
      Depth
      GR
      ILD_log10
      DeltaPHI
      PHIND
      PE
      NM_M
      RELPOS

      0
      3
      A1 SH
      SHRIMPLIN
      2793.0
      77.45
      0.664
      9.9
      11.915
      4.6
      1
      1.000

      1
      3
      A1 SH
      SHRIMPLIN
      2793.5
      78.26
      0.661
      14.2
      12.565
      4.1
      1
      0.979

      2
      3
      A1 SH
      SHRIMPLIN
      2794.0
      79.05
      0.658
      14.8
      13.050
      3.6
      1
      0.957

      3
      3
      A1 SH
      SHRIMPLIN
      2794.5
      86.10
      0.655
      13.9
      13.115
      3.5
      1
      0.936

      4
      3
      A1 SH
      SHRIMPLIN
      2795.0
      74.58
      0.647
      13.5
      13.300
      3.4
      1
      0.915

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      466
      8
      C LM
      SHRIMPLIN
      3026.0
      17.92
      0.808
      1.5
      9.730
      5.4
      2
      0.049

      467
      8
      C LM
      SHRIMPLIN
      3026.5
      16.51
      0.814
      2.0
      8.275
      5.5
      2
      0.039

      468
      8
      C LM
      SHRIMPLIN
      3027.0
      21.94
      0.820
      3.3
      7.680
      5.4
      2
      0.029

      469
      8
      C LM
      SHRIMPLIN
      3027.5
      32.54
      0.821
      2.5
      7.445
      5.8
      2
      0.020

      470
      8
      C LM
      SHRIMPLIN
      3028.0
      47.70
      0.814
      1.6
      7.340
      5.8
      2
      0.010

471 rows × 11 columns

```python
len(vocab)
```

    114

Functions to generate input and target sequence
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``get_batch()`` function generates the input and target sequence for
the transformer model. It subdivides the source data into chunks of
length ``bptt``. For the language modeling task, the model needs the
following words as ``Target``. For example, with a ``bptt`` value of 2,
we’d get the following two Variables for ``i`` = 0:

It should be noted that the chunks are along dimension 0, consistent
with the ``S`` dimension in the Transformer model. The batch dimension
``N`` is along dimension 1.

```python
bptt = 10
shift = 1
def get_batch(source, i):
    seq_len = min(bptt, len(source) - shift - i)
    data = source[i:i+seq_len]
    target = source[i+shift:i+shift+seq_len].reshape(-1)
    return data, target
```

Initiate an instance
--------------------

The model is set up with the hyperparameter below. The vocab size is
equal to the length of the vocab object.

```python
ntokens = len(vocab.stoi) # the size of vocabulary
emsize = 200 # embedding dimension
nhid = 200 # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2 # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 4 # the number of heads in the multiheadattention models
dropout = 0.2 # the dropout value
model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout).to(device)
```

Run the model
-------------

`CrossEntropyLoss <https://pytorch.org/docs/master/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss>`__
is applied to track the loss and
`SGD <https://pytorch.org/docs/master/optim.html?highlight=sgd#torch.optim.SGD>`__
implements stochastic gradient descent method as the optimizer. The initial
learning rate is set to 5.0. `StepLR <https://pytorch.org/docs/master/optim.html?highlight=steplr#torch.optim.lr_scheduler.StepLR>`__ is
applied to adjust the learn rate through epochs. During the
training, we use
`nn.utils.clip_grad_norm\_ <https://pytorch.org/docs/master/nn.html?highlight=nn%20utils%20clip_grad_norm#torch.nn.utils.clip_grad_norm_>`__
function to scale all the gradient together to prevent exploding.

```python
import time
import matplotlib.pyplot as plt

criterion = nn.CrossEntropyLoss()
lr = 5.0 # learning rate
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)

def train():
    model.train() # Turn on the train mode
    total_loss = 0.
    start_time = time.time()
    src_mask = model.generate_square_subsequent_mask(bptt).to(device)
    for batch, i in enumerate(range(0, train_data.size(0) - 1, bptt)):
        data, targets = get_batch(train_data, i)
        optimizer.zero_grad()
        if data.size(0) != bptt:
            src_mask = model.generate_square_subsequent_mask(data.size(0)).to(device)
        output = model(data, src_mask)
        loss = criterion(output.view(-1, ntokens), targets)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        log_interval = 200
        if batch % log_interval == 0 and batch > 0:
            cur_loss = total_loss / log_interval
            elapsed = time.time() - start_time
            print('| epoch {:3d} | {:5d}/{:5d} batches | '
                  'lr {:02.2f} | ms/batch {:5.2f} | '
                  'loss {:5.2f} | ppl {:8.2f}'.format(
                    epoch, batch, len(train_data) // bptt, scheduler.get_last_lr()[0],
                    elapsed * 1000 / log_interval,
                    cur_loss, math.exp(cur_loss)))
            total_loss = 0
            start_time = time.time()

def evaluate(eval_model, data_source, plot=False):
    eval_model.eval() # Turn on the evaluation mode
    total_loss = 0.
    src_mask = model.generate_square_subsequent_mask(bptt).to(device)
    with torch.no_grad():
        for i in range(0, data_source.size(0) - 1, bptt):
            data, targets = get_batch(data_source, i)
            if data.size(0) != bptt:
                src_mask = model.generate_square_subsequent_mask(data.size(0)).to(device)
            output = eval_model(data, src_mask)
            output_flat = output.view(-1, ntokens)
            if plot:
                print(output.shape)
                plt.plot(range(len(output_flat)), targets, label="Targets", color="gray")
                plt.plot(range(len(output_flat)), output_flat, label="Output", color="red")
                ax = plt.gca()
                # this does the plotting, we plot gamma ray as a function of depth. note we subtract the kelly bushing elevation (EKB) to get subsea depth
                #ax.invert_yaxis()
                plt.legend()
                return output
            total_loss += len(data) * criterion(output_flat, targets).item()
    return total_loss / (len(data_source) - 1)

def evaluate_batch(eval_model, data_source, batch_id, plot=False):
    eval_model.eval() # Turn on the evaluation mode
    total_loss = 0.
    src_mask = model.generate_square_subsequent_mask(bptt).to(device)
    with torch.no_grad():
        data, targets = get_batch(data_source, batch_id)
        if data.size(0) != bptt:
            src_mask = model.generate_square_subsequent_mask(data.size(0)).to(device)
        output = eval_model(data, src_mask)
        output_flat = output.view(-1, ntokens)
        if plot:
            print(output.shape)
            plt.plot(range(len(output_flat)), targets, label="Targets", color="gray")
            plt.plot(range(len(output_flat)), output_flat, label="Output", color="red")
            ax = plt.gca()
            # this does the plotting, we plot gamma ray as a function of depth. note we subtract the kelly bushing elevation (EKB) to get subsea depth
            #ax.invert_yaxis()
            plt.legend()
    return output
```

```python
print(ntokens)
```

    114

Loop over epochs. Save the model if the validation loss is the best
we've seen so far. Adjust the learning rate after each epoch.

```python
best_val_loss = float("inf")
epochs = 20 # The number of epochs
best_model = None

for epoch in range(1, epochs + 1):
    epoch_start_time = time.time()
    train()
    val_loss = evaluate(model, val_data)
    print('-' * 89)
    print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '
          'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),
                                     val_loss, math.exp(val_loss)))
    print('-' * 89)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_model = model

    scheduler.step()
```

    -----------------------------------------------------------------------------------------
    | end of epoch   1 | time:  0.11s | valid loss  5.17 | valid ppl   175.98
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   2 | time:  0.02s | valid loss  5.68 | valid ppl   292.43
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   3 | time:  0.02s | valid loss  6.50 | valid ppl   662.73
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   4 | time:  0.02s | valid loss  6.38 | valid ppl   589.83
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   5 | time:  0.02s | valid loss  6.86 | valid ppl   952.01
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   6 | time:  0.02s | valid loss  6.64 | valid ppl   767.07
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   7 | time:  0.02s | valid loss  7.15 | valid ppl  1269.79
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   8 | time:  0.02s | valid loss  6.96 | valid ppl  1052.02
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   9 | time:  0.02s | valid loss  7.32 | valid ppl  1512.16
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  10 | time:  0.02s | valid loss  7.17 | valid ppl  1301.13
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  11 | time:  0.02s | valid loss  7.43 | valid ppl  1681.50
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  12 | time:  0.02s | valid loss  7.48 | valid ppl  1772.45
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  13 | time:  0.02s | valid loss  7.49 | valid ppl  1784.64
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  14 | time:  0.02s | valid loss  7.58 | valid ppl  1952.40
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  15 | time:  0.02s | valid loss  7.67 | valid ppl  2147.00
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  16 | time:  0.02s | valid loss  7.68 | valid ppl  2162.68
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  17 | time:  0.02s | valid loss  7.77 | valid ppl  2357.89
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  18 | time:  0.02s | valid loss  7.74 | valid ppl  2303.75
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  19 | time:  0.02s | valid loss  7.78 | valid ppl  2389.90
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  20 | time:  0.02s | valid loss  7.78 | valid ppl  2388.30
    -----------------------------------------------------------------------------------------

Evaluate the model with the test dataset
-------------------------------------

Apply the best model to check the result with the test dataset.

```python
test_loss = evaluate(best_model, test_data)
print('=' * 89)
print('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(
    test_loss, math.exp(test_loss)))
print('=' * 89)
```

    =========================================================================================
    | End of training | test loss  7.78 | test ppl  2388.30
    =========================================================================================

```python
test_data.shape
```

    torch.Size([10, 20])

```python
output = evaluate_batch(best_model, test_data, 0, plot=True)
```

```python
output.view(-1, ntokens).shape
```

    torch.Size([180, 114])

```python
output.shape
```

    torch.Size([9, 20, 114])

# Now Lets try this out on a bigger set of data

```python
output_flat = output.view(-1, ntokens)
```

```python
output.size(-1)
```

    114

```python
get_batch(test_data, 0)[0]
```

    tensor([[ 93,  97,  79,  95,   5,  64,  15,  37,  37,  38,  86,  17,  29,  28,
              41,  19,   2,  70,  14,  76],
            [ 14,  58,   7,  59,  73,  59,  74,  94,  98,  79,  43,  28,  52,  52,
              53,  55,  30,  56,  31,  47],
            [ 69,  82,  82, 107, 103,  73,   2,   2,  48,  96, 101, 102, 100,  49,
              78,  14,   7,   7,   2,   8],
            [  4,   6,   4,   8,   4,  34,   8,  24,  47,  46,  23,  70,   8,   7,
              34,  35,   6,   2,   6,   2],
            [  2,  34,  35,  71,  58,  92,  72,  91,   4,  13,  13,   6,   2,   3,
               2,   2,  35,   3,  69,  12],
            [  3,   3,   4,   4,   2,   2,   6,   4,  66,  25,  83,  38,  38,  60,
              80,  80,  37,  51,  29,  44],
            [ 11,  39,  85,  63,  50,  51,  26,  26,  39,  28,  27,  53,  65,  27,
              39,  65,  17,  29,  40,  16],
            [ 16,  17,  54,  89,  16,  40,  29,  55,  18,  29,  17,  18,   9,  21,
              99, 108, 109, 110, 104,  49],
            [ 12,  44,  29,  40,  55,  66,  42,  90,  30,  55,  30,   9,  18,   9,
              30,  18,  43,  10,  19,  21]])

```python
def convert_outpupt_to_tensor(output):
    imm = [[vocab.itos[torch.argmax(prob).item()] for prob in result] for result in output]
    ret = []
    for row in imm:
        new_row = []
        for i in row:
            if i == '<pad>':
                new_row.append(0)
            elif i == '<unk>':
                new_row.append(-1)
            else:
                new_row.append(int(i))
        ret.append(new_row)

    return torch.tensor(ret)
convert_outpupt_to_tensor(output)
```

    tensor([[ 64,  87,  90,  46,  50,  75,  74,  77,  77,  75,  80, 107,  80,  77,
              68,  62,  15,  59,  61,  76],
            [ 61,  35,  34,  76,  50,  76,  64,  66,  89,  60,  57,  77,  67,  67,
              76,  84,  86,  33,  87,  91],
            [ 74,  90,  19,  15,  50,  87,  15,  73,  89,  74,  75,  92,  55,  87,
              76,  92,  20,  34,  73,  29],
            [ 15,  91,  94,  15,  32,  20,  15,  17,  91,  59,  87,  59,  29,  34,
              23,  55,  91,  34,  91,  15],
            [ 15,  20,  55,  81,  35,  28,  89,  32,  32,  97,  85,  69,  15,  44,
              34,  15,  55,  95,  30,  89],
            [ 64,  14,  32,  15,  73,  73,  19,  15,  46,  61,  75,  75,  64,  60,
              92,  54,  77,  67,  70,  83],
            [ 61,  62,  61,  67,  83,  87, 115, 115,  78,  68, 115,  76,  43,  62,
              62,  43, 103,  67,  80,  84],
            [ 84, 103,  80,  58,  78,  80,  15,  17,  44,  67, 103,  44,  78,  87,
              92,  84,  83,  67,  87,  87],
            [ 16,  83,  94,  80,  17,  46,  77,  91,  86,  17,  96,  78,  44,  53,
              86,  44,  57,  83,  62,  15]])

```python
len(output[0])
```

    20

```python
wells = pd.read_csv("export_csv/2018.csv", index_col=[1,0])
wells.head()
```

    /home/fastfist/.local/share/virtualenvs/transformers-kgQS_UHh/lib/python3.9/site-packages/numpy/lib/arraysetops.py:580: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
      mask |= (ar1 == a)

      gamma
      County

      well_id
      depth

      355
      161.0
      NaN
      SCOTT

      161.5
      NaN
      SCOTT

      162.0
      NaN
      SCOTT

      162.5
      NaN
      SCOTT

      163.0
      NaN
      SCOTT

```python
import matplotlib.pyplot as plt
import matplotlib as mpl
import numpy as np

cmap = mpl.cm.bone_r  # select our color map
def plot_well(well):
    fig = plt.figure(figsize=(2,10))

    x = well["gamma"].values
    x[np.isnan(x)] = 0 # set NaNs to 0
    y = well.index.values #% assign the depth values to y

    plt.plot(x, y, color="gray")
    ax = plt.gca()
    # this does the plotting, we plot gamma ray as a function of depth. note we subtract the kelly bushing elevation (EKB) to get subsea depth
    ax.invert_yaxis()

    # let's plot the fake tops from the csv file as a horizontal line, don't forget to subtract the KB elevation for subsea depth
    # ax.axis("off")
    # this gets rid of the axis so they look clean
    ax.set_title("Predictions")
    # and let's go ahead and add titles to each log for fun
```

```python
well = wells.loc[314]
plot_well(well)
```

```python
well
```

      gamma
      County

      depth

      195.0
      72.7791
      ELLIS

      195.5
      74.0550
      ELLIS

      196.0
      75.5007
      ELLIS

      196.5
      76.7649
      ELLIS

      197.0
      77.9105
      ELLIS

      ...
      ...
      ...

      3918.5
      0.0000
      ELLIS

      3919.0
      0.0000
      ELLIS

      3919.5
      0.0000
      ELLIS

      3920.0
      0.0000
      ELLIS

      3920.5
      0.0000
      ELLIS

7452 rows × 2 columns

```python
x = well["gamma"]
counter = Counter()
for line in x:
    line = str(int(line))
    counter.update(tokenizer(line))
#print(counter)
vocab = Vocab(counter)

def data_process(raw_text_iter):
  data = [torch.tensor([vocab.stoi[token] for token in tokenizer(str(item))],
                       dtype=torch.long) for item in raw_text_iter]
  # return torch.cat(data)
  return torch.cat(tuple(filter(lambda t: t.numel() > 0, data)))

#storage = Counter(x.drop_duplicates().to_numpy(dtype = 'int32').tolist())
#vocab =Vocab(storage)
# Lets break it down
gamma = well["gamma"]
x_train, x_val, x_test = gamma[:1000:5], gamma[1000:1250:5], gamma[1250:3899:5]
train_data = data_process(x_train.to_numpy(dtype = 'int64'))
val_data = data_process(x_val.to_numpy(dtype = 'int64'))
test_data = data_process(x_test.to_numpy(dtype = 'int64'))

batch_size = 10
eval_batch_size = 10
train_data = batchify(train_data, batch_size)
val_data = batchify(val_data, eval_batch_size)
test_data = batchify(test_data, eval_batch_size)
```

```python
1001/10
```

    100.1

```python
def conv(a:torch.tensor):
    return torch.tensor([int(vocab.itos[x]) for x in a.view(-1)])
```

```python
x_test
```

    depth
    1250.0     86.7215
    1252.5    100.7469
    1255.0     94.2474
    1257.5     93.0782
    1260.0     76.2082
                ...
    3887.5      0.0000
    3890.0      0.0000
    3892.5      0.0000
    3895.0      0.0000
    3897.5      0.0000
    Name: gamma, Length: 1060, dtype: float64

```python
test_data.view(-1).shape
```

    torch.Size([1060])

```python
ntokens = len(vocab.stoi) # the size of vocabulary
emsize = 200 # embedding dimension
nhid = 200 # the dimension of the feedforward network model in nn.TransformerEncoder
nlayers = 2 # the number of nn.TransformerEncoderLayer in nn.TransformerEncoder
nhead = 4 # the number of heads in the multiheadattention models
dropout = 0.2 # the dropout value
model = TransformerModel(ntokens, emsize, nhead, nhid, nlayers, dropout).to(device)
```

```python
best_val_loss = float("inf")
epochs = 100 # The number of epochs
best_model = None

for epoch in range(1, epochs + 1):
    epoch_start_time = time.time()
    train()
    val_loss = evaluate(model, val_data)
    print('-' * 89)
    print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '
          'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),
                                     val_loss, math.exp(val_loss)))
    print('-' * 89)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_model = model

    scheduler.step()
```

    -----------------------------------------------------------------------------------------
    | end of epoch   1 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   2 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   3 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   4 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   5 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   6 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   7 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   8 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch   9 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  10 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  11 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  12 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  13 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  14 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  15 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  16 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  17 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  18 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  19 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  20 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  21 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  22 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  23 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  24 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  25 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  26 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  27 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  28 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  29 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  30 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  31 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  32 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  33 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  34 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  35 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  36 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  37 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  38 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  39 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  40 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  41 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  42 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  43 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  44 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  45 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  46 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  47 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  48 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  49 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  50 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  51 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  52 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  53 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  54 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  55 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  56 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  57 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  58 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  59 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  60 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  61 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  62 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  63 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  64 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  65 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  66 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  67 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  68 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  69 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  70 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  71 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  72 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  73 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  74 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  75 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  76 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  77 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  78 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  79 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  80 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  81 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  82 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  83 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  84 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  85 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  86 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  87 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  88 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  89 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  90 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  91 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  92 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  93 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  94 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  95 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  96 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  97 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  98 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch  99 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------
    -----------------------------------------------------------------------------------------
    | end of epoch 100 | time:  0.02s | valid loss  5.40 | valid ppl   221.60
    -----------------------------------------------------------------------------------------

```python
test_loss = evaluate(best_model, test_data)
print('=' * 89)
print('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(
    test_loss, math.exp(test_loss)))
print('=' * 89)
```

    =========================================================================================
    | End of training | test loss  5.37 | test ppl   215.23
    =========================================================================================

```python
output = evaluate_batch(best_model, test_data, 50)
```

```python
output.shape
```

    torch.Size([0, 106, 159])

```python
test_data.size()
```

    torch.Size([10, 106])

```python
ntokens
```

    159

```python
output
```

    tensor([], size=(0, 106, 159))

```python
data_source = test_data
full_output = []
src_mask = torch.ones(bptt,bptt)
with torch.no_grad():
    for i in range(0, data_source.size(0), bptt):
        data, targets = get_batch(data_source, i)
        if data.size(0) != bptt:
            src_mask = model.generate_square_subsequent_mask(data.size(0)).to(device)
        output = model(data, src_mask)
        full_output.append(output)
full_output = torch.cat(full_output)
full_output.shape
```

    torch.Size([9, 106, 159])

```python
full_output = convert_outpupt_to_tensor(full_output)
full_output
```

    tensor([[ 37,  30, 201,  93, 187,  94,  49,  15, 106,  15, 106,  93, 201, 104,
             108,  37,  49, 201,  55, 174,  89,  58, 190,  25,  50,  53,  77,  32,
             190,  25, 106, 128,  53,  93,  28,  50,  50,  32,  53,  24,  32,  77,
              25,  77, 174,  25,  93, 106,  93,  55,  15,  94, 174, 174,  15,  55,
              89, 187, 201, 102, 105, 106,  15, 145,  93, 106, 190, 190,  32, 106,
             190, 201, 190, 145,  25,  15,  49, 165,  58,  25, 174, 143, 201, 106,
             108,  93, 102, 201,  37, 174, 105,  54,  37,  28, 201, 130, 102, 316,
             190,  58,  93,  89,  77,  89,  77,  89],
            [ 94,  81,  64,  72,  62,  98,  55, 106, 100,  28,  85,  37,  37,  60,
              68,  72,   0,  28, 108,  49,  93,  79,  79, 100,  98,  79,  67,  53,
             187, 137,  22,  36,  72,  19, 104, 193,  52,  90,  77, 174,  62, 106,
              60, 143,  54, 100, 143, 107, 193,  85,  30, 106,  51,  93,  82, 266,
             108,  80,  32,  37, 190,  50, 106, 108,  22, 134, 190,  85,  85, 123,
             190,  32,  36, 145, 145,  36,  55, 225,  22,  36,  81, 145, 225,  30,
              72,  72,  80, 149,  60, 132,  36, 145,  80, 108,  73, 132, 165,  77,
              85,  93, 136, 105,  35, 136, 108,  96],
            [ 82, 140, 201, 145, 132, 145,  64,  37, 266, 106,  37,  63,  16, 176,
              28, 106, 134, 123,  55,  93,  52, 149,  85,  76, 316,   0,  84,  37,
              77, 146, 109, 165,  72, 121,  79,  52,  28,  84, 106, 106, 157,  99,
              91, 131, 174, 117, 266,  99, 199, 106, 174,  68,  72,  72,  93,  77,
              79,  99, 131,  34, 128, 106, 139,  86,  28,  28,  26,  72,  37, 316,
             120, 157, 106, 106,  28, 106, 127,  77,  64, 117,  34, 145, 115, 174,
             106,  72, 145, 266, 138, 316,  84,  86,  19,  34, 176,  77, 117, 117,
              72, 316, 157, 132,  28,  89,  -1,  67],
            [128,  34, 203, 157, 117, 106,  79, 176, 187,  72, 266, 132, 176,  60,
              54,  43,  72,  52,  79,  64, 110, 132,  28,  79,  72, 165,  79,  96,
             316,  52,  37,  19, 266,  28,  60,  -1,  20, 203,  35, 145,  64, 121,
              72, 266, 106,  84,  24, 190,  -1,  77,  37,  80, 190, 199,  64, 145,
              72, 123, 137,  60,  51,  85, 190,  85,  79, 106,  67,  -1,  77,  -1,
              -1,  96,  53,  79,  84, 203,  49,  35,  51,  55, 107,  39,  93, 137,
              26, 132, 117,  55, 140,  91,  -1, 165,  28,  52,  96, 123, 165,  74,
              58, 123,  58,  82, 108, 190,  49,  85],
            [ 77,  58,  47,  43,  72, 190,  19, 117,  28, 109,  24,  34, 174, 117,
              49,  28,  16, 100,  49,  49,  22, 102,  37,  99,  86,  19,  -1,  93,
              58,  64,  77,  28,  16,  34,  67, 117,  37,  64,  26,  24, 101, 121,
              22, 143, 174,  37, 165, 111, 106, 161, 140,  51,  72,  32, 106,  96,
              55, 149,  24, 137, 190, 104, 132,  85, 165,  72,  65,  54, 135,  93,
             128, 146,  53,  41, 132, 106,  72, 143, 132,  52,  28, 187, 203,  80,
              63,  50, 128, 225,  37, 174, 108,  72, 128,  51,  47, 108, 165,  58,
              55, 106, 316,  53,  49,  -1, 174,  89],
            [ 32,  82,  96, 201, 111,  72, 121,  37,  28, 146, 225, 106, 143,  73,
              77,  55,  43,  84, 117,  99, 132,  66,  52,  32, 266,  60, 139,  82,
             190, 146, 140, 111, 106,  28, 140,  58,  72, 117, 193,  -1, 109,  37,
              60,  34, 199,  89, 117,  84,  79, 108, 266, 316, 266,  93, 106,  28,
              34,  77,  72,  91,  -1,  58,  28,  51, 140, 140, 145, 111,  79,  67,
              72,  74,  51, 157, 132, 111,  81, 126,  15,  54,  24, 107,  22,  39,
             187,  28,  56, 117,  -1, 132,  -1, 106,  28, 132, 106,  -1, 121, 174,
              64,  -1,  67,  84,  54,  -1,  17,  -1],
            [187, 132, 121,  79,  58, 108,  64,  24,  55,  32, 201,  36,  24,  72,
             145,  49, 111, 174, 117,  64,  37,  89, 137,  -1,  56,  64, 106,  35,
             124,  89,  66,  28, 104,  28, 111, 106, 203,  85,  85, 108, 132, 106,
             128,  58,  43, 100,  66, 106,  -1,  89,  58,  89, 137, 145, 117,  34,
              57, 107,  -1, 190,  67,  85,  78, 106,  28,  85, 108, 266,  36,  93,
              64,  -1, 106, 106,  79,  77,  37,  58, 132, 143,  64, 131, 187,  64,
              -1,  85,  89,  36,  73, 145,  28,  84,  72, 117,  64,  84, 143, 129,
              22,  52, 143,  60, 132,  64,  28, 140],
            [ 77,  -1,  80,  28, 106, 203,  37, 174, 117, 149, 109, 106,  85,  86,
             143,  55,  37,  28,  85,  17, 137, 190,  72, 201, 145, 273, 106, 106,
             109,  56,  77, 123, 106, 121, 108,  58,  37,  94, 176,  28,  90,  30,
              60,  64,  -1,  -1,  24, 106,  84, 117,  24, 106,  72, 190, 176,  34,
              -1,  63, 137,  84,  67, 131, 128, 203,  22, 106,  28, 203,  99,  84,
             190, 174, 143,  34, 187,  84,  34,  64,  16,  22, 174,  24,  24, 140,
             145, 106, 145, 165, 135,  80,  94, 316, 108,  83, 109, 117,  -1, 100,
              77, 128,  34, 128, 106,  22,  -1, 121],
            [ 53, 138, 106,  55, 117, 187,  52,  24, 143, 106, 174,  -1, 174, 145,
              91,  28,  84,  32,  90,  66,  71, 266, 145, 100,  28,  89,  51,  24,
              -1, 117,  72, 138, 106,  96, 138, 106,  66,  89, 128,  17,  82,  94,
              86, 111,  -1, 100, 145,  58,  17, 187,  32,  66,  60,  72,  39,  79,
             203,  17, 266, 201,  86, 132, 190, 128,  24,  22,  28,  68,  84,  91,
              -1, 187,  86, 132,  73,  61,  60,  66,  80, 199, 109, 187,  72,  72,
             132,  79,  80,  80, 120,  91,  28,  64, 125,  54, 176,  72,  64,  64,
              58, 106, 190, 225,  -1, 132, 106,  28]])

```python
full_output.shape
```

    torch.Size([9, 106])

```python
test_data.shape
```

    torch.Size([10, 106])

```python
plt.plot(range(1, len(full_output.view(-1)) + 1), full_output.view(-1), label="Output")
plt.plot(range(len(test_data.view(-1))), conv(test_data.view(-1)), label="Target")
plt.legend()
```

# fo

```python
test_data.view(-1)[106:].shape
```

    torch.Size([954])

```python
full_output.view(-1).shape
```

    torch.Size([954])

```python
well[1250:3899:5].index.values.shape
```

    (1060,)

```python
torch.tensor(conv(test_data.view(-1)))[:20]
```

    <ipython-input-100-5cbfae581997>:1: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      torch.tensor(conv(test_data.view(-1)))[:20]

    tensor([ 86, 100,  94,  93,  76,  65,  97,  73,  75,  73,  87,  93,  94, 109,
             83,  86,  97,  94,  78,  91])

```python
test_data[:20]
```

    tensor([[ 29,  77,  65,  ...,  96,  89,  96],
            [ 89,  89,   7,  ...,  57,  22,  25],
            [ 48,  20,  65,  ...,  54,   8,  19],
            ...,
            [ 57,  31,  58,  ...,   2,   8,  45],
            [153,  59,  46,  ...,  61,  21,  34],
            [  4,   7,   6,  ...,  44,  44,  44]])

```python
z = well[1250::5][106:]
z
```

      gamma
      County

      depth

      1250.0
      86.7215
      ELLIS

      1252.5
      100.7469
      ELLIS

      1255.0
      94.2474
      ELLIS

      1257.5
      93.0782
      ELLIS

      1260.0
      76.2082
      ELLIS

      ...
      ...
      ...

      3910.0
      0.0000
      ELLIS

      3912.5
      0.0000
      ELLIS

      3915.0
      0.0000
      ELLIS

      3917.5
      0.0000
      ELLIS

      3920.0
      0.0000
      ELLIS

1069 rows × 2 columns

```python
z[z["gamma"].to_numpy(dtype="int64") == 266]
```

      gamma
      County

      depth

      3370.0
      266.9463
      ELLIS

```python
results = pd.DataFrame({"Output": [t.item() for t in full_output.view(-1)], "Input": conv(test_data.view(-1))[106:] }, index=pd.Index(name="depth",data=well[1250:3899:5].index.values[106:]))
results
```

      Output
      Input

      depth

      1515.0
      37
      22

      1517.5
      30
      22

      1520.0
      201
      47

      1522.5
      93
      28

      1525.0
      187
      22

      ...
      ...
      ...

      3887.5
      225
      0

      3890.0
      -1
      0

      3892.5
      132
      0

      3895.0
      106
      0

      3897.5
      28
      0

954 rows × 2 columns

```python
data
```

    tensor([[ 29,  77,  65,  62,  32,  59,  68,  25,  23,  25,   9,  62,  65,  85,
              22,  29,  68,  65,  13,  24,  54,  28,  27,   3,   5,  71,  57,  20,
              17,   3,  23,  30,  71,  62,  14,   5,   5,  70,  71,  66,  20,  57,
               3,  74,  24,   3,  15,   9,  15,  13,  25,  59,  16,  16,  25,  13,
              54,  32,  65,  69,  72,   9,  25,  48,  62,   9,  27,  27,  20,   9,
              17,  65,  17,  48,   3,  25,  68,  42,  28,   3,  16,  88,  65,   9,
              22,  62,  69,  65,  29,  16,  72,  79,  29,  14,  65, 111,  69,  10,
              76,  90, 104,  96,  89,  96,  89,  96],
            [ 89,  89,   7,  75,  89,  50,  13,  16,  25,  14,  24,  29,  37,  38,
              68,   9,  16,  14,  22,  68,  62,  69,  69,  62,  79,  69,  69,  79,
              94, 100,  84,  81,   9,  17,  85,  80,  98,  42,  57,  72,  80,  16,
              38,  88,  79,  62,  88,  66,  74,  24,  77,  23,  27,  15,   3,  49,
              22,   5,  17,  37,  27,   5,  16,  30,  59,  59,  27,  24,  24,  29,
              20,  17,  71,  48,  48,  81,  13,  77,  66,  81,  48,  48,   5,  77,
               9,   9,   5,   3,  38,   9,  13,  48,   5,  22,  13,  14,  42,  57,
              24,  62,   9,  53,  38,  57,  22,  25],
            [ 48,  20,  65,  48,  14,  48,   7,  37,  70,  16,  63,  22,  36,  37,
              43,  23,  59,  29,  13,  62,  53,   3,  47,  13,  13,  16,  52,  29,
              63,  15,  60,  67,  52,  47,  60,  53,  43,  18,  21,  73,  39,   2,
              26,  12,  40,  39,  49,   2,  31,  21,  40,  76,  35,  75,   6,  19,
              51,   2,  12,  12,  35,  21,  64,  34,  34,  43,  11,  35,  63,  67,
              10,  39,  21,  21,  34,  21,  43,  33,   7,   4,  12,  58,   6,  40,
              21,  31,  41,  46,  18,  10,  18,  34,  12,  12,  35,  19,  39,  39,
              35,  10,  39,  61,  43,  54,   8,  19],
            [ 35,  12,   4,  39,  41,  46,  51,  35,  50,  41,  46,  61,  35,  64,
               2,  25,  52,  53,  51,   7,  56,  61,  43,  51,  52,  67,  51,  25,
              60,  53,  37,  12,  95,  36,  64,  31,  90,  45,  90,  30,   7,  45,
              52,  49, 111,  18,  11,  27,   8,  33,  29,  58, 116,  73,   7,  20,
               9,  29,  38,  64,  28,  91,  78,  47,  25,  23,  19,   8,  19,   8,
               8,  25,  15,  51,  18,  45,  17,  38,   5,  13,  66,  17,  62,  87,
              67,  33,   4,  13,  20,  26,  31,  58,  14,  53, 113,  72,  42, 128,
              28,  72,  28,  48,  22,  27,   3,  24],
            [ 57,  28,  14,  25,   9,  78,  17,   4,   3,  10,  11,  12,  40,   4,
              10,  34,  36,  36,   3,   3,  66,  25,  37,   2,  34,  12,   8,  15,
              28,   7,  47,  43,  36,  12,  19,  41,  29,   7,  11,  11,  67,  45,
               2,   6,  40,  37,  42,  56,  21,  54,  20,  27,   9,  70,  23,  25,
              13,   3,  79,  38,  18,  82,  33,  49,  67,   9,  23,  28,  17,  62,
              30,  48,  79,  63,  33,  23,  52,   6,  33,  53,  43,  56,  30,   5,
              57,   5,  30,   5,  37,  22,  30,  31,  35,  27, 133,  66,  42,  15,
              13,  16,  60,  15,  10,  31,  72,  92],
            [ 17,   3,  25,  65,   2,  41,  61,  37,  67,  15,  77,  21,   6, 102,
              88,  13,  42,  18,  41,   2,  33,  32,  53,  20,  49,  64,  49,   3,
              54,  15,  20,  14,  23,  30,  20,  28,   9,  23,  45,  31,  10,  63,
              64,  12,  31,  54,   4,  18,  51,  22,  49,  60,  95,  50,  21,  43,
              12,  19,  41,  26,   8,  15,  43,  27,  20,  20,  48,  56,  51,  19,
              41,  75,  90,  39,  61,  75,  89,  89, 104,   2,  11,  87,  66,  17,
              84,  34,  95,  41,  31,  47,  31,  21,  34,  61,  21,  31,  45,  72,
               7,   8,  19,  18,  28,   8,  56,  31],
            [ 64,  33,  45,  51,  15,  22,   7,  11,  13,  17,  65,  81,  11,   9,
              71,   3,  56,  72,  59,   7,  37,  54,  38,  71, 114,  97,  46,  38,
             121,  87,  55,  43,  82,  36,  56,  23,  45,  24,  91,  22,  33,  46,
              68,  28,  42,  36,  55,  16,  71,  74,  28,  74,  71,  48,  80,  85,
              94,  66,  71,  78,  93,  24,  71,  86,  84,  91, 110,  98,  86,  91,
              97, 108,  86,  16,  51,  19,  37,  28,  33,   6,   7,  53,  56,  84,
             108,  91,  74,  81, 102,  71,  14,  18,   9,  23,   7,  18,   6,  64,
              66,  53,   6,  64,  33,   7,  43,  20],
            [ 57,  31,  58,  43,  63,  45,  37,  22,  80, 120,  92, 106,  91,  34,
               6,  13,  63,  43,  24,  79,  65,  78,   9,  65,  71,  80,  86,  86,
              92, 105, 101, 102, 100,  97,  22,  15,  29,  62,  12,  43,  42,  93,
              64,   7,   8,   8,  11,  46,  18,   4,  11,  46,   9,  76,  35,  12,
               8,  34,  56,  11,  69,  12,  68,  45,   2,  21,  43,  45,   2,  55,
              18,  58,   6,  12,  56,  18,  12,   7,  36,   2,  40,  11,  11,  20,
             145,  46,  48,  42,  17,  58,  83,  10,  22,  19,  10,   4,  31,  40,
              63,  35,  12,  68,  23,   2,   8,  45],
            [153,  59,  46,  13,  39,  50,  40,  11,   6,  21,  40,  31,  40,  71,
              26,  34,  55,  17,  86,  32,  63,  49,  56,  36,  34,  54,  27,  11,
               8,   4, 152,  99,  47, 136,  99,  23,  32,  74,  68,  54, 107,  83,
              34,  14,  31,  36,  48,  28,  56,  50,  70,  32,  64,  52,  15,  51,
              45,  79,  70,  65,  34,  61,  76,  35,  11,   2,  43,  74,  18,  26,
               8,  64,  34,  61,   5, 115,  64,  32,  58,  82,  10,  64,  52,  41,
              61,  51,  58,  58,  10,  26,  34,   7,  68,  28,  37,  52,   7,   7,
             138,  21,  78,  77,  31,  61,  21,  34]])

```python
depth = results.index.values
plt.plot(depth, results["Input"], label="Target")
plt.plot(depth, results["Output"], label="Output")
# plt.plot(range(bptt, bptt + len(full_output.view(-1))), full_output.view(-1), label="Output")
plt.legend()
```

```python
full_output = convert_outpupt_to_tensor(output)
```

```python
[vocab.itos[target] for target in targets][0]
```

    '22'

```python
test_data
```

    tensor([[ 29,  77,  65,  ...,  96,  89,  96],
            [ 89,  89,   7,  ...,  57,  22,  25],
            [ 48,  20,  65,  ...,  54,   8,  19],
            ...,
            [ 57,  31,  58,  ...,   2,   8,  45],
            [153,  59,  46,  ...,  61,  21,  34],
            [  4,   7,   6,  ...,  44,  44,  44]])

```python
well[1250::5].head(100)
```

      gamma
      County

      depth

      1250.0
      86.7215
      ELLIS

      1252.5
      100.7469
      ELLIS

      1255.0
      94.2474
      ELLIS

      1257.5
      93.0782
      ELLIS

      1260.0
      76.2082
      ELLIS

      ...
      ...
      ...

      1487.5
      19.1712
      ELLIS

      1490.0
      95.1153
      ELLIS

      1492.5
      44.0031
      ELLIS

      1495.0
      30.0393
      ELLIS

      1497.5
      25.0105
      ELLIS

100 rows × 2 columns

```python
gamma = well[::5]["gamma"]
depth = well[::5].index.values
result_gamma = well[1250::5]["gamma"]
result_depth = well[1250::5].index.values
```

```python
import matplotlib.pyplot as plt
```

```python
fig = plt.figure(figsize=(3,10))

x = well["gamma"].values
# x[np.isnan(x)] = 0 # set NaNs to 0
y = well.index.values #% assign the depth values to y

plt.plot(results["Output"], results.index.values, label="Output", color="red")
plt.plot(well[:1000:5]["gamma"], well[:1000:5].index.values, label="Train", color="gray")
plt.plot(well[1000:1250:5]["gamma"], well[1000:1250:5].index.values, label="Validation", color="yellow")
plt.plot(well[1250::5]["gamma"], well[1250::5].index.values, label="Target", color="blue")
ax = plt.gca()
# this does the plotting, we plot gamma ray as a function of depth. note we subtract the kelly bushing elevation (EKB) to get subsea depth
ax.invert_yaxis()
plt.legend()
```

```python
def plot_well(well):
    fig = plt.figure(figsize=(2,10))

    x = well["gamma"].to_numpy(dtype='int64')
    x[np.isnan(x)] = 0 # set NaNs to 0
    y = well.index.values #% assign the depth values to y

    plt.plot(x, y, color="gray", )
    ax = plt.gca()
    # this does the plotting, we plot gamma ray as a function of depth. note we subtract the kelly bushing elevation (EKB) to get subsea depth
    ax.invert_yaxis()

    # let's plot the fake tops from the csv file as a horizontal line, don't forget to subtract the KB elevation for subsea depth
    # ax.axis("off")
    # this gets rid of the axis so they look clean
    ax.set_title("Predictions")
    # and let's go ahead and add titles to each log for fun
```

```python
def is_float(i):
    try:
        float(i)
        return True
    except:
        return False
```

```python
import pandas as pd
wells = pd.read_csv("export_csv/2016.csv", index_col=[1, 0])
```

    /home/fastfist/.local/share/virtualenvs/transformers-kgQS_UHh/lib/python3.9/site-packages/IPython/core/interactiveshell.py:3165: DtypeWarning: Columns (2) have mixed types.Specify dtype option on import or set low_memory=False.
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    /home/fastfist/.local/share/virtualenvs/transformers-kgQS_UHh/lib/python3.9/site-packages/numpy/lib/arraysetops.py:580: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
      mask |= (ar1 == a)

```python
z = wells['gamma'].unique()
```

```python
for i in z:
    try:
        float(i)
    except:
        print("oops" + i)
```

    oopsWICHITA
    oopsRAWLINS
    oopsRUSH

```python
wells.loc[0]
```

      gamma
      County

      depth

      0.0
      258.3637
      SCOTT

      0.5
      172.4415
      SCOTT

      1.0
      133.0665
      SCOTT

      1.5
      134.8826
      SCOTT

      2.0
      156.8733
      SCOTT

      ...
      ...
      ...

      4707.0
      NaN
      SCOTT

      4707.5
      NaN
      SCOTT

      4708.0
      NaN
      SCOTT

      4708.5
      NaN
      SCOTT

      4709.0
      NaN
      SCOTT

9419 rows × 2 columns

```python
z = wells.dropna().loc[0]
plot_well(z)
```

```python
wells
```

      gamma
      County

      well_id
      depth

      2282
      0.0
      64.6673
      TREGO

      0.5
      73.8969
      TREGO

      1.0
      62.0081
      TREGO

      1.5
      54.4317
      TREGO

      2.0
      58.8753
      TREGO

      ...
      ...
      ...
      ...

      0
      4707.0
      NaN
      SCOTT

      4707.5
      NaN
      SCOTT

      4708.0
      NaN
      SCOTT

      4708.5
      NaN
      SCOTT

      4709.0
      NaN
      SCOTT

20205274 rows × 2 columns