# KAIST AI605 Assignment 3: Transformer

TA in charge: Jaehyeong Jo (harryjo97@kaist.ac.kr)

**Due date**:  Nov 22 (Mon) 11:00pm, 2021

## Your Submission
If you are a KAIST student, you will submit your assignment via [KLMS](https://klms.kaist.ac.kr). If you are a NAVER student, you will submit via [Google Form](https://forms.gle/aGZZ86YpCdv2zEVt9).

You need to submit both (1) a PDF of this notebook, and (2) a link to CoLab for execution (.ipynb file is also allowed).

Use in-line LaTeX (see below) for mathematical expressions. Collaboration among students is allowed but it is not a group assignment so make sure your answer and code are your own. Make sure to mention your collaborators in your assignment with their names and their student ids.

## Grading
The entire assignment is out of 20 points. You can obtain up to 3 bonus points (i.e. max score is 23 points). For every late day, your grade will be deducted by 2 points (KAIST students only). You can use one of your no-penalty late days (7 days in total). Make sure to mention this in your submission. You will receive a grade of zero if you submit after 7 days.

## Environment
You will only use Python 3.7 and PyTorch 1.9, which is already available on Colab:

# **<<This is a Late Submission !!! (5 days)>>**

```python
from platform import python_version
import torch

print("python", python_version())
print("torch", torch.__version__)
```

    python 3.8.3
    torch 1.7.0

## 1. Attention Layer

We will first start with going over a few concepts that you learned in your high school statistics class. The variance of a random variable $X$, $\text{Var}(X)$ is defined as $\text{E}[(X-\mu)^2]$ where $\mu$ is the mean of $X$. Furthermore, given two independent random variables $X$ and $Y$ and a constant $a$,
$$ \text{Var}(X+Y) = \text{Var}(X) + \text{Var}(Y),$$
$$ \text{Var}(aX) = a^2\text{Var}(X),$$
$$ \text{Var}(XY) = \text{E}(X^2)\text{E}(Y^2) - [\text{E}(X)]^2[\text{E}(Y)]^2.$$

> **Problem 1.1** *(3 points)* Suppose we are given two sets of $n$ random variables, $X_1 \dots X_n$ and $Y_1 \dots Y_n$, where all of these $2n$ variables are mutually independent and have a mean of $0$ and a variance of $1$. Prove that
$$\text{Var}\left(\sum_i^n X_i Y_i\right) = n.$$

> **Answer 1.1**

\begin{align}
\text{Var}\big(\sum_i^n X_i Y_i\big) &= \sum_i^n\text{Var}(X_i Y_i) ~~ \text{(by indep.)} \\
&= \sum_i^n E(X_i^2)E(Y_i^2) - E(X_i)^2 E(Y_i)^2 \\
&= \sum_i^n 1\cdot 1 - 0 \cdot 0 \\
&= n
\end{align}

In Lecture 11 and 12, we discussed how the attention is computed in Transformer via the following equation,
$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$
> **Problem 1.2** *(3 points)*  Suppose $Q$ and $K$ are matrices of independent variables each of which has a mean of $0$ and a variance of $1$. Using what you learned from Problem 1.1., show that
$$\text{Var}\left(\frac{QK^\top}{\sqrt{d_k}}\right) = 1.$$

> **Answer 1.2**

Let $Q\in \mathbb{R}^{B\times d_k}$ and $K\in \mathbb{R}^{B\times d_k}$.
For every $\text{Var}(Q_i K_j^\top), \forall i,j\in[B]$, since $Q_i$ and $K_j^\top$ are mutually independent and have zero mean and unit variance, the following holds:

\begin{align}
\text{Var}\big(\frac{Q_i K_j^\top}{\sqrt{d_k}}\big) &= \frac{1}{d_k} \text{Var}\big(Q_i K_j^\top \big) \\
&= \frac{1}{d_k} \text{Var}\big(\sum_{k=1}^{d_k} Q_{ik} K_{kj}^\top \big) \\
&= \frac{1}{d_k} \cdot d_k ~~~ \text{(Problem 1.1)} \\
&= 1
\end{align}

> **Problem 1.3** *(4 points)* What would happen if the assumption that the variance of $Q$ and $K$ is $1$ does not hold? Consider each case of it being higher and lower than $1$ and conjecture what it implies, respectively.

>**Answer 1.3**

If the variance of $Q$ and $K$ are higher than $1$,  $\text{Var}(QK^T) \ge d_k$. Therefore, scaling with $\sqrt{d_k}$ may not be a good choice. Therefore, scaling with $\sqrt{d_k}$ is not good enough; considering the softmax operation in Attention formula, the temperature scale of $\sqrt{d_k}$ makes a sharp distribution, attending only a few points. On the other hand, if the variance of $Q$ and $K$ are smaller than $1$, $\text{Var}(QK^T) \le d_k$. In this case, considering the softmax operation in Attention formula, the temperature scale of $\sqrt{d_k}$ makes a flat distribution, giving every similar attention scores.

## 2. Transformer

In this section, you will implement Transformer for a few tasks that are simpler than machine translation. First, go through [Annotated Transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html) and make sure you understand every block of the code. Then, you will reuse these code where appropriate to create models for following three tasks. Note that we do not provide a separate training or evaluation data, so it is your job to be able to create these in a reasonable manner.

# Problem 2.1

> **Problem 2.1** *(4 points)* Create a model that takes a random set of input symbols from a vocabulary of digits (i.e. 0, 1, ... , 8, 9) as the input and generate back the same symbols. Instead of varying length, we fix the length to 32. Make sure to report that your model's accuracy (gives credit only if the entire output sequence is correct) goes above 90%. Note that a similar problem is also in Annotated Transformer, and copying code is allowed.

```python
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import math, copy, time
from torch.autograd import Variable
import matplotlib.pyplot as plt
import seaborn
seaborn.set_context(context="talk")
%matplotlib inline
```

```python
class EncoderDecoder(nn.Module):
    """
    A standard Encoder-Decoder architecture. Base for this and many
    other models.
    """
    def __init__(self, encoder, decoder, src_embed, tgt_embed, generator):
        super(EncoderDecoder, self).__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.src_embed = src_embed
        self.tgt_embed = tgt_embed
        self.generator = generator

    def forward(self, src, tgt, src_mask, tgt_mask):
        "Take in and process masked src and target sequences."
        return self.decode(self.encode(src, src_mask), src_mask,
                            tgt, tgt_mask)

    def encode(self, src, src_mask):
        return self.encoder(self.src_embed(src), src_mask)

    def decode(self, memory, src_mask, tgt, tgt_mask):
        return self.decoder(self.tgt_embed(tgt), memory, src_mask, tgt_mask)

class Generator(nn.Module):
    "Define standard linear + softmax generation step."
    def __init__(self, d_model, vocab):
        super(Generator, self).__init__()
        self.proj = nn.Linear(d_model, vocab)

    def forward(self, x):
        return F.log_softmax(self.proj(x), dim=-1)

def clones(module, N):
    "Produce N identical layers."
    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])

class Encoder(nn.Module):
    "Core encoder is a stack of N layers"
    def __init__(self, layer, N):
        super(Encoder, self).__init__()
        self.layers = clones(layer, N)
        self.norm = LayerNorm(layer.size)

    def forward(self, x, mask):
        "Pass the input (and mask) through each layer in turn."
        for layer in self.layers:
            x = layer(x, mask)
        return self.norm(x)

class LayerNorm(nn.Module):
    "Construct a layernorm module (See citation for details)."
    def __init__(self, features, eps=1e-6):
        super(LayerNorm, self).__init__()
        self.a_2 = nn.Parameter(torch.ones(features))
        self.b_2 = nn.Parameter(torch.zeros(features))
        self.eps = eps

    def forward(self, x):
        mean = x.mean(-1, keepdim=True)
        std = x.std(-1, keepdim=True)
        return self.a_2 * (x - mean) / (std + self.eps) + self.b_2

class SublayerConnection(nn.Module):
    """
    A residual connection followed by a layer norm.
    Note for code simplicity the norm is first as opposed to last.
    """
    def __init__(self, size, dropout):
        super(SublayerConnection, self).__init__()
        self.norm = LayerNorm(size)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, sublayer):
        "Apply residual connection to any sublayer with the same size."
        return x + self.dropout(sublayer(self.norm(x)))

class EncoderLayer(nn.Module):
    "Encoder is made up of self-attn and feed forward (defined below)"
    def __init__(self, size, self_attn, feed_forward, dropout):
        super(EncoderLayer, self).__init__()
        self.self_attn = self_attn
        self.feed_forward = feed_forward
        self.sublayer = clones(SublayerConnection(size, dropout), 2)
        self.size = size

    def forward(self, x, mask):
        "Follow Figure 1 (left) for connections."
        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))
        return self.sublayer[1](x, self.feed_forward)

class Decoder(nn.Module):
    "Generic N layer decoder with masking."
    def __init__(self, layer, N):
        super(Decoder, self).__init__()
        self.layers = clones(layer, N)
        self.norm = LayerNorm(layer.size)

    def forward(self, x, memory, src_mask, tgt_mask):
        for layer in self.layers:
            x = layer(x, memory, src_mask, tgt_mask)
        return self.norm(x)

class DecoderLayer(nn.Module):
    "Decoder is made of self-attn, src-attn, and feed forward (defined below)"
    def __init__(self, size, self_attn, src_attn, feed_forward, dropout):
        super(DecoderLayer, self).__init__()
        self.size = size
        self.self_attn = self_attn
        self.src_attn = src_attn
        self.feed_forward = feed_forward
        self.sublayer = clones(SublayerConnection(size, dropout), 3)

    def forward(self, x, memory, src_mask, tgt_mask):
        "Follow Figure 1 (right) for connections."
        m = memory
        x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, tgt_mask))
        x = self.sublayer[1](x, lambda x: self.src_attn(x, m, m, src_mask))
        return self.sublayer[2](x, self.feed_forward)

def subsequent_mask(size):
    "Mask out subsequent positions."
    attn_shape = (1, size, size)
    subsequent_mask = np.triu(np.ones(attn_shape), k=1).astype('uint8')
    return torch.from_numpy(subsequent_mask) == 0
```

```python
def attention(query, key, value, mask=None, dropout=None):
    "Compute 'Scaled Dot Product Attention'"
    d_k = query.size(-1)
    scores = torch.matmul(query, key.transpose(-2, -1)) \
             / math.sqrt(d_k)
    if mask is not None:
        scores = scores.masked_fill(mask == 0, -1e9)
    p_attn = F.softmax(scores, dim = -1)
    if dropout is not None:
        p_attn = dropout(p_attn)
    return torch.matmul(p_attn, value), p_attn

class MultiHeadedAttention(nn.Module):
    def __init__(self, h, d_model, dropout=0.1):
        "Take in model size and number of heads."
        super(MultiHeadedAttention, self).__init__()
        assert d_model % h == 0
        # We assume d_v always equals d_k
        self.d_k = d_model // h
        self.h = h
        self.linears = clones(nn.Linear(d_model, d_model), 4)
        self.attn = None
        self.dropout = nn.Dropout(p=dropout)

    def forward(self, query, key, value, mask=None):
        "Implements Figure 2"
        if mask is not None:
            # Same mask applied to all h heads.
            mask = mask.unsqueeze(1)
        nbatches = query.size(0)

        # 1) Do all the linear projections in batch from d_model => h x d_k
        query, key, value = \
            [l(x).view(nbatches, -1, self.h, self.d_k).transpose(1, 2)
             for l, x in zip(self.linears, (query, key, value))]

        # 2) Apply attention on all the projected vectors in batch.
        x, self.attn = attention(query, key, value, mask=mask,
                                 dropout=self.dropout)

        # 3) "Concat" using a view and apply a final linear.
        x = x.transpose(1, 2).contiguous() \
             .view(nbatches, -1, self.h * self.d_k)
        return self.linears[-1](x)

class PositionwiseFeedForward(nn.Module):
    "Implements FFN equation."
    def __init__(self, d_model, d_ff, dropout=0.1):
        super(PositionwiseFeedForward, self).__init__()
        self.w_1 = nn.Linear(d_model, d_ff)
        self.w_2 = nn.Linear(d_ff, d_model)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x):
        return self.w_2(self.dropout(F.relu(self.w_1(x))))

class Embeddings(nn.Module):
    def __init__(self, d_model, vocab):
        super(Embeddings, self).__init__()
        self.lut = nn.Embedding(vocab, d_model)
        self.d_model = d_model

    def forward(self, x):
        return self.lut(x) * math.sqrt(self.d_model)

class PositionalEncoding(nn.Module):
    "Implement the PE function."
    def __init__(self, d_model, dropout, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        # Compute the positional encodings once in log space.
        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2) *
                             -(math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + Variable(self.pe[:, :x.size(1)],
                         requires_grad=False)
        return self.dropout(x)
```

```python
def make_model(src_vocab, tgt_vocab, N=6,
               d_model=512, d_ff=2048, h=8, dropout=0.1):
    "Helper: Construct a model from hyperparameters."
    c = copy.deepcopy
    attn = MultiHeadedAttention(h, d_model)
    ff = PositionwiseFeedForward(d_model, d_ff, dropout)
    position = PositionalEncoding(d_model, dropout)
    model = EncoderDecoder(
        Encoder(EncoderLayer(d_model, c(attn), c(ff), dropout), N),
        Decoder(DecoderLayer(d_model, c(attn), c(attn),
                             c(ff), dropout), N),
        nn.Sequential(Embeddings(d_model, src_vocab), c(position)),
        nn.Sequential(Embeddings(d_model, tgt_vocab), c(position)),
        Generator(d_model, tgt_vocab))

    # This was important from their code.
    # Initialize parameters with Glorot / fan_avg.
    for p in model.parameters():
        if p.dim() > 1:
            nn.init.xavier_uniform(p)
    return model

class Batch:
    "Object for holding a batch of data with mask during training."
    def __init__(self, src, trg=None, pad=0):
        self.src = src
        self.src_mask = (src != pad).unsqueeze(-2)
        if trg is not None:
            self.trg = trg[:, :-1]
            self.trg_y = trg[:, 1:]
            self.trg_mask = \
                self.make_std_mask(self.trg, pad)
            self.ntokens = (self.trg_y != pad).data.sum()

    @staticmethod
    def make_std_mask(tgt, pad):
        "Create a mask to hide padding and future words."
        tgt_mask = (tgt != pad).unsqueeze(-2)
        tgt_mask = tgt_mask & Variable(
            subsequent_mask(tgt.size(-1)).type_as(tgt_mask.data))
        return tgt_mask

class NoamOpt:
    "Optim wrapper that implements rate."
    def __init__(self, model_size, factor, warmup, optimizer):
        self.optimizer = optimizer
        self._step = 0
        self.warmup = warmup
        self.factor = factor
        self.model_size = model_size
        self._rate = 0

    def step(self):
        "Update parameters and rate"
        self._step += 1
        rate = self.rate()
        for p in self.optimizer.param_groups:
            p['lr'] = rate
        self._rate = rate
        self.optimizer.step()

    def rate(self, step = None):
        "Implement `lrate` above"
        if step is None:
            step = self._step
        return self.factor * \
            (self.model_size ** (-0.5) *
            min(step ** (-0.5), step * self.warmup ** (-1.5)))

def get_std_opt(model):
    return NoamOpt(model.src_embed[0].d_model, 2, 4000,
            torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))

class LabelSmoothing(nn.Module):
    "Implement label smoothing."
    def __init__(self, size, padding_idx, smoothing=0.0):
        super(LabelSmoothing, self).__init__()
        self.criterion = nn.KLDivLoss(size_average=False)
        self.padding_idx = padding_idx
        self.confidence = 1.0 - smoothing
        self.smoothing = smoothing
        self.size = size
        self.true_dist = None

    def forward(self, x, target):
        assert x.size(1) == self.size
        true_dist = x.data.clone()
        true_dist.fill_(self.smoothing / (self.size - 2))
        true_dist.scatter_(1, target.data.unsqueeze(1), self.confidence)
        true_dist[:, self.padding_idx] = 0
        mask = torch.nonzero(target.data == self.padding_idx)
        if mask.dim() > 0:
            true_dist.index_fill_(0, mask.squeeze(), 0.0)
        self.true_dist = true_dist
        return self.criterion(x, Variable(true_dist, requires_grad=False))

class SimpleLossCompute:
    "A simple loss compute and train function."
    def __init__(self, generator, criterion, opt=None):
        self.generator = generator
        self.criterion = criterion
        self.opt = opt

    def __call__(self, x, y, norm):
        seq_len = y.size(-1)
        x = self.generator(x)
        pred = x.data.max(dim=-1)[1]
        correct = ((pred == y).sum(dim=-1) == seq_len).sum().item()
        loss = self.criterion(x.contiguous().view(-1, x.size(-1)),
                              y.contiguous().view(-1)) / norm
        loss.backward()
        if self.opt is not None:
            self.opt.step()
            self.opt.optimizer.zero_grad()
        return loss.data * norm, correct
```

```python
def data_gen(V, batch, nbatches):
    "Generate random data for a src-tgt copy task."
    for i in range(nbatches):
        data = torch.from_numpy(np.random.randint(1, V, size=(batch, 33)))
        data[:, 0] = 1
        src = Variable(data, requires_grad=False)
        tgt = Variable(data, requires_grad=False)
        yield Batch(src, tgt, 0)
```

```python
def run_epoch(data_iter, model, loss_compute, epoch):
    "Standard Training and Logging Function"
    start = time.time()
    total_tokens = 0
    total_loss = 0
    tokens = 0
    acc = 0
    cnt = 0
    for i, batch in enumerate(data_iter):
        out = model.forward(batch.src, batch.trg,
                            batch.src_mask, batch.trg_mask)
        loss, correct = loss_compute(out, batch.trg_y, batch.ntokens)
        total_loss += loss
        total_tokens += batch.ntokens
        tokens += batch.ntokens
        acc += correct
        cnt += batch.trg_y.shape[0]
        if (i+1) % 20 == 0:
            elapsed = time.time() - start
            print("Epoch: %d Step: %d Loss: %f Tokens per Sec: %f" %
                    (epoch, i+1, loss / batch.ntokens, tokens / elapsed))
            start = time.time()
            tokens = 0
    return total_loss / total_tokens, acc / cnt
```

```python
# Train the simple copy task.
V = 10
criterion = LabelSmoothing(size=V, padding_idx=0, smoothing=0.0)
model = make_model(V, V, N=2)
model_opt = NoamOpt(model.src_embed[0].d_model, 1, 400,
        torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))

for epoch in range(100):
    model.train()
    run_epoch(data_gen(V, 30, 20), model,
              SimpleLossCompute(model.generator, criterion, model_opt), epoch)
    model.eval()
    loss, acc = run_epoch(data_gen(V, 30, 5), model,
                    SimpleLossCompute(model.generator, criterion, None), epoch)
    print('Validation Loss: {:.4f}, Validation Acc: {:.2f}%'.format(loss, acc * 100))
```

    /home/sungnyun/anaconda3/lib/python3.8/site-packages/torch/nn/_reduction.py:44: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.
      warnings.warn(warning.format(ret))
    <ipython-input-5-5fe0956476b4>:20: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
      nn.init.xavier_uniform(p)

    Epoch: 0 Step: 20 Loss: 2.292718 Tokens per Sec: 6041.841309
    Validation Loss: 2.1984, Validation Acc: 0.00%
    Epoch: 1 Step: 20 Loss: 2.213340 Tokens per Sec: 6070.613281
    Validation Loss: 2.1488, Validation Acc: 0.00%
    Epoch: 2 Step: 20 Loss: 2.113729 Tokens per Sec: 6122.917969
    Validation Loss: 2.1139, Validation Acc: 0.00%
    Epoch: 3 Step: 20 Loss: 2.154241 Tokens per Sec: 6077.809570
    Validation Loss: 2.0846, Validation Acc: 0.00%
    Epoch: 4 Step: 20 Loss: 2.106472 Tokens per Sec: 6212.707520
    Validation Loss: 2.0769, Validation Acc: 0.00%
    Epoch: 5 Step: 20 Loss: 2.050679 Tokens per Sec: 6183.625977
    Validation Loss: 2.0213, Validation Acc: 0.00%
    Epoch: 6 Step: 20 Loss: 2.006129 Tokens per Sec: 6050.239258
    Validation Loss: 1.9740, Validation Acc: 0.00%
    Epoch: 7 Step: 20 Loss: 1.999506 Tokens per Sec: 6232.646973
    Validation Loss: 1.9442, Validation Acc: 0.00%
    Epoch: 8 Step: 20 Loss: 1.708790 Tokens per Sec: 6243.300293
    Validation Loss: 1.4965, Validation Acc: 0.00%
    Epoch: 9 Step: 20 Loss: 1.723271 Tokens per Sec: 6220.234863
    Validation Loss: 1.6099, Validation Acc: 0.00%
    Epoch: 10 Step: 20 Loss: 1.380073 Tokens per Sec: 6154.426270
    Validation Loss: 1.1224, Validation Acc: 0.00%
    Epoch: 11 Step: 20 Loss: 1.907574 Tokens per Sec: 6223.563965
    Validation Loss: 1.8242, Validation Acc: 0.00%
    Epoch: 12 Step: 20 Loss: 1.677559 Tokens per Sec: 6269.504883
    Validation Loss: 1.5412, Validation Acc: 0.00%
    Epoch: 13 Step: 20 Loss: 1.519670 Tokens per Sec: 6226.383301
    Validation Loss: 1.3235, Validation Acc: 0.00%
    Epoch: 14 Step: 20 Loss: 1.207654 Tokens per Sec: 6252.952148
    Validation Loss: 0.9522, Validation Acc: 0.00%
    Epoch: 15 Step: 20 Loss: 0.944293 Tokens per Sec: 6227.490234
    Validation Loss: 0.7400, Validation Acc: 0.00%
    Epoch: 16 Step: 20 Loss: 1.121010 Tokens per Sec: 6222.080078
    Validation Loss: 0.8646, Validation Acc: 0.00%
    Epoch: 17 Step: 20 Loss: 0.689292 Tokens per Sec: 6214.712402
    Validation Loss: 0.4624, Validation Acc: 0.00%
    Epoch: 18 Step: 20 Loss: 0.622735 Tokens per Sec: 6210.918945
    Validation Loss: 0.3827, Validation Acc: 0.67%
    Epoch: 19 Step: 20 Loss: 0.550777 Tokens per Sec: 6191.287109
    Validation Loss: 0.2856, Validation Acc: 3.33%
    Epoch: 20 Step: 20 Loss: 0.402400 Tokens per Sec: 6125.035156
    Validation Loss: 0.1697, Validation Acc: 14.67%
    Epoch: 21 Step: 20 Loss: 0.254299 Tokens per Sec: 6029.789062
    Validation Loss: 0.0869, Validation Acc: 39.33%
    Epoch: 22 Step: 20 Loss: 0.151851 Tokens per Sec: 5955.750488
    Validation Loss: 0.0604, Validation Acc: 44.00%
    Epoch: 23 Step: 20 Loss: 0.131954 Tokens per Sec: 5991.244629
    Validation Loss: 0.0429, Validation Acc: 71.33%
    Epoch: 24 Step: 20 Loss: 0.142635 Tokens per Sec: 5923.452637
    Validation Loss: 0.0282, Validation Acc: 71.33%
    Epoch: 25 Step: 20 Loss: 0.103245 Tokens per Sec: 5965.190918
    Validation Loss: 0.0268, Validation Acc: 80.67%
    Epoch: 26 Step: 20 Loss: 0.107378 Tokens per Sec: 5924.616699
    Validation Loss: 0.0497, Validation Acc: 60.00%
    Epoch: 27 Step: 20 Loss: 0.088820 Tokens per Sec: 5887.970703
    Validation Loss: 0.0381, Validation Acc: 72.00%
    Epoch: 28 Step: 20 Loss: 0.111645 Tokens per Sec: 5734.285645
    Validation Loss: 0.0332, Validation Acc: 70.00%
    Epoch: 29 Step: 20 Loss: 0.084944 Tokens per Sec: 5711.217773
    Validation Loss: 0.0112, Validation Acc: 92.00%
    Epoch: 30 Step: 20 Loss: 0.075890 Tokens per Sec: 5587.509766
    Validation Loss: 0.0220, Validation Acc: 78.00%
    Epoch: 31 Step: 20 Loss: 0.058663 Tokens per Sec: 5637.533203
    Validation Loss: 0.0080, Validation Acc: 93.33%
    Epoch: 32 Step: 20 Loss: 0.052396 Tokens per Sec: 5440.664062
    Validation Loss: 0.0275, Validation Acc: 76.67%
    Epoch: 33 Step: 20 Loss: 0.057778 Tokens per Sec: 5551.412109
    Validation Loss: 0.0200, Validation Acc: 81.33%
    Epoch: 34 Step: 20 Loss: 0.119597 Tokens per Sec: 5515.170898
    Validation Loss: 0.0142, Validation Acc: 87.33%
    Epoch: 35 Step: 20 Loss: 0.104406 Tokens per Sec: 5296.281250
    Validation Loss: 0.0178, Validation Acc: 89.33%
    Epoch: 36 Step: 20 Loss: 0.041032 Tokens per Sec: 4613.054199
    Validation Loss: 0.0083, Validation Acc: 92.00%
    Epoch: 37 Step: 20 Loss: 0.055340 Tokens per Sec: 4179.380371
    Validation Loss: 0.0274, Validation Acc: 78.67%
    Epoch: 38 Step: 20 Loss: 0.063703 Tokens per Sec: 4418.728516
    Validation Loss: 0.0120, Validation Acc: 88.00%
    Epoch: 39 Step: 20 Loss: 0.072006 Tokens per Sec: 4192.512695
    Validation Loss: 0.0061, Validation Acc: 95.33%
    Epoch: 40 Step: 20 Loss: 0.049078 Tokens per Sec: 4066.876953
    Validation Loss: 0.0119, Validation Acc: 90.00%
    Epoch: 41 Step: 20 Loss: 0.066630 Tokens per Sec: 3994.974365
    Validation Loss: 0.0112, Validation Acc: 90.67%
    Epoch: 42 Step: 20 Loss: 0.044727 Tokens per Sec: 3870.378418
    Validation Loss: 0.0167, Validation Acc: 86.00%
    Epoch: 43 Step: 20 Loss: 0.026363 Tokens per Sec: 3604.613281
    Validation Loss: 0.0068, Validation Acc: 92.00%
    Epoch: 44 Step: 20 Loss: 0.019874 Tokens per Sec: 3566.242188
    Validation Loss: 0.0051, Validation Acc: 96.00%
    Epoch: 45 Step: 20 Loss: 0.043552 Tokens per Sec: 3579.613525
    Validation Loss: 0.0069, Validation Acc: 95.33%
    Epoch: 46 Step: 20 Loss: 0.045021 Tokens per Sec: 3526.715576
    Validation Loss: 0.0072, Validation Acc: 94.67%
    Epoch: 47 Step: 20 Loss: 0.060334 Tokens per Sec: 3318.684082
    Validation Loss: 0.0208, Validation Acc: 80.67%
    Epoch: 48 Step: 20 Loss: 0.017947 Tokens per Sec: 3484.596924
    Validation Loss: 0.0036, Validation Acc: 96.67%
    Epoch: 49 Step: 20 Loss: 0.039517 Tokens per Sec: 3452.861328
    Validation Loss: 0.0100, Validation Acc: 88.00%
    Epoch: 50 Step: 20 Loss: 0.031835 Tokens per Sec: 3398.649414
    Validation Loss: 0.0099, Validation Acc: 93.33%
    Epoch: 51 Step: 20 Loss: 0.016612 Tokens per Sec: 3538.441650
    Validation Loss: 0.0098, Validation Acc: 91.33%
    Epoch: 52 Step: 20 Loss: 0.025570 Tokens per Sec: 3634.822998
    Validation Loss: 0.0070, Validation Acc: 93.33%
    Epoch: 53 Step: 20 Loss: 0.053554 Tokens per Sec: 3588.864990
    Validation Loss: 0.0238, Validation Acc: 80.00%
    Epoch: 54 Step: 20 Loss: 0.040647 Tokens per Sec: 3496.999512
    Validation Loss: 0.0022, Validation Acc: 98.00%
    Epoch: 55 Step: 20 Loss: 0.028060 Tokens per Sec: 3448.916260
    Validation Loss: 0.0019, Validation Acc: 98.00%
    Epoch: 56 Step: 20 Loss: 0.044370 Tokens per Sec: 3358.057373
    Validation Loss: 0.0084, Validation Acc: 95.33%
    Epoch: 57 Step: 20 Loss: 0.037871 Tokens per Sec: 3381.295410
    Validation Loss: 0.0077, Validation Acc: 94.00%
    Epoch: 58 Step: 20 Loss: 0.015135 Tokens per Sec: 3324.985840
    Validation Loss: 0.0046, Validation Acc: 94.00%
    Epoch: 59 Step: 20 Loss: 0.024038 Tokens per Sec: 3315.074463
    Validation Loss: 0.0040, Validation Acc: 96.67%
    Epoch: 60 Step: 20 Loss: 0.067089 Tokens per Sec: 3333.395508
    Validation Loss: 0.0073, Validation Acc: 91.33%
    Epoch: 61 Step: 20 Loss: 0.017708 Tokens per Sec: 3268.923096
    Validation Loss: 0.0053, Validation Acc: 97.33%
    Epoch: 62 Step: 20 Loss: 0.014599 Tokens per Sec: 3341.951172
    Validation Loss: 0.0017, Validation Acc: 98.67%
    Epoch: 63 Step: 20 Loss: 0.025191 Tokens per Sec: 3496.516357
    Validation Loss: 0.0018, Validation Acc: 98.67%
    Epoch: 64 Step: 20 Loss: 0.026372 Tokens per Sec: 3407.743652
    Validation Loss: 0.0071, Validation Acc: 95.33%
    Epoch: 65 Step: 20 Loss: 0.006839 Tokens per Sec: 3438.436279
    Validation Loss: 0.0043, Validation Acc: 95.33%
    Epoch: 66 Step: 20 Loss: 0.004360 Tokens per Sec: 3355.526855
    Validation Loss: 0.0027, Validation Acc: 97.33%
    Epoch: 67 Step: 20 Loss: 0.017144 Tokens per Sec: 3384.563965
    Validation Loss: 0.0010, Validation Acc: 99.33%
    Epoch: 68 Step: 20 Loss: 0.014897 Tokens per Sec: 3278.699707
    Validation Loss: 0.0017, Validation Acc: 98.67%
    Epoch: 69 Step: 20 Loss: 0.040671 Tokens per Sec: 3218.296631
    Validation Loss: 0.0040, Validation Acc: 96.00%
    Epoch: 70 Step: 20 Loss: 0.024125 Tokens per Sec: 3299.912354
    Validation Loss: 0.0037, Validation Acc: 98.00%
    Epoch: 71 Step: 20 Loss: 0.013309 Tokens per Sec: 3357.751953
    Validation Loss: 0.0033, Validation Acc: 95.33%
    Epoch: 72 Step: 20 Loss: 0.019307 Tokens per Sec: 3485.481201
    Validation Loss: 0.0034, Validation Acc: 98.67%
    Epoch: 73 Step: 20 Loss: 0.022947 Tokens per Sec: 3528.917236
    Validation Loss: 0.0009, Validation Acc: 99.33%
    Epoch: 74 Step: 20 Loss: 0.013794 Tokens per Sec: 3420.325684
    Validation Loss: 0.0011, Validation Acc: 100.00%
    Epoch: 75 Step: 20 Loss: 0.031032 Tokens per Sec: 3348.551270
    Validation Loss: 0.0064, Validation Acc: 93.33%
    Epoch: 76 Step: 20 Loss: 0.028775 Tokens per Sec: 3435.060059
    Validation Loss: 0.0029, Validation Acc: 97.33%
    Epoch: 77 Step: 20 Loss: 0.010420 Tokens per Sec: 3289.468994
    Validation Loss: 0.0007, Validation Acc: 98.67%
    Epoch: 78 Step: 20 Loss: 0.027443 Tokens per Sec: 3389.405518
    Validation Loss: 0.0008, Validation Acc: 100.00%
    Epoch: 79 Step: 20 Loss: 0.017726 Tokens per Sec: 3352.237793
    Validation Loss: 0.0082, Validation Acc: 90.00%
    Epoch: 80 Step: 20 Loss: 0.013686 Tokens per Sec: 3336.656738
    Validation Loss: 0.0006, Validation Acc: 100.00%
    Epoch: 81 Step: 20 Loss: 0.014543 Tokens per Sec: 3355.456055
    Validation Loss: 0.0020, Validation Acc: 98.00%
    Epoch: 82 Step: 20 Loss: 0.008503 Tokens per Sec: 3329.368164
    Validation Loss: 0.0010, Validation Acc: 99.33%
    Epoch: 83 Step: 20 Loss: 0.020764 Tokens per Sec: 3447.933350
    Validation Loss: 0.0006, Validation Acc: 100.00%
    Epoch: 84 Step: 20 Loss: 0.019289 Tokens per Sec: 3430.149658
    Validation Loss: 0.0010, Validation Acc: 99.33%
    Epoch: 85 Step: 20 Loss: 0.001286 Tokens per Sec: 3373.646484
    Validation Loss: 0.0020, Validation Acc: 98.00%
    Epoch: 86 Step: 20 Loss: 0.015896 Tokens per Sec: 3399.944336
    Validation Loss: 0.0013, Validation Acc: 99.33%
    Epoch: 87 Step: 20 Loss: 0.004553 Tokens per Sec: 3353.735840
    Validation Loss: 0.0004, Validation Acc: 100.00%
    Epoch: 88 Step: 20 Loss: 0.009902 Tokens per Sec: 3435.781006
    Validation Loss: 0.0052, Validation Acc: 94.67%
    Epoch: 89 Step: 20 Loss: 0.023687 Tokens per Sec: 3421.135742
    Validation Loss: 0.0050, Validation Acc: 96.67%
    Epoch: 90 Step: 20 Loss: 0.020975 Tokens per Sec: 3297.374268
    Validation Loss: 0.0007, Validation Acc: 99.33%
    Epoch: 91 Step: 20 Loss: 0.039933 Tokens per Sec: 3293.447998
    Validation Loss: 0.0002, Validation Acc: 100.00%
    Epoch: 92 Step: 20 Loss: 0.003902 Tokens per Sec: 3363.353760
    Validation Loss: 0.0178, Validation Acc: 90.00%
    Epoch: 93 Step: 20 Loss: 0.006548 Tokens per Sec: 3430.113281
    Validation Loss: 0.0008, Validation Acc: 99.33%
    Epoch: 94 Step: 20 Loss: 0.008626 Tokens per Sec: 3401.906494
    Validation Loss: 0.0105, Validation Acc: 94.00%
    Epoch: 95 Step: 20 Loss: 0.021188 Tokens per Sec: 3405.308350
    Validation Loss: 0.0028, Validation Acc: 97.33%
    Epoch: 96 Step: 20 Loss: 0.002150 Tokens per Sec: 3404.058838
    Validation Loss: 0.0003, Validation Acc: 100.00%
    Epoch: 97 Step: 20 Loss: 0.001807 Tokens per Sec: 3443.018555
    Validation Loss: 0.0001, Validation Acc: 100.00%
    Epoch: 98 Step: 20 Loss: 0.007447 Tokens per Sec: 3385.051758
    Validation Loss: 0.0014, Validation Acc: 98.00%
    Epoch: 99 Step: 20 Loss: 0.020616 Tokens per Sec: 3402.310059
    Validation Loss: 0.0004, Validation Acc: 100.00%

# Problem 2.2

> **Problem 2.2** *(6 points)* Now, we will implement a bit more useful function, so-called spelling error correction. Your job is to create a model whose input is a word with spelling errors, and the output is the spelling-corrected word. Here, your vocabulary will be character instead of word. You can create your own training data by using an existing text corpus as the target and inject noise into it to use it as the input. You are free to use whichever text corpus you like. If you can't think of one, please use context data in SQuAD Dataset (see Assignment 2). Report accuracy in your own evaluation data (you will receive full credit as long as both the evaluation data and the accuracy are reasonable), and also show 5 examples where it succeeds at correcting spelling.

> **Answer 2.2** I used SQuAD validation set. The training set is the words from 'context' in SQuAD, and the evaluation set is the words from 'question' in SQuAD. Evaluation accuracy for spelling check is **21.3%**. Also, see the code below for 5 examples of correcting spelling.

```python
from datasets import load_dataset
from pprint import pprint

squad_dataset = load_dataset('squad')
pprint(squad_dataset['validation'][0])
```

    Reusing dataset squad (/home/sungnyun/.cache/huggingface/datasets/squad/plain_text/1.0.0/d6ec3ceb99ca480ce37cdd35555d6cb2511d223b9150cce08a837ef62ffea453)

      0%|          | 0/2 [00:00<?, ?it/s]

    {'answers': {'answer_start': [177, 177, 177],
                 'text': ['Denver Broncos', 'Denver Broncos', 'Denver Broncos']},
     'context': 'Super Bowl 50 was an American football game to determine the '
                'champion of the National Football League (NFL) for the 2015 '
                'season. The American Football Conference (AFC) champion Denver '
                'Broncos defeated the National Football Conference (NFC) champion '
                'Carolina Panthers 24–10 to earn their third Super Bowl title. The '
                "game was played on February 7, 2016, at Levi's Stadium in the San "
                'Francisco Bay Area at Santa Clara, California. As this was the '
                '50th Super Bowl, the league emphasized the "golden anniversary" '
                'with various gold-themed initiatives, as well as temporarily '
                'suspending the tradition of naming each Super Bowl game with '
                'Roman numerals (under which the game would have been known as '
                '"Super Bowl L"), so that the logo could prominently feature the '
                'Arabic numerals 50.',
     'id': '56be4db0acb8001400a502ec',
     'question': 'Which NFL team represented the AFC at Super Bowl 50?',
     'title': 'Super_Bowl_50'}

```python
import copy
from tqdm import tqdm

words = list()
test_words = list()

for i in tqdm(range(len(squad_dataset['validation']))):
    context = squad_dataset['validation'][i]['context'].split(' ')
    question = squad_dataset['validation'][i]['question'].split(' ')
    for word, test_word in zip(context, question):
        if word not in words:
            words.append(word)
        if test_word not in test_words:
            test_words.append(test_word)

#answer = copy.deepcopy(words)
print(len(words), len(test_words))
```

    100%|██████████| 10570/10570 [00:07<00:00, 1355.23it/s]

    8467 15634

```python
vocab = list([chr(i) for i in range(ord('a'),ord('z')+1)])
vocab += ['UNK', 'PAD']
char2id = {char: i for i, char in enumerate(vocab)}
id2char = {i : char for i, char in enumerate(vocab)}
```

```python
from torch.utils.data import Dataset
from torch.utils.data import DataLoader
import random

class SentenceBatch(Dataset):
    def __init__(self, src, target, pad=27):
        src = torch.Tensor(src)
        target = torch.Tensor(target)
        assert src.shape == target.shape

        self.src = torch.ones(src.size(0), src.size(1)+1)
        self.src[:, 1:] = src
        self.src = self.src.long()

        self.target = torch.ones(target.size(0), target.size(1)+1)
        self.target[:, 1:] = target
        self.target = self.target.long()

        self.src_mask = (self.src != pad).unsqueeze(-2)

        self.trg = self.target[:, :-1]
        self.trg_y = self.target[:, 1:]
        self.trg_mask = self.make_std_mask(self.trg, pad)

    def __len__(self):
        return len(self.src)

    def __getitem__(self, idx):
        src = self.src[idx]
        trg = self.trg[idx]
        trg_y = self.trg_y[idx]
        src_mask = torch.tensor(self.src_mask[idx], dtype=torch.int64)
        trg_mask = torch.tensor(self.trg_mask[idx], dtype=torch.int64)

        return src, trg, trg_y, src_mask, trg_mask

    @staticmethod
    def make_std_mask(trg, pad):
        "Create a mask to hide padding and future words."
        trg_mask = (trg != pad).unsqueeze(-2)
        trg_mask = trg_mask & Variable(
            subsequent_mask(trg.size(-1)).type_as(trg_mask.data))
        return trg_mask

length = 32
train_src = []
train_trg = []

for word in tqdm(words):
    trg_id = [char2id[char] if char in char2id else 26 for char in word.lower()]
    wordlen = len(trg_id)
    src_id = trg_id[:]
    if wordlen >= 2:
        rand1 = random.randint(0, wordlen-1)
        rand2 = random.randint(0, 25)
        src_id[rand1] = rand2
    if wordlen < length:
        src_id += (length - wordlen) * [27]
        trg_id += (length - wordlen) * [27]
    elif wordlen > length:
        src_id = src_id[:length]
        trg_id = trg_id[:length]
    train_src.append(src_id)
    train_trg.append(trg_id)

test_src = []
test_trg = []

for word in tqdm(test_words):
    trg_id = [char2id[char] if char in char2id else 26 for char in word.lower()]
    wordlen = len(trg_id)
    src_id = trg_id[:]
    if wordlen >= 2:
        rand1 = random.randint(0, wordlen-1)
        rand2 = random.randint(0, 25)
        src_id[rand1] = rand2
    if wordlen < length:
        src_id += (length - wordlen) * [27]
        trg_id += (length - wordlen) * [27]
    elif wordlen > length:
        src_id = src_id[:length]
        trg_id = trg_id[:length]
    test_src.append(src_id)
    test_trg.append(trg_id)
```

    100%|██████████| 8467/8467 [00:00<00:00, 298580.56it/s]
    100%|██████████| 15634/15634 [00:00<00:00, 295495.71it/s]

```python
trainset = SentenceBatch(train_src, train_trg)
testset = SentenceBatch(test_src, test_trg)

train_loader = DataLoader(trainset, batch_size=256, shuffle=True)
test_loader = DataLoader(testset, batch_size=256)
```

```python
class SimpleLossCompute:
    "A simple loss compute and train function."
    def __init__(self, criterion, opt=None):
        self.criterion = criterion
        self.opt = opt

    def __call__(self, x, y, norm):
        loss = self.criterion(x.contiguous().view(-1, x.size(-1)),
                              y.contiguous().view(-1)) / norm
        loss.backward()
        if self.opt is not None:
            self.opt.step()
            self.opt.optimizer.zero_grad()
        return loss.data * norm

```

```python
V = 28
criterion = LabelSmoothing(size=V, padding_idx=27, smoothing=0.0)
model = make_model(V, V, N=2)
model_opt = NoamOpt(model.src_embed[0].d_model, 1, 400,
        torch.optim.Adam(model.parameters(), lr=0, betas=(0.9, 0.98), eps=1e-9))
loss_compute = SimpleLossCompute(criterion, opt=model_opt)
model.cuda()

for epoch in range(30):
    model.train()
    total_loss = 0
    total_ntokens = 0
    correct = 0
    total = 0

    for i, batch in enumerate(train_loader):
        src, trg, trg_y, src_mask, trg_mask = batch
        src, trg, trg_y, src_mask, trg_mask = src.cuda(), trg.cuda(), trg_y.cuda(), src_mask.cuda(), trg_mask.cuda()
        out = model.generator(model.forward(src, trg, src_mask, trg_mask))

        pred = out.data.max(dim=-1)[1]
        correct += ((pred*src_mask.squeeze()[:,1:] == (trg_y*src_mask.squeeze()[:,1:])).sum(dim=-1) == 32).cpu().sum().item()
        loss = loss_compute(out, trg_y, trg_y.size(0)*trg_y.size(1))

        total_loss += loss
        total_ntokens += trg_y.size(0)*trg_y.size(1)
        total += trg.size(0)

    print('Epoch: {:d}, Train Loss: {:.4f}, Train Acc: {:.2f}%'.format(epoch, total_loss/total_ntokens, correct/total*100))

    correct = 0
    total = 0
    model.eval()
    with torch.no_grad():
        for i, batch in enumerate(test_loader):
            src, trg, trg_y, src_mask, trg_mask = batch
            src, trg, trg_y, src_mask, trg_mask = src.cuda(), trg.cuda(), trg_y.cuda(), src_mask.cuda(), trg_mask.cuda()
            out = model.generator(model.forward(src, trg, src_mask, trg_mask))

            pred = out.data.max(dim=-1)[1]
            correct += ((pred*src_mask.squeeze()[:,1:] == (trg_y*src_mask.squeeze()[:,1:])).sum(dim=-1) == 32).cpu().sum().item()
            total += trg.size(0)
    print('Epoch: {:d}, Test Acc: {:.2f}%'.format(epoch, correct/total*100))
```

    <ipython-input-5-5fe0956476b4>:20: UserWarning: nn.init.xavier_uniform is now deprecated in favor of nn.init.xavier_uniform_.
      nn.init.xavier_uniform(p)
    <ipython-input-12-874b4246d396>:33: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      src_mask = torch.tensor(self.src_mask[idx], dtype=torch.int64)
    <ipython-input-12-874b4246d396>:34: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      trg_mask = torch.tensor(self.trg_mask[idx], dtype=torch.int64)

    Epoch: 0, Train Loss: 0.6295, Train Acc: 4.71%
    Epoch: 0, Test Acc: 3.33%
    Epoch: 1, Train Loss: 0.3621, Train Acc: 6.40%
    Epoch: 1, Test Acc: 5.51%
    Epoch: 2, Train Loss: 0.2315, Train Acc: 9.31%
    Epoch: 2, Test Acc: 9.05%
    Epoch: 3, Train Loss: 0.1734, Train Acc: 12.63%
    Epoch: 3, Test Acc: 12.05%
    Epoch: 4, Train Loss: 0.1554, Train Acc: 14.99%
    Epoch: 4, Test Acc: 12.51%
    Epoch: 5, Train Loss: 0.1491, Train Acc: 16.13%
    Epoch: 5, Test Acc: 13.94%
    Epoch: 6, Train Loss: 0.1416, Train Acc: 17.40%
    Epoch: 6, Test Acc: 15.32%
    Epoch: 7, Train Loss: 0.1414, Train Acc: 18.29%
    Epoch: 7, Test Acc: 14.95%
    Epoch: 8, Train Loss: 0.1413, Train Acc: 18.25%
    Epoch: 8, Test Acc: 12.43%
    Epoch: 9, Train Loss: 0.1467, Train Acc: 18.29%
    Epoch: 9, Test Acc: 14.98%
    Epoch: 10, Train Loss: 0.1424, Train Acc: 18.13%
    Epoch: 10, Test Acc: 15.98%
    Epoch: 11, Train Loss: 0.1490, Train Acc: 16.90%
    Epoch: 11, Test Acc: 13.94%
    Epoch: 12, Train Loss: 0.1487, Train Acc: 17.01%
    Epoch: 12, Test Acc: 13.51%
    Epoch: 13, Train Loss: 0.1438, Train Acc: 17.54%
    Epoch: 13, Test Acc: 14.53%
    Epoch: 14, Train Loss: 0.1595, Train Acc: 15.40%
    Epoch: 14, Test Acc: 15.16%
    Epoch: 15, Train Loss: 0.1424, Train Acc: 17.44%
    Epoch: 15, Test Acc: 15.46%
    Epoch: 16, Train Loss: 0.1315, Train Acc: 20.16%
    Epoch: 16, Test Acc: 16.48%
    Epoch: 17, Train Loss: 0.1363, Train Acc: 18.80%
    Epoch: 17, Test Acc: 16.03%
    Epoch: 18, Train Loss: 0.1388, Train Acc: 19.36%
    Epoch: 18, Test Acc: 15.94%
    Epoch: 19, Train Loss: 0.1295, Train Acc: 20.68%
    Epoch: 19, Test Acc: 17.23%
    Epoch: 20, Train Loss: 0.1286, Train Acc: 21.19%
    Epoch: 20, Test Acc: 17.55%
    Epoch: 21, Train Loss: 0.1253, Train Acc: 21.63%
    Epoch: 21, Test Acc: 18.54%
    Epoch: 22, Train Loss: 0.1203, Train Acc: 22.50%
    Epoch: 22, Test Acc: 18.17%
    Epoch: 23, Train Loss: 0.1138, Train Acc: 24.03%
    Epoch: 23, Test Acc: 20.30%
    Epoch: 24, Train Loss: 0.1103, Train Acc: 25.55%
    Epoch: 24, Test Acc: 19.32%
    Epoch: 25, Train Loss: 0.1128, Train Acc: 25.11%
    Epoch: 25, Test Acc: 19.73%
    Epoch: 26, Train Loss: 0.1176, Train Acc: 23.77%
    Epoch: 26, Test Acc: 20.49%
    Epoch: 27, Train Loss: 0.1148, Train Acc: 25.17%
    Epoch: 27, Test Acc: 16.85%
    Epoch: 28, Train Loss: 0.1154, Train Acc: 24.88%
    Epoch: 28, Test Acc: 20.54%
    Epoch: 29, Train Loss: 0.0999, Train Acc: 29.66%
    Epoch: 29, Test Acc: 21.30%

```python
correct = 0
total = 0
cnt = 0
model.eval()
with torch.no_grad():
    for i, batch in enumerate(test_loader):
        src, trg, trg_y, src_mask, trg_mask = batch
        src, trg, trg_y, src_mask, trg_mask = src.cuda(), trg.cuda(), trg_y.cuda(), src_mask.cuda(), trg_mask.cuda()
        out = model.generator(model.forward(src, trg, src_mask, trg_mask))

        pred = out.data.max(dim=-1)[1]
        correct = ((pred*src_mask.squeeze()[:,1:] == (trg_y*src_mask.squeeze()[:,1:])).sum(dim=-1) == 32)
        idx = (correct == 1).nonzero()[-5:]

        src = src[idx].squeeze()[:, 1:].cpu().numpy()
        pred = pred[idx].squeeze().cpu().numpy()
        src_mask = src_mask[idx].squeeze()[:, 1:].cpu().numpy()
        length = src_mask.sum(-1)

        for j in range(len(src)):
            print(f'Source {j}: ', [id2char[id_] for id_ in src[j]][:length[j]])
            print(f'Target {j}: ', [id2char[id_] for id_ in pred[j]][:length[j]])

        break
```

    Source 0:  ['p', 'e', 'r', 'k', 'o', 'r', 'm', 'e', 'd']
    Target 0:  ['p', 'e', 'r', 'f', 'o', 'r', 'm', 'e', 'd']
    Source 1:  ['l', 'i', 's', 'r']
    Target 1:  ['l', 'i', 's', 't']
    Source 2:  ['s', 't', 'a', 't', 'i', 'o', 'v']
    Target 2:  ['s', 't', 'a', 't', 'i', 'o', 'n']
    Source 3:  ['UNK', 'UNK', 'e']
    Target 3:  ['UNK', 'UNK', 'UNK']
    Source 4:  ['c', 'a', 'm', 'r']
    Target 4:  ['c', 'a', 'm', 'e']

    <ipython-input-12-874b4246d396>:33: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      src_mask = torch.tensor(self.src_mask[idx], dtype=torch.int64)
    <ipython-input-12-874b4246d396>:34: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
      trg_mask = torch.tensor(self.trg_mask[idx], dtype=torch.int64)
    <ipython-input-17-f5501da0ae34>:13: UserWarning: This overload of nonzero is deprecated:
    	nonzero()
    Consider using one of the following signatures instead:
    	nonzero(*, bool as_tuple) (Triggered internally at  /opt/conda/conda-bld/pytorch_1603729096996/work/torch/csrc/utils/python_arg_parser.cpp:882.)
      idx = (correct == 1).nonzero()[-5:]

```python

```


```python
from os import chdir
from pathlib import Path
if Path.cwd().stem == "notes": chdir('../')
```

```python
import sys
```

```python
sys.version
```

    '3.8.0 (default, Oct 28 2019, 16:14:01) \n[GCC 8.3.0]'

```python
import matplotlib
```

```python
import matplotlib.pyplot as plt
```

```python
from mpl_toolkits.mplot3d import Axes3D
```

```python
import numpy
```

```python
from sympy import *
```

$\left(x-a\right)^2+\left(y-b\right)^2=r^2$

If $i$ and $j$ represent the horizontal and vertical distances from the starting point $P$ to the center of the circle, then $c_1 = p_1+i$ and $c_2=p_2+j$.

That was convenenient. If, on the other hand, all we have is the radius of the circle...

$$\left(p_1-a\right)^2+\left(p_2-b\right)^2=r^2$$

$$\left(q_1-a\right)^2+\left(q_2-b\right)^2=r^2$$

So, with numbers obtained from the `gcode` block in question capitalized, we need to solve the following system of equations for $a$ and $b$.

$$\left(X_0-a\right)^2+\left(Y_0-b\right)^2=R^2$$

$$\left(X-a\right)^2+\left(Y-b\right)^2=R^2$$

$X_0$ and $Y_0$ represent the current tool position in the equation above, and are therefore known quantities at the time of computation.

```python
x, y, z, X, X0, Y, Y0, Z, Z0, R = symbols("x y z X X0 Y Y0 Z Z0 R")
```

```python
e1 = (x-X0)**2 + (y-Y0)**2
```

```python
e2 = (x-X)**2 + (y-Y)**2
```

[Here](https://math.stackexchange.com/questions/1781438/finding-the-center-of-a-circle-given-two-points-and-a-radius-algebraically) is the answer from Stack Exchange.

That question / answer also references [this](http://mathforum.org/library/drmath/view/54490.html) newsgroup message from who knows when?

```python
init_printing()
```

```python
help(init_printing)
```

    Help on function init_printing in module sympy.interactive.printing:

    init_printing(pretty_print=True, order=None, use_unicode=None, use_latex=None, wrap_line=None, num_columns=None, no_global=False, ip=None, euler=False, forecolor=None, backcolor='Transparent', fontsize='10pt', latex_mode='plain', print_builtin=True, str_printer=None, pretty_printer=None, latex_printer=None, scale=1.0, **settings)
        Initializes pretty-printer depending on the environment.

        Parameters
        ==========

        pretty_print : boolean, default=True
            If True, use pretty_print to stringify or the provided pretty
            printer; if False, use sstrrepr to stringify or the provided string
            printer.
        order : string or None, default='lex'
            There are a few different settings for this parameter:
            lex (default), which is lexographic order;
            grlex, which is graded lexographic order;
            grevlex, which is reversed graded lexographic order;
            old, which is used for compatibility reasons and for long expressions;
            None, which sets it to lex.
        use_unicode : boolean or None, default=None
            If True, use unicode characters;
            if False, do not use unicode characters;
            if None, make a guess based on the environment.
        use_latex : string, boolean, or None, default=None
            If True, use default LaTeX rendering in GUI interfaces (png and
            mathjax);
            if False, do not use LaTeX rendering;
            if None, make a guess based on the environment;
            if 'png', enable latex rendering with an external latex compiler,
            falling back to matplotlib if external compilation fails;
            if 'matplotlib', enable LaTeX rendering with matplotlib;
            if 'mathjax', enable LaTeX text generation, for example MathJax
            rendering in IPython notebook or text rendering in LaTeX documents;
            if 'svg', enable LaTeX rendering with an external latex compiler,
            no fallback
        wrap_line : boolean
            If True, lines will wrap at the end; if False, they will not wrap
            but continue as one line. This is only relevant if ``pretty_print`` is
            True.
        num_columns : int or None, default=None
            If int, number of columns before wrapping is set to num_columns; if
            None, number of columns before wrapping is set to terminal width.
            This is only relevant if ``pretty_print`` is True.
        no_global : boolean, default=False
            If True, the settings become system wide;
            if False, use just for this console/session.
        ip : An interactive console
            This can either be an instance of IPython,
            or a class that derives from code.InteractiveConsole.
        euler : boolean, optional, default=False
            Loads the euler package in the LaTeX preamble for handwritten style
            fonts (http://www.ctan.org/pkg/euler).
        forecolor : string or None, optional, default=None
            DVI setting for foreground color. None means that either 'Black',
            'White', or 'Gray' will be selected based on a guess of the IPython
            terminal color setting. See notes.
        backcolor : string, optional, default='Transparent'
            DVI setting for background color. See notes.
        fontsize : string, optional, default='10pt'
            A font size to pass to the LaTeX documentclass function in the
            preamble. Note that the options are limited by the documentclass.
            Consider using scale instead.
        latex_mode : string, optional, default='plain'
            The mode used in the LaTeX printer. Can be one of:
            {'inline'|'plain'|'equation'|'equation*'}.
        print_builtin : boolean, optional, default=True
            If ``True`` then floats and integers will be printed. If ``False`` the
            printer will only print SymPy types.
        str_printer : function, optional, default=None
            A custom string printer function. This should mimic
            sympy.printing.sstrrepr().
        pretty_printer : function, optional, default=None
            A custom pretty printer. This should mimic sympy.printing.pretty().
        latex_printer : function, optional, default=None
            A custom LaTeX printer. This should mimic sympy.printing.latex().
        scale : float, optional, default=1.0
            Scale the LaTeX output when using the ``png`` or ``svg`` backends.
            Useful for high dpi screens.
        settings :
            Any additional settings for the ``latex`` and ``pretty`` commands can
            be used to fine-tune the output.

        Examples
        ========

        >>> from sympy.interactive import init_printing
        >>> from sympy import Symbol, sqrt
        >>> from sympy.abc import x, y
        >>> sqrt(5)
        sqrt(5)
        >>> init_printing(pretty_print=True) # doctest: +SKIP
        >>> sqrt(5) # doctest: +SKIP
          ___
        \/ 5
        >>> theta = Symbol('theta') # doctest: +SKIP
        >>> init_printing(use_unicode=True) # doctest: +SKIP
        >>> theta # doctest: +SKIP
        \u03b8
        >>> init_printing(use_unicode=False) # doctest: +SKIP
        >>> theta # doctest: +SKIP
        theta
        >>> init_printing(order='lex') # doctest: +SKIP
        >>> str(y + x + y**2 + x**2) # doctest: +SKIP
        x**2 + x + y**2 + y
        >>> init_printing(order='grlex') # doctest: +SKIP
        >>> str(y + x + y**2 + x**2) # doctest: +SKIP
        x**2 + x + y**2 + y
        >>> init_printing(order='grevlex') # doctest: +SKIP
        >>> str(y * x**2 + x * y**2) # doctest: +SKIP
        x**2*y + x*y**2
        >>> init_printing(order='old') # doctest: +SKIP
        >>> str(x**2 + y**2 + x + y) # doctest: +SKIP
        x**2 + x + y**2 + y
        >>> init_printing(num_columns=10) # doctest: +SKIP
        >>> x**2 + x + y**2 + y # doctest: +SKIP
        x + y +
        x**2 + y**2

        Notes
        =====

        The foreground and background colors can be selected when using 'png' or
        'svg' LaTeX rendering. Note that before the ``init_printing`` command is
        executed, the LaTeX rendering is handled by the IPython console and not SymPy.

        The colors can be selected among the 68 standard colors known to ``dvips``,
        for a list see [1]_. In addition, the background color can be
        set to  'Transparent' (which is the default value).

        When using the 'Auto' foreground color, the guess is based on the
        ``colors`` variable in the IPython console, see [2]_. Hence, if
        that variable is set correctly in your IPython console, there is a high
        chance that the output will be readable, although manual settings may be
        needed.

        References
        ==========

        .. [1] https://en.wikibooks.org/wiki/LaTeX/Colors#The_68_standard_colors_known_to_dvips

        .. [2] https://ipython.readthedocs.io/en/stable/config/details.html#terminal-colors

        See Also
        ========

        sympy.printing.latex
        sympy.printing.pretty

```python
from enum import IntEnum

class CircularPlane(IntEnum):
    XY = 17
    ZX = 18
    YZ = 19

```

```python
xa = (X - X0) / 2
```

```python
ya = (Y - Y0) / 2
```

```python
x0, y0 = X0 + xa, Y0 + ya
```

```python
a = sqrt(xa**2 + ya**2)
```

```python
b = sqrt(R**2 - a**2)
```

```python
x3 = x0 + b*ya / a
```

```python
x4 = x0 - b*ya / a
```

```python
y3 = y0 - b*xa / a
```

```python
y4 = y0 + b*xa / a
```

```python
x3
```

```python
y3
```

```python
x3.simplify()
```

```python
y3.simplify()
```

```python
params = dict(zip([X0, Y0, X, Y, R], [1, 4, 5, 1, 13/2]))
```

```python
x3 = x3.subs(params)
```

```python
y3 = y3.subs(params)
```

```python
x4 = x4.subs(params)
```

```python
y4 = y4.subs(params)
```

```python
T1, T2 = symbols("T1, T2")
```

```python
C1 = Point3D(x3.subs(params), y3.subs(params))
```

```python
x3, y3, x4, y4
```

```python
from IPython.display import Image
Image("png/circles0000.png")
```

$$AC_1=0$$

```python
args1 = [-x3, -y3]
args2 = [1/params[R]] * 3
Q0 = Point3D(params[X0], params[Y0], 0).translate(*args1).scale(*args2).evalf()
Q = Point3D(params[X], params[Y], 0).translate(*args1).scale(*args2).evalf()
print("1/R:", 1/params[R])
print("Q0:", Q0)
print("Q:", Q)
```

    1/R: 0.15384615384615385
    Q0: Point3D(0.246153846153846, 0.969230769230769, 0)
    Q: Point3D(0.861538461538462, 0.507692307692308, 0)

Get the angles...

```python
theta1 = atan(Q0.y / Q0.x)
theta1
```

```python
def rad2deg(t):
    return ((t % pi) * 180 / pi).evalf()

def deg2rad(t):
    return (pi * t / 180 % (pi / 2)).evalf()
```

```python
from anglr import Angle
```

```python
dir(Angle)
```

    ['__abs__',
     '__add__',
     '__class__',
     '__complex__',
     '__delattr__',
     '__dict__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__float__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__int__',
     '__le__',
     '__lt__',
     '__module__',
     '__mul__',
     '__ne__',
     '__neg__',
     '__new__',
     '__pos__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__rmul__',
     '__round__',
     '__setattr__',
     '__sizeof__',
     '__str__',
     '__sub__',
     '__subclasshook__',
     '__truediv__',
     '__weakref__',
     'angle_between',
     'angle_between_clockwise',
     'angle_to',
     'angle_within',
     'arcminutes',
     'arcseconds',
     'degrees',
     'dump',
     'gradians',
     'hours',
     'normalized',
     'vector',
     'x',
     'y']

```python
help(Angle.__init__)
```

    Help on function __init__ in module anglr:

    __init__(self, value=0, mode='radians')
        Creates an `Angle` instance representing the angle `value` in the angular unit specified by `mode`.

        `mode` can be "radians", "degrees", "gradians", "hours", "arcminutes", "arcseconds", or "vector" (see `angle_instance.vector` for more info).

```python
P = Point3D()
Q
P.translate(*-Q)
```

```python
rad2deg(theta1)
```

```python
theta2 = atan(Q.y / Q.x)
rad2deg(theta2)
```

```python
help(Angle.angle_between)
```

    Help on function angle_between in module anglr:

    angle_between(self, angle)
        Returns a new `Angle` instance that represents the smallest of the two possible angles between `Angle` instance to `angle` on the unit circle (this is always non-negative).

```python
t1 = Angle(theta1)
t2 = Angle(theta2)
t3 = t1.angle_between_clockwise(t2)
```

```python
from math import pi as PI
PI
```

```python
t3.radians > PI
```

$\displaystyle \text{True}$

```python
import matplotlib.path
matplotlib.path.Path.arc(t1.degrees, t2.degrees).vertices.transpose()[0]
```

    array([ 0.24615385,  0.13495359,  0.01971801, -0.09449899, -0.20871598,
           -0.32021988, -0.42412041, -0.52802095, -0.62277666, -0.70423183,
           -0.785687  , -0.85263315, -0.90213422, -0.95163529, -0.98295688,
           -0.99472531, -1.00649374, -0.99853442, -0.97119641, -0.9438584 ,
           -0.8975473 , -0.83429418, -0.77104106, -0.69178434, -0.6       ,
           -0.50821566, -0.40526542, -0.29566438, -0.18606334, -0.07143754,
            0.04318584,  0.15780922,  0.27072965,  0.37699473,  0.48325981,
            0.58129301,  0.66679486,  0.75229671,  0.82399871,  0.8787562 ,
            0.9335137 ,  0.97051431,  0.98813529,  1.00575627,  1.0037362 ,
            0.98216368,  0.96059115,  0.91978621,  0.86153846])

```python
help(map)
```

    Help on class map in module builtins:

    class map(object)
     |  map(func, *iterables) --> map object
     |
     |  Make an iterator that computes the function using arguments from
     |  each of the iterables.  Stops when the shortest iterable is exhausted.
     |
     |  Methods defined here:
     |
     |  __getattribute__(self, name, /)
     |      Return getattr(self, name).
     |
     |  __iter__(self, /)
     |      Implement iter(self).
     |
     |  __next__(self, /)
     |      Implement next(self).
     |
     |  __reduce__(...)
     |      Return state information for pickling.
     |
     |  ----------------------------------------------------------------------
     |  Static methods defined here:
     |
     |  __new__(*args, **kwargs) from builtins.type
     |      Create and return a new object.  See help(type) for accurate signature.

```python
help(numpy.full)
```

    Help on function full in module numpy:

    full(shape, fill_value, dtype=None, order='C')
        Return a new array of given shape and type, filled with `fill_value`.

        Parameters
        ----------
        shape : int or sequence of ints
            Shape of the new array, e.g., ``(2, 3)`` or ``2``.
        fill_value : scalar
            Fill value.
        dtype : data-type, optional
            The desired data-type for the array  The default, `None`, means
             `np.array(fill_value).dtype`.
        order : {'C', 'F'}, optional
            Whether to store multidimensional data in C- or Fortran-contiguous
            (row- or column-wise) order in memory.

        Returns
        -------
        out : ndarray
            Array of `fill_value` with the given shape, dtype, and order.

        See Also
        --------
        full_like : Return a new array with shape of input filled with value.
        empty : Return a new uninitialized array.
        ones : Return a new array setting values to one.
        zeros : Return a new array setting values to zero.

        Examples
        --------
        >>> np.full((2, 2), np.inf)
        array([[inf, inf],
               [inf, inf]])
        >>> np.full((2, 2), 10)
        array([[10, 10],
               [10, 10]])

```python
import numpy
public(numpy)
```

```python
public(numpy.ndarray)
```

```python
help(numpy.ndarray.transpose)
```

```python
public(matplotlib.path.Path)
```

```python
from enum import Enum
class Quadrants(Enum):
    I = "I"
    II = "II"
    III = "III"
    IV = "IV"

class Directions(Enum):
    CW = 0
    CCW = 1

def quadrant(P):
    if P.x >= 0:
        if P.y > 0:
            return Quadrants.I
        else: return Quadrants.IV
    else:
        if P.y > 0:
            return Quadrants.II
        else:
            return Quadrants.III

def get_angle(P):
    return atan(P.y / P.x)

def rotate(*args, radians=(pi/2).evalf(), degrees=90):
    pass

def direction(P1, P2):
    """ Return the direction travelled to reach P2 from P1 via the shortest path.
    """
    t1 = get_angle(P1)
    t2 = get_angle(P2)
    if not quadrant(P1) == Quadrants.I:
        pass
    if quadrant(P1) == Quadrants.I:
        if quadrant(P2) == Quadrants.I:
            if t1 - t2 > 0:
                return Directions.CW
            else:
                return Directions.CCW
        if quadrant(P2) == Quadrants.II:
            return CCW
        if quadrant(P2) == Quadrants.III:
            if t2 < t1 + 180:
                return Directions.CCW
            else:
                return Directions.CCW
        if quadrant(P2) == Quadrants.IV:
            return Directions.CW
```

```python
rotate
```

```python
T1 = Matrix([[1/params[R], 0, -x3], [0, 1/params[R], -y3]])
```

$\begin{matrix}
  -1 & 3 \\
  2 & -4
 \end{matrix}$

```python
v3 = Matrix([x3, y3, 1])
v4 = Matrix([x4, y4, 1])
```

```python
v3
```

```python
T1 * v3
```

```python
T1 * v4
```

```python
import matplotlib.pyplot as plt
```

```python
plt.plot([v3[0], v3[1]])
```

```python
v3
```

```python
pwd
```

```python
from os import chdir
```

```python
chdir('../../')
```

```python
from py.startup import *
```

```python
import mpl_toolkits.mplot3d
plt.ion()
```

```python
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')

```

```python

```


```python
import copy
import pathlib
import yaml

import astropy.coordinates as coord
import astropy.table as at
import astropy.units as u
import matplotlib as mpl
import matplotlib.pyplot as plt
%matplotlib inline
import numpy as np
from scipy.optimize import minimize
from scipy.integrate import simpson, quad

from solaroid.simulate import invt_sample_z
from solaroid.likelihood import ln_normal, ln_two_sech2
from solaroid.integrate import log_simpson
from solaroid.gram_schmidt import gram_schmidt
```

```python
_path = pathlib.Path('../src/static/')
with open(_path / 'fiducial-density.yml', 'r') as f:
    pars = yaml.safe_load(f.read())
pars
```

    {'f': 0.75, 'h1': 120, 'h2': 400}

---

# The numerical way

Pre-compute integrals:

```python
def test_vector_ip(obj1, obj2):
    return np.dot(obj1, obj2)
```

```python
# basis_vecs = gram_schmidt([], test_vector_ip)
# basis_vecs = gram_schmidt([np.array([1, 3, 4.]), np.polynomial.Polynomial.basis(1)],
#                           test_vector_ip)

def func(z, n):
    return np.polynomial.Polynomial.basis(n)(z)

basis_vecs = gram_schmidt([lambda z: func(z, 0), lambda z: func(z, 1), lambda z: func(z, 2)],
                          test_vector_ip)
```

```python
vecs = np.array([
    [1.4, 6.1, 3.3],
    [4.2, 3.3, 0.5],
    [11.3, 5, 3.]
])
basis_vecs = gram_schmidt(vecs, test_vector_ip)
print(basis_vecs)

for i, vec1 in enumerate(basis_vecs):
    for j, vec2 in enumerate(basis_vecs):
        assert np.isclose(vec1 @ vec2, int(i == j))
```

    [array([0.19787121, 0.86215314, 0.46641071]), array([ 0.93267068, -0.01918324, -0.36021855]), array([ 0.30161629, -0.50628448,  0.80790076])]

```python
def test_legendre_ip(f1, f2):
    grid = np.linspace(-1, 1, 1025)  # BOO YA
    return simpson(f1(grid) * f2(grid) / 2, x=grid)
```

```python
funcs = [np.polynomial.Polynomial.basis(deg) for deg in range(6)]
basis_funcs = gram_schmidt(funcs, test_legendre_ip)

grid = np.linspace(-1, 1, 128)
for func in basis_funcs:
    plt.plot(grid, func(grid))
```

```python
def test_hermite_ip(f1, f2):
    grid = np.linspace(-10, 10, 1025)  # BOO YA
    return simpson(f1(grid) * f2(grid) * np.exp(-grid**2 / 2), x=grid)

funcs = [np.polynomial.Polynomial.basis(deg) for deg in range(7)]
basis_funcs = gram_schmidt(funcs, test_hermite_ip)

grid = np.linspace(-3, 3, 128)
for func in basis_funcs:
    plt.plot(grid, func(grid))
```

```python
from scipy.integrate import trapezoid
```

```python
def test_func_ip(f1, f2, measure_func, grid, scale):
    # TODO: or use quad() with the function handle
    return simpson(f1(grid) * f2(grid) * measure_func(scale * grid) * scale,
                   x=grid)
#     return trapezoid(f1(grid) * f2(grid) * measure_func(grid),
#                      x=grid)

def test_func_ip_quad(f1, f2, measure_func, scale):
    return quad(lambda z: f1(z) * f2(z) * measure_func(scale * z) * scale, -np.inf, np.inf)[0]
```

```python
funcs = [np.polynomial.Polynomial.basis(deg) for deg in range(15)]

measure_func = lambda z: np.exp(ln_two_sech2(z, **pars))
grid = np.arctanh(np.linspace(-1+1e-8, 1-1e-8, 8192))

# sech2_basis_funcs = gram_shit(funcs, test_func_ip, args=(measure_func, grid, pars['h2']))
sech2_basis_funcs = gram_schmidt(funcs, test_func_ip_quad, args=(measure_func, pars['h2']))
```

    /Users/apricewhelan/projects/solar-velocity/solaroid/likelihood.py:16: RuntimeWarning: overflow encountered in cosh
      lnterm1 = np.log(f) - 2 * np.log(np.cosh(z / (2 * h1))) - np.log(4 * h1)
    /Users/apricewhelan/projects/solar-velocity/solaroid/likelihood.py:17: RuntimeWarning: overflow encountered in cosh
      lnterm2 = np.log(1 - f) - 2 * np.log(np.cosh(z / (2 * h2))) - np.log(4 * h2)

```python
sech2_basis_funcs[8]
```

$x \mapsto \text{0.17649673019143625}\color{LightGray}{ + \text{0.0}\,x} - \text{0.35883555867865524}\,x^{2}\color{LightGray}{ + \text{0.0}\,x^{3}} + \text{0.0262440636114066}\,x^{4}\color{LightGray}{ + \text{0.0}\,x^{5}} - \text{0.0003610816145934385}\,x^{6}\color{LightGray}{ + \text{0.0}\,x^{7}} + \text{1.0561003250808217e-06}\,x^{8}$

```python
plot_grid = np.linspace(-10, 10, 1024)
for func in sech2_basis_funcs:
    plt.plot(plot_grid, func(plot_grid), marker='')
# plt.xlim(-5000, 5000)
plt.xlim(-10, 10)
plt.ylim(-5, 5)
```

```python
plot_grid = np.linspace(-10, 10, 1024)
for func in sech2_basis_funcs[:8]:
    plt.plot(
        plot_grid * pars['h2'] / 1e3,
        func(plot_grid) * measure_func(plot_grid * pars['h2']),
        marker=''
    )
# plt.xlim(-5000, 5000)
plt.xlim(-3, 3)
plt.xlabel('$z$ [kpc]')
```

```python

```

---

# The analytic way

See the `Orthogonal-basis.nb` mathematica notebook.

Term 0:
$$
\begin{align}
f_0(z) &= a_{00}\\
\int \textrm{d}z \, f_0(z)^2 \, \rho(z) &= 1\\
\int \textrm{d}z \, a_{00}^2 \, \rho(z) &= 1\\
\end{align}
$$
so $a_{00}=1$

---

Term 1:
$$
\begin{align}
f_1(z) &= a_{10} + a_{11} \, z\\
\int \textrm{d}z \, f_1(z)^2 \, \rho(z) &= 1\\
\int \textrm{d}z \, f_0(z) \, f_1(z) \, \rho(z) &= 0\\
\end{align}
$$

Mathematica says:
$$
\begin{align}
a_{10} &= 0\\
a_{11} &= \frac{\sqrt{3}}{\pi} \, \left( h_2^2 + f\, (h_1 - h_2)\,(h_1 + h_2) \right)^{-1/2}
\end{align}
$$

---

Term 2:
$$
\begin{align}
f_2(z) &= a_{20} + a_{21} \, z + a_{22} \, z^2\\
\int \textrm{d}z \, f_2(z)^2 \, \rho(z) &= 1\\
\int \textrm{d}z \, f_0(z) \, f_2(z) \, \rho(z) &= 0\\
\int \textrm{d}z \, f_1(z) \, f_2(z) \, \rho(z) &= 0\\
\end{align}
$$

Mathematica says:
$$
\begin{align}
a_{20} &= \frac{1}{3}(-22\,f\,h_1^2\,\pi^2 - 22\,h_2^2\,\pi^2 + 22\,f\,h_2^2\,\pi^2)\\
a_{21} &= 0\\
a_{22} &= \textrm{huge expression}
\end{align}
$$

```python
def ln_density(z, **kw):
    return ln_two_sech2(z, **kw)
```

```python
def f1(z, h1, h2, f):
    return np.sqrt(3) / np.pi * (h2**2 + f*(h1-h2)*(h1+h2))**-0.5 * z
```

```python
alpha = 0.5
zgrid = np.arange(-100, 100, 1e-3) * 1e3
ln_dens0 = ln_density(zgrid, **pars)
dens1 = f1(zgrid, **pars)
dens = alpha * np.exp(ln_dens0) + (1-alpha) * (dens1)

plt.plot(zgrid, dens)
```

```python
zgrid = np.linspace(-1000, 1000, 1024)
simpson(f1(zgrid, **pars), x=zgrid)
```

    0.0

```python
from scipy.integrate import quad
```

```python
quad(lambda z: f1(z, **pars), -100, 100)
```

    (0.0, 2.705621306299946e-13)

```python

```

```python

```


# Modelo de Reverchon

El modelo de Reverchon resulta de realizar los balances de masa sobre el lecho de extracción utilizando la supocición de flujo pisto en el interior del lecho, despreciando la disperción axial y en las que se considera que tanto el flujo del fluido, la presión, temperatura se mantienen constantes.

Estos balances de masa se muestran en las ecuaciones 1 y 2, los cuales son ecuaciones diferenciales que se resuelven con las condicones iniciales de la ecuación 3

\begin{equation}
uV \frac{\partial c_c}{\partial t}+eV \frac{\partial c_c}{\partial t}+ AK(q-q^*) = 0
\end{equation}

\begin{equation}
(1-e)VuV^* \frac{\partial c_q}{\partial t}= -AK(q-q^*)
\end{equation}

\begin{eqnarray}
C = 0 & q=q_0 %%& t = 0 &  c(0,t) & h=0
\end{eqnarray}

El equilibrio de fases se modela por medio de la supocicón que se cumple la ley de henry como se muestra en la ecuación 4

\begin{equation}
c = kq^*
\end{equation}

Una simplificación habitual, es la de aproximar el el lecho de extracción como una serie de $(n)$ subdivisiones del lecho de extracción supercritica, permitiendo convertir los balances de masa en un sistema de $(2n)$ ecuaciones diferenciales ordinarias como se muestra en las ecuaciones 6 y 8

\begin{equation}
D_n = \left( \frac{v}{n}
\right)  \left( e  \left( \frac{dC_n}{dt} \right) +(1-e) \left( \frac{dq_n}{dt} \right)
\right) = 0
\end{equation}

\begin{equation}
\left(\frac{W}{p} \right)
\left( C_n- C_{n-1} \right) + D_n = 0
\end{equation}

\begin{equation}
\left( \frac{dq_n}{dt} \right)= - \left(\frac{1}{t_i}\right) \left(q_n-q_n^* \right)
\end{equation}

condiciones iniciales:

\begin{eqnarray}
C_n = C_n^0 & q_n = q_0
\end{eqnarray}

```python

```

## Nomenclatura

$a_0$ Surface, $\frac{m^2}{m^3}$

```python

```

```python

```


# 线性代数

`numpy` 和 `scipy` 中，负责进行线性代数部分计算的模块叫做 `linalg`。

```python
import numpy as np
import numpy.linalg
import scipy as sp
import scipy.linalg
import matplotlib.pyplot as plt
from scipy import linalg

%matplotlib inline
```

## numpy.linalg VS scipy.linalg

一方面`scipy.linalg` 包含 `numpy.linalg` 中的所有函数，同时还包含了很多 `numpy.linalg` 中没有的函数。

另一方面，`scipy.linalg` 能够保证这些函数使用 BLAS/LAPACK 加速，而 `numpy.linalg` 中这些加速是可选的。

因此，在使用时，我们一般使用 `scipy.linalg` 而不是 `numpy.linalg`。

我们可以简单看看两个模块的差异：

```python
print "number of items in numpy.linalg:", len(dir(numpy.linalg))
print "number of items in scipy.linalg:", len(dir(scipy.linalg))
```

    number of items in numpy.linalg: 36
    number of items in scipy.linalg: 115

## numpy.matrix VS 2D numpy.ndarray

线性代数的基本操作对象是矩阵，而矩阵的表示方法主要有两种：`numpy.matrix` 和 2D `numpy.ndarray`。

### numpy.matrix

`numpy.matrix` 是一个矩阵类，提供了一些方便的矩阵操作：
- 支持类似 `MATLAB` 创建矩阵的语法
- 矩阵乘法默认用 `*` 号
- `.I` 表示逆，`.T` 表示转置

可以用 `mat` 或者 `matrix` 来产生矩阵：

```python
A = np.mat("[1, 2; 3, 4]")
print repr(A)

A = np.matrix("[1, 2; 3, 4]")
print repr(A)
```

    matrix([[1, 2],
            [3, 4]])
    matrix([[1, 2],
            [3, 4]])

转置和逆：

```python
print repr(A.I)
print repr(A.T)
```

    matrix([[-2. ,  1. ],
            [ 1.5, -0.5]])
    matrix([[1, 3],
            [2, 4]])

矩阵乘法：

```python
b = np.mat('[5; 6]')
print repr(A * b)
```

    matrix([[17],
            [39]])

### 2 维 numpy.ndarray

虽然 `numpy.matrix` 有着上面的好处，但是一般不建议使用，而是用 2 维 `numpy.ndarray` 对象替代，这样可以避免一些不必要的困惑。

我们可以使用 `array` 复现上面的操作：

```python
A = np.array([[1,2], [3,4]])
print repr(A)
```

    array([[1, 2],
           [3, 4]])

逆和转置：

```python
print repr(linalg.inv(A))
print repr(A.T)
```

    array([[-2. ,  1. ],
           [ 1.5, -0.5]])
    array([[1, 3],
           [2, 4]])

矩阵乘法：

```python
b = np.array([5, 6])

print repr(A.dot(b))
```

    array([17, 39])

普通乘法：

```python
print repr(A * b)
```

    array([[ 5, 12],
           [15, 24]])

`scipy.linalg` 的操作可以作用到两种类型的对象上，没有区别。

## 基本操作

### 求逆

矩阵 $\mathbf{A}$ 的逆 $\mathbf{B}$ 满足：$\mathbf{BA}=\mathbf{AB}=I$，记作 $\mathbf{B} = \mathbf{A}^{-1}$。

事实上，我们已经见过求逆的操作，`linalg.inv` 可以求一个可逆矩阵的逆：

```python
A = np.array([[1,2],[3,4]])

print linalg.inv(A)

print A.dot(scipy.linalg.inv(A))
```

    [[-2.   1. ]
     [ 1.5 -0.5]]
    [[  1.00000000e+00   0.00000000e+00]
     [  8.88178420e-16   1.00000000e+00]]

### 求解线性方程组

例如，下列方程组
$$
\begin{eqnarray*}
x + 3y + 5z & = & 10 \\
2x + 5y + z & = & 8  \\
2x + 3y + 8z & = & 3
\end{eqnarray*}
$$
的解为：
$$
\begin{split}\left[\begin{array}{c} x\\ y\\ z\end{array}\right]=\left[\begin{array}{ccc} 1 & 3 & 5\\ 2 & 5 & 1\\ 2 & 3 & 8\end{array}\right]^{-1}\left[\begin{array}{c} 10\\ 8\\ 3\end{array}\right]=\frac{1}{25}\left[\begin{array}{c} -232\\ 129\\ 19\end{array}\right]=\left[\begin{array}{c} -9.28\\ 5.16\\ 0.76\end{array}\right].\end{split}
$$

我们可以使用 `linalg.solve` 求解方程组，也可以先求逆再相乘，两者中 `solve` 比较快。

```python
import time

A = np.array([[1, 3, 5],
              [2, 5, 1],
              [2, 3, 8]])
b = np.array([10, 8, 3])

tic = time.time()

for i in xrange(1000):
    x = linalg.inv(A).dot(b)

print x
print A.dot(x)-b
print "inv and dot: {} s".format(time.time() - tic)

tic = time.time()

for i in xrange(1000):
    x = linalg.solve(A, b)

print x
print A.dot(x)-b
print "solve: {} s".format(time.time() - tic)
```

    [-9.28  5.16  0.76]
    [  0.00000000e+00  -1.77635684e-15  -8.88178420e-16]
    inv and dot: 0.0353579521179 s
    [-9.28  5.16  0.76]
    [  0.00000000e+00  -1.77635684e-15  -1.77635684e-15]
    solve: 0.0284671783447 s

### 计算行列式

方阵的行列式为
$$
\left|\mathbf{A}\right|=\sum_{j}\left(-1\right)^{i+j}a_{ij}M_{ij}.
$$

其中 $a_{ij}$ 表示 $\mathbf{A}$ 的第 $i$ 行 第 $j$ 列的元素，$M_{ij}$ 表示矩阵 $\mathbf{A}$ 去掉第 $i$ 行 第 $j$ 列的新矩阵的行列式。

例如，矩阵
$$
\begin{split}\mathbf{A=}\left[\begin{array}{ccc} 1 & 3 & 5\\ 2 & 5 & 1\\ 2 & 3 & 8\end{array}\right]\end{split}
$$
的行列式是：
$$
\begin{eqnarray*} \left|\mathbf{A}\right| & = & 1\left|\begin{array}{cc} 5 & 1\\ 3 & 8\end{array}\right|-3\left|\begin{array}{cc} 2 & 1\\ 2 & 8\end{array}\right|+5\left|\begin{array}{cc} 2 & 5\\ 2 & 3\end{array}\right|\\  & = & 1\left(5\cdot8-3\cdot1\right)-3\left(2\cdot8-2\cdot1\right)+5\left(2\cdot3-2\cdot5\right)=-25.\end{eqnarray*}
$$

可以用 `linalg.det` 计算行列式：

```python
A = np.array([[1, 3, 5],
              [2, 5, 1],
              [2, 3, 8]])

print linalg.det(A)
```

    -25.0

### 计算矩阵或向量的模

矩阵的模定义如下：
$$
\begin{split}\left\Vert \mathbf{A}\right\Vert =\left\{ \begin{array}{cc} \max_{i}\sum_{j}\left|a_{ij}\right| & \textrm{ord}=\textrm{inf}\\ \min_{i}\sum_{j}\left|a_{ij}\right| & \textrm{ord}=-\textrm{inf}\\ \max_{j}\sum_{i}\left|a_{ij}\right| & \textrm{ord}=1\\ \min_{j}\sum_{i}\left|a_{ij}\right| & \textrm{ord}=-1\\ \max\sigma_{i} & \textrm{ord}=2\\ \min\sigma_{i} & \textrm{ord}=-2\\ \sqrt{\textrm{trace}\left(\mathbf{A}^{H}\mathbf{A}\right)} & \textrm{ord}=\textrm{'fro'}\end{array}\right.\end{split}
$$
其中，$\sigma_i$ 是矩阵的奇异值。

向量的模定义如下：
$$
\begin{split}\left\Vert \mathbf{x}\right\Vert =\left\{ \begin{array}{cc} \max\left|x_{i}\right| & \textrm{ord}=\textrm{inf}\\ \min\left|x_{i}\right| & \textrm{ord}=-\textrm{inf}\\ \left(\sum_{i}\left|x_{i}\right|^{\textrm{ord}}\right)^{1/\textrm{ord}} & \left|\textrm{ord}\right|<\infty.\end{array}\right.\end{split}
$$

`linalg.norm` 可以计算向量或者矩阵的模：

```python
A = np.array([[1, 2],
              [3, 4]])

print linalg.norm(A)

print linalg.norm(A,'fro') # frobenius norm 默认值

print linalg.norm(A,1) # L1 norm 最大列和

print linalg.norm(A,-1) # L -1 norm 最小列和

print linalg.norm(A,np.inf) # L inf norm 最大行和
```

    5.47722557505
    5.47722557505
    6
    4
    7

### 最小二乘解和伪逆

#### 问题描述

所谓最小二乘问题的定义如下：

假设 $y_i$ 与 $\mathbf{x_i}$ 的关系可以用一组系数 $c_j$ 和对应的模型函数 $f_j(\mathbf{x_i})$ 的模型表示：

$$
y_{i}=\sum_{j}c_{j}f_{j}\left(\mathbf{x}_{i}\right)+\epsilon_{i}
$$

其中 $\epsilon_i$ 表示数据的不确定性。最小二乘就是要优化这样一个关于 $c_j$ 的问题：
$$
J\left(\mathbf{c}\right)=\sum_{i}\left|y_{i}-\sum_{j}c_{j}f_{j}\left(x_{i}\right)\right|^{2}
$$

其理论解满足：
$$
\frac{\partial J}{\partial c_{n}^{*}}=0=\sum_{i}\left(y_{i}-\sum_{j}c_{j}f_{j}\left(x_{i}\right)\right)\left(-f_{n}^{*}\left(x_{i}\right)\right)
$$

改写为：
$$
\begin{eqnarray*} \sum_{j}c_{j}\sum_{i}f_{j}\left(x_{i}\right)f_{n}^{*}\left(x_{i}\right) & = & \sum_{i}y_{i}f_{n}^{*}\left(x_{i}\right)\\ \mathbf{A}^{H}\mathbf{Ac} & = & \mathbf{A}^{H}\mathbf{y}\end{eqnarray*}
$$

其中：
$$
\left\{ \mathbf{A}\right\} _{ij}=f_{j}\left(x_{i}\right).
$$

当 $\mathbf{A^HA}$ 可逆时，我们有：
$$
\mathbf{c}=\left(\mathbf{A}^{H}\mathbf{A}\right)^{-1}\mathbf{A}^{H}\mathbf{y}=\mathbf{A}^{\dagger}\mathbf{y}
$$

矩阵 $\mathbf{A}^{\dagger}$ 叫做 $\mathbf{A}$ 的伪逆。

#### 问题求解

注意到，我们的模型可以写为：
$$
\mathbf{y}=\mathbf{Ac}+\boldsymbol{\epsilon}.
$$

在给定 $\mathbf{y}$ 和 $\mathbf{A}$ 的情况下，我们可以使用 `linalg.lstsq` 求解 $\mathbf c$。

在给定 $\mathbf{A}$ 的情况下，我们可以使用 `linalg.pinv` 或者 `linalg.pinv2` 求解 $\mathbf{A}^{\dagger}$。

####  例子

假设我们的数据满足：
$$
\begin{align}
y_{i} & =c_{1}e^{-x_{i}}+c_{2}x_{i} \\
z_{i} & = y_i + \epsilon_i
\end{align}
$$

其中 $x_i = \frac{i}{10},\ i = 1,\dots,10$，$c_1 = 5, c_2 = 2$，产生数据

```python
c1, c2 = 5.0, 2.0
i = np.r_[1:11]
xi = 0.1*i
yi = c1*np.exp(-xi) + c2*xi
zi = yi + 0.05 * np.max(yi) * np.random.randn(len(yi))
```

构造矩阵 $\mathbf A$：

```python
A = np.c_[np.exp(-xi)[:, np.newaxis], xi[:, np.newaxis]]
print A
```

    [[ 0.90483742  0.1       ]
     [ 0.81873075  0.2       ]
     [ 0.74081822  0.3       ]
     [ 0.67032005  0.4       ]
     [ 0.60653066  0.5       ]
     [ 0.54881164  0.6       ]
     [ 0.4965853   0.7       ]
     [ 0.44932896  0.8       ]
     [ 0.40656966  0.9       ]
     [ 0.36787944  1.        ]]

求解最小二乘问题：

```python
c, resid, rank, sigma = linalg.lstsq(A, zi)

print c
```

    [ 4.87016856  2.19081311]

其中 `c` 的形状与 `zi` 一致，为最小二乘解，`resid` 为 `zi - A c` 每一列差值的二范数，`rank` 为矩阵 `A` 的秩，`sigma` 为矩阵 `A` 的奇异值。

查看拟合效果：

```python
xi2 = np.r_[0.1:1.0:100j]
yi2 = c[0]*np.exp(-xi2) + c[1]*xi2

plt.plot(xi,zi,'x',xi2,yi2)
plt.axis([0,1.1,3.0,5.5])
plt.xlabel('$x_i$')
plt.title('Data fitting with linalg.lstsq')
plt.show()
```

### 广义逆

`linalg.pinv` 或 `linalg.pinv2` 可以用来求广义逆，其区别在于前者使用求最小二乘解的算法，后者使用求奇异值的算法求解。

## 矩阵分解

### 特征值和特征向量

#### 问题描述

对于给定的 $N \times N$ 矩阵 $\mathbf A$，特征值和特征向量问题相当与寻找标量 $\lambda$ 和对应的向量 $\mathbf v$ 使得：
$$
\mathbf{Av} = \lambda \mathbf{v}
$$

矩阵的 $N$ 个特征值（可能相同）可以通过计算特征方程的根得到：
$$
\left|\mathbf{A} - \lambda \mathbf{I}\right| = 0
$$

然后利用这些特征值求（归一化的）特征向量。

#### 问题求解

- `linalg.eig(A)`
    - 返回矩阵的特征值与特征向量
- `linalg.eigvals(A)`
    - 返回矩阵的特征值
- `linalg.eig(A, B)`
    - 求解 $\mathbf{Av} = \lambda\mathbf{Bv}$ 的问题

#### 例子

矩阵为
$$
\begin{split}\mathbf{A}=\left[\begin{array}{ccc} 1 & 5 & 2\\ 2 & 4 & 1\\ 3 & 6 & 2\end{array}\right].\end{split}
$$

特征多项式为：
$$
\begin{eqnarray*} \left|\mathbf{A}-\lambda\mathbf{I}\right| & = & \left(1-\lambda\right)\left[\left(4-\lambda\right)\left(2-\lambda\right)-6\right]-\\  &  & 5\left[2\left(2-\lambda\right)-3\right]+2\left[12-3\left(4-\lambda\right)\right]\\  & = & -\lambda^{3}+7\lambda^{2}+8\lambda-3.\end{eqnarray*}
$$

特征根为：
$$
\begin{eqnarray*} \lambda_{1} & = & 7.9579\\ \lambda_{2} & = & -1.2577\\ \lambda_{3} & = & 0.2997.\end{eqnarray*}
$$

```python
A = np.array([[1, 5, 2],
              [2, 4, 1],
              [3, 6, 2]])

la, v = linalg.eig(A)

print la

# 验证是否归一化
print np.sum(abs(v**2),axis=0)

# 第一个特征值
l1 = la[0]
# 对应的特征向量
v1 = v[:, 0].T

# 验证是否为特征值和特征向量对
print linalg.norm(A.dot(v1)-l1*v1)
```

    [ 7.95791620+0.j -1.25766471+0.j  0.29974850+0.j]
    [ 1.  1.  1.]
    3.23301824835e-15

### 奇异值分解

#### 问题描述

$M \times N$ 矩阵 $\mathbf A$ 的奇异值分解为：
$$
\mathbf{A=U}\boldsymbol{\Sigma}\mathbf{V}^{H}
$$

其中 $\boldsymbol{\Sigma}, (M \times N)$ 只有对角线上的元素不为 0，$\mathbf U, (M \times M)$ 和 $\mathbf V, (N \times N)$ 为正交矩阵。

其具体原理可以查看维基百科：
https://en.wikipedia.org/wiki/Singular_value_decomposition

#### 问题求解

- `U,s,Vh = linalg.svd(A)`
    - 返回 $U$ 矩阵，奇异值 $s$，$V^H$ 矩阵
- `Sig = linalg.diagsvd(s,M,N)`
    - 从奇异值恢复 $\boldsymbol{\Sigma}$ 矩阵

#### 例子

奇异值分解：

```python
A = np.array([[1,2,3],[4,5,6]])

U, s, Vh = linalg.svd(A)
```

$\boldsymbol{\Sigma}$ 矩阵：

```python
M, N = A.shape
Sig = linalg.diagsvd(s,M,N)

print Sig
```

    [[ 9.508032    0.          0.        ]
     [ 0.          0.77286964  0.        ]]

检查正确性：

```python
print A
print U.dot(Sig.dot(Vh))
```

    [[1 2 3]
     [4 5 6]]
    [[ 1.  2.  3.]
     [ 4.  5.  6.]]

### LU 分解

$M \times N$ 矩阵 $\mathbf A$ 的 `LU` 分解为：
$$
\mathbf{A}=\mathbf{P}\,\mathbf{L}\,\mathbf{U}
$$

$\mathbf P$ 是 $M \times M$ 的单位矩阵的一个排列，$\mathbf L$ 是下三角阵，$\mathbf U$ 是上三角阵。

可以使用 `linalg.lu` 进行 LU 分解的求解：

具体原理可以查看维基百科：
https://en.wikipedia.org/wiki/LU_decomposition

```python
A = np.array([[1,2,3],[4,5,6]])

P, L, U = linalg.lu(A)

print P
print L
print U

print P.dot(L).dot(U)
```

    [[ 0.  1.]
     [ 1.  0.]]
    [[ 1.    0.  ]
     [ 0.25  1.  ]]
    [[ 4.    5.    6.  ]
     [ 0.    0.75  1.5 ]]
    [[ 1.  2.  3.]
     [ 4.  5.  6.]]

### Cholesky 分解

`Cholesky` 分解是一种特殊的 `LU` 分解，此时要求 $\mathbf A$ 为 Hermitian 正定矩阵 （$\mathbf A = \mathbf{A^H}$）。

此时有：
$$
\begin{eqnarray*} \mathbf{A} & = & \mathbf{U}^{H}\mathbf{U}\\ \mathbf{A} & = & \mathbf{L}\mathbf{L}^{H}\end{eqnarray*}
$$
即
$$
\mathbf{L}=\mathbf{U}^{H}.
$$

可以用 `linalg.cholesky` 求解。

### QR 分解

$M×N$ 矩阵 $\mathbf A$ 的 `QR` 分解为：
$$
\mathbf{A=QR}
$$

$\mathbf R$ 为上三角形矩阵，$\mathbf Q$ 是正交矩阵。

维基链接：
https://en.wikipedia.org/wiki/QR_decomposition

可以用 `linalg.qr` 求解。

### Schur 分解

对于 $N\times N$ 方阵 $\mathbf A$, `Schur` 分解要求找到满足下式的矩阵：
$$
\mathbf{A=ZTZ^H}
$$

其中 $\mathbf Z$ 是正交矩阵，$\mathbf T$ 是一个上三角矩阵。

维基链接：
https://en.wikipedia.org/wiki/Schur_decomposition

```python
A = np.mat('[1 3 2; 1 4 5; 2 3 6]')

print A

T, Z = linalg.schur(A)

print T, Z

print Z.dot(T).dot(Z.T)
```

    [[1 3 2]
     [1 4 5]
     [2 3 6]]
    [[ 9.90012467  1.78947961 -0.65498528]
     [ 0.          0.54993766 -1.57754789]
     [ 0.          0.51260928  0.54993766]] [[ 0.36702395 -0.85002495 -0.37782404]
     [ 0.63681656 -0.06646488  0.76814522]
     [ 0.67805463  0.52253231 -0.51691576]]
    [[ 1.  3.  2.]
     [ 1.  4.  5.]
     [ 2.  3.  6.]]

## 矩阵函数

考虑函数 $f(x)$ 的泰勒展开：
$$
f\left(x\right)=\sum_{k=0}^{\infty}\frac{f^{\left(k\right)}\left(0\right)}{k!}x^{k}
$$

对于方阵，矩阵函数可以定义如下：
$$
f\left(\mathbf{A}\right)=\sum_{k=0}^{\infty}\frac{f^{\left(k\right)}\left(0\right)}{k!}\mathbf{A}^{k}
$$

这也是计算矩阵函数的最好的方式。

### 指数和对数函数

#### 指数

指数可以定义如下：
$$
e^{\mathbf{A}}=\sum_{k=0}^{\infty}\frac{1}{k!}\mathbf{A}^{k}
$$

`linalg.expm3` 使用的是泰勒展开的方法计算结果：

```python
A = np.array([[1, 2], [3, 4]])

print linalg.expm3(A)
```

    [[  51.96890355   74.73648784]
     [ 112.10473176  164.07363531]]

另一种方法先计算 A 的特征值分解：
$$
\mathbf{A}=\mathbf{V}\boldsymbol{\Lambda}\mathbf{V}^{-1}
$$

然后有（正交矩阵和对角阵的性质）：
$$
e^{\mathbf{A}}=\mathbf{V}e^{\boldsymbol{\Lambda}}\mathbf{V}^{-1}
$$

`linalg.expm2` 使用的就是这种方法：

```python
print linalg.expm2(A)
```

    [[  51.9689562    74.73656457]
     [ 112.10484685  164.07380305]]

最优的方法是用 [`Padé` 近似](https://en.wikipedia.org/wiki/Pad%C3%A9_approximant) 实现，`Padé` 近似往往比截断的泰勒级数准确，而且当泰勒级数不收敛时，`Padé` 近似往往仍可行，所以多用于在计算机数学中。

`linalg.expm` 使用的就是这种方法：

```python
print linalg.expm(A)
```

    [[  51.9689562    74.73656457]
     [ 112.10484685  164.07380305]]

#### 对数

指数的逆运算，可以用 `linalg.logm` 实现：

```python
print A
print linalg.logm(linalg.expm(A))
```

    [[1 2]
     [3 4]]
    [[ 1.  2.]
     [ 3.  4.]]

### 三角函数

根据欧拉公式，其定义为：
$$
\begin{eqnarray*} \sin\left(\mathbf{A}\right) & = & \frac{e^{j\mathbf{A}}-e^{-j\mathbf{A}}}{2j}\\ \cos\left(\mathbf{A}\right) & = & \frac{e^{j\mathbf{A}}+e^{-j\mathbf{A}}}{2}.\end{eqnarray*}
$$

正切函数定义为：
$$
\tan\left(x\right)=\frac{\sin\left(x\right)}{\cos\left(x\right)}=\left[\cos\left(x\right)\right]^{-1}\sin\left(x\right)
$$

因此矩阵的正切函数定义为：
$$
\left[\cos\left(\mathbf{A}\right)\right]^{-1}\sin\left(\mathbf{A}\right).
$$

具体实现：
- `linalg.sinm`
- `linalg.cosm`
- `linalg.tanm`

### 双曲三角函数

\begin{eqnarray*} \sinh\left(\mathbf{A}\right) & = & \frac{e^{\mathbf{A}}-e^{-\mathbf{A}}}{2}\\ \cosh\left(\mathbf{A}\right) & = & \frac{e^{\mathbf{A}}+e^{-\mathbf{A}}}{2}\\ \tanh\left(\mathbf{A}\right) & = & \left[\cosh\left(\mathbf{A}\right)\right]^{-1}\sinh\left(\mathbf{A}\right).\end{eqnarray*}

具体实现：
- `linalg.sinhm`
- `linalg.coshm`
- `linalg.tanhm`

## 特殊矩阵

`Scipy` 提供了一些特殊矩阵的实现，具体可以参考：

http://docs.scipy.org/doc/scipy/reference/tutorial/linalg.html#special-matrices


# Version information

Print the versions of the most important packages in the same form as in `environment.yml`. Print also the path of the Python interpreter to see that we are using the correct one.

```python
import sys
print(sys.executable)

import platform
import numpy as np
import matplotlib as mpl
import scipy
import sympy as sp
import pandas as pd
import skfem

print(f'  - python={platform.python_version()}')
print(f'  - numpy={np.__version__}')
print(f'  - matplotlib={mpl.__version__}')
print(f'  - scipy={scipy.__version__}')
print(f'  - sympy={sp.__version__}')
print(f'  - pandas={pd.__version__}')
print(f'  - pip:')
print(f'    - scikit-fem={skfem.__version__}')
```

# Testing more complicated widgets

```python
from ipywidgets import interact
@interact(theta=(0, 2.0))
def hello_world(theta):
    print(f'Hello world version {theta}')
```

```python
from ipycanvas import Canvas
canvas = Canvas(width=200, height=200)
canvas.fill_rect(25, 25, 100, 100)
canvas
```