# Configuration

```python
from traitlets.config.manager import BaseJSONConfigManager
path = "/home/ucl/nbconfig"
cm = BaseJSONConfigManager(config_dir=path)

cm.update('livereveal', {
        'theme': 'sky',
        'transition': 'zoom',
        'start_slideshow_at': 'selected',
        'controls': True
})
```

    {'controls': True,
     'start_slideshow_at': 'selected',
     'theme': 'sky',
     'transition': 'zoom'}

```javascript
%%javascript
require(['base/js/utils'],
function(utils) {
   utils.load_extensions('calico-spell-check', 'calico-document-tools', 'calico-cell-tools');
});
```

    <IPython.core.display.Javascript object>

```python
%load_ext tikzmagic
```

```python
%%html
<style>
.red { color: #E41A1C; }
.orange { color: #FF7F00 }
.yellow { color: #FFC020 }
.green { color: #4DAF4A }
.blue { color: #377EB8; }
.purple { color: #984EA3 }

h1 {
    color: #377EB8;
}

ctb_global_show div.ctb_hideshow.ctb_show {
    display: inline;
}

div.tabContent {
    padding: 0px;
    background: #ffffff;
    border: 0px;
}

.left {
    float: left;
    width: 50%;
    vertical-align: text-top;
}

.right {
    margin-left: 50%;
    vertical-align: text-top;
}

.small {
    zoom: 0.9;
    -ms-zoom: 0.9;
    -webkit-zoom: 0.9;
    -moz-transform:  scale(0.9,0.9);
    -moz-transform-origin: left center;
}

.verysmall {
    zoom: 0.75;
    -ms-zoom: 0.75;
    -webkit-zoom: 0.75;
    -moz-transform:  scale(0.75,0.75);
    -moz-transform-origin: left center;
}

.tiny {
    zoom: 0.6;
    -ms-zoom: 0.6;
    -webkit-zoom: 0.6;
    -moz-transform:  scale(0.6,0.6);
    -moz-transform-origin: left center;
}

.rendered_html blockquote {
    border-left-width: 0px;
    padding: 15px;
    margin: 0px;
    width: 100%;
}

.rendered_html th {
    padding: 0.5em;
    border: 0px;
}

.rendered_html td {
    padding: 0.25em;
    border: 0px;
}

#for reveal
.aside .controls, .reveal .controls {
    display: none !important;
    width: 0px !important;
    height: 0px !important;
}

.rise-enabled .reveal .slide-number {
    right: 25px;
    bottom: 25px;
    font-size: 200%;
    color: #377EB8;
}

.rise-enabled .reveal .progress span {
    background: #377EB8;
}

.present .top {
    position: fixed !important;
    top: 0 !important;
}

.present .rendered_html * + p, .present .rendered_html p, .present .rendered_html * + br, .present .rendered_html br {
    margin: 0.5em 0;
}

.present tr, .present td {
    border: 0px;
    padding: 0.35em;
}

.present th {
    border: 1px;
}

present .prompt {
    min-width: 0px !important;
    transition-duration: 0s !important;
}

.prompt {
    min-width: 0px !important;
    transition-duration: 0s !important;
}

.rise-enabled .cell li {
    line-height: 135%;
}

</style>

%load_ext tikzmagic
```

%load_ext tikzmagic

Recurrent Neural Networks

Pasquale Minervini

p.minervini@cs.ucl.ac.uk

University College London

Statistical Natural Language Processing Course, UCL
9th December 2016

# Language Models (LMs)

A LM computes a **probability** for a **sequence of words**

$$p(w_{1}, \ldots, w_{d})$$

Useful in a miriad of NLP tasks, such as machine translation:

$$p(\text{I}, \text{go}, \text{home}) > p(\text{I}, \text{go}, \text{house})$$

# $n$-Gram Language Models

In *$n$-gram language models*, the probability $p(w_{1}, \ldots, w_{d})$ of observing the sentence $(w_{1}, \ldots, w_{d})$ is **approximated** as:

$$p(w_{1}, \ldots, w_{d}) = \prod_{i=1}^{d} p(w_{i} \mid w_{1}, \ldots, w_{i - 1}) \\ \approx \prod_{i=1}^{d} p(w_{i} \mid w_{i - (n - 1)}, \ldots, w_{i - 1})$$

Example with a **bigram** ($n = 2$) **language model**:

$$p(\text{I}, \text{go}, \text{home}) \approx p(\text{I} \mid \text{[START]}) \; p(\text{go} \mid \text{I}) \; p(\text{home} \mid \text{go})$$

# A Recurrent Neural Network LM

Given a sequence of **word vectors** (e.g. one-hot, word2vec, GloVe), each representing a word $w \in V$ in the vocabulary $V$:

$$(\mathbf{x}_{1}, \ldots, \mathbf{x}_{t - 1}, \mathbf{x}_{t}, \mathbf{x}_{t + 1}, \ldots, \mathbf{x}_{d})$$

At each single time step $t$, the hidden state $\mathbf{h}_t$ and output $\hat{\mathbf{y}}_t$ are calculated as follows:

$$
\begin{aligned}
\mathbf{h}_t & = \text{sigmoid}(\mathbf{W}^h \mathbf{h}_{t-1}+ \mathbf{W}^x \mathbf{x}_t) \in \mathbb{R}^{d_{h}}\\
\hat{\mathbf{y}}_{t} & = \text{softmax}(\mathbf{W}^o \mathbf{h}_{t}) \in \mathbb{R}^{|V|}\\
\end{aligned}
$$

where $\mathbf{y}_{t} \in \mathbb{[0, 1]}^{|V|}$ is a **probability distribution** over words in $V$.

The probability that the $t$-th word in the sequence is $w_{j}$ is given by:

$$p(w_{j} \mid x_{t}, \ldots, x_{1}) = \hat{\mathbf{y}}_{t, j}$$

# Example

Consider the word sequence $(\text{I}, \text{go}, \text{home}) \rightarrow (x_{1}, x_{2}, x_{3})$

Reminder: $\mathbf{h}_t = \text{sigmoid}(\mathbf{W}^h \mathbf{h}_{t-1}+ \mathbf{W}^x \mathbf{x}_t + \mathbf{b})$

$$
\begin{aligned}
\mathbf{h}_1 = \text{sigmoid}(\mathbf{W}^h \mathbf{h}_{0} + \mathbf{W}^x \mathbf{x}_1) &\;& \hat{\mathbf{y}}_{1} = \text{softmax}(\mathbf{W}^o \mathbf{h}_{1}) \\
\mathbf{h}_2 = \text{sigmoid}(\mathbf{W}^h \mathbf{h}_{1} + \mathbf{W}^x \mathbf{x}_2) &\;& \hat{\mathbf{y}}_{2} = \text{softmax}(\mathbf{W}^o \mathbf{h}_{2}) \\
\mathbf{h}_3 = \text{sigmoid}(\mathbf{W}^h \mathbf{h}_{2} + \mathbf{W}^x \mathbf{x}_3) &\;& \hat{\mathbf{y}}_{3} = \text{softmax}(\mathbf{W}^o \mathbf{h}_{3}) \\
\end{aligned}
$$

$$p(\text{I}, \text{go}, \text{home}) = \hat{\mathbf{y}}_{1, [\text{I}]} \; \hat{\mathbf{y}}_{2, [\text{go}]} \; \hat{\mathbf{y}}_{3, [\text{home}]}$$

- Initial state: $\mathbf{h}_{0} \in \mathbb{R}^{d_{h}}$
- Input matrix: $\mathbf{W}^x \in \mathbb{R}^{d_{h} \times d_{x}}$
- Transition matrix: $\mathbf{W}^h \in \mathbb{R}^{d_{h} \times d_{h}}$
- Output matrix: $\mathbf{W}^o \in \mathbb{R}^{|V| \times d_{h}}$

# Recurrent Neural Networks

```python
%%tikz -l arrows -s 1000,400 -sc 0.65

\newcommand{\lstm}{
\definecolor{nice-red}{HTML}{E41A1C}
\definecolor{nice-orange}{HTML}{FF7F00}
\definecolor{nice-yellow}{HTML}{FFC020}
\definecolor{nice-green}{HTML}{4DAF4A}
\definecolor{nice-blue}{HTML}{377EB8}
\definecolor{nice-purple}{HTML}{984EA3}

%lstm first step

%lstm module box
\draw[line width=3pt, color=black!50] (0,0) rectangle (3,3);
}

\lstm
\node[] at (0.5,-1.25) {$\mathbf{x}_t$};
\node[] at (-1.5,2) {$\mathbf{h}_{t-1}$};
\node[] at (4.25,2) {$\mathbf{h}_t$};
\node[] at (2.5,5) {$\mathbf{h}_t$};

\draw[ultra thick, ->, >=stealth'] (0.5,-0.75) -- (0.5,0);
\draw[ultra thick, ->, >=stealth'] (-0.75,2) -- (0,2);
\draw[ultra thick, ->, >=stealth'] (3,2) -- (3.75,2);
\draw[ultra thick, ->, >=stealth'] (2.5,3) -- (2.5,4.75);

\path[line width=3pt, ->, >=stealth', color=nice-red] (4, 2.5) edge[bend right=0, in=-110, out=-70] (-1.75, 2.5);

\node[] at (1.5,2) {$f_\theta(\mathbf{x}_t, \mathbf{h}_{t-1})$};

```

\begin{align}
\mathbf{h}_t &= f_{\theta}(\mathbf{x}_{t}, \mathbf{h}_{t - 1}) \\
    f_{\theta} \; & \text{is a } \textbf{transition function} \text { with parameters } \theta\\
    \theta \; & \text{can be } \textbf{learned from data}\\
\\
\\
& \text{Remind - Simple Recurrent Neural Network} \\
\mathbf{h}_t &= \text{sigmoid}(\mathbf{W}^h \mathbf{h}_{t-1}+ \mathbf{W}^x \mathbf{x}_t)
\end{align}

# Objective Function

Recall that $\mathbf{y}_{t} \in \mathbb{R}^{|V|}$ is a probability distribution over the vocabulary $V$.

We can train a RNN by minimizing the **cross-entropy loss**, predicting **words** instead of classes:

$$
\begin{aligned}
J_{t} = - \sum_{i = 1}^{|V|} \mathbf{y}_{t, i} \log \hat{\mathbf{y}}_{t, i}, \quad \mathbf{y}_{t, i} = \left\{\begin{array}{ll}1 \; \text{if the $t$-th word is $w_{i}$,}\\0 \, \text{otherwise.}\end{array} \right.
\end{aligned}
$$

# Evaluating Language Models

Evaluation - negative of average log-probability over corpus:

$$J = - \frac{1}{T} \sum_{t = 1}^{T} \sum_{j = 1}^{|V|} \mathbf{y}_{t, j} \log \hat{\mathbf{y}}_{t, j} = \frac{1}{T} J_{t}$$

Or also **perplexity**:

$$PP(w_1,\ldots,w_T) = \sqrt[T]{\prod_{i = 1}^T \frac{1}{p(w_i | w_{1}, \ldots, w_{i-1})}}$$

# Sequence-to-Sequence Models

Recurrent Neural Networks are extremely powerful and flexible
- They can also learn to **generate** sequences

Seq2Seq models are composed by:
- **Encoder** - Gets the input and outputs $\mathbf{v} \in \mathbb{R}^{d}$
- **Decoder** - Gets $\mathbf{v}$ and generates the output sequence

Seq2Seq models are widely popular in e.g.:
- *Neural Machine Translation*
- *Text Summarization*
- *Learning to Execute*

```python
%%tikz -l arrows -s 1000,400 -sc 0.65

\definecolor{nice-red}{HTML}{E41A1C}
\definecolor{nice-orange}{HTML}{FF7F00}
\definecolor{nice-yellow}{HTML}{FFC020}
\definecolor{nice-green}{HTML}{4DAF4A}
\definecolor{nice-blue}{HTML}{377EB8}
\definecolor{nice-purple}{HTML}{984EA3}

\newcommand{\lstm}{

%lstm first step

%lstm module box
\draw[line width=3pt, color=black!50] (0,0) rectangle (3,3);

\draw[ultra thick, ->, >=stealth'] (0.5,-0.75) -- (0.5,0);
\draw[ultra thick, ->, >=stealth'] (-0.75,2) -- (0,2);
\draw[ultra thick, ->, >=stealth'] (3,2) -- (3.75,2);
\draw[ultra thick, ->, >=stealth'] (2.5,3) -- (2.5,3.75);
}

%\lstm

%\node[] at (0.5,-1.25) {$\mathbf{x}_t$};
%\node[] at (-1.5,2) {$\mathbf{h}_{t-1}$};
%\node[] at (4.25,2) {$\mathbf{h}_t$};
%\node[] at (2.5,5) {$\mathbf{h}_t$};
%\path[line width=3pt, ->, >=stealth', color=nice-blue] (4, 2.5) edge[bend right=0, in=-110, out=-70] (-1.75, 2.5);
%\node[] at (1.5,2) {$f_\theta(\mathbf{x}_t, \mathbf{h}_{t-1})$};

\foreach \x/\w in {0/I, 1/like, 2/neural, 3/networks} {
    \begin{scope}[shift={(\x*3.75,0)}]
        \lstm
        \node[font=\LARGE, text height=1.5ex, color=nice-red] at (0.5,-1.5) {\bf\w};
    \end{scope}
}

\foreach \x/\w/\t in {4/EOS/Ich, 5/Ich/mag, 6/mag/neuronale, 7/neuronale/Netze, 8/Netze/EOS} {
    \begin{scope}[shift={(\x*3.75,0)}]
        \lstm
        \node[font=\LARGE, text height=1.5ex] at (0.5,-1.5) {\bf\w};
        \node[font=\LARGE, text height=1.5ex, color=nice-blue] at (2.5,4.5) {\bf\t};
    \end{scope}
}

\node[font=\Huge, color=nice-red] at (16.5,1.5) {$\mathbf{v}$};
```

# Problem - Training RNNs is Hard

- **Vanishing** and **exploding** gradients [Pascanu et al. 2013].

Why? Multiply the same matrix $\mathbf{W}^{h}$ at each time step during forward propagation. The norm of the gradient might either tend to 0 (**vanish**) or be too large (**explode**).

# Related Problem - Long-Term Dependencies

Words from time steps far away are hardly considered when training to predict the next word.

Example:
- John walked to the hallway.
- Mary walked in too.
- Daniel moved to the garden.
- John said "Hi" to \_\_\_\_.

A RNN is very likely to e.g. put an uniform probability distributions over nouns in $V$, and a low probability everywhere else.

It's an issue with language modeling, question answering, and many other tasks.

# Vanishing/Exploding Gradients - Solutions

Several solutions in the literature:

- Bound the gradient to a threshold (**Gradient Clipping**)
[Pascanu et al. 2013]

- Use $\text{ReLU}(x) = \max(0, x)$ (**Re**ctified **L**inear **U**nits) or similar non-linearities instead of $\text{sigmoid}(x)$ or $\text{tanh}(x)$
[Glorot et al. 2011].

- Clever Initialization of the Transition Matrix ($\mathbf{W}^h = \mathbf{I}$)
[Socher et al. 2013, Le et al. 2015].

- Use different recurrent models that favour backpropagation
LSTM[Hochreiter et al. 1997], GRU[Chung et al. 2014].

# Long Short-Term Memory (LSTM) Networks

- Can adaptively learn what to **keep** (store) into memory (gate $\mathbf{i}_{t}$), **forget** (gate $\mathbf{f}_{t}$) and **output** (gate $\mathbf{o}_{t}$)

```python
%%tikz -l arrows -s 1000,400 -sc 0.65

\newcommand{\lstm}{
\definecolor{nice-red}{HTML}{E41A1C}
\definecolor{nice-orange}{HTML}{FF7F00}
\definecolor{nice-yellow}{HTML}{FFC020}
\definecolor{nice-green}{HTML}{4DAF4A}
\definecolor{nice-blue}{HTML}{377EB8}
\definecolor{nice-purple}{HTML}{984EA3}

%lstm first step

%lstm module box
\draw[line width=3pt, color=black!50] (-6,-3) rectangle (1.5,5.25);
\draw[ultra thick] (0,0) rectangle (1,2);

%memory ct
\draw[ultra thick, color=nice-purple, fill=nice-purple!10] (0,0) rectangle (1,2);

%non-linearities
\foreach \w/\h/\color in {-2/4.25/nice-blue,-2/1/nice-red,-2/-1/nice-green,0.5/-2/nice-yellow,0.5/3/black} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, yshift=-0.5cm, color=\color] plot [domain=-0.3:0.3](\x, {(0.8/(1+exp(-15*\x))+0.1)});
        \draw[ultra thick, color=\color] (0,0) circle (0.5cm);
    \end{scope}
}

%tanh
\draw[thick, color=black] (0.25,3) -- (0.75,3);
\draw[thick, color=nice-yellow] (0.25,-2) -- (0.75,-2);

%component-wise multiplications
\foreach \w/\h in {-1/1,0.5/-1,0.5/4.25} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, color=black] (0,0) circle (0.05cm);
        \draw[ultra thick, color=black] (0,0) circle (0.5cm);
    \end{scope}
}

%vector concat
\begin{scope}[shift={(-4,1)},scale=0.5]
    \draw[ultra thick,yshift=0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick,yshift=-0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick] (0,0) circle (0.5cm);
\end{scope}

\foreach \fx/\fy/\tx/\ty in {
    -5/-3.5/-5/0.85, %xt
    -5/0.85/-4.2/0.85,
    -6.5/4.25/-5/4.25, %ht1
    -5/4.25/-5/1.15,
    -5/1.15/-4.2/1.15,
    -3.75/1/-3/1, %H
    -3/4.25/-3/-2,
    -3/-2/0.25/-2, %i
    0.5/-1.75/0.5/-1.25,
    -3/-1/-2.25/-1, %it
    -1.75/-1/0.25/-1,
    -3/1/-2.25/1, %ft
    -1.75/1/-1.25/1,
    -0.75/1/0/1,
    -3/4.25/-2.25/4.25, %ot
    -1.75/4.25/0.25/4.25,
    0.5/2/0.5/2.75, %ct
    -5.5/2/-5.1/2, %ct1
    -5.5/2/-5.5/1,
    -6.5/1/-5.5/1,
    -4.9/2/-3.1/2,
    -2.9/2/-1/2,
    -1/2/-1/1.25
} {
    \draw[ultra thick] (\fx,\fy) -- (\tx,\ty);
}

\foreach \fx/\fy/\tx/\ty in {
    0.5/-0.75/0.5/0, %it
    -0.75/1/0/1, %ft
    1/1/2.25/1,
    0.5/3.25/0.5/4,
    0.75/4.25/2.25/4.25, %ht
    0.5/4.5/0.5/6
} {
    \draw[->, >=stealth', ultra thick] (\fx,\fy) -- (\tx,\ty);
}
}

%\begin{scope}[scale=0.8]
%\foreach \d in {0,1} {
%\foreach \t in {0,1,2,3,4} {
%\begin{scope}[shift={(\t*8.5+\d*5.5,\d*9.5)}]
%    \lstm
%\end{scope}
%}
%}
%\end{scope}

\lstm

%annotations
\node[] at (-5,-3.75) {$\mathbf{x}_t$};
\node[anchor=east] at (-6.5,4.25) {$\mathbf{h}_{t-1}$};
\node[anchor=east] at (-6.5,1) {$\mathbf{c}_{t-1}$};
\node[] at (0.5,6.25) {$\mathbf{h}_t$};
\node[anchor=west] at (2.25,4.25) {$\mathbf{h}_t$};
\node[anchor=west] at (2.25,1) {$\mathbf{c}_t$};
\node[xshift=0.4cm,yshift=0.25cm] at (-4,1) {$\mathbf{H}_t$};
\node[xshift=0.35cm,yshift=0.25cm] at (-2,-1) {$\mathbf{i}_t$};
\node[xshift=0.35cm,yshift=0.25cm] at (-2,1) {$\mathbf{f}_t$};
\node[xshift=0.35cm,yshift=0.25cm] at (-2,4.25) {$\mathbf{o}_t$};

%dummy node for left alignment
\node[] at (17,0) {};
```

\begin{align}
\\
\mathbf{H}_t &= \left[
 \begin{array}{*{20}c}
        \mathbf{x}_t \\
        \mathbf{h}_{t-1}
      \end{array}
\right]\\
\mathbf{i}_t &= \text{sigmoid}(\mathbf{W}^i\mathbf{H}+\mathbf{b}^i)\\
\mathbf{f}_t &= \text{sigmoid}(\mathbf{W}^f\mathbf{H}+\mathbf{b}^f)\\
\mathbf{o}_t &= \text{sigmoid}(\mathbf{W}^o\mathbf{H}+\mathbf{b}^o)\\
\mathbf{c}_t &= \mathbf{f}_t \odot \mathbf{c}_{t-1} + \mathbf{i}_t \odot \tanh(\mathbf{W}^c\mathbf{H}+\mathbf{b}^c)\\
\mathbf{h}_t &= \mathbf{o}_t \odot \tanh(\mathbf{c}_t)
\end{align}

# Sentence Encoding

```python
%%tikz -l arrows -s 1000,400 -sc 0.65

\newcommand{\lstm}{
\definecolor{nice-red}{HTML}{E41A1C}
\definecolor{nice-orange}{HTML}{FF7F00}
\definecolor{nice-yellow}{HTML}{FFC020}
\definecolor{nice-green}{HTML}{4DAF4A}
\definecolor{nice-blue}{HTML}{377EB8}
\definecolor{nice-purple}{HTML}{984EA3}

%lstm first step

%lstm module box
\draw[line width=3pt, color=black!50] (-6,-3) rectangle (1.5,5.25);
\draw[ultra thick] (0,0) rectangle (1,2);

%memory ct
\draw[ultra thick, color=nice-purple, fill=nice-purple!10] (0,0) rectangle (1,2);

%non-linearities
\foreach \w/\h/\color in {-2/4.25/nice-blue,-2/1/nice-red,-2/-1/nice-green,0.5/-2/nice-yellow,0.5/3/black} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, yshift=-0.5cm, color=\color] plot [domain=-0.3:0.3](\x, {(0.8/(1+exp(-15*\x))+0.1)});
        \draw[ultra thick, color=\color] (0,0) circle (0.5cm);
    \end{scope}
}

%tanh
\draw[thick, color=black] (0.25,3) -- (0.75,3);
\draw[thick, color=nice-yellow] (0.25,-2) -- (0.75,-2);

%component-wise multiplications
\foreach \w/\h in {-1/1,0.5/-1,0.5/4.25} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, color=black] (0,0) circle (0.05cm);
        \draw[ultra thick, color=black] (0,0) circle (0.5cm);
    \end{scope}
}

%vector concat
\begin{scope}[shift={(-4,1)},scale=0.5]
    \draw[ultra thick,yshift=0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick,yshift=-0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick] (0,0) circle (0.5cm);
\end{scope}

\foreach \fx/\fy/\tx/\ty in {
    -5/-3.5/-5/0.85, %xt
    -5/0.85/-4.2/0.85,
    -6.5/4.25/-5/4.25, %ht1
    -5/4.25/-5/1.15,
    -5/1.15/-4.2/1.15,
    -3.75/1/-3/1, %H
    -3/4.25/-3/-2,
    -3/-2/0.25/-2, %i
    0.5/-1.75/0.5/-1.25,
    -3/-1/-2.25/-1, %it
    -1.75/-1/0.25/-1,
    -3/1/-2.25/1, %ft
    -1.75/1/-1.25/1,
    -0.75/1/0/1,
    -3/4.25/-2.25/4.25, %ot
    -1.75/4.25/0.25/4.25,
    0.5/2/0.5/2.75, %ct
    -5.5/2/-5.1/2, %ct1
    -5.5/2/-5.5/1,
    -6.5/1/-5.5/1,
    -4.9/2/-3.1/2,
    -2.9/2/-1/2,
    -1/2/-1/1.25
} {
    \draw[ultra thick] (\fx,\fy) -- (\tx,\ty);
}

\foreach \fx/\fy/\tx/\ty in {
    0.5/-0.75/0.5/0, %it
    -0.75/1/0/1, %ft
    1/1/2.25/1,
    0.5/3.25/0.5/4,
    0.75/4.25/2.25/4.25, %ht
    0.5/4.5/0.5/6
} {
    \draw[->, >=stealth', ultra thick] (\fx,\fy) -- (\tx,\ty);
}
}

\begin{scope}[scale=0.8]
\foreach \d in {0} {
\foreach \t/\word in {0/A,1/wedding,2/party,3/taking,4/pictures} {
    \node[font=\Huge, anchor=west] at (\t*8.5-5.75,-4.5) {$\mathbf{v}$\_\word};
    \begin{scope}[shift={(\t*8.5+\d*5.5,\d*9.5)}]
        \lstm
    \end{scope}
}
}
\end{scope}

\node[font=\Huge, anchor=west] at (27,5.75) {$\mathbf{v}$\_Sentence};

%dummy node for left alignment
\node[] at (17,0) {};
```

# Gating

```python
%%tikz -l arrows -s 1000,400 -sc 0.65

\definecolor{nice-red}{HTML}{E41A1C}
\definecolor{nice-orange}{HTML}{FF7F00}
\definecolor{nice-yellow}{HTML}{FFC020}
\definecolor{nice-green}{HTML}{4DAF4A}
\definecolor{nice-blue}{HTML}{377EB8}
\definecolor{nice-purple}{HTML}{984EA3}

\newcommand{\lstm}{
%lstm first step

%lstm module box
\draw[line width=3pt, color=black!50] (-6,-3) rectangle (1.5,5.25);
\draw[ultra thick] (0,0) rectangle (1,2);

%memory ct
\draw[ultra thick, color=nice-purple, fill=nice-purple!10] (0,0) rectangle (1,2);

%non-linearities
\foreach \w/\h/\color in {-2/4.25/nice-blue,-2/1/nice-red,-2/-1/nice-green,0.5/-2/nice-yellow,0.5/3/black} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, yshift=-0.5cm, color=\color] plot [domain=-0.3:0.3](\x, {(0.8/(1+exp(-15*\x))+0.1)});
        \draw[ultra thick, color=\color] (0,0) circle (0.5cm);
    \end{scope}
}

%tanh
\draw[thick, color=black] (0.25,3) -- (0.75,3);
\draw[thick, color=nice-yellow] (0.25,-2) -- (0.75,-2);

%component-wise multiplications
\foreach \w/\h in {-1/1,0.5/-1,0.5/4.25} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, color=black] (0,0) circle (0.05cm);
        \draw[ultra thick, color=black] (0,0) circle (0.5cm);
    \end{scope}
}

%vector concat
\begin{scope}[shift={(-4,1)},scale=0.5]
    \draw[ultra thick,yshift=0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick,yshift=-0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick] (0,0) circle (0.5cm);
\end{scope}

\foreach \fx/\fy/\tx/\ty in {
    -5/-3.5/-5/0.85, %xt
    -5/0.85/-4.2/0.85,
    -6.5/4.25/-5/4.25, %ht1
    -5/4.25/-5/1.15,
    -5/1.15/-4.2/1.15,
    -3.75/1/-3/1, %H
    -3/4.25/-3/-2,
    -3/-2/0.25/-2, %i
    0.5/-1.75/0.5/-1.25,
    -3/-1/-2.25/-1, %it
    -1.75/-1/0.25/-1,
    -3/1/-2.25/1, %ft
    -1.75/1/-1.25/1,
    -0.75/1/0/1,
    -3/4.25/-2.25/4.25, %ot
    -1.75/4.25/0.25/4.25,
    0.5/2/0.5/2.75, %ct
    -5.5/2/-5.1/2, %ct1
    -5.5/2/-5.5/1,
    -6.5/1/-5.5/1,
    -4.9/2/-3.1/2,
    -2.9/2/-1/2,
    -1/2/-1/1.25
} {
    \draw[ultra thick] (\fx,\fy) -- (\tx,\ty);
}

\foreach \fx/\fy/\tx/\ty in {
    0.5/-0.75/0.5/0, %it
    -0.75/1/0/1, %ft
    1/1/2.25/1,
    0.5/3.25/0.5/4,
    0.75/4.25/2.25/4.25, %ht
    0.5/4.5/0.5/6
} {
    \draw[->, >=stealth', ultra thick] (\fx,\fy) -- (\tx,\ty);
}
}

\begin{scope}[scale=0.8]
\foreach \d in {0} {
\foreach \t/\word in {0/A,1/wedding,2/party,3/taking,4/pictures} {
    \node[font=\Huge, anchor=west] at (\t*8.5-5.75,-4.5) {$\mathbf{v}$\_\word};
    \begin{scope}[shift={(\t*8.5+\d*5.5,\d*9.5)}]
        \lstm
    \end{scope}
}
}
\end{scope}

\node[font=\Huge, anchor=west] at (27,5.75) {$\mathbf{v}$\_Sentence};

\draw[line width=10pt, color=nice-red, opacity=0.8] (27.6,5) -- (27.6,0.75);
\draw[line width=10pt, color=nice-red, opacity=0.8] (27.5,0.75) -- (3,0.75);
\draw[->, >=stealth', line width=10pt, color=nice-red, opacity=0.8] (2.75,0.75) -- (2.75,-3);

%dummy node for left alignment
\node[] at (17,0) {};
```

# Visualizing Gradients

RNN vs. LSTM gradients on the input matrix $\mathbf{W}^x$

- Error is generated at 128th step and propagated back, no error from other steps.

```python
%%html

<center>

</center>
```

# Stacking (Deep LSTMs)

```python
%%tikz -l arrows -s 1100,500 -sc 0.65

\definecolor{nice-red}{HTML}{E41A1C}
\definecolor{nice-orange}{HTML}{FF7F00}
\definecolor{nice-yellow}{HTML}{FFC020}
\definecolor{nice-green}{HTML}{4DAF4A}
\definecolor{nice-blue}{HTML}{377EB8}
\definecolor{nice-purple}{HTML}{984EA3}

\newcommand{\lstm}{
%lstm first step

%lstm module box
\draw[line width=3pt, color=black!50] (-6,-3) rectangle (1.5,5.25);
\draw[ultra thick] (0,0) rectangle (1,2);

%memory ct
\draw[ultra thick, color=nice-purple, fill=nice-purple!10] (0,0) rectangle (1,2);

%non-linearities
\foreach \w/\h/\color in {-2/4.25/nice-blue,-2/1/nice-red,-2/-1/nice-green,0.5/-2/nice-yellow,0.5/3/black} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, yshift=-0.5cm, color=\color] plot [domain=-0.3:0.3](\x, {(0.8/(1+exp(-15*\x))+0.1)});
        \draw[ultra thick, color=\color] (0,0) circle (0.5cm);
    \end{scope}
}

%tanh
\draw[thick, color=black] (0.25,3) -- (0.75,3);
\draw[thick, color=nice-yellow] (0.25,-2) -- (0.75,-2);

%component-wise multiplications
\foreach \w/\h in {-1/1,0.5/-1,0.5/4.25} {
    \begin{scope}[shift={(\w,\h)},scale=0.5]
        \draw[ultra thick, color=black] (0,0) circle (0.05cm);
        \draw[ultra thick, color=black] (0,0) circle (0.5cm);
    \end{scope}
}

%vector concat
\begin{scope}[shift={(-4,1)},scale=0.5]
    \draw[ultra thick,yshift=0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick,yshift=-0.2cm] (0,0) circle (0.05cm);
    \draw[ultra thick] (0,0) circle (0.5cm);
\end{scope}

\foreach \fx/\fy/\tx/\ty in {
    -5/-3.5/-5/0.85, %xt
    -5/0.85/-4.2/0.85,
    -6.5/4.25/-5/4.25, %ht1
    -5/4.25/-5/1.15,
    -5/1.15/-4.2/1.15,
    -3.75/1/-3/1, %H
    -3/4.25/-3/-2,
    -3/-2/0.25/-2, %i
    0.5/-1.75/0.5/-1.25,
    -3/-1/-2.25/-1, %it
    -1.75/-1/0.25/-1,
    -3/1/-2.25/1, %ft
    -1.75/1/-1.25/1,
    -0.75/1/0/1,
    -3/4.25/-2.25/4.25, %ot
    -1.75/4.25/0.25/4.25,
    0.5/2/0.5/2.75, %ct
    -5.5/2/-5.1/2, %ct1
    -5.5/2/-5.5/1,
    -6.5/1/-5.5/1,
    -4.9/2/-3.1/2,
    -2.9/2/-1/2,
    -1/2/-1/1.25
} {
    \draw[ultra thick] (\fx,\fy) -- (\tx,\ty);
}

\foreach \fx/\fy/\tx/\ty in {
    0.5/-0.75/0.5/0, %it
    -0.75/1/0/1, %ft
    1/1/2.25/1,
    0.5/3.25/0.5/4,
    0.75/4.25/2.25/4.25, %ht
    0.5/4.5/0.5/6
} {
    \draw[->, >=stealth', ultra thick] (\fx,\fy) -- (\tx,\ty);
}
}

\begin{scope}[scale=0.8]
\foreach \d in {0,1,2} {
\foreach \t/\word in {0/A,1/wedding,2/party,3/taking,4/pictures} {
    \node[font=\Huge, anchor=west] at (\t*8.5-5.75,-4.5) {$\mathbf{v}$\_\word};
    \begin{scope}[shift={(\t*8.5+\d*5.5,\d*9.5)}]
        \lstm
    \end{scope}
}
}
\end{scope}

\node[font=\Huge, anchor=west] at (34,20.75) {$\mathbf{v}$\_Sentence};

\draw[line width=10pt, color=nice-red, opacity=0.8] (36.4,16) -- (36.4,20);
\draw[line width=10pt, color=nice-red, opacity=0.8] (25.25,16) -- (36.5,16);
\draw[line width=10pt, color=nice-red, opacity=0.8] (25.25,8.5) -- (25.25,16);
\draw[line width=10pt, color=nice-red, opacity=0.8] (14,8.5) -- (25.25,8.5);
\draw[line width=10pt, color=nice-red, opacity=0.8] (14,8.5) -- (14,0.75);
\draw[line width=10pt, color=nice-red, opacity=0.8] (14,0.75) -- (3,0.75);
\draw[->, >=stealth', line width=10pt, color=nice-red, opacity=0.8] (2.75,0.75) -- (2.75,-3);

%dummy node for left alignment
\node[] at (17,0) {};
```

# Applications
- Language Modeling
- Machine Translation
- Question Answering
- Dialog Modeling
- Language Generation
- Sentence Summarization
- Paraphrasing
- Sentiment Analysis
- Recognizing Textual Entailment
- ...

# Learning to Execute

RNNs are **Turing-Complete** [Siegelman, 1995] - they can simulate arbitrary programs, given the proper parameters.

Learning to Execute [Zaremba and Sutskever, 2014]

# Implementing LSTM (TensorFlow)

```python
lstm = rnn_cell.BasicLSTMCell(lstm_size)
# Initial state of the LSTM memory.
state = tf.zeros([batch_size, lstm.state_size])
probabilities = []
loss_val = 0.0
for batch_of_words in words_in_dataset:
  # State is updated after processing each batch
  output, state = lstm(batch_of_words, state)
  # Output is used to make next word predictions
  scores = tf.matmul(output, out_w) + out_b
  probabilities.append(tf.nn.softmax(scores))
  loss_val += loss(probabilities, target_words)
```

- Pay attention to **batching**, **bucketization** and **padding**

# Levels of Granularity

Char-Level Language Models: Char-RNN - see e.g. [Karpathy, 2015]

# Neural "Lego Blocks"

We can combine models! Example: Show and Tell [Vinyals, 2015]

# Bidirectional RNNs

Problem - for word classification, you may need to incorporate information from both the **left** and **right** contexts of the word.

\begin{align}
\\
\\
\\
\\
\\
\\
\overleftarrow{\mathbf{h}}_t &= f_{\overleftarrow{\theta}}(\mathbf{x}_t, \overleftarrow{\mathbf{h}}_{t-1})\\
\overrightarrow{\mathbf{h}}_t &= f_{\overrightarrow{\theta}}(\mathbf{x}_t, \overrightarrow{\mathbf{h}}_{t+1})\\
\hat{\mathbf{y}}_t & = g(\overleftarrow{\mathbf{h}}_t, \overrightarrow{\mathbf{h}}_t)
\end{align}

- $\overleftarrow{\mathbf{h}}_t$ and $\overrightarrow{\mathbf{h}}_t$ represent (summarize) both the **past** and the **future** around a given sequence element.

```python

```


Table of Contents

1  Setup

2  Information Retrieval

2.0.1  Practical Example

2.0.2  Boolean Search Queries

2.0.3  Examples of Documents That Would Satisfy the Above Document-Term Matrix

2.0.3.1  Query Example (nuclear AND treaty)

2.0.3.2  Query Example (nuclear AND treaty) OR ((NOT treaty) AND (nonproliferation OR Iran))

3  Tokenizing Sentences

3.1  Definition of a Token

3.2  Examples of Tokens/Words Not Being the Identical Concepts

3.3  Definition of a Document

3.4  Why Is Tokenization Hard?

3.4.1  Sentence Boundary Detection

3.4.2  Approaches to Sentence Boundary Detection

3.4.2.1  Rules-Based

3.4.2.2  Supervised/Unsupervised Learning

3.4.3  NLTK's PunktSentenceTokenizer

3.5  Stemming

3.6  Lemmatization

3.6.1  Why Stemming?

3.6.2  Differences Between Stemming and Lemmatization

3.7  Scoring Metrics

3.7.1  Precision/Recall

3.7.2  F1 Score

3.7.3  Exercises

3.8  Removing Stopwords

3.8.1  When Should You Avoid Removing Stopwords?

3.9  Exercises

4  Vectorization Techniques

4.1  Count Vectorization

# Setup

You'll need to download/install `nltk`, and then also download several of `nltk`'s modules.

```python
!pip install nltk
```

    Requirement already satisfied: nltk in c:\users\ballwang\anaconda3.3\lib\site-packages (3.4.5)
    Requirement already satisfied: six in c:\users\ballwang\anaconda3.3\lib\site-packages (from nltk) (1.14.0)

```python
import nltk
nltk.download('punkt') # A popular NLTK sentence tokenizer
nltk.download('stopwords') # library of common English stopwords
```

    [nltk_data] Downloading package punkt to
    [nltk_data]     C:\Users\ballwang\AppData\Roaming\nltk_data...
    [nltk_data]   Unzipping tokenizers\punkt.zip.
    [nltk_data] Downloading package stopwords to
    [nltk_data]     C:\Users\ballwang\AppData\Roaming\nltk_data...
    [nltk_data]   Unzipping corpora\stopwords.zip.

    True

# Information Retrieval

> **Information retrieval (IR)** is finding material (usually documents) of an unstructured nature (usually text) that **satisfies an information need** from within large collections (usually stored on computers).

### Practical Example

A mobile app contains a gallery of about **8,000** pieces of artwork that they'd like to sell to users. Each artwork contains artwork name, artist name, and some descriptions + keywords. Produce a search algorithm that will allow users to enter in a string text and return relevant pieces of artwork.

### Boolean Search Queries

Example of a **document-term matrix**, from [Northeastern University, Fall 2006 Information Retrieval](http://www.ccs.neu.edu/home/jaa/CSG339.06F/Lectures/boolean.pdf):

### Examples of Documents That Would Satisfy the Above Document-Term Matrix

* **Document 1**

"The new cookbook was a huge hit with younger millenials."
* **Document 2**

"Iran has significant nuclear production capabilities."
* **Document 3**

"The treaty that ended the American Revolution was signed soon thereafter."

#### Query Example `(nuclear AND treaty)`

`D7` would get returned since it is the only document that contains both `nuclear` and `treaty`.

#### Query Example `(nuclear AND treaty) OR ((NOT treaty) AND (nonproliferation OR Iran))`

* `D7`: satisfies LHS (left-hand side of expression) - contains `nuclear` and `treaty`
* `D5`: satisfied RHS - contains `nonproliferation` and not `treaty`)
* `D2`: satisfies RHS - contains `Iran` and not `treaty`)

# Tokenizing Sentences

## Definition of a Token

> A token is an instance of a sequence of characters in some particular document that are grouped together as a useful semantic unit for processing. ([Stanford NLP](https://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html)):

## Examples of Tokens/Words Not Being the Identical Concepts

* Multi-word entities: `San Francisco`, `New York City`
* Phone numbers / dates: `(800) 123-4564` and `Apr. 30th, 2020`

At first, the task of tokenizing seems simple:
1. Split apart `corpus` into `documents`.
2. Split apart `documents` into `tokens`.

## Definition of a Document

A document is a distinct group of tokens. For example, in the Amazon toy product review dataset, each review can be considered a document. In *Tale of Two Cities*, a document could be either a sentence, or a paragraph, or a chapter - it all depends on the task at hand.

* **Sentiment Analysis**: each document is a review/comment/tweet
* **Authorship Classification** (determining if a piece of text was written by Person A or Person B): the entire text is a document

## Why Is Tokenization Hard?

### Sentence Boundary Detection

Deciding where one sentence ends and another sentence begins is easy for humans, but extremely complex for machines. A prime cause of this is ambiguity of punctuation marks. Particularly in English, periods can mean many things beyond denoting the end of a sentence:
* abbreviations
* decimal points
* domain names and email addresses

Furthermore, other "obvious" sentence boundaries like question marks and exclamation marks are becoming increasingly ambiguous due to slang, emoticons, or just emphasis (`"What are you doing???"` should not be split into 3 separate sentences).

### Approaches to Sentence Boundary Detection

#### Rules-Based

* Using regex: `((?<=[a-z0-9][.?!])|(?<=[a-z0-9][.?!]\"))(\s|\r\n)(?=\"?[A-Z])` (don't worry if you can't understand this yet- it includes usage of several advanced techniques such as **lookahead/behind references** which we have not covered).

#### Supervised/Unsupervised Learning

* Maximum entropy models ([UC Berkeley's SATZ Adaptive Sentence Boundary Detector](https://web.archive.org/web/20070922132340/http://elib.cs.berkeley.edu/src/satz/))
* NLTK's `Punkt` sentence tokenizer

```python
# Why not just tokenize myself?
text = "I made two purchases today! I bought a bag of grapes for $4.99, \
but then... realized John Francis already bought some at the Y.M.C.A!"
```

```python
# trying to write our own tokenizer
text.split(".")

```

    ['I made two purchases today! I bought a bag of grapes for $4',
     '99, but then',
     '',
     '',
     ' realized John Francis already bought some at the Y',
     'M',
     'C',
     'A!']

What are some of the issues you notice with the above approach?

### NLTK's `PunktSentenceTokenizer`

> `PunktSentenceTokenizer` is an **sentence boundary detection algorithm** that must be trained to be used. NLTK already includes a pre-trained version of the `PunktSentenceTokenizer` ([StackOverflow](https://stackoverflow.com/questions/35275001/use-of-punktsentencetokenizer-in-nltk))

```python
# Using NLTK sent_tokenize()
nltk.sent_tokenize(text)
```

    ['I made two purchases today!',
     'I bought a bag of grapes for $4.99, but then... realized John Francis already bought some at the Y.M.C.A!']

## Stemming

Stemming is the process of reducing inflection in words to their root forms such as mapping a group of words to the same stem even if the stem itself is not a valid word in the language [Source](https://www.datacamp.com/community/tutorials/stemming-lemmatization-python)

In Python, we can use **`nltk.stem.porter.PorterStemmer`** stem our words:

```python
stemmer = PorterStemmer()
print(stemmer.stem("caressed"))  # caress
print(stemmer.stem("athlete"))  # athlet
print(stemmer.stem("athletics"))  # athlet
print(stemmer.stem("media"))  # media
print(stemmer.stem("photography"))  # photographi
print(stemmer.stem("sexy"))  # sexi
print(stemmer.stem("journalling"))  # journal
print(stemmer.stem("Slovakia")) # slovakia
print(stemmer.stem("corpora")) # corpora
print(stemmer.stem("thieves")) # thiev
print(stemmer.stem("rocks")) # rock
```

```python
stemmer = PorterStemmer()
print(stemmer.stem("caressed"))  # caress
print(stemmer.stem("athlete"))  # athlet
print(stemmer.stem("athletics"))  # athlet
print(stemmer.stem("media"))  # media
print(stemmer.stem("photography"))  # photographi
print(stemmer.stem("sexy"))  # sexi
print(stemmer.stem("journalling"))  # journal
print(stemmer.stem("Slovakia")) # slovakia
print(stemmer.stem("corpora")) # corpora
print(stemmer.stem("thieves")) # thiev
print(stemmer.stem("rocks")) # rock
```

## Lemmatization

Lemmatization is closely related to stemming. However, while stemming looks only at the individual word itself, and considers the usage of the word (ie. part of speech, is this word a noun, a verb, etc.). For example, if we compared the [outputs of stemming and lemmatization certain ambiguous French words](https://blog.bitext.com/lemmatization-vs-stemming):

```python
from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
print(lemmatizer.lemmatize("caressed")) #caressed
print(lemmatizer.lemmatize("athlete")) #athlete
print(lemmatizer.lemmatize("athletics")) #athletics
print(lemmatizer.lemmatize("media"))
print(lemmatizer.lemmatize("photography")) #photography
print(lemmatizer.lemmatize("sexy")) #sexy
print(lemmatizer.lemmatize("journalling")) #journalling
print(lemmatizer.lemmatize("Slovakia")) #Slovakia
print(lemmatizer.lemmatize("corpora")) # corpus
print(lemmatizer.lemmatize("thieves")) # thief
print(lemmatizer.lemmatize("rocks")) #rock
```

### Why Stemming?
- smaller and faster
- simplicity in "good enough"
- can often **provide higher recall (coverage)** if you are using it for text searching: `drives` and `drivers` will likely shorten to `driv`, which may be useful if your search engine wants to make sure to get all relevant documents, even at the cost of surfacing a few irrelevant documents
- could potentially be more useful for predictive models that tend to overfit

### Differences Between Stemming and Lemmatization
> **Stemming** usually refers to a crude heuristic process that chops off the ends of words in the hope of achieving this goal correctly most of the time, and often includes the removal of derivational affixes. **Lemmatization** usually refers to doing things properly with the use of a vocabulary and morphological analysis of words, normally aiming to remove inflectional endings only and to return the base or dictionary form of a word, which is known as the lemma . If confronted with the token `saw`, stemming might return just `s`, whereas lemmatization would attempt to return either `see` or `saw` depending on whether the use of the token was as a verb or a noun. ([Stanford NLP Group](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html))

## Scoring Metrics

### Precision/Recall

**Recall:** What percent of the positive classes did the model successfully predict?
**Precision:** When a model predicted a positive class, what percentage of the time was it correct?

In terms of NLP / stemming / lemmatization:

**Recall**: After processing (tokenizing, stemming/lemmatizing) the data, what percent of the relevant search results were surfaced? Ie. - when a user searches for `blue jeans`, did all the results returned include all the relevant items (blue-ish colored denim pants)?

**Precision**: After processing (tokenizing, stemming/lemmatizing) the data, what percent of the results returned were relevant?

### F1 Score

The F1 score of a model represents the harmonic mean between precision and recall, and is defined as

$$
\begin{equation}
F_{1} = 2 * \frac{P * R}{P + R}
\end{equation}
$$

An F1 score is often a good measure
* our dataset target is class imbalanced (ie. 96% positive, 4% negative)
* when we want to balance optimizing for both precision and recall

### Exercises

Calculate:

1. Overall **accuracy**: $\frac{?}{?}$

```python
accuracy = (10+20)/(10+4+2+20)
accuracy
```

    0.8333333333333334

2. **Precision:** $\frac{?}{?}$

```python
precision = 10/12
precision
```

    0.8333333333333334

3. **Recall:** $\frac{?}{?}$

```python
recall = 10/(10+4)
recall
```

    0.7142857142857143

4. **F1 Score**

```python
#good F1 score: 0-1 depend on use case, whether it's 80% or 90%
#if customer churn, this is good score
#if it's for internet security, this is not a good score
#auc > 0.7, it's workable
f1 = 2 * precision*recall / (precision + recall)
f1
```

    0.7692307692307692

## Removing Stopwords

It's your call ultimately if you want to remove stopwords. There are advantages and disadvantages to both approaches. You will first need to run `nltk.download("stopwords")` to download the set of stopwords for NLTK:

```python
# see the set of words NLTK considers stopwords
from nltk.corpus import stopwords
print(set(stopwords.words("english")))
```

    {"you've", "isn't", 'that', "you're", 'ain', 'any', 'an', 'will', "it's", 'own', 'ours', "wasn't", 'this', "hadn't", "aren't", 'won', 'who', 'doing', "she's", 'few', "doesn't", 'further', 'her', "hasn't", 'our', 'you', 'll', 'until', 've', 'other', "you'd", 'his', 'your', 'into', 'yourselves', "shouldn't", 'doesn', "you'll", 'is', 'there', 'he', 'am', 'most', 's', 'should', 'being', 'them', 'does', 'up', 'some', 'against', 'was', 'we', 'having', 're', 'those', 'about', 'haven', 'down', "don't", 'weren', 'shouldn', 'if', 'above', 'me', "mustn't", 'did', 'had', 'hers', 'between', 'again', "didn't", 'under', 'before', 'all', 'now', 'as', 'she', 'were', 'aren', "weren't", "that'll", 'from', 'same', 'through', 'don', 'here', 'nor', 't', 'd', 'off', 'while', "couldn't", 'o', 'their', 'its', 'the', 'than', "mightn't", 'been', 'shan', 'can', 'such', "needn't", 'i', 'at', 'yours', 'how', 'to', 'or', 'just', 'both', 'm', 'wouldn', 'couldn', 'after', 'himself', 'needn', 'over', 'out', "should've", 'no', 'why', 'but', 'where', 'below', 'by', 'has', 'because', 'didn', 'and', 'him', 'so', 'very', 'ourselves', 'more', 'not', 'during', 'yourself', 'are', 'have', 'mustn', 'wasn', 'itself', 'do', 'for', 'a', 'in', 'on', "shan't", 'only', 'isn', "wouldn't", 'myself', 'theirs', 'themselves', 'too', 'y', "won't", 'hadn', 'when', 'they', 'my', 'what', 'of', 'with', 'once', 'these', 'each', 'ma', 'mightn', 'hasn', "haven't", 'then', 'be', 'which', 'whom', 'it', 'herself'}

```python
# iterate through the Pandas dataframe, and drop the columns that reflect stopwords

```

### When Should You Avoid Removing Stopwords?

The below example is taken from a Medium article by Gagandeep Singh, ["Why you should avoid removing STOPWORDS"](https://towardsdatascience.com/why-you-should-avoid-removing-stopwords-aa7a353d2a52).

```python
reviews = [
    ("The product is really very good", "Positive"),
    ("The products seems to be good", "Positive"),
    ("Good product. I really liked it", "Positive"),
    ("I didn’t like the product", "Negative"),
    ("The product is not good.", "Negative")
]
```

```python
cleaned_reviews = []
nltk_stopwords = set(stopwords.words('english'))

# This iterates through each of the reviews, splitting the review into distinct tokens
# Then it checks each token for whether or not it is a stopword, before adding them back into a "cleaned_review"
for review in reviews:
    words = review[0].split()
    new_words = []
    for word in words:
        if word in nltk_stopwords:
            continue
        new_words.append(word)
    cleaned_review = " ".join(new_words)
    cleaned_reviews.append((cleaned_review, review[1]))
```

```python
cleaned_reviews
```

    [('The product really good', 'Positive'),
     ('The products seems good', 'Positive'),
     ('Good product. I really liked', 'Positive'),
     ('I didn’t like product', 'Negative'),
     ('The product good.', 'Negative')]

Singh:
> If you are working with **basic NLP techniques like BOW, Count Vectorizer or TF-IDF(Term Frequency and Inverse Document Frequency)** then removing stopwords is a good idea because stopwords act like noise for these methods. If you working with **LSTMs or other models which capture the semantic meaning and the meaning of a word depends on the context of the previous text**, then it becomes important not to remove stopwords.

## Exercises

##### 1. For each of the following statements, label them True or False. If False, briefly explain why:

A. *If the **F1 score** of a model is **1.0 (100%)**, then the accuracy of your model must also be **100%**.*

B. *Stemming **increases the size of the vocabulary** (the vocabulary is the set of all tokens found inside the corpus)*

C. *Texts processed using lemmatization will typically have higher recall than stemming.*

##### 2. Calculate precison and recall given the following results from a confusion matrix:

##### 3. Provide an example of how stemming can improve recall.

##### 4. Provide an example of when stemming might reduce precision.

# Vectorization Techniques

## Count Vectorization

You can use **`sklearn.feature_extraction.text.CountVectorizer`** to easily convert your corpus into a bag of words matrix:

```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
data_corpus = ["John likes to watch movies. Mary likes movies too.",
"John also likes to watch football games. Mary does not like football much."]
X = vectorizer.fit_transform(data_corpus)
```
Note that the output `X` here is not your traditional Numpy matrix! Calling **`type(X)`** here will yield **`<class 'scipy.sparse.csr.csr_matrix'>`**, which is a **CSR ([compressed sparse row format matrix](https://docs.scipy.org/doc/scipy-0.14.0/reference/generated/scipy.sparse.csr_matrix.html))**. To convert it into an actual matrix, call the `toarray()` method:

```python
X.toarray()
```
Your output will be

```
array([[0, 0, 0, 0, 1, 0, 2, 1, 2, 0, 0, 1, 1, 1],
       [1, 1, 2, 1, 1, 1, 1, 1, 0, 1, 1, 1, 0, 1]], dtype=int64)
```
Notice that using **`X.shape`** $\rightarrow$ `(2,14)`, indicating a total vocabulary size $V$ of 14. To get what word each of the 14 columns corresponds to, use **`vectorizer.get_feature_names()`**:
```
['also', 'does', 'football', 'games', 'john', 'like', 'likes', 'mary', 'movies', 'much', 'not', 'to', 'too', 'watch']
```

Notice, however, that as the vocabulary size $V$ increases, the percent of the matrix taken up by zero values increases:

```python
corpus = [
    "Some analysts think demand could drop this year because a large number of homeowners take on remodeling projectsafter buying a new property. With fewer homes selling, home values easing, and mortgage rates rising, they predict home renovations could fall to their lowest levels in three years.",

          "Most home improvement stocks are expected to report fourth-quarter earnings next month.",

         "The conversation boils down to how much leverage management can get out of its wide-ranging efforts to re-energize operations, branding, digital capabilities, and the menu–and, for investors, how much to pay for that.",

    "RMD’s software acquisitions, efficiency, and mix overcame pricing and its gross margin improved by 90 bps Y/Y while its operating margin (including amortization) improved by 80 bps Y/Y. Since RMD expects the slower international flow generator growth to continue for the next few quarters, we have lowered our organic growth estimates to the mid-single digits."
]

X = vectorizer.fit_transform(corpus).toarray()
```

```python
from sklearn.feature_extraction.text import CountVectorizer
# list of text documents
text = ["It's still early, so box-office disappointments are still among the highest-grossing movies of the year.",
        "That movie was terrific", "You love cats",
        "Pay for top executives at big US companies is vastly higher than what everyday workers make, and a new report from The Wall Street Journal has found that CEOs have hit an eye-popping milestone in the size of their monthly paychecks."]
```

```python
# see the outputted vectors
```

```python
# load vectorized corpus into Pandas dataframe
```


# メモ

実験場、メモ置き場、です

```
# 実験 ファイルを一度全部読み込む
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

const string  str = R"(まだあげ初めし前髪の
林檎のもとに見えしとき
前にさしたる花櫛(はなぐし)の
花ある君と思ひけり)";

int main () {
  ofstream ofs("temp.txt");
  string line;
  stringstream ss(str);
  while (getline(ss, line)) {
      ofs << line << endl;
  }

  ifstream ifs("temp.txt");

  string myStream((istreambuf_iterator<char>(ifs)), istreambuf_iterator<char>());
  cout << myStream;

  // stringstream myStream;
  // myStream << ifs.rdbuf();
  // cout << myStream.str() << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    まだあげ初めし前髪の
    林檎のもとに見えしとき
    前にさしたる花櫛(はなぐし)の
    花ある君と思ひけり

```
# 初恋で実験
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

const string  str = R"(まだあげ初めし前髪の
林檎のもとに見えしとき
前にさしたる花櫛(はなぐし)の
花ある君と思ひけり)";

int main () {
  ofstream ofs("temp.txt");
  string line;
  stringstream ss(str);
  while (getline(ss, line)) {
      ofs << line << endl;
  }

  ifstream ifs("temp.txt");
  while (getline (ifs, line)) {
    cout << line << endl;
  }
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

        まだあげ初めし前髪の
    林檎のもとに見えしとき
    前にさしたる花櫛(はなぐし)の
    花ある君と思ひけり

```
# How to convert vector to set?
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

int main () {
    vector<int> vec = {1, 2, 3, 1, 2};
    set<int> s(vec.begin(), vec.end());
    for (auto item : s) {
        cout << item << endl;
    }
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    1
    2
    3

```
# words を作る。words(str) => vector<str>
# string を単語に分解する
# stringstream を使って直接 words にする
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

const string str = R"(  Now I need a drink.
this is a pen. pen pen)";

vector<string> words(const string &str) {
    string str02;
    str02 = regex_replace(str, regex("[ .,\n]+"), " ");
    string item;
    vector<string> elems = {};
    stringstream ss(str02);
    while (getline(ss, item, ' ')) {
        if (item == "") continue;
        elems.push_back(item);
    }
    return elems;
}

string formatVecStr(vector<string> vec){
    if (vec.empty()) return "";
    string str = "";
    str +="{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) str += ", ";
        str = str + '"' + item + '"';
        isFirst = false;
    };
    str += "}";
    return str;
}

int main() {
    vector<string> wordvec;
    wordvec = words(str);
    cout << formatVecStr(wordvec) << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {"Now", "I", "need", "a", "drink", "this", "is", "a", "pen", "pen", "pen", "これは本です。"}

```
# とりあえず動いて入るが、ダメなので書き直す
# stringstream を使って直接 words にするように書き直す
# 単語に分解する
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

const string str = R"(   Now I need a drink.
this
is a
pen. pen pen)";

void split(const string &s, char delim, vector<string> &elems) {
    stringstream ss(s);
    string item;
    while (getline(ss, item, delim)) {
        elems.push_back(item);
    }
}

vector<string> split(const string &s, char delim) {
    vector<string> elems;
    split(s, delim, elems);
    return elems;
}

string formatVecStr(vector<string> vec){
    string str = "";
    str +="{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) str += ", ";
        str = str + '"' + item + '"';
        isFirst = false;
    };
    str += "}";
    return str;
}

int main() {
  string str02;
  str02 = regex_replace(str, regex("[ .,\n]+"), " ");
  vector<string> words = split(str02, ' ');
  for (auto& item : words){
      cout << item << " ";
  }
  cout << endl;
  cout << formatVecStr(words) << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

     Now I need a drink this is a pen pen pen
    {"", "Now", "I", "need", "a", "drink", "this", "is", "a", "pen", "pen", "pen"}

```
# アイデア lines を作る
# 複数の行からなるテキストで云々。
# vector<str> lines() と言う関数でもとのテキストを行単位の
# 返り値が {"this is a pen.", "これは本です。"}
# 元のテキストが
# const string  str = R"(this is a pen.
# これは本です。)";
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

string formatVecStr(vector<string> vec){
    string str = "";
    str +="{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) str += ", ";
        str = str + '"' + item + '"';
        isFirst = false;
    };
    str += "}";
    return str;
}

vector<string> lines(string str){
    vector<string> vec = {};
    stringstream ss(str);
    string line = "";
    while (getline(ss, line)){
        vec.push_back(line);
    }
    return vec;
}

int main () {
    const string  str = R"(this is a pen.
これは本です。)";
    vector<string> vec = lines(str);
    cout << formatVecStr(vec) << endl;
}
```

```
# heredocument から変数への入力
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;
const string  str = R"(
2 3
tiis is a ペン
)";
int main(){
    stringstream ss(str);
    int N, M;
    ss >> N >> M;
    vector<string> vec(4);
    for (int i = 0; i < 4; i++) {
        ss >> vec.at(i);
    }
    cout << N << " " << M << endl;
    for (int i = 0; i < 4; i++) {
        cout << vec.at(i) << " ";
    }
    cout << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    2 3
    tiis is a ペン

```
# 実験 vector の連結
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

string formatVecStr(vector<string> vec){
    string str = "";
    str +="{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) str += ", ";
        str = str + '"' + item + '"';
        isFirst = false;
    };
    str += "}";
    return str;
}

int main(){
    vector<string> vec = {"this", "is", "a", "pen"};
    vector<string> vec02 = {"that", "is", "not", "a", "pen"};
    vec.insert(vec.end(), vec02.begin(), vec02.end());
    cout << formatVecStr(vec) << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {"this", "is", "a", "pen", "that", "is", "not", "a", "pen"}

```
# c++ formatVecStr を作る
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

string formatVecStr(vector<string> vec){
    string str = "";
    str +="{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) str += ", ";
        str += item;
        isFirst = false;
    };
    str += "}";
    return str;
}

int main(){
    vector<string> vec = {"this", "is", "a", "pen"};
    cout << formatVecStr(vec) << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {this, is, a, pen}

```
# tail recursion 末尾再帰
# fact :: Int -> Int
# fact n = fact' n 1
#     where
#         fact' 0 y = y
#         fact' x y = fact' (x - 1) (x * y)
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

unsigned long  fact02 (unsigned long  n, unsigned long  acc){
    if (n==0) return acc;
    else {
        return fact02((n-1), n*acc);
    }
}

unsigned long  fact (unsigned long  n) {
    return fact02(n, 1);
}

int main () {
    unsigned long  n; cin >> n;
    cout << fact (n) << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; echo 50 |./a.out
```

    15188249005818642432

```
# heredocument
# 隣接する文字列の連結を利用した方法
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;
/*
const char *str = "1"
                  "  2\n"
                  "    3"; */
const string str =
"1"
"  2\n"
"    3";
int main(){
    cout << str << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    1  2
        3

```
# heredocument
# c++ の R"( を利用した方法
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;
const string  str = R"(
まだあげ初めし前髪の
林檎のもとに見えしとき
前にさしたる花櫛(はなぐし)の
花ある君と思ひけり
)";
int main(){
    cout << str << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    まだ生えそめし前髪の
    まだあげ初めし前髪の
    林檎のもとに見えしとき
    前にさしたる花櫛(はなぐし)の
    花ある君と思ひけり

```
# 数字の桁読み
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

string conv2en(long num){
    string result = "";
    if (num == 0) return "0";
    if (num > 999999999999) {
        result += to_string(num / 1000000000000);
        result += "trillion";
        num = (num % 1000000000000);
    }
    if (num > 999999999) {
        result += to_string(num / 1000000000);
        result += "billion";
        num = (num % 1000000000);
    }
    if (num > 999999) {
        result += to_string(num / 1000000);
        result += "million";
        num = (num % 1000000);
    }
    if (num > 999) {
        result += to_string(num / 1000);
        result += "thousand";
        num = (num % 1000);
    }
    if (num > 0) {
        result += to_string(num);
    }
    return result;
}
string conv2ja(long num){
    string result = "";
    if (num == 0) return "0";
    if (num > 9999999999999999) {
        result += to_string(num / 10000000000000000);
        result += "京";
        num = (num % 10000000000000000);
    }
    if (num > 999999999999) {
        result += to_string(num / 1000000000000);
        result += "兆";
        num = (num % 1000000000000);
    }
    if (num > 99999999) {
        result += to_string(num / 100000000);
        result += "億";
        num = (num % 100000000);
    }
    if (num > 9999) {
        result += to_string(num / 10000);
        result += "万";
        num = (num % 10000);
    }
    if (num > 0) {
        result += to_string(num);
    }
    return result;
}

int main() {
    long num;
    cin >> num;
    cout << conv2ja(num) << endl;
    cout << conv2en(num) << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; echo 12345678901234567 | ./a.out
!g++ temp.cpp; echo 0000 | ./a.out
!g++ temp.cpp; echo 10000 | ./a.out
!g++ temp.cpp; echo 100000000 | ./a.out
!g++ temp.cpp; echo 1000000000000 | ./a.out
!g++ temp.cpp; echo 10000000000000000 | ./a.out
!g++ temp.cpp; echo 100000000000000000000 | ./a.out
!g++ temp.cpp; echo 1000000000000000000 | ./a.out
```

    1京2345兆6789億123万4567
    12345trillion678billion901million234thousand567
    0
    0
    1万
    10thousand
    1億
    100million
    1兆
    1trillion
    1京
    10000trillion
    922京3372兆368億5477万5807
    9223372trillion36billion854million775thousand807
    100京
    1000000trillion

```
# how to cout a vector in c++ answer
# printVectorElements は template<typename T> としている
# これで int と string に対応できているが、cout 出来ない型に対応できているわけではない
# あと末尾に余計な `;` が付く
# 私の vecIntToStr の方が良いが名称が違うね => formatVecInt2Str
%%writefile temp.cpp
#include <iostream>
#include <vector>

using std::cout; using std::cin;
using std::endl; using std::string;
using std::vector;

template<typename T>
void printVectorElements(vector<T> *vec)
{
    for (auto i = 0; i < vec->size(); ++i) {
        cout << vec->at(i) << "; ";
    }
    cout << endl;
}

int main() {
    /* vector<string> str_vec = {"bit", "nibble",
                              "byte", "char",
                              "int", "long",
                              "long long", "float",
                              "double", "long double"}; */
    // vector<int> str_vec = {1,2,3}; //=> 1; 2; 3; と出力された
    // vector<vector<int>> str_vec = {vector<int>{1,2},vector<int>{3,4}}; // エラー

    printVectorElements(&str_vec);

    return EXIT_SUCCESS;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

```
# c++ formatVecInt2Str を作る
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

string formatVecInt2Str(vector<int> vec){
    string str = "";
    str +="{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) str += ", ";
        str += to_string(item);
        isFirst = false;
    };
    str += "}";
    return str;
}

int main(){
    vector<int> vec = {1,2,3,4,5,6};
    string str;
    str = formatVecInt2Str(vec);
    cout << str << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {1, 2, 3, 4, 5, 6}

```
# reverseVec を作る c++
# 必要ないみたい ブランクでも通る
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

void vecCout(vector<int> vec){
    cout << "{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) cout << ", ";
        cout << item;
        isFirst = false;
    };
    cout << "}" << endl;
}

int main() {
    vector<int> vec = {3,2,1};
    vecCout(vec);
    cout << endl;
    reverse(vec.begin(),vec.end());
    vecCout(vec);
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {3, 2, 1}

    {1, 2, 3}

```
# bitset を vector<int> にする
# vector を出力する
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

void vecCout(vector<int> vec){
    cout << "{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) cout << ", ";
        cout << item;
        isFirst = false;
    };
    cout << "}" << endl;
}

vector<int> toNumVec(bitset<4> bs) {
    vector<int> vec;
    for (int i = 0; i < bs.size(); i++){
        if(bs.test(i)) vec.push_back(i);
    }
    return vec;
}

int main() {
    bitset<4> bs("1110");
    vector<int> vec;
    vec = toNumVec(bs);
    reverse(vec.begin(),vec.end());
    vecCout(vec);
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {3, 2, 1}

```
# vector を出力する
%%writefile temp.cpp
#include <bits/stdc++.h>
using namespace std;

void vecCout(vector<int> vec){
    cout << "{";
    bool isFirst = true;
    for (auto item : vec){
        if(!isFirst) cout << ", ";
        cout << item;
        isFirst = false;
    };
    cout << "}" << endl;
}

int main() {
    vector<int> vec = {1,2,3};
    vecCout(vec);
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    {1, 2, 3}

```
# 関数で vector を返り値にする
# str2vec で文字列を vector にする
# 文字列は utf-8 が前提
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

int count_byte(char c){
    if ((c & 0x80) == 0) return 1;
    else if ((c & 0xe0) == 0xc0) return 2;
    else if ((c & 0xf0) == 0xe0) return 3;
    else return 4;
}

vector<string> str2vec(string &str){
    vector<string> vec;
    for (int i = 0 ; i < str.size(); i += count_byte(str.at(i))){
        vec.push_back(str.substr(i,count_byte(str.at(i))));
    }
    return vec;
}

int main() {
    string str;
    getline(cin, str);
    vector<string> vec;
    vec = str2vec(str);
    for (auto& item : vec) {
        cout << item;
    };
    cout << endl;
    cout << "文字列の長さは " << vec.size() << " です。" << endl;
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; echo "This is 本" | ./a.out
```

    This is 本
    文字列の長さは 9 です。

```
# コード表示
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;
int main(){
    string str1 = "aあ";
    for (int i=0; i < str1.size(); i++){
      cout << "0x" << hex << (str1[i] & 0xff) << endl; // 0xff は 2桁にするおまじないか?
    }
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    0x61
    0xe3
    0x81
    0x82

```
# qiita c++で、UTF-8のbyte数を判定することでマルチバイトの文字列をカウントする
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;
int count_byte(char c){
    if ((c & 0x80) == 0) return 1;
    else if ((c & 0xe0) == 0xc0) return 2;
    else if ((c & 0xf0) == 0xe0) return 3;
    else return 4;
}
int main(){
    string str4 = "b";
    string str5 = "©";
    string str6 = "亜";
    printf("%sのbyte数は%d\n",str4.c_str(),count_byte(str4[0]));
    printf("%sのbyte数は%d\n",str5.c_str(),count_byte(str5[0]));
    printf("%sのbyte数は%d\n",str6.c_str(),count_byte(str6[0]));
}

```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    bのbyte数は1
    ©のbyte数は2
    亜のbyte数は3

```
# 文字数カウント
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;
int count_byte(char c){
    if ((c & 0x80) == 0) return 1;
    else if ((c & 0xe0) == 0xc0) return 2;
    else if ((c & 0xf0) == 0xe0) return 3;
    else return 4;
}
int count_char(string str){
    int count = 0;
    /*
    char c = *strings;
    while(c != '\0'){
        count += 1;
        strings += count_byte(c);
        c = *strings;
    } *//*
    for (int i = 0 ; i < str.size(); i++){
        count += 1;
        i += count_byte(str.at(i)) -1;
    }*/
    for (int i = 0 ; i < str.size(); i += count_byte(str.at(i))){
        count += 1;
//        cout << i << ":" << count_byte(str.at(i)) << endl;
    }
    return count;
}
int main (){
string strs2 = "著作権マーク（ちょさくけんマーク）またはコピーライトマーク（copyright mark）とは、大文字のCを丸で囲んだ記号（©）であり、音声録音[1]以外の作品の著作権表示に使用される記号である。";
printf("strs2 の文字数は%d\n",count_char(strs2)); // 98 になったら成功らしい
string str = "a亜";
cout << str << "の文字数は" <<  count_char(str) << endl; // 2 が正解
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    strs2 の文字数は98
    a亜の文字数は2

```
# C++: UTF-8 の文字列を1文字ずつ表示する (std::string)
# と言う記事
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

void print_each_char(std::string);

int main()
{
    string str = "wats𠮷野家";
    print_each_char(str);
}

void print_each_char(std::string str)
{
    using namespace std;

    int pos;
    unsigned char lead;
    int char_size;

   for (pos = 0; pos < str.size(); pos += char_size) {

        lead = str[pos];

        if (lead < 0x80) {
            char_size = 1;
        } else if (lead < 0xE0) {
            char_size = 2;
        } else if (lead < 0xF0) {
            char_size = 3;
        } else {
            char_size = 4;
        }

        cout << str.substr(pos, char_size) << '\n';
    }

}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    w
    a
    t
    s
    𠮷
    野
    家

```
# iterator version
#include <iostream>
#include <string>

void print_each_char(std::string);

int main(void)
{
    using namespace std;
    string str = u8"𠮷野家のコピペ";
    print_each_char(str);

    return 0;
}

void print_each_char(std::string str)
{
    using namespace std;

    unsigned char lead;
    int char_size = 0;

    for (string::iterator it = str.begin(); it != str.end(); it += char_size) {

        lead = *it;

        if (lead < 0x80) {
            char_size = 1;
        } else if (lead < 0xE0) {
            char_size = 2;
        } else if (lead < 0xF0) {
            char_size = 3;
        } else {
            char_size = 4;
        }

        cout << str.substr(distance(str.begin(), it), char_size) << '\n';

    }

}
```

```
# tuple version
#include <iostream>
#include <string>

std::tuple<std::string, int> utf8_next_char(std::string str, int str_size, int pos);

int main(void)
{
    using namespace std;

    string str = u8"あいうえお";
    tuple<string, int> t;
    int str_size = str.size();

    int pos = 0;
    string buf;
    int buf_size = 0;

    for (int pos = 0; pos < str_size; pos += buf_size) {
        t = utf8_next_char(str, str_size, pos);
        buf = get<0>(t);
        buf_size = get<1>(t);
        cout << buf << '\n';
    }

    return 0;
}

std::tuple<std::string, int> utf8_next_char(std::string str, int str_size, int pos)
{
    using namespace std;
    unsigned char c;

    string buf;
    int buf_size;

    if (pos >= str_size) {
        return make_tuple(u8"?", pos);
    }

    c = str[pos];

    if (c < 0x80) {
        buf_size = 1;
    } else if (c < 0xE0) {
        buf_size = 2;
    } else if (c < 0xF0){
        buf_size = 3;
    } else {
        buf_size = 4;
    }

    buf = str.substr(pos, buf_size);

    return  make_tuple(buf, buf_size);
}
```

```
# この記事を書いた人は constexpr の練習、と言っているが const にしても動く
# https://qiita.com/Kogia_sima/items/fc5094776ba9761e186f
# 文字列をわざわざ `u""` としなくても動く
# wchar を使っていない
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

const size_t utf8_strlen(const char* str) {
  size_t count = 0;

  while (*str != '\0') {
    uint8_t lead = static_cast<uint8_t>(*(str++));
    ptrdiff_t secondary_chars =
        (lead < 0x80) ? 0 :
        ((lead >> 5) == 0b110) ? 1 :
        ((lead >> 4) == 0b1110) ? 2 :
        ((lead >> 3) == 0b11110) ? 3 :
        throw std::out_of_range("invalid UTF code point was detected.");

    while (secondary_chars) {
      if (-65 < *(str++)) {
        throw std::out_of_range("invalid UTF code point was detected.");
      }

      --secondary_chars;
    }

    ++count;
  }

  return count;
}

int main() {
  // constexpr const char str[] = u8"UTF-8だよ〜ん";
  const char str[] = "UTF-8だよ〜ん";
  const size_t length = utf8_strlen(str);

  printf("%lu\n", length);
}
```

    Overwriting temp.cpp

```
# const を取ったらどうなるか実験  => 動いた
# この記事を書いた人は constexpr の練習、と言っているが const にしても動く
# https://qiita.com/Kogia_sima/items/fc5094776ba9761e186f
# 文字列をわざわざ `u""` としなくても動く
# wchar を使っていない。size_t は使っている
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

size_t utf8_strlen(const char* str) {
  size_t count = 0;

  while (*str != '\0') {
    uint8_t lead = static_cast<uint8_t>(*(str++));
    ptrdiff_t secondary_chars =
        (lead < 0x80) ? 0 :
        ((lead >> 5) == 0b110) ? 1 :
        ((lead >> 4) == 0b1110) ? 2 :
        ((lead >> 3) == 0b11110) ? 3 :
        throw std::out_of_range("invalid UTF code point was detected.");

    while (secondary_chars) {
      if (-65 < *(str++)) {
        throw std::out_of_range("invalid UTF code point was detected.");
      }

      --secondary_chars;
    }

    ++count;
  }

  return count;
}

int main() {
  // constexpr const char str[] = u8"UTF-8だよ〜ん";
  char str[] = "UTF-8だよ〜ん";
  size_t length = utf8_strlen(str);

  printf("%lu\n", length);
}
```

    Overwriting temp.cpp

```
# char* str を string str にしたらどうか #=> エラーは出たがアウトプットは合っている
# const を取ったらどうなるか実験  => 動いた
# この記事を書いた人は constexpr の練習、と言っているが const にしても動く
# https://qiita.com/Kogia_sima/items/fc5094776ba9761e186f
# 文字列をわざわざ `u""` としなくても動く
# wchar を使っていない。size_t は使っている
%%writefile temp.cpp
# include <bits/stdc++.h>
using namespace std;

size_t utf8_strlen(const char* str) {
  size_t count = 0;

  while (*str != '\0') {
    uint8_t lead = static_cast<uint8_t>(*(str++));
    ptrdiff_t secondary_chars =
        (lead < 0x80) ? 0 :
        ((lead >> 5) == 0b110) ? 1 :
        ((lead >> 4) == 0b1110) ? 2 :
        ((lead >> 3) == 0b11110) ? 3 :
        throw std::out_of_range("invalid UTF code point was detected.");

    while (secondary_chars) {
      if (-65 < *(str++)) {
        throw std::out_of_range("invalid UTF code point was detected.");
      }

      --secondary_chars;
    }

    ++count;
  }

  return count;
}

int main() {
  // constexpr const char str[] = u8"UTF-8だよ〜ん";
  // char str[] = "UTF-8だよ〜ん";
  string str = "UTF-8だよ〜ん";
  size_t length = utf8_strlen(str);

  printf("%lu\n", length);
}
```

    Overwriting temp.cpp

```
!g++ temp.cpp; ./a.out
```

    temp.cpp: In function ‘int main()’:
    temp.cpp:34:34: error: cannot convert ‘std::__cxx11::string {aka std::__cxx11::basic_string<char>}’ to ‘const char*’ for argument ‘1’ to ‘size_t utf8_strlen(const char*)’
       size_t length = utf8_strlen(str);
                                      ^
    9

```
%%html
<style>
    tr td:nth-child(2) {
        color: red;
    }
</style>

<table border="1">
<thead>
<tr><th>基本的な型 </th><th> 値の例</th></tr>
</thead>
<tbody>
<tr><td>整数 </td><td> 1</td></tr>
<tr><td>整数(16進数) </td><td> 0x1F</td></tr>
<tr><td>小数 </td><td> 1.0</td></tr>
<tr><td>論理値 </td><td> true</td></tr>
<tr><td><span style="color: red;">アトム</span></td><td> :atom</td></tr>
<tr><td>文字列 </td><td> "elixir"</td></tr>
<tr><td>リスト </td><td> [1, 2, 3]</td></tr>
<tr><td>タプル </td><td> {1, 2, 3}</td></tr>
</tbody>
</table>
```

基本的な型 	 値の例

整数 	 1

整数(16進数) 	 0x1F

小数 	 1.0

論理値 	 true

アトム	 :atom

文字列 	 "elixir"

リスト 	 [1, 2, 3]

タプル 	 {1, 2, 3}

```
import matplotlib.pyplot as plt
fig = plt.figure(figsize=(6,4))
ax = fig.add_subplot()
x = [0,6,7,8,9,11,12,13,14,15,16,17,22]
y = [0,22,26,29.5,33,40.5,44,47.5,51,54.5,59,62,80]
ax.plot(x,y)
plt.show()
```

```
# 三角関数 trigonometry
import math
math.tan(24/180*math.pi)
```

    0.4452286853085361

15.5. A bit of number theory with SymPy

https://ipython-books.github.io/155-a-bit-of-number-theory-with-sympy/

をなぞってみよう。

合同算術を sypy でできるか。 という課題。 number theory に道具立てはあるみたいなのだが、合同 $\equiv$ が表示できないし、よくわからないのでウロウロしているところ

chinese remainder theorem

と言う問題を解いている。

# チェックマーク check mark
$\checkmark$
$✓$

✓

```
from sympy.ntheory.modular import *
import sympy.ntheory
dir(sympy.ntheory)
```

```
from sympy import *
print(isprime (2017))
print(nextprime(2017))
print(nextprime(8))
#=> from sympy.ntheory としないでも nextprime とかが入る。
```

    True
    2027
    11

```
from sympy import *
# prime(n) は n 番目の素数
print(prime(1000))
print(prime(1))
# primepi は n 以下の素数の数
print(primepi(2017))
print(primepi(3)) #=> {2,3} => 2 個
print(primepi(10)) #=> 2,3,5,7 => 4個

```

    7919
    2
    306
    2
    4

```
import numpy as np
import matplotlib.pyplot as plt
x = np.arange(2, 10000)
fig, ax = plt.subplots(1, 1, figsize=(6, 4))
ax.plot(x, list(map(primepi, x)), '-k',
        label='$\pi(x)$')
ax.plot(x, x / np.log(x), '--k',
        label='$x/\log(x)$')
ax.legend(loc=2)
```

```
print(factorint(1998))
```

    {2: 1, 3: 3, 37: 1}

8.  Finally, a small problem. A lazy mathematician is counting his marbles. When they are arranged in three rows, the last column contains one marble. When they form four rows, there are two marbles in the last column, and there are three with five rows. How many marbles are there? (Hint: The lazy mathematician has fewer than 100 marbles.)

```
from sympy.ntheory.modular import solve_congruence
solve_congruence((1, 3), (2, 4), (3, 5))
```

    (58, 60)

(58,60) の意味は congruence を満足させるのは 58 を始めとして、それに 60 を足した数はすべて。

```
# 課題
# y= ax*b になるような散布図を描く
# そのために
# 1. y=ax+b のグラフを matplotlibで描く => ベース
# randomの使い方を考える

```

```
import matplotlib.pyplot as plt
import numpy as np
fig, ax = plt.subplots()

data = {'a': np.arange(50),
        'c': np.random.randint(0, 50, 50),
        'd': np.random.randn(50)}
data['b'] = data['a'] + 10 * np.random.randn(50)
data['d'] = np.abs(data['d']) * 100

# ax.scatter('a', 'b', c='c', s='d', data=data)
ax.scatter(data['a'], data['b'], c=data['c'], s=data['d'])
# ax.set_xlabel('entry a')
# ax.set_ylabel('entry b')
# ax.axis('scaled')
# ax.set_axis_off()
plt.show()
```

```
import matplotlib.pyplot as plt
import numpy as np
# fig, ax = plt.subplots()

data = {'a': np.arange(3),
        'c': np.random.randint(0, 50, 3),
        'd': np.random.randn(3)}
data['b'] = data['a'] + 10 * np.random.randn(3)
data['d'] = np.abs(data['d']) * 100

print(data['a'])
# ax.scatter('a', 'b', c='c', s='d', data=data)
# ax.set_xlabel('entry a')
# ax.set_ylabel('entry b')
# plt.show()
```

    [0 1 2]

```
import matplotlib.pyplot as plt
import numpy as np
# fig, ax = plt.subplots()

print (10*np.random.randn(3))

# x = [1,2,3]
# y = x + np.random.randn(3)

# ax.scatter(x,y)
# plt.show()
```

    [-4.72102573  8.84976573  2.39627047]

```
from sympy.abc import *
from sympy import *
expr = 5 * sqrt(2) - 7
expr = expr**Rational(1,3)
display(expr)
solve((sqrt(2)*x +y)**3 - (expr)**3)
```

$\displaystyle \sqrt[3]{-7 + 5 \sqrt{2}}$

    [{x: nan}]

# このノートの読み方
1. colab で開く
1. 上の方のメニューでドライブにコピーを押す
1. セルを実行しながら読む

# 媒介変数

媒介変数 matplotlib グラフ

媒介変数 Parameter

http://www.yamamo10.jp/yamamoto/comp/Python/library/Matplotlib/implicit/index.php

陰関数 implicit function

---
集合と位相をなぜ学ぶのか

数学の基礎として根づくまでの歴史

藤田博司

なかなかいい題名だが云々。

すこし読んでみよう。

フーリエ級数から始まっている

熱伝導方程式

偏微分方程式

$$
\frac{\partial u}{\partial t} = \frac{\partial^2 u}{\partial^2 x} \tag {1.1}
$$

これはいいが、そのあとのフーリエ級数の説明が読めない。

あ、わからない、と言うより、読めない、はいいね!!!!

それにしよう。

```
# sympy フーリエ で検索
from sympy import *
import mpmath
# fourier 5回級数展開
l = [-mpmath.pi, mpmath.pi]
f0 = lambda x:sign(sin(x))
cs = mpmath.fourier(f0, l, 5)
f1 = lambda x: mpmath.fourierval(cs, l, x)
mpmath.plot([f0, f1], xlim=l)
```

# 数式番号を振る
markdown 数式番号

[Jupyter Notebook]Markdown の数式に数式番号を振って、参照する

メモ 記事には `\label{a}` が使えると書いてあるが、実際には使えない。

$$y'+ f(x)y = g(x) \tag {1.5}$$

$$y'+ f(x)y = g(x) \tag{1.2}$$

$\ref{a}$ はラベルです。

$$
\frac{\partial u}{\partial t} = \alpha \nabla ^2 u \tag{1}
$$

$(\ref{a})$は熱拡散方程式です。

$
\begin{equation}
\frac{\partial u}{\partial t} = \alpha \nabla ^2 u \tag{2}
\end{equation}
$

$(\ref{b})$も熱拡散方程式です。

$$ x \tag {8.5}$$

$(\ref{c})$も熱拡散方程式です。

$$
\frac{\partial u}{\partial t} = \alpha \nabla ^2 u tag{1}
$$
$(\ref{d})$は熱拡散方程式です。

# 大波括弧による連立方程式 cases, align, array

$$\begin{cases}
x = u + 1 \\
y = v + 2
\end{cases}$$

$$
\left \{
\begin{align}
  x &=& y \\
  y &=& this is a  pen
\end{align}
\right .
$$

$
\left \{
\begin{array}{rrrr}
    3 x + & y+& 2z=& 9 \\
    x - & 2 y + & + z =& 8\\
    -2x-& y+& z=& -3
\end{array}
\right . \\
$

```
# 3元連立1次方程式 p.89
%%latex
\displaystyle
\left \{
\begin{array}{rrrr}
    3 x + & y+& 2z=& 9 \\
    x - & 2 y + & + z =& 8\\
    -2x-& y+& z=& -3
\end{array}
\right . \\
を解く。
```

\displaystyle
\left \{
\begin{array}{rrrr}
    3 x + & y+& 2z=& 9 \\
    x - & 2 y + & + z =& 8\\
    -2x-& y+& z=& -3
\end{array}
\right . \\
を解く。

# スペース実験
$A\;{}\;{}\;{}\;{}\;{}\;{}\;{}\;{}\;{}\;{}B$
$A\ {}\ {}\ {}\ {}\ {}\ {}\ {}\ {}\ {}\ {}B$

$A\;{}\;{}\;{}\;{}\;{}\;{}\;{}\;{}\;{}\;{}B$
$A\:{}\:{}\:{}\:{}\:{}\:{}\:{}\:{}\:{}\:{}B$

$A\quad{}\quad{}B$
$A\qquad B$
$A\quad\quad B$

# イプシロンデルタ論法

latex で論理式、論理記号文を書いていて、テキストによって括弧 (丸括弧 parenthesis、角括弧 bracket、波括弧 braces) の使い方に揺れがあったりして、あとカンマの使い方、どういう場合に省略するのか。 と考えていてイプシロンデルタ論法の数式で比べよう、と思いつきました。

日本の離散数学などのテキストでは次のような表示になっている。

$\forall\varepsilon \gt 0 \;\;\exists\delta \gt 0 \;\; \forall x \in \mathbf R \;\;(0 \lt |x - a| \lt \delta\implies |f(x) - f(a)| \lt \varepsilon)$

日本語のウィキペディアでは次のような表示になっていた。

${}^{\forall} \varepsilon \gt 0,\;^{\exists} \delta \gt 0 ;\;^{\forall} x \in \mathbf R\;[0 \lt |x - a| \lt \delta\implies |f(x) - f(a)| \lt \varepsilon]$

英語では epsilon-delta definition of limit と言うそうです。

英語のウィキペディアでは次のような表示になっていた。

$\forall\varepsilon \gt 0,\;\exists\delta \gt 0 ;\;\forall x \in
\mathbf R,\;0 \lt |x - a| \lt \delta\implies |f(x) - f(a)| \lt \varepsilon$

たぶんどれも正しいのだとして、自分の好みとわかりやすさでは日本のテキストの表示がよいと思う。 カンマやセミコロンで明確に区別できるのかもしれないが、カンマやセミコロンは論理記号式以外でも使われるので、いずれにしても文脈で確認しなくてはいけないので、それならばスペースがよい。 命題を角括弧 bracket で囲むのは話としてはわかるが通常の丸括弧 parenthesis で十分だと思う。 その証拠に英語のウィキペディアでは囲ってもいない。 囲まないルールはわかりにくくなると思うのでなんらかの方法で囲った方がよいと思う。

あと、全称記号 $\forall$ と 存在記号 $\exists$ を小さく前上付きにするのはみんなが同じルールならばわかりやすいとは思うが揺れがあるのだったら普通の大きさで変数の前につけて問題ないと思う。

latex でのソースは以下である。

日本ののテキスト
```
$\forall\varepsilon \gt 0 \;\;\exists\delta \gt 0 \;\; \forall x \in \mathbf R \;\;(0 \lt |x - a| \lt \delta\implies |f(x) - f(a)| \lt \varepsilon)$
```
日本語のウィキペディア
```
${}^{\forall} \varepsilon \gt 0,\;^{\exists} \delta \gt 0 ;\;^{\forall} x \in \mathbf R\;[0 \lt |x - a| \lt \delta\implies |f(x) - f(a)| \lt \varepsilon]$
```
英語のウィキペディア
```
$\forall\varepsilon \gt 0,\;\exists\delta \gt 0 ;\;\forall x \in
\mathbf R,\;0 \lt |x - a| \lt \delta\implies |f(x) - f(a)| \lt \varepsilon$
```

$\forall \varepsilon > 0,\  \exists \delta > 0 ;\; \forall x \in \mathbf R \ (0 < |\:x - a\:| < \delta \: \Rightarrow \: |\:f(x) - f(a)\:| < \epsilon)$

```
import matplotlib.pyplot as plt
from matplotlib_venn import venn3
A = set([9,3,6])
B = set([2,4,6,8])
C = set([0,5,1,7])
v = venn3([A,B,C], ('P', 'Q', 'U'))

v.get_label_by_id('100').set_text('\n'.join(map(str,A-B)))
v.get_label_by_id('110').set_text('\n'.join(map(str,A&B)))
v.get_label_by_id('010').set_text('\n'.join(map(str,B-A)))
v.get_label_by_id('001').set_text('\n'.join(map(str,C)))
v.get_patch_by_id('001').set_color('white')
plt.axis('on')
plt.show()
```

```
import matplotlib.pyplot as plt
from matplotlib_venn import venn3
A = set([9,3,6])
B = set([2,4,6,8])
C = set([0,5,1,7])
venn3([A,B,C], ('P', 'Q', 'U'))

# v.get_label_by_id('100').set_text('\n'.join(map(str,A-B)))
# v.get_label_by_id('110').set_text('\n'.join(map(str,A&B)))
# v.get_label_by_id('010').set_text('\n'.join(map(str,B-A)))
# v.get_label_by_id('001').set_text('\n'.join(map(str,C)))
# v.get_patch_by_id('001').set_color('white')
# plt.axis('on')
plt.show()
```

```
# ベン図 venn diagram
from matplotlib_venn import venn3
import matplotlib.pyplot as plt

A = set([1, 2])
B = set([2, 3, 4])
C = set([1,2,3,4,5,6,7,8,9,10,11])

venn3(subsets=(A,B,C))
plt.show()

```

```
# sin 3α を sin α の式に変形する
# 式変形による証明フォーマットは次のようにする
# まず latex で、 次に sympy で書いてみる
%%latex
\displaystyle

\sin 3 α\\
= \{単純変形により\}\\
\quad \sin(2α + α)\\
= \{加法定理~ \sin(α + β) = \sin α \cos β + \cos α \sin β ~により\}\\
\quad \sin 2α \cos α + \cos 2α \sin α \\
= \{公式 \sin 2α = 2 \sin α \cos α ~により\}\\
\quad 2 \sin α \cos^2 α + \cos 2α \sin α \\
= \{公式 \cos 2α = \cos^2 α − \sin^2 α ~により\}\\
\quad 2 \sin α \cos^2 α + (\cos^2 α − \sin^2 α) \sin α \\
= \{\sin α~で括って\}\\
\quad \sin α (2 \cos^2 α+ \cos^2α - \sin^2 α)\\
= \{単純計算\}\\
\quad \sin α (3 \cos^2 α - \sin^2 α)\\
= \{公式 \sin^2 α+ \cos^2 α= 1 \Longrightarrow \cos^2 α = 1 - \sin^2 α~ により\}\\
\quad \sin α (3 - 3 \sin^2 α - \sin^2 α)\\
= \{単純計算\}\\
\quad \sin α (3 - 4 \sin^2 α)\\
```

\displaystyle

\sin 3 α\\
= \{単純変形により\}\\
\quad \sin(2α + α)\\
= \{加法定理~ \sin(α + β) = \sin α \cos β + \cos α \sin β ~により\}\\
\quad \sin 2α \cos α + \cos 2α \sin α \\
= \{公式 \sin 2α = 2 \sin α \cos α ~により\}\\
\quad 2 \sin α \cos^2 α + \cos 2α \sin α \\
= \{公式 \cos 2α = \cos^2 α − \sin^2 α ~により\}\\
\quad 2 \sin α \cos^2 α + (\cos^2 α − \sin^2 α) \sin α \\
= \{\sin α~で括って\}\\
\quad \sin α (2 \cos^2 α+ \cos^2α - \sin^2 α)\\
= \{単純計算\}\\
\quad \sin α (3 \cos^2 α - \sin^2 α)\\
= \{公式 \sin^2 α+ \cos^2 α= 1 \Longrightarrow \cos^2 α = 1 - \sin^2 α~ により\}\\
\quad \sin α (3 - 3 \sin^2 α - \sin^2 α)\\
= \{単純計算\}\\
\quad \sin α (3 - 4 \sin^2 α)\\

```
# sympy で書いてみる実験
from sympy import *
from IPython.display import Markdown
init_printing()
α, β = symbols('α β')
display (sin (3*α))
display(Markdown("$\sin (3α)$ を $\sin(α)$ だけで表示する。"))
display(Markdown("まず公式"))
display (Eq(sin(α+β),expand(sin(α+β),trig=True)))
display(Markdown("より、$α = 2α,~β = α$ とおくと"))
display(Eq(sin (3*α),(sin(2*α)*cos(α)+cos(2*α)*sin(α))))
display(Markdown("項の順序はとりあえず気にしない。"))
display(Markdown("また同じ公式"))
display (Eq(sin(α+β),expand(sin(α+β),trig=True)))
display(Markdown("で、$β = α$ とおくと"))
display (Eq(sin(2*α),expand(sin(2*α),trig=True)))
display(Markdown("を用いて $\sin(2α)$ を消去すると"))
display(Eq(sin (3*α),(expand(sin(2*α)*cos(α),trig=True)+cos(2*α)*sin(α))))
display(Markdown("$\cos(2α)$について公式"))
display (Eq(cos(2*α),expand(cos(2*α),trig=True)))
display(Markdown("を用いて $\cos(2α)$ を消去すると"))
display(Eq(sin (3*α),(expand(sin(2*α)*cos(α)+cos(2*α)*sin(α),trig=True))))
display(Markdown("次に公式を使って $\cos^2(α)$ を消すために公式を変形する。"))
display(Eq((sin(α))**2+(cos(α))**2,1))
display(Eq((cos(α))**2,1-(sin(α))**2))
display(Markdown("を代入する"))
display(Eq(sin (3*α),4*sin(α)*(1-(sin(α))**2) - sin(α)))
display(Eq(sin (3*α),expand(4*sin(α)*(1-(sin(α))**2) - sin(α))))

```

```
from IPython.display import Math, display
import sympy as sy
x = sy.symbols('x')
expression = sy.sin(x)
display(Math('f(x) =' + sy.latex(expression)))
display(Math('\\left[ \\frac{3 \\mu_2^2+\\sigma_2^2\\log(2)}{2\\mu_2} \\right]'))
```

$$f(x) =\sin{\left (x \right )}$$

$$\left[ \frac{3 \mu_2^2+\sigma_2^2\log(2)}{2\mu_2} \right]$$

```
from sympy import *
init_printing()

x, f = symbols('x, f')
mu = symbol
display(sin(x))
display(Eq(f(x), sin(x)))
mu = symbols(r'\mu')
display(3*mu)
# expression = sy.sin(x)
# display(Math('f(x) =' + sy.latex(expression)))
# display(Math('\\left[ \\frac{3 \\mu_2^2+\\sigma_2^2\\log(2)}{2\\mu_2} \\right]'))
```

```latex
%%latex
\left[ \frac{3 \mu_2^2+ \sigma_2^2 \log(2)}{ 2\mu_2} \right]
```

\left[ \frac{3 \mu_2^2+ \sigma_2^2 \log(2)}{ \mu_2} \right]

$$
\left[ \frac{3 \mu_2^2+ \sigma_2^2 \log(2)}{ 2 \mu_2} \right]
$$

# sympy の latex output display とか

```
from sympy import *
init_printing()
a, b = symbols('a, b')
a
```

```
from sympy import *
display(a**3)
```

```
from sympy import *
init_printing()
a, b = symbols('a_{i-1} n_3')
display (a)
display (b)
```

```
xx = ["even" if i % 2 == 0 else "odd" for i in range(10)]

print(xx)
```

    ['even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd', 'even', 'odd']

マークダウンでドルマークを表示する方法

`$` => $5 ドル

が確実か。

`$` => $10 ドル

もとりあえず colab では使える。こちらは覚えやすい。

£4ポンド

は `£` としたら出た。

€20ユーロ

¥100円

https://dev.w3.org/html5/html-author/charref

character entity reference chart
というページがあった。

$ T $$

これは $ マークで囲むとインライン、$$ で囲むと段落を変えて、センタリングして表示する。

$$
x = 3
$$

とか。

これは \\$ マークで囲むとインライン、\\$\\$ で囲むと段落を変えて、センタリングして表示する。

これは $ マークで囲むとインライン、$$ で囲むと段落を変えて、センタリングして表示する。

#  Decision Tree
https://medium.com/@rnbrown/creating-and-visualizing-decision-trees-with-python-f8e8fa394176

```

```

```
import sklearn.datasets as datasets
import pandas as pd
iris=datasets.load_iris()
df=pd.DataFrame(iris.data, columns=iris.feature_names)
y=iris.target
```

```
from sklearn.tree import DecisionTreeClassifier
dtree=DecisionTreeClassifier()
dtree.fit(df,y)
```

    DecisionTreeClassifier(ccp_alpha=0.0, class_weight=None, criterion='gini',
                           max_depth=None, max_features=None, max_leaf_nodes=None,
                           min_impurity_decrease=0.0, min_impurity_split=None,
                           min_samples_leaf=1, min_samples_split=2,
                           min_weight_fraction_leaf=0.0, presort='deprecated',
                           random_state=None, splitter='best')

```
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
dot_data = StringIO()
export_graphviz(dtree, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())
```

# meshgrid を理解する。

```
import numpy as np
x=np.linspace(-1,1,100)
y=np.linspace(-1,1,100)
x, y = np.meshgrid(x, y)
# print(x)
z = np.exp(-4*(x**2+y**2))
# print(z)

import matplotlib.pyplot as plt
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x,y,z, cmap=cm.jet)
plt.show()
```

```
import numpy as np
Z = np.zeros((5,5), [('x',float),('y',float)])
Z['x'], Z['y'] = np.meshgrid(np.linspace(0,1,5),
                             np.linspace(0,1,5))
print(Z)
```

    [[(0.  , 0.  ) (0.25, 0.  ) (0.5 , 0.  ) (0.75, 0.  ) (1.  , 0.  )]
     [(0.  , 0.25) (0.25, 0.25) (0.5 , 0.25) (0.75, 0.25) (1.  , 0.25)]
     [(0.  , 0.5 ) (0.25, 0.5 ) (0.5 , 0.5 ) (0.75, 0.5 ) (1.  , 0.5 )]
     [(0.  , 0.75) (0.25, 0.75) (0.5 , 0.75) (0.75, 0.75) (1.  , 0.75)]
     [(0.  , 1.  ) (0.25, 1.  ) (0.5 , 1.  ) (0.75, 1.  ) (1.  , 1.  )]]

```
# import for colormaps
import matplotlib.pyplot as plt
from matplotlib import cm

x=np.linspace(-10,10, num=50)
y=np.linspace(-10,10, num=50)

x, y = np.meshgrid(x, y)

z = np.exp(-0.1*x**2-0.1*y**2)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x,y,z, cmap=cm.jet)
plt.show()

```

```
# 実験

# import for colormaps
from matplotlib import cm

# x=np.linspace(-10,10, num=100)
# y=np.linspace(-10,10, num=100)
x=np.linspace(-1,1, 100)
y=np.linspace(-1,1, 100)
x, y = np.meshgrid(x, y)

# D = np.sqrt(X*X+Y*Y)
# z = np.exp(-0.1*x**2-0.1*y**2)
# sigma, mu = 1.0, 0.0
# G = np.exp(-( (D-mu)**2 / ( 2.0 * sigma**2 ) ) )
z = np.exp(-(x**2+y**2)**2/2.0)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot_surface(x,y,z, cmap=cm.jet)
plt.show()
```

```
# 実験
import matplotlib.pyplot as plt
import numpy as np
import math

def f(x):
    return (math.exp(-x**2/2)) / math.sqrt(2*math.pi)

n = np.linspace(-5.0, 5.0, 50)

p = []
for i in range(len(n)):
    p.append(f(n[i]))

# グラフに表示
plt.plot(n, p)
plt.show()

```

# html で変数を扱う

```
# css を使ってhtmlで変数を使う。
# span を使えば自由な位置に決まった文字列を入れることができる。
%%html
<style>
    .parson1:before {
        content: "田中";
    }

    p.age1:after {
        content: "36歳";
    }
</style>
<h1>サンプル</h1>
<p class="parson1">は男の子</p>
<p class="parson1 age1">の年齢は</p>
<p>私の彼氏は<span class="parson1"></span>君だよ</p>
```

サンプル

は男の子

の年齢は

私の彼氏は君だよ

```
# javascript を使って html で変数を使う
%%html
<span class="name1″></span>は男です<br />
<span class="name1″></span>の年齢は30歳です

```

は男です

の年齢は30歳です

```
# 実験
%%html
<!-- <button>Click me</button> -->
<div>
<!-- <span class="name1"></span>は男です<br> -->
<span class="name1"></span>の年齢は30歳です
<div>

```

の年齢は30歳です

```
# ここで for loop を使いたくないので、for loop を reduce にする研究を始める
```

# ここで for loop を使いたくないので、for loop を reduce にする研究を始める

```
# 0-9 の合計 45 を for loop を使わずに計算する
%%script node
const add = (a, b) => a + b;
sum = Array.from(Array(10),(a,b) => b).reduce(add)

console.log(sum)
```

    45

```
# 1-0 の合計 55 を for loop を使わずに計算する
%%script node
const add = (a, b) => a + b;
sum = Array.from(Array(10),(a,b) => b+1).reduce(add)

console.log(sum)
```

    55

```
# 説明
# const add = (a, b) => a + b;
# は function を使わずに関数を定義している
%%script node
const add = (a, b) => a + b;
console.log(add(1,2));
function add2 (a, b) {return a + b};
console.log(add2(2,3));

```

    3
    5

```
# 実験 複数行
%%script node
const addmult = (a, b) => {
    sum = a+b;
    product = a*b;
    return sum * product
};
console.log(addmult(2,3)); // 30
```

    30

```
# Array.from は Array を作る
%%script node
console.log(Array.from('foo'));
console.log(Array.from([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11], x => x + x));
console.log(Array.from(Array(10), (a, b) => b));
```

    [ 'f', 'o', 'o' ]
    [
       2,  4,  6,  8, 10,
      12, 14, 16, 18, 20,
      22
    ]
    [
      0, 1, 2, 3, 4,
      5, 6, 7, 8, 9
    ]

```
# colab で Array を console.log で出力すると一行に出力されない
# JSON.stringifyを使えばとりあえず一行になる
%%script node
console.log(Array.from(Array(10), (a, b) => b));
console.log(JSON.stringify(Array.from(Array(10), (a, b) => b)));
console.log(JSON.stringify(Array.from({length:10}, (a, b) => b)));
console.log(Array.from({length: 10}));
```

```
%%script node
//reduce
const  initModel = 0
const update = ( model, msg) =>  model + msg
const sum = msgs => msgs.reduce(update,  initModel)

const items = [0, 1, 2]
console.log(sum(items))  //  3
```

    3

```
const range = (start, stop) => Array.from({ length: (stop - start) + 1}, (_, i) => start + i);
```

# JavaScriptでJSONからテーブルを動的に生成

```
%%html

<style>
    div#paging {
        text-align: center;
        /* スクロールに対してページングボックスの位置を固定 */
        position: fixed;
        /* 位置を指定 */
        bottom: 0;
        right: 45%;
    }

    div#pagingbox {
        background: #FFF;
    }

    th {
        /* ヘッダ背景塗りつぶし */
        background: #eee;
    }

    th,
    td {
        /* 枠線を1本線指定 */
        border: solid 1px;
        width: auto;
    }

    table {
        /* 枠線を1本線指定 */
        border: solid 1px;
        border-collapse: collapse;
        white-space: nowrap;
    }

    footer {
        position: fixed;
        width: 100%;
        background-color: #C0C0C0;
        padding: 15px 0;
        bottom: 0;
        /*下に固定*/
    }
</style>

<!--テーブル生成位置-->
<div id='maintable'></div>

<!--ページングボタン配置-->
<footer>
    <div id="paging">
        <table>
            <tr>
                <tb><button id="prevbtn" type="button">
                        << /button>
                </tb>
                <tb>
                    <span id="currentpage">currentpage</span>
                    /
                    <span id="lastpage">lastpage</span>
                </tb>
                <tb><button id="nextbtn" type="button">></button></tb>
            </tr>
        </table>
    </div>
</footer>

```

                <tb><button id="prevbtn" type="button">
                        << /button>
                </tb>
                <tb>
                    currentpage
                    /
                    lastpage
                </tb>
                <tb><button id="nextbtn" type="button">></button></tb>

```
# 実験 アラビア語 1. temp.ipynb の王貞治
# css.ipynb の w3schools
# headerはどうでもよい
# アラビア語のフォントを大きくする。 フォントを自由に変える。
# jsonを使う
# 専用のサイトにするか => arabic table
# arabic01は duolingo 専用にできるのではないか
%%html

<style>
    div#paging {
        text-align: center;
        /* スクロールに対してページングボックスの位置を固定 */
        position: fixed;
        /* 位置を指定 */
        bottom: 0;
        right: 45%;
    }

    div#pagingbox {
        background: #FFF;
    }

    th {
        /* ヘッダ背景塗りつぶし */
        background: #eee;
    }

    th,
    td {
        /* 枠線を1本線指定 */
        border: solid 1px;
        width: auto;
    }

    table {
        /* 枠線を1本線指定 */
        border: solid 1px;
        border-collapse: collapse;
        white-space: nowrap;
    }
</style>

<!--テーブル生成位置-->
<div id='maintable'></div>

```

# いまここ

https://qiita.com/ttatsf/items/846ff861bd45fca0832c

```
# you may be able to define a subclass of sympy.Symbol that implements subs() properly for summations
# 添字をつけたいので subscription を検索してて関係ないのがひっかかったけど面白いのでとっておく
import sympy
a_seq = [-1, 3, 23, 8]
n, r = sympy.symbols('n, r')
a_n = sympy.Function('a')(n)
terms = 4
short_expr = sympy.Sum(a_n * r**n, (n, 0, terms - 1))
coeffed_short_expr = short_expr.doit().subs(
    (a_n.subs(n, i), a_seq[i]) for i in range(terms)) # 8*r**3 + 23*r**2 + 3*r - 1
func_short_expr = sympy.lambdify(r, coeffed_short_expr, 'numpy')
display(short_expr)
display(coeffed_short_expr)
func_short_expr(1)
```

# いまここ

$$
\require{AMScd}
\begin{CD}
    A @>{f}>> B \\
@V{g}VV    @VVV \\
    C   @>>>  D
\end{CD}
$$

$$
\begin{array}{ccccccccc}
0 & \xrightarrow{i} & A & \xrightarrow{f} & B & \xrightarrow{q} & C & \xrightarrow{d} & 0\\
\downarrow & \searrow & \downarrow & \nearrow & \downarrow & \searrow & \downarrow & \nearrow & \downarrow\\
0 & \xrightarrow{j} & D & \xrightarrow{g} & E & \xrightarrow{r} & F & \xrightarrow{e} & 0
\end{array}
$$

$$
\begin{array}{}
1 & \xrightarrow{i} & A \\
\downarrow & \searrow{i} & \downarrow \\
0 & \xrightarrow{j} & D & \xrightarrow{g} & E
\end{array}
$$

```
\begin{array}{ccccccccc}
0 & \xrightarrow{i} & A & \xrightarrow{f} & B & \xrightarrow{q} & C & \xrightarrow{d} & 0\\

\downarrow & \searrow & \downarrow & \nearrow & \downarrow & \searrow & \downarrow & \nearrow & \downarrow\\

0 & \xrightarrow{j} & D & \xrightarrow{g} & E & \xrightarrow{r} & F & \xrightarrow{e} & 0
\end{array}
```

$$
\begin{array}{ccccccccc}
0 & \xrightarrow{i} & A & \xrightarrow{f} & B & \xrightarrow{q} & C & \xrightarrow{d} & 0 \\
\downarrow & \searrow & \downarrow & \nearrow & \downarrow & \searrow & \downarrow & \nearrow & \downarrow \\
0 & \xrightarrow{j} & D & \xrightarrow{g} & E & \xrightarrow{r} & F & \xrightarrow{e} & 0
\end{array}
$$

**メモ**

github と colab では begin{array} の書き方の厳密さというかルールの適用に差があるが、きっちり書けばどちらも通る。

colab では \require{AMScd} は通るが、github ではエラーになってそれを含む latex は平文になる。

colab では array の中に空行があったり、最終行に \\ がついていると対応せず rendering せず平文になるが、githubでは処理する。

逆に array に {ccc} がついていなかったり、{ccc} の数より多く & が振られていると colab ではそのまま適当に rendering してくれるが
github では rendering しない。

```

```