# Tutorial 17 - LSTM with Keras and TensorFlow

## Setup GPU & TensorFlow

```python
# Choose to the GPU number you want to use,
# otherwise you will get a Python error
# e.g. USE_GPU = 4
USE_GPU = X # YOUR_CHOICE
```

```python
# Import TensorFlow
import tensorflow as tf

# Print the installed TensorFlow version
print(f'TensorFlow version: {tf.__version__}\n')

# Get all GPU devices on this server
gpu_devices = tf.config.list_physical_devices('GPU')

# Print the name and the type of all GPU devices
print('Available GPU Devices:')
for gpu in gpu_devices:
    print(' ', gpu.name, gpu.device_type)

# Set only the GPU specified as USE_GPU to be visible
tf.config.set_visible_devices(gpu_devices[USE_GPU], 'GPU')

# Get all visible GPU  devices on this server
visible_devices = tf.config.get_visible_devices('GPU')

# Print the name and the type of all visible GPU devices
print('\nVisible GPU Devices:')
for gpu in visible_devices:
    print(' ', gpu.name, gpu.device_type)

# Set the visible device(s) to not allocate all available memory at once,
# but rather let the memory grow whenever needed
for gpu in visible_devices:
    tf.config.experimental.set_memory_growth(gpu, True)
```

## LSTM with Keras and TensorFlow

So far, the neural networks that we have examined have always had _forward connections_, meaning that each hidden layer always connects to the next hidden layer, and the final hidden layer always connects to the output layer. This manner to connect layers is the reason that these networks are called “feedforward”.

### LEARNING OBJECTIVES

* understand the anatomy of LSTM networks
* learn the available recurrent layer types in TensorFlow
* learn how to use recurrent layers sequence prediction

__Recurrent neural networks__ are more flexible, as backward connections are also allowed. A recurrent connection links a neuron in a layer to either a previous layer or the neuron itself.  Most recurrent neural network architectures maintain state in the recurrent connections.  Feedforward neural networks don’t maintain any state.  A recurrent neural network’s state acts as a sort of short-term memory for the neural network.  Consequently, a recurrent neural network will not always produce the same output for a given input.

Recurrent neural networks do not force the connections to flow only from one layer to the next, from input layer to output layer.

A recurrent connection occurs when a connection is formed between a neuron and one of the following other types of neurons:

* The neuron itself
* A neuron on the same level
* A neuron on a previous level

Recurrent connections can never target the input neurons or the bias neurons.
The processing of recurrent connections can be challenging. Because the recurrent links create endless loops, the neural network must have some way to know when to stop.  A neural network that entered an endless loop would not be useful.  To prevent endless loops, we can calculate the recurrent connections with the following three approaches:

* Context neurons
* Calculating output over a fixed number of iterations
* Calculating output until neuron output stabilizes

The __context neuron__ is a special neuron type that remembers its input and provides that input as its output the next time that we calculate the network.  For example, if we gave a context neuron 0.5 as input, it would output 0.  Context neurons always output 0 on their first call.  However, if we gave the context neuron a 0.6 as input, the output would be 0.5.  We never weight the input connections to a context neuron, but we can weight the output from a context neuron just like any other connection in a network.

Context neurons allow us to calculate a neural network in a single feedforward pass.  Context neurons usually occur in layers.  A layer of context neurons will always have the same number of context neurons as neurons in its source layer, as demonstrated by Figure 1.CTX.

**Figure 1.CTX: Context Layers**

As you can see from the above layer, two hidden neurons that are labeled hidden 1 and hidden 2 directly connect to the two context neurons.  The dashed lines on these connections indicate that these are not weighted connections. These weightless connections are never dense.  If these connections were dense, hidden 1 would be connected to both hidden 1 and hidden 2.  However, the direct connection simply joins each hidden neuron to its corresponding context neuron.  The two context neurons form dense, weighted connections to the two hidden neurons.  Finally, the two hidden neurons also form dense connections to the neurons in the next layer.  The two context neurons would form two connections to a single neuron in the next layer, four connections to two neurons, six connections to three neurons, and so on.

You can combine context neurons with the input, hidden, and output layers of a neural network in many different ways.

### Understanding LSTM

Long Short Term Neural Network (LSTM) are a type of recurrent unit that is often used with deep neural networks.[[Cite:hochreiter1997long]](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.676.4320)  For TensorFlow, LSTM can be thought of as a layer type that can be combined with other layer types, such as dense.  LSTM makes use two activation function types internally.

The first type of activation function is the sigmoid.  This activation function type is used form gates inside of the unit.  The sigmoid  function is given by the following equation:

$ \mbox{S}(t) = \frac{1}{1 + e^{-t}} $

The second type of transfer function is the hyperbolic tangent (tanh) function.  This function is used to scale the output of the LSTM, similarly to how other activation functions have been used in this course.

Both of these two functions compress their output to a specific range.  For the sigmoid function, this range is 0 to 1.  For the hyperbolic tangent function, this range is -1 to 1.

__LSTM maintains an internal state and produces an output.__

The following diagram shows an LSTM unit over three time slices: the current time slice (t), as well as the previous (t-1) and next (t+1) slice, as demonstrated by Figure 2.LSTM.

**Figure 2.LSTM: LSTM Layers**

The values $\hat{y}$ are the output from the unit, the values ($x$) are the input to the unit and the values $c$ are the context values.  Both the output and context values are always fed to the next time slice.  The context values allow the network to maintain state between calls.  Figure 10.ILSTM shows the internals of a LSTM layer.

**Figure 3.ILSTM: Inside a LSTM Layer**

LSTM is made up of three gates:

* Forget Gate ($f_t$) - Controls if/when the context is forgotten. (MC)
* Input Gate ($i_t$) - Controls if/when a value should be remembered by the context. (M+/MS)
* Output Gate ($o_t$) - Controls if/when the remembered value is allowed to pass from the unit. (RM)
Mathematically, the above diagram can be thought of as the following:
**These are vector values.**
First, calculate the forget gate value.  This gate determines if the short term memory is forgotten.  The value $b$ is a bias, just like the bias neurons we saw before.  Except LSTM has a bias for every gate: $b_t$, $b_i$, and $b_o$.

$ f_t = S(W_f \cdot [\hat{y}_{t-1}, x_t] + b_f) $

$ i_t = S(W_i \cdot [\hat{y}_{t-1},x_t] + b_i) $

$ \tilde{C}_t = \tanh(W_C \cdot [\hat{y}_{t-1},x_t]+b_C) $

$ C_t = f_t \cdot C_{t-1}+i_t \cdot \tilde{C}_t $

$ o_t = S(W_o \cdot [\hat{y}_{t-1},x_t] + b_o ) $

$ \hat{y}_t = o_t \cdot \tanh(C_t) $

### Recurrent layers available in TensorFlow

Built-in RNN layers: a simple example
There are three built-in RNN layers in Keras:

[keras.layers.SimpleRNN](https://www.tensorflow.org/api_docs/python/tf/keras/layers/SimpleRNN), a fully-connected RNN where the output from previous timestep is to be fed to next timestep.

[keras.layers.GRU](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GRU), first proposed in Cho et al., 2014.

[keras.layers.LSTM](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LSTM), first proposed in Hochreiter & Schmidhuber, 1997.

#### Stateful vs. Stateless LSTM

1. **Stateless**: LSTM updates parameters on **batch 1** and then initiates cell states (meaning - memory, usually with zeros) for **batch 2**
2. **Stateful**: it uses batch 1 last output cell sates as initial states for batch 2.

#### When to use which?
----------------

- When  sequences in batches are related to each other (e.g. prices of one commodity), we should better use **stateful** mode
- Else, when one sequence represents a complete sentence, we  should go with **stateless** mode

### Simple LSTM Example in TensorFlow

The following code creates the LSTM network.  This is an example of RNN classification.  The following code trains on a data set (x) with a max sequence size of 6 (columns) and 6 training elements (rows)

```python
from tensorflow.keras.preprocessing import sequence
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding
from tensorflow.keras.layers import LSTM
import numpy as np
```

```python
max_features = 4 # 0,1,2,3 (total of 4)
x = [
    [[0],[1],[1],[0],[0],[0]],
    [[0],[0],[0],[2],[2],[0]],
    [[0],[0],[0],[0],[3],[3]],
    [[0],[2],[2],[0],[0],[0]],
    [[0],[0],[3],[3],[0],[0]],
    [[0],[0],[0],[0],[1],[1]]
]
x = np.array(x,dtype=np.float32)

print (f"Input size is: {x.shape[0]} example sequences and  {x.shape[1]} sequence length")
y = np.array([1,2,3,2,3,1],dtype=np.int32)

# Convert y2 to dummy variables
y2 = np.zeros((y.shape[0], max_features),dtype=np.float32)
y2[np.arange(y.shape[0]), y] = 1.0
print('One-hot encoded representation for the outputs:', y2)
```

```python
print('Build model...')
model = Sequential()
model.add(LSTM(units =128, dropout=0.2, recurrent_dropout=0.2,stateful=False,name ='first_lstm', input_shape=(None,1)))  # inputs: A 3D tensor with shape [batch, timesteps, feature].
model.add(Dense(4, activation='sigmoid'))
model.summary()
```

```python
# try using different optimizers and different optimizer configs
model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

print('Train...')
model.fit(x,y2,epochs=20)
pred = model.predict(x)
predict_classes = np.argmax(pred,axis=1)
print("Predicted classes: {}",predict_classes)
print("Expected classes: {}",predict_classes)
```

```python
first_layer = model.get_layer('first_lstm' )
first_layer.states
```

```python
def runit(model, inp):
    inp = np.array(inp,dtype=np.float32)
    pred = model.predict(inp)
    return np.argmax(pred[0])

print( runit( model, [[[0],[0],[0],[0],[0],[1]]] ))

```

How to cumpute the number of parameters for a LSTM layer?
--------------------------------

1. To decide how to handle the memory each LSTM Cell has <bold>3 Gates</bold>:
    - input (what to let in),
    - forget (what to forget) and
    - output (what to write to the output)
2. LSTM **Cell State** is its **memory**
3. LSTM Hidden State is equivalent to the Cell output:
    - lstm_hidden_state_size (number of neurons = memory cells) = lstm_outputs_size
4. Parameters:
    - weights for the inputs (lstm_inputs_size)
    - weights for the outputs (lstm_outputs_size)
    - bias variable
5.  Result from previous point - for all 3 Gates and for Cell State ( = 4)

    \begin{equation}
          \textbf{PARAMETERS} = \textbf4 \times \textbf{ LSTM outputs size} \times (\textbf{weights LSTM inputs size} + \textbf{weights LSTM outputs size} + 1 \textbf{ bias variable})
    \end{equation}

## Further reading on LSTM/recurrent neural networks.

* [Understanding LSTM Networks](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)

```python

```

Week 17/17 - LSTM with Keras and TensorFlow.ipynb


**Report Submission Information (must be completed before submitting report!)**

* Student 1 Full Name and Number :
* Student 2 Full Name and Number :
* Workshop day : e.g., Wednesday
* Workshop time : e.g., 12pm

#  Workshop 1 – Optimisation [2 weeks]

## Objectives:

* Learn how to formulate optimisation problems in practice.
* Familiarise yourself with practical software tools used for optimisation.
* Solve optimisation problems using Python Scipy and Matlab.
* Connect theoretical knowledge and practical usage by doing it yourself.

> __Common objectives of all workshops:__
> Gain hands-on experience and learn by doing! Understand how theoretical knowledge discussed in lectures relates to practice. Develop motivation for gaining further theoretical and practical knowledge beyond the subject material.

## Overview:
Another name for the field of “Optimisation” is “Mathematical Optimisation.” As the name indicates optimisation is an area of applied mathematics. It is possible to study optimisation entirely from a mathematical perspective. However, engineers are interested in solving real-world problems in a principled way. Many engineering problems can be and are formulated as optimisation problems. In those cases, mathematical optimisation provides a solid theoretical foundation for solving them in a principled way.

In this workshop, you will learn how to formulate and solve optimisation problems in practice. This will give you a chance to connect theoretical knowledge and practical usage by doing it yourself. You will familiarise yourself with practical optimisation tools for Python. These are chosen completely for educational reasons (simplicity, accessibility, cost). While the underlying mathematics is timeless, optimisation software evolves with time, and can be diverse. Fortunately, once you learn one or two, it should be rather easy to learn others now and in the future, because software designers often try to make it user friendly and take into account what people already know.

> In the future, you should consider and learn serious optimisation software for scalability and reliability. They can be complex and/or expensive but they get the job done for serious engineering. Teaching such software takes too much time and is beyond the scope of this subject.

## Workshop Preparation: [before you arrive to the lab]

You can come to the workshops as you are or you can prepare beforehand to learn much more!
We will give you a lot of time to finish the tasks but those are the bare minimums. Just like in the lectures, the topics we cover in the workshops are quite deep and we can only do so much in two hours. There is much more to learn and coming prepared to the workshop is one of the best ways to gain more knowledge! For example, there are a few questions in each workshop which you can answer beforehand.

> __Self-learning__ is one of the most important skills that you should acquire as a student. Today, self-learning is much easier than it used to be thanks to a plethora of online resources.
For this workshop, start by exploring the resource mentioned in the preparation steps below.

### Workshop Preparation Steps:

1. Common step for all workshops: read the Workshop Manual (Jupyter Notebook) beforehand!
2. Review relevant lecture slides on optimisation.
3. Read/check relevant reading material and links from LMS/Resources-Reading
4. Check the embedded links below hints and background.
5. _\[optional\]_ _You can start with workshop tasks and questions_

## Tasks and Questions:

Follow the procedures described below, perform the given tasks and answer the workshop questions __on the Python notebook itself!__ The marks associated with each question are clearly stated. Keep your answers to the point and complete to get full marks! Ensure that your code is clean and appropriately commented.

__The resulting notebook will be your Workshop Report!__

> __The goal is to learn__, NOT blindly follow the procedures in the fastest possible way! __Do not simply copy-paste answers (from Internet, friends, etc.). You can and should use all available resources but only to develop your own understanding. If you copy-paste, you will pay the price in the final exam!__

# Section 1: Convex Functions

Remember the definition of convex and concave functions from lecture slides. Functions are mathematical objects but they are used in engineering in very practical ways, for example, to represent the relationship between two quantities. Let's draw a function!

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt

# define function f(x)
def f(x):
    return 3*(x-1)**2

# define x and y
x = np.linspace(-20, 20, 100) # 100 equally spaced points on interval [-20,20]
y = f(x) # call function f(x) and set y to the function's return value

# Plot the function y=f(x)
plt.plot(x,y)
plt.xlabel('x')
plt.ylabel('y')
plt.title('$y=3(x-1)^2$')
plt.show()
```

    <IPython.core.display.Javascript object>

### Question 1.1. (1 pt)
Plot one concave and one nonconvex functions of your choosing (preferably in 3D). Provide their formulas below.

```python
''' Answer as code here '''
from mpl_toolkits.mplot3d.axes3d import Axes3D
from matplotlib import cm
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
# Plot convex figure
ax = fig.add_subplot(1, 2, 1, projection='3d')
X=np.arange(-10,10,1)
Y=np.arange(-10,10,1)
X, Y = np.meshgrid(X, Y)
Z = (2*X)**2 +(2*Y)**2
surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1,
cmap=cm.jet,linewidth=0.01, antialiased=False)
ax.set_zlim3d(0,1000)

# Plot non-convex figure
ax = fig.add_subplot(1, 2, 2, projection='3d')
X=np.arange(-10,10,1)
Y=np.arange(-10,10,1)
X, Y = np.meshgrid(X, Y)
Z = -(2*X)**2 +(2*Y)**2
surf = ax.plot_surface(X, Y, Z, rstride=1, cstride=1,
cmap=cm.jet,linewidth=0.01, antialiased=False)
ax.set_zlim3d(-1000,0)
fig.colorbar(surf, shrink=0.5, aspect=5)
plt.show()

```

**Answer as text here**

Convex Formula:

$$ z = 2x^2 + 2y^2$$

Non-convex Formula:

$$ z = -2x^2 + 2y^2$$

### Question 1.2. (1 pts)
How would you determine whether a single or multi-variate continuously differentiable function is convex or not?
> Note that the question becomes very tricky if you have a **parametric** multivariate polynomials of degree four or higher!

> *[Optional]* An interesting paper (for those who wish to go deeper) http://web.mit.edu/~a_a_a/Public/Publications/convexity_nphard.pdf

**Answer as text here**

Single continously differential function is convex if and only if:
\begin{equation}
\frac{\partial^2 f}{\partial^2 f} \geq 0 \\
\frac{\partial^2 f}{\partial^2 f} > 0 \Rightarrow strictly \ convex (slope \ increasing)
\end{equation}

Multi-variate continously differential function is convex if and only if eigin values of Hessian matrix is positive:
$$\triangledown^2 f(x) = H = Hessian \ matrix $$

### Question 1.3. (2 pts)
Why are convex optimisation problems considered to be easy to solve? Consider optimality conditions of unconstrained functions in your answer. Plot first and second order derivative functions of the concave and non-convex functions you have chosen above (as part of Question 1.1) to further support your argument.

**Answer as text here**

Convex optimisation problem are easier to solve because of existance of global maximum or minimum.

```python
''' Answer as code here '''
from mpl_toolkits.mplot3d.axes3d import Axes3D
import matplotlib.pyplot as plt
import numpy as np

x = Symbol('x')
y = Symbol('y')
z = (2*x )**2 +(2*y)**2
dzdx = z.diff(x)
dzdy = z.diff(y)
div_z = [dzdx, dzdy]
dzdxx = dzdx.diff(x)
dzdxy = dzdx.diff(y)
dzdyx = dzdy.diff(x)
dzdyy = dzdy.diff(y)
div2_z = [[dzdxx, dzdxy],[dzdyx, dzdyy]]

fig_2 = plt.figure()
ax_1 = fig_2.add_subplot(1,2,1, projection='3d')
X=np.arange(-10,10,1)
Y=np.arange(-10,10,1)
X,Y = np.meshgrid(X,Y)
# Z = div_z[0]+div_z[1]
Z = (2*X)**2 +(2*Y)**2
surf = ax_1.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.jet,linewidth=0.01, antialiased=False)
Z = 8*X
surf = ax_1.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.Purples,linewidth=0.01, antialiased=False)
Z = 8*Y
surf = ax_1.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.Oranges,linewidth=0.01, antialiased=False)
Z = 8+0*X*Y
surf = ax_1.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.Oranges,linewidth=0.01, antialiased=False)

ax_1.set_zlim3d(0,1000)
plt.show()

ax_2 = fig_2.add_subplot(122, aspect='auto')
X=np.arange(-10,10,1)
Y=np.arange(-10,10,1)
X, Y = np.meshgrid(X, Y)
Z = -(2*X)**2 -div_z[1]
#surf = ax_2.plot_surface(X, Y, Z, rstride=1, cstride=1,cmap=cm.jet,linewidth=0.01, antialiased=False)
ax.set_zlim3d(-1000,0)
fig.colorbar(surf, shrink=0.5, aspect=5)

```

# Section 2: Unconstrained Optimisation

## 2.1 _Example_. Aloha communication protocol

**Aloha** is a well-known random access or _MAC_ (Media/multiple Access Control) communication protocol. It enables multiple nodes sharing a broadcast channel without any additional signaling in a distributed manner. Unlike _FDMA_ or _TDMA_ (frequency or time-division multiple access), the channel is not divided into segments beforehand and collisions of packets due to simultaneous transmissions by nodes are allowed. In slotted Aloha, the nodes can only transmit at the beginning of time slots, which are kept by a global/shared clock. See [Aloha](https://en.wikipedia.org/wiki/ALOHAnet#Slotted_ALOHA) for further background information.

### Slotted Aloha Efficiency

For an $N$-node slotted Aloha system, where each node transmits with a probability $p$, the throughput of the system is given by
$$ S(p) = N p (1-p)^{N-1}$$

### Question 2.1. (2 pts)
Formally define the optimisation problem to find the optimal probability $p$ that maximises the throughput. Clearly identify the objective and decision variable(s). Is the objective convex or concave? Show through derivation.

Note that, there is the constraint $0 \leq p \leq 1$ on probability $p$ but we will ignore it for now.

**Answer as text here**

Optimization problem:
$$ \max_p S(p) = Np(1-P)^{N-1}$$ where $S(p)$ is objective and $p$ is decision variable.

Concave or Convex:
\begin{equation}
\begin{split}
S^{'}(p) &= \frac{\partial}{\partial p}Np(1 - p)^{N - 1} \\
&= N(1-p)^{N-1} + Np(N-1)(1-p)^{N-2}(-1) \\
&= N(1-p)^{N-2}(1-p - p(N-1)) \\
&= N(1-p)^{N-2}(1-p - pN + p)) \\
&= N(1-p)^{N-2}(1-pN)
\end{split}
\end{equation}

\begin{equation}
\begin{split}
S^{''}(p) &= \frac{\partial}{\partial p}S^{'}(p) \\
&= N(N-2)(1-p)^{N-3}(-1)(1-pN)+ N(1-p)^{N-2}(-N) \\
&= -N(1-p)^{N-3}((N-2)(1-pN) + N(1-p)) \\
&= -N(1-p)^{N-3}(N-pN^2-2+2pN+N-pN) \\
&= -N(1-p)^{N-3}(2N-pN^2-2+pN) \\
&= N(1-p)^{N-3}(p(N^2-N)+2-2N) \\
&\Rightarrow \ convex \ or \ concave \ depends \ on \ vlaue \ of \ p \ and \ N
\end{split}
\end{equation}

### Question 2.2. (2 pts)
Plot the performance function and its derivative for $N=10$ nodes. Is the objective function convex or concave? Determine using mathematical methods. Investigate the property of the derivative function of the objective. What do you call such functions?

**Answer as text here**

Concave or Convex with $N=10$:
\begin{equation}
\begin{split}
S^{''}(p) &= N(1-p)^{N-3}(p(N^2-N)+2-2N) \\
&= 10(1-p)^{7}(90p-18)
\end{split}
\end{equation}
$\Rightarrow$ the function is quasi-convex since it depends on $p$ for above value to be positive or negative.

```python
''' Answer as code here '''
```

    ' Answer as code here '

### Question 2.3. (2 pts)
Find the optimal probability $p$ for $N=10$ nodes. Use an appropriate package from [Scipy](https://docs.scipy.org/doc/scipy/reference/optimize.html). Cross-check your answer with a mathematical formula that you should derive by hand.

```python
from scipy import optimize
''' Answer as code here '''
```

    ' Answer as code here '

**Answer as text here**

## 2.2 Gradient Descent Algorithms

### Question 2.4. (10 pts)
Write your own gradient algorithm (with constant step size) to solve the problem
$$ \min_x x^T Q x + r^T x,$$
where $x,\, r \in \mathbb{R}^2$ and $Q$ is a $2\times 2$ positive definite matrix of your choice. Cross-check your answers using one of the standard optimisation packages, e.g. scipy or cvxpy.

1. If Q is positive definite, then what type of optimisation problem is this? Give your answer using mathematical tools learned in classroom. Would anything change if $Q$ were not positive definite? Plot both cases and comment.
2. Focusing on positive definite $Q$, what happens if you choose your fixed step size too large or too small? Observe and comment.
3. Choose $Q$ in such a way that is has a low [condition number] (https://www.encyclopediaofmath.org/index.php/Condition_number), [see also.](https://calculus.subwiki.org/wiki/Gradient_descent_with_constant_learning_rate_for_a_quadratic_function_of_multiple_variables) Next, choose a $Q$ with a high condition number. Compare the performance of your algorithm in both cases, and comment.
4. Now, solve both versions of the problem using (b) diminishing step size (c) Armijo rule/wolf test (line search). Discuss stopping criteria for all variants.
5. Plot your trajectories clearly showing the iterations of the gradient algorithm. You should also show either by displaying the level sets of the objective or the objective function itself (if resorting to a 3D plot).

**Answer as text here**
**Solution** at: https://en.wikipedia.org/wiki/Gradient_descent

Question: 2.4.1

Since $Q$ is positive definite, then $$x^TQx>0 \Rightarrow convex \ optimization$$ if $Q$ is not positive difinite then the problem is not convex optimization.

Question: 2.4.2,

If the fixed step size $\alpha$ is too small, it will take long time to converge and if the $\alpha$ is too big, we may miss the mimum point.

```python
''' Answer as code here '''
```

    ' Answer as code here '

###  Question 2.5. (2 pts)
Choose one of the versions of the problem and algorithms in **Question 2.4** that finds the correct solution.

1. Use objective function itself or a norm of its gradient as a descent (Lyapunov) function that establishes the convergence of the solution algorithm $A(x)$. Plot the value of the chosen descent function versus the trajectory or steps.
2. Calculate and plot $||x(n)-x^*||$ over iterations $n=0,\ldots$, to establish that your solution algorithm leads to a pseudo-contraction.

**Answer as text here**

```python
''' Answer as code here '''
```

    ' Answer as code here '

# Section 3: Constrained Optimisation

## 3.1 _Example_. Waterfilling in Communications

_by Robert Gowers, Roger Hill, Sami Al-Izzi, Timothy Pollington and Keith Briggs.
From the book by Boyd and Vandenberghe, Convex Optimization, Example 5.2 page 245._

$$\min_{x} \sum_{i=1}^N -\log(\alpha_i + x_i)$$

$$\text{subject to } x_i \geq 0, \; \forall i, \text{ and } \sum_{i=1}^N x_i = P $$

This problem arises in information/communication theory, in allocating power to a
set of $n$ communication channels. The variable $x_i$ represents the transmitter power
allocated to the _i-th_ channel, and $\log(\alpha_i + x_i)$ gives the capacity or communication rate of the channel, where $\alpha_i>0$ represents the floor above the baseline at which power can be added to the channel. The problem is to allocate a total power of one to the channels,
in order to maximize the total communication rate.

This can be solved using a classic [water filling algorithm](https://en.wikipedia.org/wiki/Water_filling_algorithm).

###  Question 3.1. (4 pts)

1. (1 pt) Is the problem in Example 3.1 convex? Formally explain/argue why or why not. What does this imply regarding the solution?
2. (2 pts) Solve the problem above for $N=8$ and a randomly chosen $\alpha$ vector. You can use for example _Cvxpy_ package. Cross-check your answer with another software (package), e.g. Matlab or Scipy.
3. (1 pt) Write the Lagrangian, KKT conditions, and find numerically the Lagrange multipliers associated with the solution (using the software package/function). Which constraints are active? Explain and discuss briefly.

**Answer as text here**

**Solution** at https://www.cvxpy.org/examples/applications/water_filling_BVex5.2.html

Question 3.1.1

Convex Optimization:
\begin{equation}
\begin{split}
f(x_i) &= \sum_{i=1}^N - \log(\alpha_i + x_i) \\
\Rightarrow f^{'}(x_i) &= \frac{\partial}{\partial x_i} \sum_{i=1}^N - \log(\alpha_i + x_i)
= -\sum_{i=1}^N \frac{1}{(\alpha_i + x_i)ln(10)}
= -\sum_{i=1}^N \frac{(\alpha_i + x_i)^{-1}}{ln(10)} \\
\end{split}
\end{equation}

\begin{equation}
\begin{split}
\Rightarrow f^{''}(x_i) = -\sum_{i=1}^N (-1) \frac{(\alpha_i + x_i)^{-2}}{ln(10)}
= -\sum_{i=1}^N \frac{1}{(\alpha_i + x_i)^2 ln(10)}
\end{split}
\end{equation}
Since both $x_i, \alpha_i > 0$, $\Rightarrow convex \ optimization \ problem$.

Question 3.1.3

Lagrangian condition:
\begin{equation}
\mathcal{L}(x, \lambda, \nu) = f_0 (x) + \sum_{i=1}^M \lambda_i f_i(x) + \sum_{i=1}^N \nu_i h_i(x) \\
\Rightarrow \mathcal{L}(x, \lambda, \nu) = \sum_{i=1}^N -log(\alpha_i + x_i) - \sum_{i=1}^N \lambda_i x_i + \sum_{i=1}^N \nu_i(x_i - P)
\end{equation}

KKT condition:
\begin{equation}
x^* \geq 0 \\
x^* = P \\
\lambda* \geq 0 \\
\lambda* x_i^* = 0, i=1,2..., N \\
-\frac{1}{(\alpha_i + x_i^*)} - \lambda^* + \nu^* = 0, i=1,2..., N
\end{equation}

```python
''' Answer as code here '''
#!/usr/bin/env python3
# @author: R. Gowers, S. Al-Izzi, T. Pollington, R. Hill & K. Briggs
import numpy as np
import cvxpy as cp

def water_filling(n, a, sum_x=1):
    '''
    Boyd and Vandenberghe, Convex Optimization, example 5.2 page 145
    Water-filling.

    This problem arises in information theory, in allocating power to a set of
    n communication channels in order to maximise the total channel capacity.
    The variable x_i represents the transmitter power allocated to the ith channel,
    and log(α_i+x_i) gives the capacity or maximum communication rate of the channel.
    The objective is to minimise -∑log(α_i+x_i) subject to the constraint ∑x_i = 1
    '''

    # Declare variables and parameters
    x = cp.Variable(shape=n)
    alpha = cp.Parameter(n, nonneg=True)
    alpha.value = a

    # Choose objective function. Interpret as maximising the total communication rate of all the channels
    obj = cp.Maximize(cp.sum(cp.log(alpha + x)))

    # Declare constraints
    constraints = [x >= 0, cp.sum(x) - sum_x == 0]

    # Solve
    prob = cp.Problem(obj, constraints)
    prob.solve()
    if(prob.status=='optimal'):
        return prob.status, prob.value, x.value
    else:
        return prob.status, np.nan, np.nan

# As an example, we will solve the water filling problem with 3 buckets, each with different α
np.set_printoptions(precision=3)
buckets = 3
alpha = np.array([0.8, 1.0, 1.2])

stat, prob, x = water_filling(buckets, alpha)
print('Problem status: {}'.format(stat))
print('Optimal communication rate = {:.4g} '.format(prob))
print('Transmitter powers:\n{}'.format(x))

```

## 3.2 _Example_. Economic Dispatch in Power Generation

The problem is formulated as
$$ \min_P \sum_{i=1}^N c_i P_i $$
$$\text{subject to } P_{i,max} \geq P_i \geq 0, \; \forall i, \text{ and } \sum_{i=1}^N P_i = P_{demand} $$

Here, $P_1,\ldots P_N$ are the power generated by Generators $1,\ldots,N$, $c_i$ is the per-unit generation cost of the i-_th_ generator, and $P_{demand}$ is the instantaneous power demand that needs to be satisfied by aggregate generation. More complex formulations take into account transmission, generator ramp-up and down constraints, and reactive power among other things.

### Question 3.2. (4 pts)

Let us get inspired from generation in Victoria with $N=12$ biggest generators that have more than 200MW capacity. Choose their maximum generation randomly or from the Victoria generator report if you wish to be more realistic. Generate a random cost vector $c$ varying between $10-50$ AUD per MWh. _(Optionally, you can search and find how much different generation types cost if you are interested)._ Let the demand be $P_{demand}=5000MW$.

Solve this simplified [economic dispatch](https://en.wikipedia.org/wiki/Economic_dispatch) problem defined above. The resulting [merit order](https://en.wikipedia.org/wiki/Merit_order) is the generation that would have been if there was no NEM (electricity market).

More about electricity market and generation at https://www.aemo.com.au/ See also this [NEM overview introductory document (right click to download)](./files/National_Electricity_Market_Fact_Sheet.pdf) and the [Victoria generator report as of January 2019](files/Generation_Information_VIC_January_2019.xlsx).

1. Solve the problem using _cvxpy._
2. What type of an optimisation problem is this? Briefly explain.
3. Formulate by hand the dual problem and solve it with _cvxpy._ Is there a duality gap? Explain briefly why or why not.
4. Briefly comment on simplex algorithm and solve the problem using [Scipy and simplex algorithm.](https://docs.scipy.org/doc/scipy/reference/optimize.linprog-simplex.html) Compare your results.

**Answer as text here**

```python
''' Answer as code here '''
```

    ' Answer as code here '

## 3.3 _Example_. Power Control in Wireless Communication

 *Adapted from Boyd, Kim, Vandenberghe, and Hassibi,* "[A Tutorial on Geometric Programming](https://web.stanford.edu/~boyd/papers/pdf/gp_tutorial.pdf)."

The [power control problem in wireless communications](http://winlab.rutgers.edu/~narayan/PAPERS/PC%20for%20Wireless%20Data.pdf) aims to minimise the total transmitter power available across $N$ trasmitters while concurrently achieving good (or a pre-defined minimum) performance.

The technical setup is as follows. Each transmitter $i$ transmits with a power level $P_i$ bounded below and above by a minimum and maximum level. The power of the signal received from transmitter $j$ at receiver $i$ is $G_{ij} P_{j}$, where $G_{ij} > 0$ represents the path gain (often loss) from transmitter $j$ to receiver $i$. The signal power at the intended receiver $i$ is $G_{ii} P_i$, and the interference power at receiver $i$ from other transmitters is given by $\sum_{k \neq i} G_{ik}P_k$. The (background) noise power at receiver $i$ is $\sigma_i$. Thus, the _Signal to Interference and  Noise Ratio (SINR)_ of the $i$th receiver-transmitter pair is

$$ S_i = \frac{G_{ii}P_i}{\sum_{k \neq i} G_{ik}P_k + \sigma_i }. $$

The minimum SINR represents a performance lower bound for this system, $S^{\text min}$.

The resulting optimisation problem is formulated as

$$
\begin{array}{ll}
\min_{P} & \sum_{i=1}^N P_i \\
\text{subject to} & P^{min} \leq P_i \leq P^{max}, \; \forall i \\
& \dfrac{G_{ii}P_i}{\sigma_i + \sum_{k \neq i} G_{ik}P_k} \geq S^{min} , \; \forall i \\
\end{array}
$$

### Question 3.3. (10 pts)

Let $N=6$, $P^{min}=0.1$, $P^{max}=5$, $\sigma=0.2$ (same for all). Create a random path loss matrix $G$, where off-diagonal elements are between $0.1$ and $0.9$ and the diagonal elements are equal to $1$.

1. (2 pts) Write down the Langrangian and KKT conditions of this problem.
2. (2 pts) Solve the problem first with $S^{min}=0$ using _cvxpy_. Plot the power levels and SINRs that you obtain.
3. (2 pts) What happens if you choose an $S^{min}$ that is larger? Solve the problem again and document your results. What happens if you choose a very large $S^{min}$? Observe and comment.
4. (4 pts) Solve the problem using a combination of active set and penalty function methods. Specifically, choose a penalty function to impose the $S^{min}$ constraint and use an active set method for $P^{min}$, $P^{max}$ power constraints. Choose tighter constraints to make the problem more interesting.

**Answer as text here**

```python
''' Answer as code here '''
import cvxpy as cp

# Create two scalar optimization variables.
x = cp.Variable()
y = cp.Variable()
print(x)
print(type(x))
"""
# Create two constraints.
constraints = [x + y == 1,
               x - y >= 1]

# Form objective.
obj = cp.Minimize((x - y)**2)

# Form and solve problem.
prob = cp.Problem(obj, constraints)
prob.solve()  # Returns the optimal value.
print("status:", prob.status)
print("optimal value", prob.value)
print("optimal var", x.value, y.value)
"""
```

    var28
    <class 'cvxpy.expressions.variables.variable.Variable'>

    '\n# Create two constraints.\nconstraints = [x + y == 1,\n               x - y >= 1]\n\n# Form objective.\nobj = cp.Minimize((x - y)**2)\n\n# Form and solve problem.\nprob = cp.Problem(obj, constraints)\nprob.solve()  # Returns the optimal value.\nprint("status:", prob.status)\nprint("optimal value", prob.value)\nprint("optimal var", x.value, y.value)\n'

## 3.4 (_Optional_ Bonus, 10 pts) Model Predictive Control

It is possible to formulate a **discrete-time, finite-horizon optimal-control** as a constrained optimisation problem. This is quite useful since it allows making use of powerful optimisation solvers in addressing the [control problem](https://en.wikipedia.org/wiki/Control_system). This formulation is called Model Predictive Control [(MPC).](https://en.wikipedia.org/wiki/Model_predictive_control)

Specifically, consider a system with a state vector $x_t\in {\bf R}^n$ that varies over the discrete time steps $t=0,\ldots,T$, and control actions $u_t\in {\bf R}^m$ that affect the state as part of a linear dynamical system formulated as

$$ x_{t+1} = A x_t + B u_t, $$
where $A \in {\bf R}^{n\times n}$ and $B \in {\bf R}^{n\times m}$ are system matrices.

The goal is to find the optimal actions $u_0,\ldots,u_{T-1}$ over the finite horizon $T$ by solving the optimization problems

\begin{array}{ll} \mbox{minimize} & \sum_{t=0}^{T-1} \ell (x_t,u_t) + \ell_T(x_T)\\
\mbox{subject to} & x_{t+1} = Ax_t + Bu_t\\%, \quad t=0, \ldots, T-1\\
& (x_t,u_t) \in \mathcal C, \quad x_T\in \mathcal C_T,
%, \quad \quad t=0, \ldots, T
\end{array}

where $\ell: {\bf R}^n \times {\bf R}^m\to {\bf R}$ is the stage cost, $\ell_T$ is the terminal cost,
$\mathcal C$ is the state/action constraints, and $\mathcal C_T$ is the terminal constraint.

1. Choose a simple linear dynamical system that you are interested in and formulate its state evolution as $x_{t+1} = A x_t + B u_t$. This can be a very well-known system, you don't need to be original.
2. Define the objective, i.e. ongoing (and if there are terminal) costs imposed on states and control actions (cost of good/bad states, cost of taking a control action).
3. Solve the problem over a finite horizon. Apply actions to compute and plot the evolution of states.

A recent research paper (which has won the best student paper award) using MPC formulation is [available here (right click to download).](files/MPC_paper.pdf)

### 3.5 (_Optional_  without bonus :) How do optimisation software handle non-standard problems?

_This is just for those of you, who are very interested and have spare time and bored and have nothing else to do!_

Try to solve Question 6 from Module 2, Lesson 2 (Constrained Optimisation) using the software packages (Python ones and Matlab). How do various software packages handle such non-standard situations? Observe and add to your report briefly with a sentence or two.

#  Workshop Report Submission Instructions

_You should ideally complete the workshop tasks and answer the questions within the respective session!_ The submission deadline is usually Friday, the week after. Submission deadlines will be announced on LMS.

It is **mandatory to follow all of the submissions guidelines** given below. _Don't forget the Report submission information on top of this notebook!_

1. The completed Jupyter notebook and its Pdf version (you can simply print-preview and then print as pdf from within your browser) should be uploaded to the right place in LMS by the announced deadline. _It is your responsibility to follow the announcements!_ **Late submissions will be penalised (up to 100% of the total mark depending on delay amount)!**
2. Filename should be “ELEN90061 Workshop **W: StudentID1-StudentID2** of session **Day-Time**", where **W** refers to the workshop number, **StudentID1-StudentID2** are your student numbers, **Day-Time** is your session day and time, e.g. *Tue-14*.
3. Answers to questions, simulation results and diagrams should be included in the Jupyter notebook as text, code, plots. *If you don't know latex, you can write formulas/text to a paper by hand, scan it and then include as image within Markdown cells.*
4. One report submission per group.

### Additional guidelines for your programs:

* Write modular code using functions.
* Properly indent your code. But Python forces you do that anyway ;)
* Heavily comment the code to describe your implementation and to show your understanding. No comments, no credit!
* Make the code your own! It is encouraged to find and get inspired by online examples but you should exactly understand, modify as needed, and explain your code via comments. There will be no credit for blind copy/paste even if it somehow works (and it is easier to detect it than you might think)!

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

One of the basic things we have to do in math is to express numbers. Numbers are the basic particles of essence so it makes sense to start with them first. We'll start with the *natural numbers* and work our way up through the whole spectrum of numbers finally arriving at the *complex numbers*.

We'll see that every type of number has a use. These numbers are not just cooked up because somebody figured that would be cool. No, every number serves a definite purpose and even though we might not always understand completely what that purpose is that does not mean we shouldn't use them in order to try a greater understanding.

Take for example the number $\sqrt{2}$ which is a perfectly fine number nowadays. However, how hard we might try, there's really no other way to write down this number. We could try for an approximation like $1.4142135\ldots$ but that's really just being sloppy. There *are* other ways to write this number but we will never get a *real* number out of it. Only approximations.

### natural numbers
Some people like to include the number zero in this set but we'll stick with with the numbers $1, 2, 3, \ldots, n$ where $n \gt 0$. They are basically all the whole numbers.

$$\mathbb{N} = {1, 2, 3, \ldots}$$

If we want to be really unambiguous about what we mean we could be extra explicit.

$$
\begin{align}
\mathbb{N^0} = \mathbb{N_0} & = {0, 1, 2, 3, \ldots}\\
\mathbb{N^*} = \mathbb{N^+} = \mathbb{N_1} = \mathbb{N_{\gt 0}} & = {1, 2, 3, \ldots}
\end{align}
$$

If we really want to include zero in this text we'll use a different set though.

### whole numbers
This is all of the numbers in $\mathbb{N}$ and the number zero. We don't have any cool letter for this though but we can say it's all the whole numbers $0, 1, 2, \ldots, n$ where $n \ge 0$.

### integers
Now this is an interesting set. If only because it's so prevalent in almost all math that we do. For example, computers love integers because they can be so easily represented by a sequence of bits.

Integers is the set of *whole numbers* but it also includes all the negatives of the *natural numbers*. So now were talking about $-n, \ldots, -2, -1, 0, 1, 2, \ldots, n$. This set is important enough to get its own symbol:

$$\mathbb{Z} = {-n, \ldots, -2, -1, 0, 1, 2, \ldots, n}$$

### interlude: integers
Even though we are not even half-way up our ladder of number systems those integers are already getting a bit interesting. Why do we like them so much in computing? Well, because they can easily be represented as a sequence of one and zero. How does this work though?

If we look at any integer, let's take for example $321$ and analyze what it means we can come to the insight that:

$$321 = (3 \times 100) + (2 \times 10) + (1 \times 1)$$

If we look a little deeper we can also see that $100 = 10^2$, $10 = 10^1$ and $1 = 10^0$ so in other words:

$$321 = (3 \times 10^2) + (2 \times 10^1) + (1 \times 10^0)$$

Our number system is called the *decimal* system and that's because we have base `10` numbers. There other number systems. Other sytems that are in commonly in use are *binary*, *hexadecimal* and sometimes *octal*. The binary system is popular because it aligns with electronic switches that can be either on or off. There are only two possibilities and that is what binary means. The octal system is sometimes used because it aligns nicely with the *byte* memory unit in computers but you don't see it much these days. However, the hexadecimal system is still prevalent and you'll see a lot, for example in color codes.

So how does the binary system work? Well remember that the decimal system operates on *powers* of ten so the binary system operates on power of two.

$$
\begin{align}
0 = 0 \times 2^0 & = 0\\
1 = 1 \times 2^0 & = 1 \\
2 = (1 \times 2^1) + (0 \times 2^0) & = 10 \\
3 = (1 \times 2^1) + (1 \times 2^0) & = 11 \\
4 = (1 \times 2^2) + (0 \times 2^1) + (0 \times 2^0) & = 100 \\
5 = (1 \times 2^2) + (0 \times 2^1) + (1 \times 2^0) & = 101 \\
6 = (1 \times 2^2) + (1 \times 2^1) + (0 \times 2^0) & = 110 \\
7 = (1 \times 2^2) + (1 \times 2^1) + (1 \times 2^1) & = 111
\end{align}
$$

### rational numbers
We get rational numbers when we need to express one integer as some part of another. When people started doing real math this problem soon cropped up. The easier way to deal with it is to kind of *not* deal with it and just say well it's this number expressed as some ratio of another. This might be a bit abstract so let's take an example.

When we first started doing divisions we took stuff like $\frac{3}{3} = 1$ and the world was good. As people started doing more fancy stuff with math and numbers things got a bit out of hand though. At some point we found ourselfs in the need to express something else than *integers*. As always, when math runs into a wall we'll just invent something to get over it. And as such we got *rational numbers* which are basically just *fractions* like $\frac{1}{3}$ or $\frac{1}{\pi}$.

At this point note that in order to get better numbers we just take some existing numbers we know and combine them in some way that makes sense but is somewhat unexpected. I mean, people are still drawing out proplems with triangles and such and now you're starting to abstract some of this stuff away. And it makes sense too because you don't want to lose any information. By keeping that number in its exact *ratio* $\frac{1}{3}$ you will have a clean number to calculate with.

Which leads us to the unfortunately necessary...

### real numbers
Let me start by saying that *real numbers* are unfortunately named because most of them are anything but real. Real numbers are supposed to be plotted along a line (usually the x-axis) and they are but when we *do have* to work with them they are usually just an approximation of a rational number.

We like to be pure as long as we can so we'll use rational numbers over real numbers but sometimes (especially dealing with computers) we'll *have* to convert our rational number to some *real* approximation. For most purposes you can just think of a *real* number as an *approximation* of some rational number that is to be used for real-life purposes.

### irrational numbers
These are interesting numbers because they are so called *real* numbers but we cannot express them as a rational number. As always, when we encounter such a thing in math we tend to give it a name or a convenient notation. Examples are $\pi$, our friend $\sqrt{2}$ and $e$.

Irrational numbers are awkward and in order to stay pure we sometimes can do no better than express something as a fraction of an irrational unit. Of course we could try to get a *real* number out but we have to remember that this will always be an approximiation. This might be fine though depending on our purposes.

### imaginary numbers
This is where things start to get really interesting (and strange). After hundreds of years of working on math puzzles mathematicians where getting annoyed by the fact that this $\sqrt{-1}$ kept popping up in their would-be solutions. And (at that time) there was no possible way to calculate a negative *square root* so they just gave up... Mostly. Finally they just decided to go with it and complete the calculations involving the negative square root $\sqrt{-1}$ and things turned out beautifully.

After a few more hundred years it has become so useful that we gave it a special notation and even a special algebra so it all makes sense as a number as well.

So the first thing we have to consider is that we have this new number now called $i$ which is defined as $i^2 = -1$ so $i = \sqrt{-1}$. Now we can just use regular algebra to express any negative root, for example if we need $\sqrt{-5}$ I could just go ${i \times \sqrt{5}}$ and this simple transformation allows us to actually calculate with those things.

Note that the name *imaginary* is actually a bad name. These numbers *are* real in the normal sense of that word. The fact that we describe them this way is just byproduct of the way we write math in general. It's better to look at any number and imagine it to have a so-called *imaginary* component. In a lot of cases this will just be $0 \times i$ (no imaginary component) but sometimes they do.

### complex numbers
In some sense, this is the best way to describe a number. This form of numbers is called *complex* buty they are not really that complex though. Again, this is kind of a misnomer and actually complex numbers are very easy. In fact, they are just numbers.

It's just so many numbers we deal with are on the x-axis of the *complex plane* that we don't even notice we are dealing with complex numbers at all. Thanks to a lot of evolution and schooling we can now reasonably *feel* how most numbers work up to and including rational numbers. However, *imaginary* and *complex numbers* are still a bit weird.

One of the best ways to show how complex numbers enter math is to show an innocent looking equation like: $y = x^2 + 1$. If we try to solve this for $y = 0$ we get $0 = x^2 + 1$. And going further we get $-1 = x^2 \implies x = \sqrt{-1}$. Before complex numbers there was no such thing as $\sqrt{-1}$ and we would have simply given up.

Nowadays we can say that the solution is $i$ (and $-i$ is a valid solution too).

```python

```


```python
%matplotlib inline
```

파이프라인 병렬화로 트랜스포머 모델 학습시키기
==============================================

**Author**: `Pritam Damania <https://github.com/pritamdamania87>`_
  **번역**: `백선희 <https://github.com/spongebob03>`_

이 튜토리얼은 파이프라인(pipeline) 병렬화(parallelism)를 사용하여 여러 GPU에 걸친 거대한 트랜스포머(transformer)
모델을 어떻게 학습시키는지 보여줍니다. `NN.TRANSFORMER 와 TORCHTEXT 로 시퀀스-투-시퀀스(SEQUENCE-TO-SEQUENCE) 모델링하기 <https://tutorials.pytorch.kr/beginner/transformer_tutorial.html>`__ 튜토리얼의
확장판이며 파이프라인 병렬화가 어떻게 트랜스포머 모델 학습에 쓰이는지 증명하기 위해 이전 튜토리얼에서의
모델 규모를 증가시켰습니다.

선수과목(Prerequisites):

    * `Pipeline Parallelism <https://pytorch.org/docs/stable/pipeline.html>`__
    * `NN.TRANSFORMER 와 TORCHTEXT 로 시퀀스-투-시퀀스(SEQUENCE-TO-SEQUENCE) 모델링하기 <https://tutorials.pytorch.kr/beginner/transformer_tutorial.html>`__

모델 정의하기
-------------

이번 튜토리얼에서는, 트랜스포머 모델을 두 개의 GPU에 걸쳐서 나누고 파이프라인 병렬화로 학습시켜 보겠습니다.
모델은 바로 `NN.TRANSFORMER 와 TORCHTEXT 로 시퀀스-투-시퀀스(SEQUENCE-TO-SEQUENCE) 모델링하기
<https://tutorials.pytorch.kr/beginner/transformer_tutorial.html>`__ 튜토리얼과
똑같은 모델이지만 두 단계로 나뉩니다. 대부분 파라미터(parameter)들은
`nn.TransformerEncoder <https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html>`__ 계층(layer)에 포함됩니다.
`nn.TransformerEncoder <https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html>`__ 는
`nn.TransformerEncoderLayer <https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html>`__ 의 ``nlayers`` 로 구성되어 있습니다.
결과적으로, 우리는 ``nn.TransformerEncoder`` 에 중점을 두고 있으며,
``nn.TransformerEncoderLayer`` 의 절반은 한 GPU에 두고
나머지 절반은 다른 GPU에 있도록 모델을 분할합니다. 이를 위해서 ``Encoder`` 와
``Decoder`` 섹션을 분리된 모듈로 빼낸 다음, 원본 트랜스포머 모듈을
나타내는 nn.Sequential을 빌드 합니다.

```python
import sys
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
import tempfile
from torch.nn import TransformerEncoder, TransformerEncoderLayer

if sys.platform == 'win32':
    print('Windows platform is not supported for pipeline parallelism')
    sys.exit(0)
if torch.cuda.device_count() < 2:
    print('Need at least two GPU devices for this tutorial')
    sys.exit(0)

class Encoder(nn.Module):
    def __init__(self, ntoken, ninp, dropout=0.5):
        super(Encoder, self).__init__()
        self.pos_encoder = PositionalEncoding(ninp, dropout)
        self.encoder = nn.Embedding(ntoken, ninp)
        self.ninp = ninp
        self.init_weights()

    def init_weights(self):
        initrange = 0.1
        self.encoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, src):
        # 인코더로 (S, N) 포맷이 필요합니다.
        src = src.t()
        src = self.encoder(src) * math.sqrt(self.ninp)
        return self.pos_encoder(src)

class Decoder(nn.Module):
    def __init__(self, ntoken, ninp):
        super(Decoder, self).__init__()
        self.decoder = nn.Linear(ninp, ntoken)
        self.init_weights()

    def init_weights(self):
        initrange = 0.1
        self.decoder.bias.data.zero_()
        self.decoder.weight.data.uniform_(-initrange, initrange)

    def forward(self, inp):
        # 파이프라인 결과물을 위해 먼저 배치 차원 필요합니다.
        return self.decoder(inp).permute(1, 0, 2)
```

``PositionalEncoding`` 모듈은 시퀀스 안에서 토큰의 상대적인 또는 절대적인 포지션에 대한 정보를 주입합니다.
포지셔널 인코딩은 임베딩과 합칠 수 있도록 똑같은 차원을 가집니다. 여기서
다른 주기(frequency)의 ``sine`` 과 ``cosine`` 함수를 사용합니다.

```python
class PositionalEncoding(nn.Module):

    def __init__(self, d_model, dropout=0.1, max_len=5000):
        super(PositionalEncoding, self).__init__()
        self.dropout = nn.Dropout(p=dropout)

        pe = torch.zeros(max_len, d_model)
        position = torch.arange(0, max_len, dtype=torch.float).unsqueeze(1)
        div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
        pe[:, 0::2] = torch.sin(position * div_term)
        pe[:, 1::2] = torch.cos(position * div_term)
        pe = pe.unsqueeze(0).transpose(0, 1)
        self.register_buffer('pe', pe)

    def forward(self, x):
        x = x + self.pe[:x.size(0), :]
        return self.dropout(x)
```

데이터 로드하고 배치 만들기
---------------------------

학습 프로세스는 ``torchtext`` 의 Wikitext-2 데이터셋을 사용합니다.
torchtext 데이터셋에 접근하기 전에, https://github.com/pytorch/data 을 참고하여 torchdata를 설치하시기 바랍니다.

단어 오브젝트는 훈련 데이터셋으로 만들어지고, 토큰을 텐서(tensor)로 수치화하는데 사용됩니다.
시퀀스 데이터로부터 시작하여, ``batchify()`` 함수는 데이터셋을 열(column)들로 정리하고,
``batch_size`` 사이즈의 배치들로 나눈 후에 남은 모든 토큰을 버립니다.
예를 들어, 알파벳을 시퀀스(총 길이 26)로 생각하고 배치 사이즈를 4라고 한다면,
알파벳을 길이가 6인 4개의 시퀀스로 나눌 수 있습니다:

\begin{align}\begin{bmatrix}
  \text{A} & \text{B} & \text{C} & \ldots & \text{X} & \text{Y} & \text{Z}
  \end{bmatrix}
  \Rightarrow
  \begin{bmatrix}
  \begin{bmatrix}\text{A} \\ \text{B} \\ \text{C} \\ \text{D} \\ \text{E} \\ \text{F}\end{bmatrix} &
  \begin{bmatrix}\text{G} \\ \text{H} \\ \text{I} \\ \text{J} \\ \text{K} \\ \text{L}\end{bmatrix} &
  \begin{bmatrix}\text{M} \\ \text{N} \\ \text{O} \\ \text{P} \\ \text{Q} \\ \text{R}\end{bmatrix} &
  \begin{bmatrix}\text{S} \\ \text{T} \\ \text{U} \\ \text{V} \\ \text{W} \\ \text{X}\end{bmatrix}
  \end{bmatrix}\end{align}

이 열들은 모델에 의해서 독립적으로 취급되며, 이는
``G`` 와 ``F`` 의 의존성이 학습될 수 없다는 것을 의미하지만, 더 효율적인
배치 프로세싱(batch processing)을 허용합니다.

```python
import torch
from torchtext.datasets import WikiText2
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator

train_iter = WikiText2(split='train')
tokenizer = get_tokenizer('basic_english')
vocab = build_vocab_from_iterator(map(tokenizer, train_iter), specials=["<unk>"])
vocab.set_default_index(vocab["<unk>"])

def data_process(raw_text_iter):
  data = [torch.tensor(vocab(tokenizer(item)), dtype=torch.long) for item in raw_text_iter]
  return torch.cat(tuple(filter(lambda t: t.numel() > 0, data)))

train_iter, val_iter, test_iter = WikiText2()
train_data = data_process(train_iter)
val_data = data_process(val_iter)
test_data = data_process(test_iter)

device = torch.device("cuda")

def batchify(data, bsz):
    # 데이터셋을 bsz 파트들로 나눕니다.
    nbatch = data.size(0) // bsz
    # 깔끔하게 나누어 떨어지지 않는 추가적인 부분(나머지)은 잘라냅니다.
    data = data.narrow(0, 0, nbatch * bsz)
    # 데이터를 bsz 배치들로 동일하게 나눕니다.
    data = data.view(bsz, -1).t().contiguous()
    return data.to(device)

batch_size = 20
eval_batch_size = 10
train_data = batchify(train_data, batch_size)
val_data = batchify(val_data, eval_batch_size)
test_data = batchify(test_data, eval_batch_size)
```

입력과 타겟 시퀀스를 생성하기 위한 함수들
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

``get_batch()`` 함수는 트랜스포머 모델을 위한 입력과 타겟 시퀀스를
생성합니다. 이 함수는 소스 데이터를 ``bptt`` 길이를 가진 덩어리로 세분화합니다.
언어 모델링 과제를 위해서, 모델은 다음 단어인 ``Target`` 이 필요합니다. 에를 들어 ``bptt`` 의 값이 2라면,
``i`` = 0 일 때 다음의 2 개 변수(Variable)를 얻을 수 있습니다:

변수 덩어리는 트랜스포머 모델의 ``S`` 차원과 일치하는 0 차원에 해당합니다.
배치 차원 ``N`` 은 1 차원에 해당합니다.

```python
bptt = 25
def get_batch(source, i):
    seq_len = min(bptt, len(source) - 1 - i)
    data = source[i:i+seq_len]
    target = source[i+1:i+1+seq_len].view(-1)
    # 파이프라인 병렬화를 위해 먼저 배치 차원이 필요합니다.
    return data.t(), target
```

모델 규모와 파이프 초기화
-------------------------

파이프라인 병렬화를 활용한 대형 트랜스포머 모델 학습을 증명하기 위해서
트랜스포머 계층 규모를 적절히 확장시킵니다. 4096차원의 임베딩 벡터, 4096의 은닉 사이즈,
16개의 어텐션 헤드(attention head)와 총 12 개의 트랜스포머 계층
(``nn.TransformerEncoderLayer``)를 사용합니다. 이는 최대
**1.4억** 개의 파라미터를 갖는 모델을 생성합니다.

Pipe는 `RRef <https://pytorch.org/docs/stable/rpc.html#rref>`__ 를 통해
`RPC 프레임워크 <https://pytorch.org/docs/stable/rpc.html>`__ 에 의존하는데
이는 향후 호스트 파이프라인을 교차 확장할 수 있도록 하기 때문에
RPC 프레임워크를 초기화해야 합니다. 이때 RPC 프레임워크는 오직 하나의 하나의 worker로 초기화를 해야 하는데,
여러 GPU를 다루기 위해 프로세스 하나만 사용하고 있기 때문입니다.

그런 다음 파이프라인은 한 GPU에 8개의 트랜스포머와
다른 GPU에 8개의 트랜스포머 레이어로 초기화됩니다.

Note

효율성을 위해 ``Pipe`` 에 전달된 ``nn.Sequential`` 이
   오직 두 개의 요소(2개의 GPU)로만 구성되도록 합니다. 이렇게 하면
   Pipe가 두 개의 파티션에서만 작동하고
   파티션 간 오버헤드를 피할 수 있습니다.

```python
ntokens = len(vocab) # 단어 사전(어휘집)의 크기
emsize = 4096 # 임베딩 차원
nhid = 4096 # nn.TransformerEncoder 에서 순전파(feedforward) 신경망 모델의 차원
nlayers = 12 # nn.TransformerEncoder 내부의 nn.TransformerEncoderLayer 개수
nhead = 16 # multiheadattention 모델의 헤드 개수
dropout = 0.2 # dropout 값

from torch.distributed import rpc
tmpfile = tempfile.NamedTemporaryFile()
rpc.init_rpc(
    name="worker",
    rank=0,
    world_size=1,
    rpc_backend_options=rpc.TensorPipeRpcBackendOptions(
        init_method="file://{}".format(tmpfile.name),
        # _transports와 _channels를 지정하는 것이 해결 방법이며
        # PyTorch 버전 >= 1.8.1 에서는 _transports와 _channels를
        # 지정하지 않아도 됩니다.
        _transports=["ibv", "uv"],
        _channels=["cuda_ipc", "cuda_basic"],
    )
)

num_gpus = 2
partition_len = ((nlayers - 1) // num_gpus) + 1

# 처음에 인코더를 추가합니다.
tmp_list = [Encoder(ntokens, emsize, dropout).cuda(0)]
module_list = []

# 필요한 모든 트랜스포머 블록들을 추가합니다.
for i in range(nlayers):
    transformer_block = TransformerEncoderLayer(emsize, nhead, nhid, dropout)
    if i != 0 and i % (partition_len) == 0:
        module_list.append(nn.Sequential(*tmp_list))
        tmp_list = []
    device = i // (partition_len)
    tmp_list.append(transformer_block.to(device))

# 마지막에 디코더를 추가합니다.
tmp_list.append(Decoder(ntokens, emsize).cuda(num_gpus - 1))
module_list.append(nn.Sequential(*tmp_list))

from torch.distributed.pipeline.sync import Pipe

# 파이프라인을 빌드합니다.
chunks = 8
model = Pipe(torch.nn.Sequential(*module_list), chunks = chunks)

def get_total_params(module: torch.nn.Module):
    total_params = 0
    for param in module.parameters():
        total_params += param.numel()
    return total_params

print ('Total parameters in model: {:,}'.format(get_total_params(model)))
```

모델 실행하기
-------------

손실(loss)을 추적하기 위해 `CrossEntropyLoss <https://pytorch.org/docs/master/nn.html?highlight=crossentropyloss#torch.nn.CrossEntropyLoss>`__ 가
적용되며, 옵티마이저(optimizer)로서 `SGD <https://pytorch.org/docs/master/optim.html?highlight=sgd#torch.optim.SGD>`__
는 확률적 경사하강법(stochastic gradient descent method)을 구현합니다. 초기
학습률(learning rate)은 5.0로 설정됩니다. `StepLR <https://pytorch.org/docs/master/optim.html?highlight=steplr#torch.optim.lr_scheduler.StepLR>`__ 는
에폭(epoch)에 따라서 학습률을 조절하는 데 사용됩니다. 학습하는 동안,
기울기 폭발(gradient exploding)을 방지하기 위해 모든 기울기를 함께 조정(scale)하는 함수
`nn.utils.clip_grad_norm\_ <https://pytorch.org/docs/master/nn.html?highlight=nn%20utils%20clip_grad_norm#torch.nn.utils.clip_grad_norm_>`__
을 이용합니다.

```python
criterion = nn.CrossEntropyLoss()
lr = 5.0 # 학습률
optimizer = torch.optim.SGD(model.parameters(), lr=lr)
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1.0, gamma=0.95)

import time
def train():
    model.train() # 훈련 모드로 전환
    total_loss = 0.
    start_time = time.time()
    ntokens = len(vocab)

    # 스크립트 실행 시간을 짧게 유지하기 위해서 50 배치만 학습합니다.
    nbatches = min(50 * bptt, train_data.size(0) - 1)

    for batch, i in enumerate(range(0, nbatches, bptt)):
        data, targets = get_batch(train_data, i)
        optimizer.zero_grad()
        # Pipe는 단일 호스트 내에 있고
        # forward 메서드로 반환된 ``RRef`` 프로세스는 이 노드에 국한되어 있기 때문에
        # ``RRef.local_value()`` 를 통해 간단히 찾을 수 있습니다.
        output = model(data).local_value()
        # 타겟을 파이프라인 출력이 있는
        # 장치로 옮겨야합니다.
        loss = criterion(output.view(-1, ntokens), targets.cuda(1))
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), 0.5)
        optimizer.step()

        total_loss += loss.item()
        log_interval = 10
        if batch % log_interval == 0 and batch > 0:
            cur_loss = total_loss / log_interval
            elapsed = time.time() - start_time
            print('| epoch {:3d} | {:5d}/{:5d} batches | '
                  'lr {:02.2f} | ms/batch {:5.2f} | '
                  'loss {:5.2f} | ppl {:8.2f}'.format(
                    epoch, batch, nbatches // bptt, scheduler.get_lr()[0],
                    elapsed * 1000 / log_interval,
                    cur_loss, math.exp(cur_loss)))
            total_loss = 0
            start_time = time.time()

def evaluate(eval_model, data_source):
    eval_model.eval() # 평가 모드로 전환
    total_loss = 0.
    ntokens = len(vocab)
    # 스크립트 실행 시간을 짧게 유지하기 위해 50 배치만 평가합니다.
    nbatches = min(50 * bptt, data_source.size(0) - 1)
    with torch.no_grad():
        for i in range(0, nbatches, bptt):
            data, targets = get_batch(data_source, i)
            output = eval_model(data).local_value()
            output_flat = output.view(-1, ntokens)
            # 타겟을 파이프라인 출력이 있는
            # 장치로 옮겨야합니다.
            total_loss += len(data) * criterion(output_flat, targets.cuda(1)).item()
    return total_loss / (len(data_source) - 1)
```

에폭을 반복합니다. 만약 검증 오차(validation loss)가 지금까지 관찰한 것 중 최적이라면
모델을 저장합니다. 각 에폭 이후에 학습률을 조절합니다.

```python
best_val_loss = float("inf")
epochs = 3 # 에폭 수
best_model = None

for epoch in range(1, epochs + 1):
    epoch_start_time = time.time()
    train()
    val_loss = evaluate(model, val_data)
    print('-' * 89)
    print('| end of epoch {:3d} | time: {:5.2f}s | valid loss {:5.2f} | '
          'valid ppl {:8.2f}'.format(epoch, (time.time() - epoch_start_time),
                                     val_loss, math.exp(val_loss)))
    print('-' * 89)

    if val_loss < best_val_loss:
        best_val_loss = val_loss
        best_model = model

    scheduler.step()
```

평가 데이터셋으로 모델 평가하기
-------------------------------

평가 데이터셋에서의 결과를 확인하기 위해 최고의 모델을 적용합니다.

```python
test_loss = evaluate(best_model, test_data)
print('=' * 89)
print('| End of training | test loss {:5.2f} | test ppl {:8.2f}'.format(
    test_loss, math.exp(test_loss)))
print('=' * 89)
```


# Image Style Transferの実装

--------------------------------
Gatys, Ecker, and Bethge(2016)の画像スタイル変換を実装する。

【参考文献】
L. A. Gatys, A. S. Ecker, and M. Bethge,
Image style transfer using convolutional neural networks,
In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages, 2414-2423, 2016.

```python
import numpy as np
import pandas as pd
import tensorflow as tf
```

```python
CONTENT_FILE = '/home/ishiyama/image_style_transfer/image/input/test_input_01.JPG'
STYLE_FILE = '/home/ishiyama/image_style_transfer/image/style/test_style_01.jpg'
```

```python
class Image(np.ndarray):

    """画像を扱うためのnumpy.ndarray

    XXX: 実装が大変なので一旦保留
         画像の形状の情報を属性で取り出せるようにしたい
    """

    DATA_FORMAT_CHAR = {
        'BATCH': 'N',
        'HEIGHT': 'H',
        'WIDTH': 'W',
        'CHANNEL': 'C'
    }

    def __new__(subtype,
                shape,
                dtype=float,
                buffer=None,
                offset=0,
                strides=None,
                order=None,
                data_format='NHWC'):

        super(__class__, self).__new__(subtype, shape, dtype, buffer, offset, strides, order)

        self.data_format = data_format
        num_batch, num_height, num_width, num_channel = self._get_image_shape(data_format=data_format)
        self.num_batch = num_batch
        self.num_height = num_height
        self.num_width = num_width
        self.num_channel = num_channel

    def _get_image_shape(self, data_format):
        _image_shape = self.shape
        idx_batch = self._get_index_data_format(data_format=data_format, data_type='BATCH'),
        idx_height = self._get_index_data_format(data_format=data_format, data_type='HEIGHT')
        idx_width = self._get_index_data_format(data_format=data_format, data_type='WIDTH')
        idx_channel = self._get_index_data_format(data_format=data_format, data_type='CHANNEL')
        reordered_image_shape = (_image_shape[idx_batch],
                                 _image_shape[idx_height],
                                 _image_shape[idx_width],
                                 _image_shape[idx_channel])
        return reordered_image_shape

    def _get_index_data_format(self, data_format, data_type):
        idx = data_format.find(__class__.DATA_FORMAT_CHAR[data_type])
        if idx == -1:
            raise ValueError('data type "{}" is not available.'.format(data_type))

        return idx

    @classmethod
    def reshape(self, *args):
        self = __class__(super(__class__, self).reshape(args))

```

```python
def read_images_as_jpeg(file):

    """
    JPEG Image Reader

    This function reads the content and style images as JPEG format.
    These image data must be square for now, different height and
    width will be able to supplied for future.

    Args:
        file : str. A path of the image file.

    Returns:
        A tuple. Each Elements are Tensor object of the read images.
    """

    filename_queue = tf.train.string_input_producer([file])
    reader = tf.WholeFileReader()
    key, value = reader.read(filename_queue)
    image = tf.image.decode_jpeg(value)

    # Read image is a Tensor object which tf.nn.conv2d cannot handle,
    # so convert it to numpy.ndarray.
    sess = tf.Session()
    sess.run(tf.global_variables_initializer())
    tf.train.start_queue_runners(sess)
    # Returned array's shape will be (height, width, channel).
    image_array_hwc = sess.run(image)
    new_shape = [1]
    new_shape.extend(image_array_hwc.shape)

    # return image_array_chw
    return image_array_hwc.reshape(new_shape)
```

```python
image = read_images_as_jpeg(file=CONTENT_FILE)
image.shape
```

    (1, 1200, 1600, 3)

# VGGを実装する
------------------------
画像の特徴量を抽出するアルゴリズムにはSimonyan and Zisserman(2015)で提案されたCNN(VGG19)の畳込み層とプーリング層が使われている。
ここでは、「TensorFlowで学ぶディープラーニング入門」の多層CNNの実装を参考にVGG19を構築する。

【参考文献】
K. Simonyan and A. Zisserman, Very Deep Convolutional Networks For Large-Scale Image Recognition, arXiv: 1409.1556v6, 2015
中井悦司, TensorFlowで学ぶディープラーニング入門〜畳み込みニューラルネットワーク徹底解説, マイナビ出版, 2016

### 畳み込み層を実装する

```python
import tensorflow as tf
import numpy as np
from tensorflow.examples.tutorials.mnist import input_data

def calculate_convolutional_layer(x, filter_height, filter_width, output_channels):

    """
    Executeing a convolutional layer task.

    Args:
        x                     : An image data.
        filter_height   (int) : A height of each filters.
        filter_width    (int) : A width of each filters.
        output_channels (int) : A number of channels which must be outputed.

    Returns:
        An activation of an convolutional layer.
    """

    if ((not isinstance(filter_height, int))
        or (not isinstance(filter_width, int))
        or (not isinstance(output_channels, int))):
        raise TypeError('"filter_height" and "filter_width" and "output_channels" '
                        'must be integer.')

    # TODO: 入力画像の縦、横、チャンネル数を属性で取得できるようにする
    # 例） input_channels = x.num_channels
    input_channels = int(x.shape[-1])
    W = tf.Variable(
        tf.truncated_normal(
            shape=[filter_height,
                   filter_width,
                   input_channels,
                   output_channels],
            stddev=0.1
        )
    )
    h = tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')
    b = tf.Variable(tf.constant(0.1, shape=[output_channels]))
    convoluted_image = tf.nn.relu(h + b)

    return convoluted_image
```

```python
x = tf.placeholder(tf.float32, [1, 1200, 1600, 3])
test_model = calculate_convolutional_layer(
    x=x,
    filter_height=3,
    filter_width=3,
    output_channels=1
)
sess = tf.InteractiveSession()
# tf.Session()だと、sess.runで返ってくる行列の要素がすべて0だった。
# TODO: Sessionメソッド と InteractiveSessionメソッドの違いを調べる
# sess = tf.Session()
sess.run(tf.global_variables_initializer())
test_result = sess.run(test_model, feed_dict={x: image})
```

```python
test_result.shape
```

    (1, 1200, 1600, 1)

```python
test_result
```

    array([[[[ 112.70514679],
             [ 128.16131592],
             [ 128.5138092 ],
             ...,
             [  91.39342499],
             [  92.1344223 ],
             [  75.42243958]],

            [[ 137.54698181],
             [ 134.74995422],
             [ 132.69534302],
             ...,
             [  97.32675934],
             [  98.38056183],
             [  83.37560272]],

            [[ 135.02359009],
             [ 131.20184326],
             [ 129.7742157 ],
             ...,
             [  97.85587311],
             [  98.76654816],
             [  82.42256927]],

            ...,
            [[ 133.74493408],
             [ 130.07633972],
             [ 133.27828979],
             ...,
             [ 135.64941406],
             [ 135.94631958],
             [ 114.93429565]],

            [[ 135.07643127],
             [ 133.87402344],
             [ 136.48208618],
             ...,
             [ 135.72117615],
             [ 135.72525024],
             [ 114.62847137]],

            [[ 113.70575714],
             [  80.11441803],
             [  79.86685944],
             ...,
             [  79.5676651 ],
             [  79.77925873],
             [  50.97554779]]]], dtype=float32)

### Maxプーリング層を実装する

```python
def calculate_max_pooling_layer(x, ksize, strides):

    """Wrapper function of tf.nn.max_pool.

    Args:
        x       : A Tensor produced by calculate_convolutional_layer.
        ksize   : A list of ints that has length >= 4. The size of
                  the window for each dimension of the input tensor.
        strides : A list of ints that has length >= 4. The stride
                  of the sliding window for each dimension of the
                  input tensor.

    Returns:
        A pooled image.
    """

    pooled_image = tf.nn.max_pool(x, ksize=ksize, strides=strides, padding='SAME')

    return pooled_image
```

### 畳込みとプーリング処理の途中経過を保持するクラスを実装する

```python
class ConvNetProgressHolder(object):

    """Holder of convoluted images and pooled image.

    This class is used like the struct of C language.
    This has no methods.

    Attributes:
        input_data (Tensor) : An image that is applied to convolution and pooling.
        conv (list)         : The list of convoluted images, each images are Tensor objects.
        pool (Tensor)       : A image that is pooled after convolutional layer.
    """

    def __init__(self):

        self.input_data = None
        self.conv = []
        self.pool = None

# FILTER_CONF = {
#     'height': 3,
#     'width': 3,
#     'channels': 1,
#     'num': 1
# }

FILTER_CONF = {
    'height': 3,
    'width': 3
}

def apply_vgg_network_unit(x, channels, num_conv):

    """Apply VGG Network From a Convolutional Layer to Max Pooling Layer.

    Table 1 of Simonyan and Zisserman(2015) is separated by 5 parts,
    each parts is from an input data or a pooled data at previous part
    to a maxpool.
    This function provides to apply a that part.
    This will apply recursively.

    Args:
        x (Tensor)     : An input data or A Max pooled data returned by this function.
        channels (int) : A number of channels described at Table 1 of
                         Simonyan and Zisserman(2015).
        num_conv (int) : A number of applying covolutional layers.
                         See Simonyan and Zisserman(2015) for detail.

    Returns:
        A ConvNetProgressHolder object.
    """

    if num_conv < 2:
        raise ValueError('num_conv must be >= 2.')

    conv_holder = ConvNetProgressHolder()
    conv_holder.input_data = x

    conv = calculate_convolutional_layer(
        x=conv_holder.input_data,
        filter_height=FILTER_CONF['height'],

        filter_width=FILTER_CONF['width'],
        output_channels=channels
    )
    conv_holder.conv.append(conv)

    for i in range(1, num_conv):
        conv = calculate_convolutional_layer(
            x=conv_holder.conv[i - 1],
            filter_height=FILTER_CONF['height'],
            filter_width=FILTER_CONF['width'],
            output_channels=channels
        )
        conv_holder.conv.append(conv)

    conv_holder.pool = calculate_max_pooling_layer(
        x=conv_holder.conv[i - 1],
        ksize=[1, 2, 2, 1],
        strides=[1, 2, 2, 1]
    )

    return conv_holder

```

### VGGの畳込みとプーリング層を構築する

VGGの論文に従い、複数回の畳込み処理と1回のMAXプーリング処理を1セットとして、それを5回繰り返す。
今回実装する画像変換アルゴリズムでは、この処理の途中経過を使うため、使いたい部分をリストで１つにまとめてsess.run()に投げる。

#### ※処理が重たいので実行に注意すること。
実行環境は下記の通り。
今回はテストで実行するため、チャンネル数を少なくして実行しているが、
それでも実行するとメモリをすべて使うため、20〜30秒ほど
フリーズしたかのように動作が重くなる。

OS： Linux Mint 17.1 Rebecca
CPU： Intel(R) Core(TM)2 Duo CPU L9400 @ 1.86GHz
メモリ: 4GB

```python
x = tf.placeholder(tf.float32, [1, 1200, 1600, 3])
unit1 = apply_vgg_network_unit(x=x, channels=2, num_conv=2)
unit2 = apply_vgg_network_unit(x=unit1.pool, channels=4, num_conv=2)
unit3 = apply_vgg_network_unit(x=unit2.pool, channels=8, num_conv=4)
unit4 = apply_vgg_network_unit(x=unit3.pool, channels=16, num_conv=4)
unit5 = apply_vgg_network_unit(x=unit4.pool, channels=32, num_conv=4)
sess = tf.InteractiveSession()
# sess = tf.Session()
sess.run(tf.global_variables_initializer())

# 使いたい過程をリストでまとめてsess.runに投げると特徴量抽出の
# 途中経過を取り出せる
result_unit2_conv, result_unit5_conv, result_unit5_pool = sess.run(
    [unit2.conv[1], unit5.conv[2], unit5.pool],
    feed_dict={x: image}
)
#result_list = sess.run(
#    [unit2.conv[1], unit5.conv[2], unit5.pool],
#    feed_dict={x: image}
#)
```

```python
flattened_result_list = [flatten(res) for res in result_list]
flattened_result_list
```

    [array([[ 0.        ,  0.19515181,  0.28172809, ...,  0.10160407,
              0.05559425,  0.09484372],
            [ 0.        ,  0.37957731,  0.29346544, ...,  0.11967082,
              0.08694198,  0.10553464],
            [ 0.        ,  0.        ,  0.        , ...,  0.11967082,
              0.08694198,  0.10553464],
            ...,
            [ 0.        ,  0.42349544,  0.78957283, ...,  0.11967082,
              0.08694198,  0.10553464],
            [ 0.0559251 ,  0.40405619,  0.29025877, ...,  0.11967082,
              0.08694198,  0.10553464],
            [ 0.        ,  0.        ,  0.        , ...,  0.10149642,
              0.07735725,  0.10091563]], dtype=float32),
     array([[ 0.33444834,  0.0888759 ,  0.        , ...,  0.        ,
              0.07049304,  0.2961235 ],
            [ 0.33314872,  0.16404781,  0.        , ...,  0.        ,
              0.15296367,  0.4274711 ],
            [ 0.17283794,  0.10241053,  0.        , ...,  0.03096017,
              0.23394999,  0.54884279],
            ...,
            [ 0.30813834,  0.13530649,  0.        , ...,  0.0519722 ,
              0.23881677,  0.6213997 ],
            [ 0.22297519,  0.11944702,  0.        , ...,  0.        ,
              0.20090596,  0.753025  ],
            [ 0.00371078,  0.21586302,  0.        , ...,  0.3994118 ,
              0.18777519,  0.42602578]], dtype=float32),
     array([[ 0.33444834,  0.16404781,  0.        , ...,  0.38993728,
              0.15296367,  0.4274711 ],
            [ 0.19038874,  0.10241053,  0.        , ...,  0.5832026 ,
              0.28920072,  0.54884279],
            [ 0.20619097,  0.06473988,  0.        , ...,  0.59744376,
              0.28771335,  0.57466584],
            ...,
            [ 0.21057001,  0.08940767,  0.        , ...,  0.58340442,
              0.28275216,  0.54860938],
            [ 0.30813834,  0.13530649,  0.        , ...,  0.63199806,
              0.23881677,  0.753025  ],
            [ 0.00371078,  0.21586302,  0.        , ...,  0.76431578,
              0.18777519,  0.42602578]], dtype=float32)]

```python
result_unit2_conv.shape
```

    (1, 600, 800, 4)

```python
result_unit5_conv.shape
```

    (1, 75, 100, 32)

```python
result_unit5_conv
```

    array([[[[ 0.        ,  0.        ,  0.34665611, ...,  0.01044758,
               0.00896682,  0.16145825],
             [ 0.        ,  0.03078408,  0.539891  , ...,  0.11569723,
               0.05869087,  0.        ],
             [ 0.        ,  0.13931246,  0.61979979, ...,  0.13327536,
               0.09154148,  0.        ],
             ...,
             [ 0.        ,  0.07932027,  0.59292465, ...,  0.1436328 ,
               0.09119272,  0.        ],
             [ 0.        ,  0.26379913,  0.56585687, ...,  0.12962864,
               0.04334351,  0.        ],
             [ 0.        ,  0.35073641,  0.38637096, ...,  0.22623608,
               0.        ,  0.        ]],

            [[ 0.        ,  0.        ,  0.31645674, ...,  0.        ,
               0.        ,  0.20772454],
             [ 0.        ,  0.09629722,  0.49515995, ...,  0.        ,
               0.        ,  0.        ],
             [ 0.        ,  0.36474115,  0.52087045, ...,  0.        ,
               0.        ,  0.        ],
             ...,
             [ 0.        ,  0.30093169,  0.59148234, ...,  0.        ,
               0.01584271,  0.        ],
             [ 0.        ,  0.52672178,  0.59828991, ...,  0.        ,
               0.02490016,  0.        ],
             [ 0.        ,  0.50753939,  0.41637349, ...,  0.        ,
               0.        ,  0.        ]],

            [[ 0.        ,  0.        ,  0.24866414, ...,  0.        ,
               0.0337012 ,  0.38094741],
             [ 0.        ,  0.27722761,  0.34605095, ...,  0.        ,
               0.        ,  0.        ],
             [ 0.        ,  0.52632284,  0.39138198, ...,  0.        ,
               0.03279281,  0.        ],
             ...,
             [ 0.        ,  0.47802359,  0.42127004, ...,  0.        ,
               0.00663704,  0.        ],
             [ 0.        ,  0.62312078,  0.49327278, ...,  0.        ,
               0.06904371,  0.        ],
             [ 0.        ,  0.38692412,  0.31898922, ...,  0.00153147,
               0.        ,  0.        ]],

            ...,
            [[ 0.0398218 ,  0.00273792,  0.21777621, ...,  0.        ,
               0.09281436,  0.31107682],
             [ 0.        ,  0.2537224 ,  0.34750521, ...,  0.        ,
               0.        ,  0.        ],
             [ 0.        ,  0.49294737,  0.49836549, ...,  0.        ,
               0.00452305,  0.01326401],
             ...,
             [ 0.        ,  0.49006444,  0.53520775, ...,  0.        ,
               0.        ,  0.        ],
             [ 0.        ,  0.64487153,  0.59590834, ...,  0.        ,
               0.04557376,  0.        ],
             [ 0.        ,  0.34724972,  0.39819378, ...,  0.00555965,
               0.        ,  0.        ]],

            [[ 0.12522115,  0.        ,  0.31568015, ...,  0.06195009,
               0.00610594,  0.48040614],
             [ 0.        ,  0.2129516 ,  0.47588873, ...,  0.        ,
               0.        ,  0.12605849],
             [ 0.        ,  0.40521276,  0.5584563 , ...,  0.        ,
               0.        ,  0.19377044],
             ...,
             [ 0.        ,  0.36936945,  0.61476296, ...,  0.        ,
               0.03699226,  0.09776261],
             [ 0.        ,  0.45121223,  0.70541602, ...,  0.        ,
               0.11093734,  0.        ],
             [ 0.        ,  0.31516796,  0.41281551, ...,  0.        ,
               0.        ,  0.        ]],

            [[ 0.12536027,  0.53227723,  0.18502754, ...,  0.        ,
               0.08350039,  0.15595068],
             [ 0.        ,  0.59961879,  0.18500343, ...,  0.        ,
               0.04414895,  0.01673431],
             [ 0.        ,  0.78678268,  0.20948535, ...,  0.        ,
               0.07499287,  0.21903834],
             ...,
             [ 0.        ,  0.66935784,  0.30286148, ...,  0.        ,
               0.05717379,  0.12638204],
             [ 0.        ,  0.63711232,  0.32440916, ...,  0.        ,
               0.13823158,  0.07364799],
             [ 0.        ,  0.24815798,  0.16843089, ...,  0.        ,
               0.        ,  0.14192262]]]], dtype=float32)

# 画像を合成する処理を作る
VGGによってスタイルを変換する画像とスタイルのリファレンスとなる画像の特徴量を抽出できるようになった。
今度はこの特徴量を用いて画像を実際に合成する部分を作っていく。

このアルゴリズムは変換したいスタイルに寄せるための損失関数$L_{style}$と
変換する画像の内容に寄せるための損失関数$L_{content}$をそれぞれ計算して、
\begin{equation} L_{total} = \alpha L_{content} + \beta L_{style} \end{equation}
を最適化することで画像を合成する。

----------------------------------------
# 変換したいスタイルに寄せるための損失関数$L_{style}$を実装する
変換したいスタイルに寄せるための損失関数$L_{style}$は次で計算する：
\begin{align}
    L_{style} &= \sum_{l=1}^{L} w_{l} E_{l} \\
        E_{l} &= \frac{1}{4N_{l}^{2}M_{l}^{2}} \sum_{i=1}^{N_{l}} \sum_{j=1}^{N_{l}} (G_{ij}^{l} - A_{ij}^{l})^{2}
\end{align}
ただし、$N_{l}$は層$l$の畳込みフィルターの数、
$M_{l}$は層$l$の畳込みフィルターのサイズで$M_{l} = \text{フィルターの高さ} \times \text{フィルターの幅}$、
$A_{ij}^{l}$はスタイル画像から抽出した特徴量で計算したグラム行列$A^{l}$の$(i, j)$成分、
同様に、$G_{ij}^{l}$は合成後の画像から抽出した特徴量で計算したグラム行列$G^{l}$の$(i, j)$成分である。
$G_{ij}^{l}$は次で計算する。
$G^{l}$は$(N_{l}, N_{l})$型行列で
\begin{equation}
    G_{ij}^{l} = \sum_{k=1}^{M_{l}} F_{ik}^{l}F_{jk}^{l}.
\end{equation}

```python
result_unit5_conv.shape
```

    (1, 75, 100, 32)

```python
def convert_nhwc_to_nchw(image):
    return image.transpose([0, 3, 1, 2])
```

```python
converted = convert_nhwc_to_nchw(result_unit5_conv)
converted.shape
```

    (1, 32, 75, 100)

```python
def flatten(image):
    num_batch, num_channel, num_height, num_width = image.shape
    if num_batch != 1:
        raise ValueError('Not assumed batch size has been ocurred.')
    return image.reshape([num_channel, num_height * num_width])
```

#### flattenのテスト

```python
test_data1 = np.arange(1 * 3 * 4 * 2).reshape([1, 3, 4, 2])
test_data1
```

    array([[[[ 0,  1],
             [ 2,  3],
             [ 4,  5],
             [ 6,  7]],

            [[ 8,  9],
             [10, 11],
             [12, 13],
             [14, 15]],

            [[16, 17],
             [18, 19],
             [20, 21],
             [22, 23]]]])

```python
flatten(test_data1)
```

    array([[ 0,  1,  2,  3,  4,  5,  6,  7],
           [ 8,  9, 10, 11, 12, 13, 14, 15],
           [16, 17, 18, 19, 20, 21, 22, 23]])

```python
test_data2 = np.arange(2 * 3 * 4 * 2).reshape([2, 3, 4, 2])
test_data2
```

    array([[[[ 0,  1],
             [ 2,  3],
             [ 4,  5],
             [ 6,  7]],

            [[ 8,  9],
             [10, 11],
             [12, 13],
             [14, 15]],

            [[16, 17],
             [18, 19],
             [20, 21],
             [22, 23]]],

           [[[24, 25],
             [26, 27],
             [28, 29],
             [30, 31]],

            [[32, 33],
             [34, 35],
             [36, 37],
             [38, 39]],

            [[40, 41],
             [42, 43],
             [44, 45],
             [46, 47]]]])

```python
flatten(test_data2)
```

```python
# グラム行列を計算する
def calculate_gram_matrix(x):
    return np.dot(x, x.T)
```

```python
flattened = flatten(result_unit5_conv)
gram_matrix = calculate_gram_matrix(flattened)
gram_matrix
```

    array([[  3.29813063e-02,   6.68354481e-02,   7.13971704e-02, ...,
              2.21395370e-04,   0.00000000e+00,   1.77914575e-02],
           [  6.68354481e-02,   2.83326566e-01,   9.90822017e-02, ...,
              1.52218690e-05,   0.00000000e+00,   7.55421817e-02],
           [  7.13971704e-02,   9.90822017e-02,   4.68583584e+00, ...,
              2.07738742e-01,   0.00000000e+00,   2.62595937e-02],
           ...,
           [  2.21395370e-04,   1.52218690e-05,   2.07738742e-01, ...,
              5.73648885e-02,   0.00000000e+00,   0.00000000e+00],
           [  0.00000000e+00,   0.00000000e+00,   0.00000000e+00, ...,
              0.00000000e+00,   0.00000000e+00,   0.00000000e+00],
           [  1.77914575e-02,   7.55421817e-02,   2.62595937e-02, ...,
              0.00000000e+00,   0.00000000e+00,   2.01420300e-02]], dtype=float32)

```python
gram_matrix.shape
```

    (3200, 3200)

```python

```