# Information

Python implementation of the learning framework proposed in [OrthoNet: Multilayer Network Data Clustering](https://arxiv.org/abs/1811.00821) published at *IEEE Transactions on Signal and Information Processing over Networks* (2020).

### Authors

- [Mireille El Gheche](http://www-syscom.univ-mlv.fr/~elgheche/)
- [Giovanni Chierchia](https://perso.esiee.fr/~chierchg/)
- Pascal Frossard

### License

This notebook is released under the [CeCILL-B](https://spdx.org/licenses/CECILL-B.html) license, which is fully compatible with BSD-like licenses (BSD, X11, MIT) with a strong attribution requirement.

### How to cite

- M. El Gheche, G. Chierchia, and P. Frossard. "OrthoNet: Multilayer Network Data Clustering," IEEE Transactions on Signal and Information Processing over Networks, Vol. 6, No. 1, pp. 13-23, December 2020.

```
@Article{ElGheche2019_tsipn,
  author  = {El Gheche, M. and Chierchia, G. and Frossard, P.},
  title   = {OrthoNet: Multilayer Network Data Clustering},
  journal = {IEEE Transactions on Signal and Information Processing over Networks},
  volume  = {6},
  number  = {1},
  pages   = {13--23},
  year    = {2020},
  month   = dec,
}
```

```python
import math, random
import numpy as np
import scipy

import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.optim import Adam

from utils import *
from plots import *
```

# Datasets & Graphs

In this notebook, we consider a graph $\mathcal{G}(V,E)$ consisting of a set $V$ of vertices, and a set $E$ of edges with non-negative **similarity** weights $w_{ij}$, where $W = [w_{ij}]_{i,j} \in \mathbb{R}^{N\times N}$ is the weighted adjacency matrix with $w_{i,i} = 0$ for all $i$. The degree of a vertex $i$ is defined as the sum of weights of all the edges incident to $i$. The degree matrix $D$ is then defined as the following diagonal matrix

$$
D = \begin{bmatrix}
\sum_{j=1}^N w_{1j} & 0 & \dots & 0 \\
0 & \sum_{j=1}^N w_{2j} & \dots & 0 \\
\vdots & \vdots & \ddots & \vdots \\
0 & 0 & \dots & \sum_{j=1}^N w_{Nj}
\end{bmatrix}.
$$

Based on $W$ and $D$, the Laplacian matrix of $\mathcal{G}$ is

$$
L = D - W.
$$

We further assume that each vertex of the graph $\mathcal{G}$ is associated with $M$-dimensional features, and we denote such data as

$$
X =
\begin{bmatrix}
x_1^\top\\
\vdots\\
x_N^\top
\end{bmatrix}
\in \mathbb{R}^{N\times M}.
$$

```python
sets = generate_data()
```

```python
show_datasets(sets)
```

```python
show_graphs(sets)
```

```python
results = OrderedDict()

def add_result(experiment_name, set_name, embedding, net=None):
    e = results.setdefault(experiment_name, {})
    r = e.setdefault(set_name, {})
    e[set_name]['embedding'] = embedding
    e[set_name]['net'] = net
```

# Spectral clustering

The idea behind spectral clustering is to embed the nodes of a graph as points of a low-dimensional space, where they can be trivially clustered. More specifically, for a $K$-way clustering, one aims at finding the semi-orthogonal matrix $U = \left[ u_1 \,|\dots|\, u_N \right]^\top$ such that its rows $u_n \in \mathbb{R}^K$ minimize the pairwise distances on the graph edges, namely

$$
\operatorname*{minimize}_{U\in\mathbb{R}^{N\times K}}\;\;
\operatorname{Tr}(U^\top LU) := \sum_{i=1}^N\sum_{j=1}^N w_{i,j} \|u_i - u_j\|^2
\quad{\rm s.t.}\quad
U^\top U = {\bf I_{K\times K}}.
$$

The solution to the above problem is the matrix formed by the eigenvectors associated to the $K$ smallest eigenvalues of the Laplacian matrix $L$. Each row of the solution $\bar{U}$ is a point in the low-dimensional space $\mathbb{R}^K$ representing a graph vertex. Strongly-connected vertices are mapped to close vectors in $\mathbb{R}^K$. One can then apply a $K$-mean algorithm on the rows of $\bar{U}$, so that $K$ clusters are formed by grouping together the vertices that are the most strongly connected by the graph.

```python
def laplacian(graph, normed=False):
    W = graph.copy()
    np.fill_diagonal(W,0)
    d = np.sum(W, axis=0)
    if normed:
        d = np.sqrt(d)
        D = np.diag(1/d)
        I = np.eye(W.shape[0])
        L = I - D.dot(W).dot(D)
    else:
        D = np.diag(d)
        I = np.eye(W.shape[0])
        L = D - W
    return L

def spectral_embedding(adjacency, n_clusters):
    L = laplacian(adjacency, normed=False)
    _, V = scipy.linalg.eigh(L)
    E = V[:,:K]
    return E
```

```python
for i, name in enumerate(sets):

    X = sets[name]["X"]
    y = sets[name]["y"]
    K = sets[name]["K"]
    G = sets[name]["G"]

    E = spectral_embedding(G, K)

    add_result('spectral', name, E)
```

```python
show_results(results, sets, 'spectral')
```

# OrthoNet

### Formulation

The proposed node feature embedding relies on both the node attributes $X$ and the Laplacian matrix $L=D-W$ of a given graph. The goal is to estimate a mapping $f_\theta$ that represents the features of strongly connected nodes as close points in the latent space. That is, for large similarities $w_{ij}$, we want the distance between $f_\theta(x_i)$ and $f_\theta(x_j)$ to be small, which amounts to minimizing

\begin{equation}
{\rm Tr}(Y_\theta^\top L Y_\theta) = \sum_{i=1}^N \sum_{j=1}^N w_{ij} \|f_\theta(x_i) - f_\theta(x_j)\|^2,
\end{equation}

where $Y_\theta = f_\theta(X)$. This objective is however not sufficient alone for learning an embedding that would result in effective clustering, as the sum of pairwise distances is trivially minimized by mapping all points to the same output vector. To avoid trivial solutions, we can borrow from spectral clustering the idea that the embedded points must be orthogonal to each other, yielding

\begin{equation}
\operatorname*{minimize}_{\theta\in\mathbb{R}^B}\;\;  {\rm Tr}(Y_\theta^\top L Y_\theta)
\quad{\rm s.t.} \quad
Y_\theta^\top Y_\theta = {\bf I_{K\times K}}.
\end{equation}

The matrix $Y_\theta$ represents the graph nodes as vectors of the latent space $\mathbb{R}^K$. Intuitively, nodes within the same cluster should be mapped to close vectors, while nodes from different clusters should be spaced out from each other, so that the latent space can be easily clustered. By optimizing the sum of pairwise distances over the orthogonality constraint, the rows of $Y_\theta$ tend to be split into $K$ clusters, which are formed by grouping together the more strongly connected vertexes in the graph. This is indeed similar to spectral clustering, which uses the spectrum of the Laplacian matrix to perform dimensionality reduction before clustering.

### Learning algorithm

We propose to solve the above optimization problem through gradient descent. The main difficulty arises from the orthogonality constraint, since it is enforced on the mapping to be estimated, rather than the optimization parameters, ruling out standard techniques based on alternating optimization. To circumvent this issue, we propose to use the fact that the orthogonality constraint can be enforced implicitly by using the upper triangular matrix $R_\theta^\top \in\mathbb{R}^{K\times K}$ of the QR decomposition of $Y_\theta$, defined as

\begin{equation}
Y_\theta = Q_\theta R_\theta^\top,
\end{equation}

with $Q_\theta\in\mathbb{R}^{N\times K}$ being a semi-orthogonal matrix. Indeed, when $Y_\theta$ is full rank, or equivalently $Y_\theta^\top Y_\theta$ is positive definite, its QR decomposition is unique, and $R_\theta$ is equal to the lower triangular factor of the Cholesky decomposition

\begin{equation}
Y_\theta^\top Y_\theta = R_\theta R_\theta^\top.
\end{equation}

Therefore, the semi-orthogonal factor $Q_\theta$ can be extracted from the matrix $Y_\theta$ by multiplying it with $R_\theta^{-\top}$, namely

\begin{equation}
Q_\theta = Y_\theta R_\theta^{-\top} \quad\Rightarrow\quad Q_\theta^\top Q_\theta =  {\bf I_{K\times K}}.
\end{equation}

This consideration allows us to rewrite our problem as

$$
\operatorname*{minimize}_{\theta\in\mathbb{R}^B}\;\;  J(\theta) := {\rm Tr}(R_\theta^{-1} Y_\theta^\top L Y_\theta R_\theta^{-\top})
\quad \operatorname{s.t.} \quad
R_\theta = {\rm Cholesky}(Y_\theta^\top Y_\theta).
$$

In the above reformulation, the term $Y_\theta$ is no longer a semi-orthogonal matrix. The constraint is now enforced implicitly through the factor $R_\theta$ derived from the Cholesky decomposition of $Y_\theta^\top Y_\theta$, which ensures that the product $Y_\theta R_\theta^{-\top}$ is a semi-orthogonal matrix of $\mathbb{R}^{N\times K}$. All operations involved in this reformulation are differentiable, provided that the mapping $f_\theta$ is differentiable, and thus the problem can be solved by gradient descent.

```python
def build_net(input_dim, hidden_dim, output_dim):
    random.seed(42)
    np.random.seed(42)
    torch.manual_seed(42)

    net = nn.Sequential(
        nn.Linear(input_dim, hidden_dim),
        nn.ReLU(),
        nn.Linear(hidden_dim, hidden_dim//2),
        nn.ReLU(),
        nn.Linear(hidden_dim//2, hidden_dim//3),
        nn.ReLU(),
        nn.Linear(hidden_dim//3, hidden_dim//4),
        nn.ReLU(),
        nn.Linear(hidden_dim//4, output_dim),
    )

    return net
```

```python
class OrthoNet:

    def __init__(self, model, epochs, lr):
        self.model  = model
        self.epochs = epochs
        self.lr     = lr

    def fit(self, X, graph):
        self.setup(graph)
        self.train(X)

    def setup(self, graph):
        L = laplacian(graph)
        self.L = torch.from_numpy(L.astype(np.float32))
        self.m = math.sqrt(len(L))
        self.K = self.model[-1].out_features

    def loss(self, inputs):
        z = self.model(inputs)
        M = z.t().mm(z)
        R = torch.cholesky(M, upper=True)
        self.Q = self.m * torch.inverse(R)
        y = torch.mm(z, self.Q)
        loss = y.t().mm(self.L).mm(y).trace()
        return loss

    def train(self, X):
        inputs    = torch.from_numpy(X.astype(np.float32))
        optimizer = torch.optim.Adam(self.model.parameters(), lr=self.lr, amsgrad=True)
        self.history = []
        self.model.train()
        for epoch in range(self.epochs+1):
            loss = self.loss(inputs)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()
            self.history.append(loss.item())
            if epoch == 0 or epoch % 500 == 0:
                print('Epoch {:4d}/{:d}: {:2.2f}'.format(epoch, self.epochs, loss.item()))
        self.model.eval()

    def predict(self, X, orthogonal=True):
        inputs  = torch.from_numpy(X.astype(np.float32))
        outputs = self.model(inputs)
        if orthogonal:
            outputs = torch.mm(outputs, self.Q)
        return outputs.data.numpy()
```

### Test

```python
for name in sets:

    X = sets[name]["X"]
    y = sets[name]["y"]
    K = sets[name]["K"]
    G = sets[name]["G"]

    model = build_net(X.shape[1], 250, K)

    net = OrthoNet(model, epochs=1000, lr=0.001)
    net.fit(X, G)
    E = net.predict(X)

    add_result('orthonet', name, E, net)
```

    Epoch    0/1000: 47.70
    Epoch  500/1000: 0.81
    Epoch 1000/1000: 0.73
    Epoch    0/1000: 158.99
    Epoch  500/1000: 4.47
    Epoch 1000/1000: 4.44
    Epoch    0/1000: 145.89
    Epoch  500/1000: 12.72
    Epoch 1000/1000: 10.07

```python
show_results(results, sets, 'orthonet')
```

```python

```


# Neural Network with TensorFlow

The goal of this tutorial is to show how to use python TensorFlow API to train and evaluate a simple neural network. TensorFlow is an open source software library developed by Google for the purposes of conducting machine learning and deep neural networks research.

## Tutorial content
We will cover the following topics in this tutorial:
* Installing TensorFlow
* Loading MNIST dataset
* TensorFlow Mechanism
* Building a Neural Network Model
* Building a Convolutional Net Model

## Installing TensorFlow

Please find the instruction to install TensorFlow in your environment. GPU enabled installation requires several steps for setup. I recommend you to install a CPU only version for this tutorial.
https://www.tensorflow.org/versions/r0.11/get_started/os_setup.html

```python
import matplotlib
matplotlib.use("svg")
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
matplotlib.rcParams['figure.figsize'] = (10.0, 5.0)

from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
```

    /Users/odatakuma/anaconda/envs/tensorflow/lib/python2.7/site-packages/matplotlib/__init__.py:1357: UserWarning:  This call to matplotlib.use() has no effect
    because the backend has already been chosen;
    matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
    or matplotlib.backends is imported for the first time.

      warnings.warn(_use_error_msg)

## Loading MNIST Data
We will train our neural networks to classify handwritten digits form the popular MNIST dataset that has been constructed by Yann LeCun et al. MNIST dataset is available via tensorflow.

```python
mnist = input_data.read_data_sets("MNIST_data", one_hot=True)
```

    Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
    Extracting MNIST_data/train-images-idx3-ubyte.gz
    Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
    Extracting MNIST_data/train-labels-idx1-ubyte.gz
    Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
    Extracting MNIST_data/t10k-images-idx3-ubyte.gz
    Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
    Extracting MNIST_data/t10k-labels-idx1-ubyte.gz

The MNIST data consisits of three parts: 55,000 images for training data (mnist.train), 10,000 images for test data (mnist.test), and 5,000 images for validation data (mnist.validation). Each dataset contains 28x28 images flattened to 784 feature vectors and labels which representing correct digits as 10-d one-hot vectors. To get an idea what the images in MNIST dataset look like, let's visualize examples of the digits 0-9 after reshaping the 784 feature vectors into the original 28 x 28 pixel image via matplotlib's imshow function:

```python
fig, ax = plt.subplots(nrows=2, ncols=5, sharex=True, sharey=True)
ax = ax.flatten()
for  i in range(10):
    img = mnist.train.images[mnist.train.labels.dot(np.arange(10)) == i][0].reshape(28, 28)
    ax[i].imshow(img, cmap='Greys', interpolation='nearest')
ax[0].set_xticks([])
ax[0].set_yticks([])
plt.tight_layout()
plt.show()
```

## TensorFlow Mechanism
TensorFlow represents computations as graphs in which each node represents operation of tensors. A tensor is a multi-demensional array similar to numpy array.
The common usage for TensorFlow programs is to first create a computation graph and then launch it in a Session. A Session places the graph onto Devices, such as CMUs or GPUs. In general TensorFlow uses GPU, if you have one, for as many as operations as possible.
We will describe the basic flow of TensorFlow programs by building a single neuron classifier (softmax regression) for digits recognition.
* Define a network architecture
* Define a loss function
* Define an optimization method
* Train the model
* Evaluate the model

Softmax regression with the cross entropy loss is formulated as:
\begin{align}
z & = wx + b \\
\phi_j & = \frac{e^{z_j}}{\sum_{k} e^{-z_k}} \\
L & = \log \phi_y
\end{align}
where $x$: inputs, $y$:labels, $w$: weights, $b$:bias, $z$: logits, $\phi$:probability, $L$:loss

### Define a network architecture
We start building the computation graph by creating nodes for the input and target output classes.

```python
input_size = 784
output_size = 10
x = tf.placeholder(tf.float32, [None, input_size])
y = tf.placeholder(tf.float32, [None, output_size])
```

tf.placeholder is a value that we will input when we run TensorFlow. We represent x as a 2-D tensor shaping [None, 784], which we will feed MNIST images flattened into a 784-dimensional vector. We don't specify the data size since we will use the batch gradient descent method to speed up the training process. The lablels y will consist of a 2-D tensor shaping [None, 10], where each row is a one-hot 10-dimensional vector.

Machine learning model parameters are represented as tf.Variable, which can be used and modified by the computation. We define the weights W and biases b for our softmax regression model by tf.Variable. We can now implement logits $z$, or hypothesis function, as matrix multiplication of x and W plus the bias term.

```python
W = tf.Variable(tf.zeros([input_size, output_size]))
b = tf.Variable(tf.zeros([output_size]))
z = tf.matmul(x, W) + b
```

### Define a loss function
We select the cross-entropy as our loss function. Here tf.nn.softmax_cross_entropy_with_logits applies the softmax function on logits $z$ and sums cross-entropy across all classes. tf.reduce_mean takes the average over the instances.
$$
L = \frac{1}{m} \sum_{i=1}^m \log \phi^{(i)}_{y^{(i)}}
$$

```python
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(z, y))
```

### Define an optimazation method
We will use gradient descent method to optimize our loss function. tf.train.GradientDescentOptimizer(learning_rate).minimize(loss) computes a gradient based on the loss function and updates the weights (tf.Variable) we defined in the loss fucntion.

```python
learning_rate = 0.5
train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)
```

### Train the model
Now we have our model set up for trianing. Before launching it, we have to initialize the variables we defined.

```python
init = tf.initialize_all_variables()
sess = tf.Session()
sess.run(init)
```

We can now launch our model in a Session. Session holds states such as variables until being closed. We use batch gradient descent by feeding 100 batch training examples in each iteration. You can replace any tensor in your computation graph passing it to feed_dict.

```python
batch_size = 100
for _ in range(1000):
    batch = mnist.train.next_batch(batch_size)
    sess.run(train_step, feed_dict={x: batch[0], y: batch[1]})
```

### Evaluate the model
Finally, we can evaluate the accuracy on the test data. The model's prediction can be obtained by using tf.argmax, which gives you the index of highest entry in a tensor. tf.equal enables to check if our prediction matches the truth.

```python
correct_prediction = tf.equal(tf.argmax(z, 1), tf.argmax(y, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print sess.run(accuracy, feed_dict={x: mnist.test.images, y:mnist.test.labels})

# Close the Session when we're done
sess.close()
```

    0.9179

## Build a Neural Network Model
 Neural network are modeled as collections of neurons connected in as acyclic graph. There are several hyper parameters for the design of neural network architecture such as the number of neurons and the number of layers. In addition, we need to choose an activation function which takes a single number (mostly logits) and performs a certain mathematical operation on it.
 In this section, we will build a simple neural network with one fully-connected hidden layer for MNIST digit recognition. We are going to use the ReLU (Rectified Linear Unit)  $f(z) = max(0,z)$ as an activation function, which is getting popular recently, and the cross-entropy fucntion as a loss funciton.
\begin{align}
z^1 & = w^1x + b^1 \\
a^1 & = max(0, z^1) \\
z^2 & = w^2a^1 + b^2 \\
\phi_j & = \frac{e^{z_j^2}}{\sum_{k} e^{-z_k^2}} \\
L & = \log \phi_y
\end{align}
where $z^l$: $l$ layer logits, $a^l$: $l$ layer activation

Let's begin with creating two function for model parameter creation.

```python
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)

def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)
```

Here we initialize weights with small randomized values for symmetry breaking. Also, in order to avoid dead neurons, it is good practice to initialize biases with a positive value.

Let's build 2-layer neural networks and compare the accuracy of models with different size of hidden layer.

```python
hidden_sizes = [10, 20, 30, 40, 50] # The size of hidden layer
max_steps = 10001 # The number of training steps
batch_size = 100
learning_rate = 0.5

result = []

for hsize in hidden_sizes:

    # Input-hidden layer
    W1 = weight_variable([input_size, hsize])
    b1 = bias_variable([hsize])
    z1 = tf.matmul(x, W1) + b1
    a1 = tf.nn.relu(z1)

    # Hidden-output layer
    W2 = weight_variable([hsize, output_size])
    b2 = bias_variable([output_size])
    z2 = tf.matmul(a1, W2) + b2

    # Loss function
    loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(z2, y))

    # Optimization method
    train_step = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)

    # Evaluation
    correct_prediction = tf.equal(tf.argmax(z2, 1), tf.argmax(y, 1))
    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

    with tf.Session() as sess:
        init = tf.initialize_all_variables()
        sess.run(init)
        for i in range(max_steps):
            batch = mnist.train.next_batch(batch_size)
            if i%2000 == 0:
                loss_value, train_accuracy = sess.run([loss, accuracy], feed_dict={x:batch[0], y: batch[1]})
                print("size %d, step %d, loss %g, training accuracy %g"%(hsize, i, loss_value, train_accuracy))
            sess.run(train_step, feed_dict={x: batch[0], y: batch[1]})
        test_accuracy = sess.run(accuracy, feed_dict={x: mnist.test.images, y:mnist.test.labels})
        result.append(test_accuracy)
        print("size %d, test accuracy %g"%(hsize, test_accuracy))
```

    size 10, step 0, loss 2.30693, training accuracy 0.13
    size 10, step 2000, loss 0.124664, training accuracy 0.97
    size 10, step 4000, loss 0.149537, training accuracy 0.95
    size 10, step 6000, loss 0.196914, training accuracy 0.95
    size 10, step 8000, loss 0.141455, training accuracy 0.94
    size 10, step 10000, loss 0.112663, training accuracy 0.96
    size 10, test accuracy 0.9336
    size 20, step 0, loss 2.32134, training accuracy 0.04
    size 20, step 2000, loss 0.259123, training accuracy 0.93
    size 20, step 4000, loss 0.188348, training accuracy 0.97
    size 20, step 6000, loss 0.0880904, training accuracy 0.95
    size 20, step 8000, loss 0.0330778, training accuracy 0.98
    size 20, step 10000, loss 0.163547, training accuracy 0.97
    size 20, test accuracy 0.9562
    size 30, step 0, loss 2.31416, training accuracy 0.11
    size 30, step 2000, loss 0.0774963, training accuracy 0.98
    size 30, step 4000, loss 0.115844, training accuracy 0.93
    size 30, step 6000, loss 0.0791531, training accuracy 0.96
    size 30, step 8000, loss 0.0216784, training accuracy 0.99
    size 30, step 10000, loss 0.0079687, training accuracy 1
    size 30, test accuracy 0.967
    size 40, step 0, loss 2.45583, training accuracy 0.1
    size 40, step 2000, loss 0.130733, training accuracy 0.97
    size 40, step 4000, loss 0.0475326, training accuracy 0.99
    size 40, step 6000, loss 0.0167952, training accuracy 1
    size 40, step 8000, loss 0.0646013, training accuracy 0.99
    size 40, step 10000, loss 0.00557586, training accuracy 1
    size 40, test accuracy 0.9717
    size 50, step 0, loss 2.32493, training accuracy 0.09
    size 50, step 2000, loss 0.079928, training accuracy 0.98
    size 50, step 4000, loss 0.0684385, training accuracy 0.99
    size 50, step 6000, loss 0.042688, training accuracy 0.98
    size 50, step 8000, loss 0.0386569, training accuracy 0.98
    size 50, step 10000, loss 0.0320923, training accuracy 0.98
    size 50, test accuracy 0.9721

Let's plot the accuracy of the models we trained and evaluated.

```python
plt.plot(hidden_sizes, result)
plt.xlabel('size of hidden layer')
plt.ylabel('accuracy')
plt.show()
```

As you can imagine, the more the hidden layer units are, the better the model performs. This results lead us to the idea that we can also increase the number of layers in neural networks to make it more powerful. Unfortunately, deep neural networks are often much harder to train than shallow neural networks. In order to train deep networks, we need to use several tricks.

# Build a Convolutional Network Model
In this section, we will try to improve the accuray on MNIST with a multilayer convolutional neural network model. Convolutional nets use a special architecture which take into accout the spatial structure of the images. There are three basic ideas in convolutional nets: local receptive fields, shared weights and pooling.

### Local receptive fields
In convolutional nets, it will help to think of the inputs as 28x28 square neurons instead of 784 vertical line of neurons. We only make connections between the input layer and a hidden layer in small, localized regions. If we have a 28x28 input image, and 5x5 local receptive fields which being moved by one pixel at a time (1 stride length), then there will be 24x24 neurons in the hidden layer.

### Shared weights and biases
In addition to the local receptive fields, in convolutional layer, we use the same weights and bias for each of the 24x24 hidden neurons. In other words, the hypothesis function of $j, k$th hidden neuron is:
$$
z_{jk} = \sum_{l=0}^4 \sum_{m=0}^4 w_{l,m} a_{j+l, k+m} + b
$$
This means that all the neurons in the first hidden layer detect the same feature, creating a feature map. The shared weights and bias are called a filter in TensorFlow. For image recognition, we will need more than one feature map, which is represented as a set of 5x5 shared weights and a shared bias.

### Pooling
In addition to the convoluational layer described above, convolutional nets also contain pooling layer. We use a common procedure for pooling known as max-pooling, in which each unit takes the maximum activation of 2x2 input region.

We will build the 4-layer convolutional nets consisting two convolutional and pooling layers and two fully-connected layer.

### First Convolutional Layer
Let's begin with defining input features and a filter in the first convolutional layer. The input feature tensor must be reshaped to [batch, in_height, in_width, in_channels] where in_height=28, in_width=28, in_channels=1. We can think of in_channels as the depth of the layer. If we have color images, in_channels must be 3.

Nest, we define the weights tensor of shape [filter_height, filter_width, in_channels, out_channels] where filter_height=5, filter=width=5, in_channels=1, out_channels=32, meaning 32 sets of 5x5x1 local receptive fields.

```python
x_image = tf.reshape(x, [-1,28,28,1])
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
```

We now can compute a convolution and a pooling. conv2d funciton computes convolution given 4-D input and filter tensors with stride-length 1, returning a 4-D tensor of shape [batch, 28, 28, 32] in the first convolutional layer. max_pool_2x2 fucntion computes max-pooling fo 2x2 input region, resulting in a [batch, 14, 14, 32] shaped tensor.

```python
def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')

def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                        strides=[1, 2, 2, 1], padding='SAME')

a_conv1 = tf.nn.relu(conv2d(x_image, W_conv1) + b_conv1)
a_pool1 = max_pool_2x2(a_conv1)
```

### Second Convolutional Layer

We can design a filter, convolution and pooling in the second convoluational layer in the same way. Here, we create 64 feature maps from 32 feature maps from the first layer. The output of the pooling will be a tensor of shape [batch, 7, 7, 64].

```python
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
a_conv2 = tf.nn.relu(conv2d(a_pool1, W_conv2) + b_conv2)
a_pool2 = max_pool_2x2(a_conv2)
```

### Densely Connected Layer
We will add a fully-connected layer with 1024 neurons. We reshape the tensor from the pooling layer and apply a ReLU activation to the logits.

```python
a_pool2_flat = tf.reshape(a_pool2, [-1, 7*7*64])
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
a_fc1 = tf.nn.relu(tf.matmul(a_pool2_flat, W_fc1) + b_fc1)
```

We will introduce a regularization technique called dropout, which modifies the network during training. Wiht dropout, we start by randomly deleting certain amount of the hidden neurons in the network in each batch gradient decent process. We will define the probability of dropout as placeholder. Note that the dropout should be performed only when we are training the network.

```python
keep_prob = tf.placeholder(tf.float32)
a_fc1_drop = tf.nn.dropout(a_fc1, keep_prob)
```

Finally, we add a readout layer just like the one layer of softmax regression.

```python
W_fc2 = weight_variable([1024, 10])
b_fc2 = bias_variable([10])
z_conv = tf.matmul(a_fc1_drop, W_fc2) + b_fc2
```

### Train and Evaluate the CNN model
The code for traning and evaluation of convoluational nets are almost the same as 2-layer neural networks except that we use ADAM optimizer and feed the additional parameter, keep_prob, to control dropout rate.

```python
max_steps = 20001
batch_size = 50
learning_rate = 1e-4

loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(z_conv, y))
train_step = tf.train.AdamOptimizer(learning_rate).minimize(loss)
correct_prediction = tf.equal(tf.argmax(z_conv,1), tf.argmax(y,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

with tf.Session() as sess:
    sess.run(tf.initialize_all_variables())

    for i in range(max_steps):
        batch = mnist.train.next_batch(batch_size)
        if i%2000 == 0:
            # set keep_probability 1.0 when we evaluate the model
            train_accuracy = accuracy.eval(feed_dict={
                x:batch[0], y: batch[1], keep_prob: 1.0})
            print("step %d, training accuracy %g"%(i, train_accuracy))
        train_step.run(feed_dict={x: batch[0], y: batch[1], keep_prob: 0.5})

    # set keep_probability 1.0 when we evaluate the model
    print("test accuracy %g"%accuracy.eval(feed_dict={
        x: mnist.test.images, y: mnist.test.labels, keep_prob: 1.0}))
```

    step 0, training accuracy 0.06
    step 2000, training accuracy 1
    step 4000, training accuracy 1
    step 6000, training accuracy 1
    step 8000, training accuracy 1
    step 10000, training accuracy 1
    step 12000, training accuracy 1
    step 14000, training accuracy 1
    step 16000, training accuracy 1
    step 18000, training accuracy 1
    step 20000, training accuracy 1
    test accuracy 0.9916

We obtained a test accuracy of 99.2 percent!

### Summary and references
We have learned how to quickly build, train and evaluate a fairly sophisticated neural network using TensorFlow. This tutorial covers only a few elements of neural network models. Further information is available from the following links.
1. TensorFlow: https://www.tensorflow.org/
2. Neural networks and deep learning: http://neuralnetworksanddeeplearning.com/

```python

```


```python
import numpy as np
import numba
import matplotlib.pyplot as plt
import sympy as sym
%matplotlib notebook
plt.style.use('presentation.mplstyle')
colors_cycle=plt.rcParams.get('axes.prop_cycle')
colors = [item['color'] for item in colors_cycle]
def d2np(d):

    names = []
    numbers = ()
    dtypes = []
    for item in d:
        names += item
        if type(d[item]) == float:
            numbers += (d[item],)
            dtypes += [(item,float)]
        if type(d[item]) == int:
            numbers += (d[item],)
            dtypes += [(item,int)]
        if type(d[item]) == np.ndarray:
            numbers += (d[item],)
            dtypes += [(item,np.float64,d[item].shape)]
    return np.array([numbers],dtype=dtypes)
```

```python
alpha = np.exp(2.0/3*np.pi*1j)
A_0a =  np.array([[1, 1, 1],
                  [1, alpha**2, alpha],
                  [1, alpha, alpha**2]])

A_a0 = 1/3* np.array([[1, 1, 1],
                      [1, alpha, alpha**2],
                      [1, alpha**2, alpha]])
```

## Steady state solution

```python

s = 0.035365
n = 1478
n_1 = 1500
s = (n_1 - n)/n_1

#V_abc = np.array([235.0,240*np.exp(1j*np.deg2rad(-117.9)),245*np.exp(1j*np.deg2rad(120))]).reshape(3,1)

s = 0.035365
omega_s = 2.0*np.pi*freq/N_pp
omega = omega_s*(1-s)
s_pos = (omega_s - omega)/omega_s
s_neg = 2 - s_pos

s_pos = 0.035365*1.45
s_neg = 2 - s_pos
# kersting
R_s = 0.0774
X_s = 0.1843
X_mag = 4.8384
R_r = 0.0908
X_r = 0.1843

# Kersting paper
# Phase  Frame Analysis of the Effects of Voltage Unbalance on Induction Machines
Z_b = 240**2/(18642.5)
R_r = 39.4e-3*Z_b
X_r = 80e-3*Z_b
X_mag = 2.1*Z_b
R_s = 33.6e-3*Z_b
X_s = 80e-3*Z_b

# M. Al-Badri paper:
# A Novel Full-Load Efficiency Estimation Technique for Induction Motors Operating with Unbalanced Voltages
freq = 50.0
N_pp = 1
P_n = 5592.75  # 7.5 HP
R_s = 3.0050
X_s = 2.3155
R_fe = 4521.5*1e6
X_mag = 200.20
R_r = 2.3000
X_r = 3.4559

# ABB 90 kW
P_n = 90.0e3
freq = 50.0
N_pp = 2.0
R_r = 24e-3
X_r =196e-3
X_mag = 3.7
R_fe = 111.0
R_s = 23e-3
X_s = 86e-3
U_n = 400.0

# Voltages
V = U_n/np.sqrt(3)
theta = np.deg2rad(20.0)
V_zero =  0.0*np.exp(1j*0.0)
V_neg  = V*0.05*np.exp(1j*0.0)
V_pos  =      V*np.exp(1j*theta)
V_zpn = np.array([[V_zero],[V_pos],[V_neg]])
V_abc = A_0a @ V_zpn
n_n = 1478.0

def im_pf_phasor(V_abc, omega_r):
    omega_1 = 2*np.pi*freq/N_pp
    s_pos = (omega_1 - omega_r)/omega_1
    s_neg = 2 - s_pos
    Z_r_pos = R_r/s_pos + 1j*X_r
    Z_r_neg = R_r/s_neg + 1j*X_r
    Z_mag = (R_fe * 1j*X_mag)/(R_fe + 1j*X_mag)
    Z_s = R_s + 1j*X_s
    Z_eq_pos = Z_s + Z_r_pos*Z_mag/(Z_r_pos+Z_mag)
    Z_eq_neg = Z_s + Z_r_neg*Z_mag/(Z_r_neg+Z_mag)

    V_012 = A_a0 @ V_abc
    V_s_p = V_012[1,0]
    V_s_n = V_012[2,0]

    I_s_p =  V_s_p/Z_eq_pos
    I_s_n =  V_s_n/Z_eq_neg

    E_p =  V_s_p - Z_s*I_s_p
    E_n =  V_s_n - Z_s*I_s_n

    I_r_p =  E_p/Z_r_pos
    I_r_n =  E_n/Z_r_neg

    P_s_loss = 3*R_s*(np.abs(I_s_p)**2 + np.abs(I_s_n)**2)

    P_a_p = (3*E_p*I_r_p.conj()).real
    P_a_n = (3*E_n*I_r_n.conj()).real

    P_r_p = P_a_p*s_pos
    P_r_n = P_a_n*s_neg
    P_r_loss = P_r_p + P_r_n

    P_mi_p = P_a_p - P_r_p
    P_mi_n = P_a_n - P_r_n

    P_mi = P_mi_p + P_mi_n

    Y_zpn = np.diag([1.0,1.0/Z_eq_pos,1.0/Z_eq_neg])

    Y_abc = A_0a @ Y_zpn @ A_a0
    I_abc = Y_abc @ V_abc

    P_a = P_a_p - P_a_n
    tau_mi = P_a/omega_1

    return I_abc, tau_mi

n_n_0 = 1478
p_ref = 90e3
for it in range(20):
    omega_1 = 2*np.pi*freq/N_pp
    n_n = n_n_0
    omega_r = n_n*2*np.pi/60.0
    I_abc, tau_mi = im_pf_phasor(V_abc,omega_r)
    P_mi = tau_mi*omega_r
    n_n_0 = n_n +  (P_mi - p_ref)/P_n*n_n*0.01
print(n_n)

V_avg = np.average(np.abs(V_abc))
DV_max = np.max(np.abs(np.abs(V_abc) - V_avg))
V_unbalance = DV_max/np.abs(V_avg)
print('V_unbalance % =', V_unbalance*100)
I_avg = np.average(np.abs(I_abc))
DI_max = np.max(np.abs(np.abs(I_abc) - I_avg))
I_unbalance = DI_max/np.abs(I_avg)
print('I_unbalance % =', I_unbalance*100)

S = V_abc.T @ I_abc.conj()
P = S.real
Q = S.imag
pf = P/np.abs(S)
efficiency = P_mi/P
print('I_avg',I_avg)
print('P',P)
print('P_mi',P_mi)
print('tau_mi',tau_mi)
print('Q',Q)
print('pf',pf)
#print('P_r_loss',P_r_loss)
#print('P_s_loss',P_s_loss)
print('efficiency',efficiency)
```

    1477.15936644
    V_unbalance % = 4.64619716405
    I_unbalance % = 26.7852065685
    I_avg 162.198905522
    P [[ 94710.10262697]]
    P_mi 89999.9998071
    tau_mi 581.817175844
    Q [[ 58958.25640837]]
    pf [[ 0.84894591]]
    efficiency [[ 0.95026821]]

```python
26/4
```

    6.5

```python
# ABB

N_pp = 1.0
omega_s = 2.0*np.pi*freq
L_m = X_mag/omega_s
L_s = X_s/omega_s
L_r = X_r/omega_s

omega_r = omega_s*(1-s)

V_012 = A_a0 @ V_abc
v_s_z = V_012[0,0]
v_s_p = V_012[1,0]
v_s_n = V_012[2,0]

theta_pll = np.angle(v_s_p)

v_s_dq_z = v_s_z
v_s_dq_p = v_s_p*np.exp(-1j*theta_pll)*np.sqrt(2)
v_s_dq_n = v_s_n*np.exp( 1j*theta_pll)*np.sqrt(2)

v_s_d_z = v_s_dq_z.real # ??
v_s_q_z = v_s_dq_z.imag # ??
v_s_d_p = v_s_dq_p.imag
v_s_q_p = v_s_dq_p.real
v_s_d_n = v_s_dq_n.imag
v_s_q_n = v_s_dq_n.real

# positive sequence
sigma_p = (omega_s - omega_r)/omega_s
sigma = sigma_p
v_s_q = v_s_q_p
v_s_d = v_s_d_p
i_s_d = (omega_s*v_s_q*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + v_s_d*(L_m**2*R_r*omega_s**2*sigma + R_r**2*R_s + R_s*omega_s**2*sigma**2*(L_m + L_r)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
i_s_q = (-omega_s*v_s_d*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + v_s_q*(L_m**2*R_r*omega_s**2*sigma + R_r**2*R_s + R_s*omega_s**2*sigma**2*(L_m + L_r)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
i_r_d = -L_m*omega_s*sigma*(omega_s*v_s_d*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) - v_s_q*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
i_r_q = -L_m*omega_s*sigma*(omega_s*v_s_q*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) + v_s_d*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)

psi_s_d = (L_s+L_m)*i_s_d + L_m*i_r_d
psi_s_q = (L_s+L_m)*i_s_q + L_m*i_r_q
psi_r_d = (L_r+L_m)*i_r_d + L_m*i_s_d
psi_r_q = (L_r+L_m)*i_r_q + L_m*i_s_q

i_s_d_p = i_s_d
i_s_q_p = i_s_q
i_r_d_p = i_r_d
i_r_q_p = i_r_q

psi_s_d_p = psi_s_d
psi_s_q_p = psi_s_q
psi_r_d_p = psi_r_d
psi_r_q_p = psi_r_q

# negative sequence
sigma_n = 2-sigma_p
sigma = sigma_n
v_s_q = v_s_q_n
v_s_d = v_s_d_n
i_s_d = (omega_s*v_s_q*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + v_s_d*(L_m**2*R_r*omega_s**2*sigma + R_r**2*R_s + R_s*omega_s**2*sigma**2*(L_m + L_r)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
i_s_q = (-omega_s*v_s_d*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + v_s_q*(L_m**2*R_r*omega_s**2*sigma + R_r**2*R_s + R_s*omega_s**2*sigma**2*(L_m + L_r)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
i_r_d = -L_m*omega_s*sigma*(omega_s*v_s_d*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) - v_s_q*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
i_r_q = -L_m*omega_s*sigma*(omega_s*v_s_q*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) + v_s_d*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)

psi_s_d = (L_s+L_m)*i_s_d + L_m*i_r_d
psi_s_q = (L_s+L_m)*i_s_q + L_m*i_r_q
psi_r_d = (L_r+L_m)*i_r_d + L_m*i_s_d
psi_r_q = (L_r+L_m)*i_r_q + L_m*i_s_q

i_s_d_n = i_s_d
i_s_q_n = i_s_q
i_r_d_n = i_r_d
i_r_q_n = i_r_q

psi_s_d_n = psi_s_d
psi_s_q_n = psi_s_q
psi_r_d_n = psi_r_d
psi_r_q_n = psi_r_q

#v_qr = R_r*i_qr + omega_s*sigma*(L_m*i_dr + L_m*i_ds + L_r*i_dr)
#v_dr = R_r*i_dr - omega_s*sigma*(L_m*i_qr + L_m*i_qs + L_r*i_qr)
#psi_dr = L_m*i_dr + L_m*i_ds + L_r*i_dr
#psi_qs = (R_s*i_ds - v_ds)/omega_s
#psi_ds = (-R_s*i_qs + v_qs)/omega_s
#psi_qr = L_m*i_qr + L_m*i_qs + L_r*i_qr

tau_e_p  = 3.0/2.0*N_pp*(psi_r_q_p*i_r_d_p - psi_r_d_p*i_r_q_p)
tau_e_n  = 3.0/2.0*N_pp*(psi_r_q_n*i_r_d_n - psi_r_d_n*i_r_q_n)
tau_e = tau_e_p - tau_e_n
p_m = tau_e*omega_r

p_s_p = 3/2*(v_s_d_p*i_s_d_p + v_s_q_p*i_s_q_p)
p_s_n = 3/2*(v_s_d_n*i_s_d_n + v_s_q_n*i_s_q_n)
p_s_loss =3/2*R_s*(i_s_d_p**2+i_s_d_n**2+i_s_q_p**2+i_s_q_n**2)
p_r_loss =3/2*R_r*(i_r_d_p**2+i_r_d_n**2+i_r_q_p**2+i_r_q_n**2)
p_s = p_s_p + p_s_n
rend = p_m/p_s
print('v_s_d_p = ', v_s_d_p)
print('v_s_q_p = ', v_s_q_p)
print('i_s_d_p = ', i_s_d_p)
print('i_s_q_p = ', i_s_q_p)

print('v_s_d_n = ', v_s_d_n)
print('v_s_q_n = ', v_s_q_n)
print('i_s_d_n = ', i_s_d_n)
print('tau_e = ', tau_e)

print('p_s_loss = ', p_s_loss)
print('p_r_loss = ', p_r_loss)
print('p_m = ', p_m)
print('p_s = ', p_s)
print('rend = ', rend)
```

    v_s_d_p =  -2.00971834712e-14
    v_s_q_p =  326.598632371
    i_s_d_p =  3.87911568302
    i_s_q_p =  11.8160531909
    v_s_d_n =  1.1160997896e-13
    v_s_q_n =  1.28447296598e-14
    i_s_d_n =  1.07573411545e-14
    tau_e =  16.2067487675
    p_s_loss =  697.159931603
    p_r_loss =  509.150028666
    p_m =  4582.35025799
    p_s =  5788.66021826
    rend =  0.79160808982

```python
p_s_n
```

    44.042360381628214

```python
p_r_loss*3/2
```

    673.35091291075605

```python
psi_s_d,psi_s_q,psi_r_d,psi_r_q = sym.symbols('psi_s_d,psi_s_q,psi_r_d,psi_r_q')
i_s_d,i_s_q,i_r_d,i_r_q = sym.symbols('i_s_d,i_s_q,i_r_d,i_r_q ')
di_s_d,di_s_q,di_r_d,di_r_q = sym.symbols('di_s_d,di_s_q,di_r_d,di_r_q')
L_s,L_r,L_m = sym.symbols('L_s,L_r,L_m')
R_s,R_r = sym.symbols('R_s,R_r')
omega_s,omega_r,sigma = sym.symbols('omega_s,omega_r,sigma')
v_s_d,v_s_q,v_r_d,v_r_q = sym.symbols('v_s_d,v_s_q,v_r_d,v_r_q')

eq_sd = (L_s+L_m)*i_s_d + L_m*i_r_d - psi_s_d
eq_sq = (L_s+L_m)*i_s_q + L_m*i_r_q - psi_s_q
eq_rd = (L_r+L_m)*i_r_d + L_m*i_s_d - psi_r_d
eq_rq = (L_r+L_m)*i_r_q + L_m*i_s_q - psi_r_q
dpsi_sd = v_s_d - R_s*i_s_d + omega_s*psi_s_q
dpsi_sq = v_s_q - R_s*i_s_q - omega_s*psi_s_d
dpsi_rd =  - R_r*i_r_d + sigma*omega_s*psi_r_q
dpsi_rq =  - R_r*i_r_q - sigma*omega_s*psi_r_d

s = sym.solve([  eq_sd,  eq_sq,  eq_rd,  eq_rq,
               dpsi_sd,dpsi_sq,dpsi_rd,dpsi_rq],
              [   i_s_d,   i_s_q,   i_r_d,   i_r_q,
                psi_s_d, psi_s_q,   psi_r_d, psi_r_q])

for item in s:
    print(item, '=', sym.simplify(s[item]))
```

    i_s_d = (omega_s*v_s_q*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + v_s_d*(L_m**2*R_r*omega_s**2*sigma + R_r**2*R_s + R_s*omega_s**2*sigma**2*(L_m + L_r)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    i_s_q = (-omega_s*v_s_d*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + v_s_q*(L_m**2*R_r*omega_s**2*sigma + R_r**2*R_s + R_s*omega_s**2*sigma**2*(L_m + L_r)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    i_r_d = -L_m*omega_s*sigma*(omega_s*v_s_d*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) - v_s_q*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    i_r_q = -L_m*omega_s*sigma*(omega_s*v_s_q*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) + v_s_d*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    psi_s_d = (R_s*v_s_d*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) + omega_s*v_s_q*(L_m**4*omega_s**2*sigma**2 + L_m**2*R_r*R_s*sigma - 2*L_m**2*omega_s**2*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*(L_m + L_s)**2 + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    psi_s_q = (R_s*v_s_q*(-L_m**2*omega_s**2*sigma**2*(L_m + L_r) + R_r**2*(L_m + L_s) + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)) - omega_s*v_s_d*(L_m**4*omega_s**2*sigma**2 + L_m**2*R_r*R_s*sigma - 2*L_m**2*omega_s**2*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*(L_m + L_s)**2 + omega_s**2*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    psi_r_d = L_m*R_r*(omega_s*v_s_q*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) + v_s_d*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)
    psi_r_q = -L_m*R_r*(omega_s*v_s_d*(R_r*(L_m + L_s) + R_s*sigma*(L_m + L_r)) - v_s_q*(L_m**2*omega_s**2*sigma + R_r*R_s - omega_s**2*sigma*(L_m + L_r)*(L_m + L_s)))/(L_m**4*omega_s**4*sigma**2 + 2*L_m**2*R_r*R_s*omega_s**2*sigma - 2*L_m**2*omega_s**4*sigma**2*(L_m + L_r)*(L_m + L_s) + R_r**2*R_s**2 + R_r**2*omega_s**2*(L_m + L_s)**2 + R_s**2*omega_s**2*sigma**2*(L_m + L_r)**2 + omega_s**4*sigma**2*(L_m + L_r)**2*(L_m + L_s)**2)

```python
# [1] T. Demiray, F. Milano, and G. Andersson,
# “Dynamic phasor modeling of the doubly-fed induction generator under unbalanced conditions,” 2007 IEEE Lausanne POWERTECH, Proc., no. 2, pp. 1049–1054, 2007.

@numba.jit(nopython=True, cache=True)
def dfim_alg_ctrl1(struct,i,m):
    '''
    Doubly Fed Induction Machine in with neglected dynamics and
    rotor side converter and control level 1 already implemented.
    i_rd = i_rd_ref and i_rq = i_rq_ref without dynamics
    '''

    x_idx = struct[i]['dfim_idx']
    #psi_dr = float(struct[i]['x'][x_idx+0,0])
    #psi_qr = float(struct[i]['x'][x_idx+1,0])

    L_m = struct[i]['L_m']
    L_r = struct[i]['L_r']
    L_s = struct[i]['L_s']
    R_r = struct[i]['R_r']
    R_s = struct[i]['R_s']
    N_pp = struct[i]['N_pp']

    Dt  = struct[i]['Dt']

    i_dr_ref = struct[i]['i_dr_ref']
    i_qr_ref = struct[i]['i_qr_ref']

    i_dr = i_dr_ref
    i_qr = i_qr_ref

    v_ds = struct[i]['v_ds']
    v_qs = struct[i]['v_qs']

    omega_r = struct[i]['omega_r']
    omega_s = struct[i]['omega_s']

    sigma = (omega_s - omega_r)/omega_s

    den = R_s**2 + omega_s**2*(L_m + L_s)**2
    i_qs = (-L_m*R_s*i_dr*omega_s - L_m*i_qr*omega_s**2*(L_m + L_s) + R_s*v_qs - omega_s*v_ds*(L_m + L_s))/den
    i_ds = ( L_m*R_s*i_qr*omega_s - L_m*i_dr*omega_s**2*(L_m + L_s) + R_s*v_ds + omega_s*v_qs*(L_m + L_s))/den

    v_qr = R_r*i_qr + omega_s*sigma*(L_m*i_dr + L_m*i_ds + L_r*i_dr)
    v_dr = R_r*i_dr - omega_s*sigma*(L_m*i_qr + L_m*i_qs + L_r*i_qr)
    psi_dr = L_m*i_dr + L_m*i_ds + L_r*i_dr
    psi_qs = (R_s*i_ds - v_ds)/omega_s
    psi_ds = (-R_s*i_qs + v_qs)/omega_s
    psi_qr = L_m*i_qr + L_m*i_qs + L_r*i_qr

    tau_e  = 3.0/2.0*N_pp*(psi_qr*i_dr - psi_dr*i_qr)

    struct[i]['v_dr'] = v_dr
    struct[i]['v_qr'] = v_qr

    struct[i]['i_ds'] = i_ds
    struct[i]['i_qs'] = i_qs
    struct[i]['i_dr'] = i_dr
    struct[i]['i_qr'] = i_qr

    struct[i]['psi_ds'] = psi_ds
    struct[i]['psi_qs'] = psi_qs
    struct[i]['psi_dr'] = psi_dr
    struct[i]['psi_qr'] = psi_qr

    struct[i]['tau_e'] = tau_e
    struct[i]['sigma'] = sigma

    struct[i]['p_s'] = 3.0/2.0*(v_ds*i_ds + v_qs*i_qs)
    struct[i]['q_s'] = 3.0/2.0*(v_ds*i_qs - v_qs*i_ds)

    struct[i]['p_r'] = 3.0/2.0*(v_dr*i_dr + v_qr*i_qr)
    struct[i]['q_r'] = 3.0/2.0*(v_dr*i_qr - v_qr*i_dr)

    return tau_e

@numba.jit(nopython=True, cache=True)
def wecs_mech_1(struct,i,m):

    x_idx = struct[i]['mech_idx']
    omega_t = struct[i]['x'][x_idx,0]  # rad/s
    tau_t   = struct[i]['tau_t']
    tau_r   = struct[i]['tau_r']

    J_t  = struct[i]['J_t']
    N_tr = struct[i]['N_tr']
    Dt   = struct[i]['Dt']

    domega_t = 1.0/J_t*(tau_t - N_tr*tau_r)

    omega_r = N_tr*omega_t

    struct[i]['f'][x_idx,0] = domega_t

    struct[i]['omega_r'] = omega_r
    struct[i]['omega_t'] = omega_t

    return omega_t

```

## Controler level 2

```python
@numba.jit(nopython=True, cache=True)
def dfim_ctrl2(struct,i,m):
    '''
    Control level 2 for DFIM for stator active and reactive power.

    '''

    x_idx = struct[i]['ctrl2r_idx']
    xi_p_s = float(struct[i]['x'][x_idx+0,0])
    xi_q_s = float(struct[i]['x'][x_idx+1,0])

    K_r_p = struct[i]['K_r_p']
    K_r_i = struct[i]['K_r_i']

    p_s_ref = struct[i]['p_s_ref']
    q_s_ref = struct[i]['q_s_ref']

    p_s = struct[i]['p_s']
    q_s = struct[i]['q_s']

    S_b = struct[i]['S_b']

    omega_r = struct[i]['omega_r']
    omega_s = struct[i]['omega_s']
    R_r = struct[i]['R_r']

    I_b = S_b/(np.sqrt(3)*690.0)
    sigma = (omega_s - omega_r)/omega_s

    error_p_s = (p_s_ref - p_s)/S_b
    error_q_s = (q_s_ref - q_s)/S_b
    dxi_p_s = error_p_s
    dxi_q_s = error_q_s

    struct[i]['f'][x_idx+0,0] = dxi_p_s
    struct[i]['f'][x_idx+1,0] = dxi_q_s

    struct[i]['i_dr_ref'] = -I_b*(K_r_p*error_p_s + K_r_i*xi_p_s)
    struct[i]['i_qr_ref'] = -I_b*(K_r_p*error_q_s + K_r_i*xi_q_s)

    return struct[0]['i_dr_ref'],struct[0]['i_qr_ref']

```

```python
Omega_b = 2.0*np.pi*50.0
omega_r = Omega_b/2
2.0e6/omega_r
```

    12732.395447351626

```python
Omega_b = 2.0*np.pi*50.0
S_b = 2.0e6
U_b = 690.0
Z_b = U_b**2/S_b
#nu_w =np.linspace(0.1,15,N)
H = 2.0
N_pp = 2
N_tr = 20
# H = 0.5*J*Omega_t_n**2/S_b
S_b = 2.0e6
Omega_t_n = Omega_b/N_pp/N_tr
J_t = 2*H*S_b/Omega_t_n**2

#Z_b = 1.0
#Omega_b = 1.0
d =dict(S_b = S_b,
        Omega_b = Omega_b,
        R_r = 0.01*Z_b,
        R_s = 0.01*Z_b,
        L_r = 0.08*Z_b/Omega_b,
        L_s = 0.1*Z_b/Omega_b,
        L_m = 3.0*Z_b/Omega_b,
        N_pp = N_pp,
        psi_ds = 0.0,
        psi_qs = 0.0,
        p_s = 0.0,
        q_s = 0.0,
        p_r = 0.0,
        q_r = 0.0,
        psi_dr = 0.0,
        psi_qr = 0.0,
        p_s_ref = 0.0,
        q_s_ref = 0.0,
        i_ds   = 0.0,
        i_qs   = 0.0,
        i_dr   = 0.0,
        i_qr   = 0.0,
        i_dr_ref   = 0.0,
        i_qr_ref   = 0.0,
        v_ds   = 0.0,
        v_qs   = 0.0,
        v_dr   = 0.0,
        v_qr   = 0.0,
        omega_r   = Omega_b/N_pp,
        omega_s   = Omega_b/N_pp,
        sigma   = 0.0,
        tau_e   = 0.0,
        x = np.zeros((3,1)),
        f = np.zeros((3,1)),
        Dt = 0.0,
        J_t = J_t,
        omega_t = 0.0,
        tau_t = 0.0,
        tau_r = 0.0,
        N_tr = N_tr,
        K_r_p = 0.02,
        K_r_i = 20.0,
        dfim_idx = 0,
        mech_idx = 0,
        ctrl2r_idx = 1
        )

struct = d2np(d)
struct = np.hstack((struct[0],np.copy(struct[0])))

#wecs_mech_1(struct,0)

dfim_alg_ctrl1(struct,0,0)
dfim_ctrl2(struct,0,0)
dfim_alg_ctrl1(struct,1,0)
dfim_ctrl2(struct,1,0)
print(struct[0]['p_s']/1e6,struct[0]['q_s']/1e6,struct[0]['tau_e'])
print(struct[1]['p_s']/1e6,struct[0]['q_s']/1e6,struct[0]['tau_e'])
```

    0.0 0.0 0.0
    0.0 0.0 0.0

```python
struct = d2np(d)
struct = np.hstack((struct[0],np.copy(struct[0])))
sys_d = dict(x = np.zeros((6,1)),
             f = np.zeros((6,1)))

sys_struct = d2np(sys_d)

@numba.jit(nopython=True, cache=True)
def f_eval(sys_struct,struct):

    for i in range(2):
        struct[i]['x'][:,0] = sys_struct[0]['x'][3*i:3*(i+1),0]
        wecs_mech_1(struct,i,2)
        dfim_ctrl2(struct,i,2)
        dfim_alg_ctrl1(struct,i,2)
        sys_struct[0]['f'][3*i:3*(i+1),:] = struct[i]['f']
    return 0

```

```python
@numba.jit(nopython=True, cache=True)
def run(sys_struct,struct):
    N_steps = 1000
    N_states = 6

    Dt = 10.0e-3
    Omega_r = np.zeros((N_steps,1))
    Omega_t = np.zeros((N_steps,1))
    P_s_1 = np.zeros((N_steps,1))
    Q_s_1 = np.zeros((N_steps,1))
    P_r_1 = np.zeros((N_steps,1))
    Q_r_1 = np.zeros((N_steps,1))
    P_s_2 = np.zeros((N_steps,1))
    Q_s_2 = np.zeros((N_steps,1))
    P_r_2 = np.zeros((N_steps,1))
    Q_r_2 = np.zeros((N_steps,1))
    V_dr = np.zeros((N_steps,1))
    V_qr = np.zeros((N_steps,1))
    I_dr = np.zeros((N_steps,1))
    I_qr = np.zeros((N_steps,1))
    I_ds = np.zeros((N_steps,1))
    I_qs = np.zeros((N_steps,1))
    Tau_e = np.zeros((N_steps,1))
    Tau_t = np.zeros((N_steps,1))
    T     = np.zeros((N_steps,1))
    X = np.zeros((N_steps,N_states))
    p_ref = 0.0
    q_ref = 0.0
    xi_p = 0.0
    xi_q = 0.0

    struct[0]['x'][:,0] = np.copy(sys_struct[0]['x'][0:3,0])
    struct[1]['x'][:,0] = np.copy(sys_struct[0]['x'][3:6,0])

    for it in range(N_steps):
        t = Dt*float(it)

        # perturbations and references

        struct[0]['p_s_ref'] = 0.0
        struct[1]['p_s_ref'] = 0.0
        struct[0]['q_s_ref'] = 0.0
        struct[1]['q_s_ref'] = 0.0

        if t>1.0:
            Omega_t_b = struct[0]['Omega_b']/struct[0]['N_tr']/struct[0]['N_pp']
            struct[0]['tau_t'] = 1.5e6/Omega_t_b+np.random.normal(500e3,100e3)/Omega_t_b
        if t>1.5:
            struct[0]['p_s_ref'] = 1.0e6
        if t>3.0:
            struct[1]['p_s_ref'] = 1.50e6
        if t>4.0:
            struct[0]['q_s_ref'] = 0.5e6
        if t>5.0:
            struct[1]['q_s_ref'] = -0.7e6
        ## solver

        f_eval(sys_struct,struct)
        f1 = np.copy(sys_struct[0]['f'])
        x1 = np.copy(sys_struct[0]['x'])

        sys_struct[0]['x'][:]= np.copy(x1 + Dt*f1)
        f_eval(sys_struct,struct)
        f2 = np.copy(sys_struct[0]['f'])

        sys_struct[0]['x'][:]= np.copy(x1 + 0.5*Dt*(f1 + f2))

        for i in range(2):
            struct[i]['x'][:,0] = sys_struct[0]['x'][3*i:3*(i+1),0]

        struct[0]['tau_r'] = struct[0]['tau_e']
        struct[1]['tau_r'] = struct[1]['tau_e']

        T[it,0] = t
        P_s_1[it,0] = float(struct[0]['p_s'])
        Q_s_1[it,0] = float(struct[0]['q_s'])
        P_r_1[it,0] = float(struct[0]['p_r'])
        Q_r_1[it,0] = float(struct[0]['q_r'])
        P_s_2[it,0] = float(struct[1]['p_s'])
        Q_s_2[it,0] = float(struct[1]['q_s'])
        P_r_2[it,0] = float(struct[1]['p_r'])
        Q_r_2[it,0] = float(struct[1]['q_r'])

        I_dr[it,0] = float(struct[0]['i_dr'])
        I_qr[it,0] = float(struct[0]['i_qr'])
        I_ds[it,0] = float(struct[0]['i_ds'])
        I_qs[it,0] = float(struct[0]['i_qs'])
        Omega_r[it,0] = float(struct[0]['omega_r'])
        Omega_t[it,0] = float(struct[0]['omega_t'])

        V_dr[it,0] = float(struct[0]['v_dr'])
        V_qr[it,0] = float(struct[0]['v_qr'])
        Tau_e[it,0] = float(struct[0]['tau_e'])
        Tau_t[it,0] = float(struct[0]['tau_t'])
        X[it,:] = sys_struct[0]['x'][:].T
    return T,X,Tau_e,P_s_1,Q_s_1,P_r_1,Q_r_1,P_s_2,Q_s_2,P_r_2,Q_r_2,V_dr,V_qr,Omega_r,Omega_t,I_dr,I_qr,I_ds,I_qs,Tau_t
%timeit run(sys_struct, struct)
```

    1.04 ms ± 21.5 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
sys_struct['x'][:]= np.zeros((6,1))
struct['v_qs'] = 0.0
struct['v_ds'] = 690.0*np.sqrt(2.0/3.0)
struct['tau_t'] = 0.0
sys_struct[0]['x'][0,0] = Omega_b*0.9/struct[0]['N_tr']/struct[0]['N_pp']
sys_struct[0]['x'][3,0] = Omega_b*1.1/struct[1]['N_tr']/struct[0]['N_pp']
#%timeit run(sys_struct, struct)
T,X,Tau_e,P_s_1,Q_s_1,P_r_1,Q_r_1,P_s_2,Q_s_2,P_r_2,Q_r_2,V_dr,V_qr,Omega_r,Omega_t,I_dr,I_qr,I_ds,I_qs,Tau_t = run(sys_struct, struct)
```

```python
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 4), sharex = True)
fig.set_tight_layout(False)
axes.plot(T,Tau_e/1e3)
axes.set_xlabel('Time (s)')
axes.set_ylabel('DFIM Torque (kNm)')
fig.savefig('dfim_tau_e.svg', bbox_inches='tight')
```

    <IPython.core.display.Javascript object>

```python
from matplotlib import rcParams
plt.style.use('presentation.mplstyle')
#rcParams.update({'figure.tight_layout':False})

fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(8, 6), sharex = True)
fig.set_tight_layout(False)
axes[0].plot(T,P_s_1/1e6, label='$\sf p_{s1}$')
axes[0].plot(T,Q_s_1/1e6, label='$\sf q_{s1}$')
#axes[0].plot(T,P_s_2/1e6, label='$\sf p_{s2}$')
#axes[0].plot(T,Q_s_2/1e6, label='$\sf q_{s2}$')

axes[1].plot(T,P_r_1/1e6, label='$\sf p_{r1}$')
axes[1].plot(T,Q_r_1/1e6, label='$\sf q_{r1}$')
#axes[1].plot(T,P_r_2/1e6, label='$\sf p_{r2}$')
#axes[1].plot(T,Q_r_2/1e6, label='$\sf q_{r2}$')

axes[0].legend(loc='best')
axes[1].legend(loc='best')

axes[0].set_ylabel('Stator powers (MVA)')
axes[1].set_ylabel('Rotor powers (MVA)')
axes[1].set_xlabel('Time (s)')
axes[0].set_ylim([-0.1,1.1])
#axes[0].set_xlim([0,3.0])

fig.savefig('dfim_pq_s_pq_r.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

```python
fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(8, 6))
fig.set_tight_layout(False)

axes[0].plot(T,Omega_t*60/(2*np.pi))

axes[1].plot(T,Tau_t/20.0/1000, label='$\sf \\tau_t$')
axes[1].plot(T,Tau_e/1000, label='$\sf \\tau_e$')
#axes[1].plot(T,P_r_2/1e6, label='$\sf p_{r2}$')
#axes[1].plot(T,Q_r_2/1e6, label='$\sf q_{r2}$')

axes[1].legend(loc='best')

axes[0].set_ylabel('Rotor speed (rev/m)')
axes[1].set_ylabel('Torques (kNm)')
axes[1].set_xlabel('Time (s)')
#axes[0].set_ylim([0,2.5])
#axes[0].set_xlim([0,3.0])

fig.savefig('dfim_omega_taus.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

```python
fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(8, 6), sharex = True)
fig.set_tight_layout(False)
axes[0].plot(T,V_dr, label='$\sf v_{dr}$')
axes[0].plot(T,V_qr, label='$\sf v_{qr}$')

axes[1].plot(T,P_r_1/1e6, label='$\sf p_{r}$', color = colors[2])
axes[1].plot(T,Q_r_1/1e6, label='$\sf q_{r}$', color = colors[3])
axes[1].plot(T,(P_r_1**2+Q_r_1**2)**0.5/1e6, label='$\sf s_{r}$', color = colors[4])

axes[0].legend(loc='best')
axes[1].legend(loc='best')

axes[0].set_ylabel('Rotor voltages (V)')
axes[1].set_ylabel('Rotor powers (MVA)')
axes[1].set_xlabel('Time (s)')
#axes[0].set_ylim([0,2.5])
#axes[0].set_xlim([0,3.0])

fig.savefig('dfim_rotor_v_powers.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

```python
fig, axes = plt.subplots(nrows=2, ncols=1, figsize=(8, 7), sharex = True)
fig.set_tight_layout(False)

axes[0].plot(T,I_dr/1000, label='$\sf i_{dr}$')
axes[0].plot(T,I_qr/1000, label='$\sf i_{qr}$')
axes[1].plot(T,I_ds/1000, label='$\sf i_{ds}$')
axes[1].plot(T,I_qs/1000, label='$\sf i_{qs}$')

axes[0].legend()
axes[1].legend()

axes[0].set_ylabel('Stator currents (kA)')
axes[1].set_ylabel('Rotor currents (kA)')
axes[1].set_xlabel('Time (s)')
#axes[0].set_ylim([0,2.5])
#axes[0].set_xlim([0,3.0])

fig.savefig('dfim_i_s_i_r.svg', bbox_inches='tight')

```

    <IPython.core.display.Javascript object>

```python

```


```
# default_exp system_response
```

```
%load_ext autoreload
%autoreload 2
```

```
# Makes it possible to do symbolic maths and use a control system lib
import sympy
sympy.init_printing()

# Let's also ignore some warnings here due to sympy using an old matplotlib function to render Latex equations.
import warnings
warnings.filterwarnings('ignore')
```

```
#export
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
```

```
%matplotlib inline
```

# System Response

This notebook analyses how different systems might respond to different classes of inputs.

-----------------------

## The Dominant Pole Approximation

Lower order (1st and 2nd) are weel understood and easy to characterize (speed of system, oscillations, damping…, but his is much more difficult with higher order systems.

One way to make many such systems easier to think about is to approximate the system by a lower order system using a technique called the dominant pole approximation. This approximation assumes that the slowest part of the system dominates the response, and that the faster part(s) of the system can be ignored.

Let's take some examples and verify how a few systems behaves.

We will use Sympy to do it, so we need to define some variables first.

```
t, K, tau = sympy.symbols('t, K, tau',real=True, positive=True)
s = sympy.Symbol('s')
u = sympy.Heaviside(t)

def L(f):
    return sympy.laplace_transform(f, t, s, noconds=True)
def invL(F):
    return sympy.inverse_laplace_transform(F, s, t)
```

```
def evaluate(f, times):
    res = []
    for time in times:
        res.append(f.evalf(subs={t:time}).n(chop=1e-5))
    return res
```

```
q1 = (-1.53283653303955 + 6.08703605256546e-17*sympy.I)
q2 = (-1.53283653303955 + 6.08703605256546e-5*sympy.I)

print(q1.n(chop=1e-5))
print(q2.n(chop=1e-5))
```

    -1.53283653303955
    -1.53283653303955 + 6.08703605256546e-5*I

- When we have a BIBO stable system, every mode of the system is an exponentially dumped signal.
- Beyond the initial transient, the main effect is driven by the slowest modes

```
fig, axs = plt.subplots(1,1,figsize=(15,7))

plt.plot(-0.1, 0, marker='.', markersize=25)
plt.plot([-10, -10], [-17.32, 17.32], marker='.', markersize=25, linestyle='')
plt.plot(-15.5, 0, marker='.', markersize=25)
plt.plot([-20, -20], [-34.64, 34.64], marker='.', markersize=25, linestyle='')

axs.set_xlim([-24, 1])
axs.set_ylim([-35, 35])
axs.set_xlabel('Re')
axs.set_ylabel('Img')

plt.legend(['G1', 'G2', 'G3', 'G4'])
plt.grid()

```

We can now verify what output is associated to each of these systems:

```
fig, axs = plt.subplots(1,4,figsize=(15,7))

time = np.linspace(0,10,100)

# Define our systems
G1 = 1/(s + 0.1)
zeta, w_n = 0.5, 10
G2 = w_n**2/(s**2 + 2*zeta*w_n*s + w_n**2)
G3 = 15.5/(s + 15.5)
zeta, w_n = 0.5, 10
G4 = w_n**2/(s**2 + 2*zeta*w_n*s + w_n**2)

axs[0].plot(time, evaluate(invL(G1), time), color='blue', linewidth=3), axs[0].set_ylim(-.5, 1), axs[0].set_title('G1'), axs[0].set_xlabel('time (s)')
axs[1].plot(time, evaluate(invL(G2), time), color='orange', linewidth=3), axs[1].set_ylim(-.5, 1), axs[1].set_title('G2'), axs[1].set_xlabel('time (s)')
axs[2].plot(time, evaluate(invL(G3), time), color='green', linewidth=3), axs[2].set_ylim(-.5, 1), axs[2].set_title('G3'), axs[2].set_xlabel('time (s)')
axs[3].plot(time, evaluate(invL(G4), time), color='red', linewidth=3), axs[3].set_ylim(-.5, 1), axs[3].set_title('G4'), axs[3].set_xlabel('time (s)')

fig.tight_layout()
```

- Depending on how far they are in the $s$ plane their influence in time is different.
- **Beyond the initial transient, the slower modes are those the matter**

### Example 1, second order

Let's consider when we have:

```
G1 = 0.1/(s+0.1)
G2 = 1/(s+1)
```

In this case, one pole is 10 times bigger than the other. The entire system is:

```
G = G1*G2
print(G)
```

    0.1/((s + 0.1)*(s + 1))

And we can plot their output:

```
fig, ax = plt.subplots(1,1,figsize=(8,5))

time = np.linspace(0,40,100)

ax.plot(time, evaluate(invL(G), time), linewidth=3)
ax.plot(time, evaluate(invL(G1), time), linewidth=3)
ax.plot(time, evaluate(invL(G2), time), linewidth=3)

ax.set_ylim(-.1, 0.3)
ax.set_xlabel('time (s)')
ax.grid()
fig.legend(['y_g(t)', 'y_g1(t)', 'y_g2(t)']);
```

And we can also calculate the step response:

```
fig, ax = plt.subplots(1,1,figsize=(8,5))

time = np.linspace(0,40,100)

ax.plot(time, evaluate(invL(G*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G1*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G2*1/s), time), linewidth=3)

ax.set_ylim(-.1, 1.1)
ax.set_xlabel('time (s)')
ax.grid()
fig.legend(['y_g(t)', 'y_g1(t)', 'y_g2(t)']);
```

The system $$G(s) = \frac{0.1}{(s+0.1)(s+1)} \approx \frac{0.1}{(s+0.1)}$$

### Example 2, Second order, Pole dominates but not as strongly

```
G1 = 0.2/(s+0.2)
G2 = 1/(s+1)
G=G1*G2
print('G:', G)
```

    G: 0.2/((s + 0.2)*(s + 1))

```
fig, ax = plt.subplots(1,1,figsize=(8,5))

time = np.linspace(0,40,100)

ax.plot(time, evaluate(invL(G*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G1*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G2*1/s), time), linewidth=3)

ax.set_ylim(-.1, 1.1)
ax.set_xlabel('time (s)')
ax.grid()
fig.legend(['y_g(t)', 'y_g1(t)', 'y_g2(t)']);
```

## Example 3, Second Order, neither pole dominates

```
G1 = 1.25/(s+1.25)
G2 = 1/(s+1)
G=G1*G2
print('G:', G)
```

    G: 1.25/((s + 1)*(s + 1.25))

```
fig, ax = plt.subplots(1,1,figsize=(8,5))

time = np.linspace(0,10,100)

ax.plot(time, evaluate(invL(G*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G1*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G2*1/s), time), linewidth=3)

ax.set_ylim(-.1, 1.1)
ax.set_xlabel('time (s)')
ax.grid()
fig.legend(['y_g(t)', 'y_g1(t)', 'y_g2(t)']);
```

### Simplifying Higher Order System

The dominant pole approximation can also be applied to higher order systems. Here we consider a third order system with one real root, and a pair of complex conjugate roots.

- In this case the test for the dominant pole compare $\alpha$ against $\xi \omega_n$.
- This is because $\xi \omega_n$ is the real part of the complex conjugate root
- We only compare the real parts of the roots when determining dominance because it is the real part that determines how fast the response decreases.
- Note, that as with the previous case, that the DC gain of the exact system and the two approximate systems are equal. Note, again, that H(0) is unchanged for the exact and approximate transfer funtions. This is necessary to ensure that the final value of the step response (which is determined by H(0) is unchanged).

## Example 5, Third order, Real Pole Dominates

```
alpha = 0.1 # real pole that dominates
```

$$G(s) = \frac{\alpha \omega_n^2}{(s+\alpha)(s^2+2\xi\omega_n s + \omega_n^2)} = \frac{17}{(s+1)(s^2+2s+17)}$$

$$ \approx G_dp(s) = \frac{\alpha}{s+\alpha} = \frac{0.1}{s+0.1}$$

```
G1 = 0.1/(s+0.1)
G2 = 17/(s**2+2*s+17)

G=G1*G2
print('G:', G)
```

    G: 1.7/((s + 0.1)*(s**2 + 2*s + 17))

```
fig, ax = plt.subplots(1,1,figsize=(8,5))

time = np.linspace(0,40,100)

ax.plot(time, evaluate(invL(G*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G1*1/s), time), linewidth=3)
ax.plot(time, evaluate(invL(G2*1/s), time), linewidth=3)

ax.set_ylim(-.1, 1.6)
ax.set_xlabel('time (s)')
ax.grid()
fig.legend(['y_g(t)', 'y_g1(t)', 'y_g2(t)']);
```

**Dominant pole approximation can simplify systems analysis**

The dominant pole approximation is a method for approximating a (more complicated) high order system with a (simpler) system of lower order if the location of the real part of some of the system poles are sufficiently close to the origin compared to the other poles.

--------------------

## First-order and second-order systems

- Given that the main effect is driven by the slowest modes, then the dynamics of the system can be approximated by the modes associated to the dominant poles
- Typically this ends up being a first-order system (real dominant pole) or a second-oder system (complex-conjugated poles), possibly with a constant delay
- It is hence important to understand the response of the first-order and second-order systems as they can be representive of a broader class.

### Step response of first order systems

Given a system:
$$G(s) = \frac{1}{1+\tau s}$$

with pole in $s = \frac{-1}{\tau} $

and given a input: $$u(t) = 1(t)$$

the output of the system is:
$$ y(t) = 1 - e^{-\frac{t}{\tau}}$$

This is obtained:

$$\frac{1}{1+\tau s}\frac{1}{s} = \frac{1/\tau}{s + 1/\tau} \frac{1}{s} = \frac{A}{s + 1/\tau} + \frac{B}{s}$$

where $A = -1$, $B = 1$

We can verify it with Sympy:

```
F = 1/(K*s + 1)*1/s
invL(F)
```

We can then plot the output $y(t)$:

```
time = np.linspace(0,20,100)

tau = 2

y_t = 1 - np.exp(-time/tau)
```

```
fig = plt.figure(figsize=(10,5))

plt.plot(time, y_t, linewidth=3)
plt.title('Step response')
plt.xlabel('time (s)')
plt.grid()
```

We call:

- $\tau$: time constant - Characterises completely the response of the system
- Settling time: the time it takes to get to 95% (or other times 90%, 98%) of the steady state value:
    - $t_s = -\tau ln(0.05)$

```
#y_t = -1 # Desired steady state value
final_value_pc = 0.95 # percentage of y_t

# final_value = 1 - np.exp(-t/tau)
# np.log(1-final_value) = -t/tau

t_s = -tau*np.log(1-final_value_pc)

print('Time to get to {}% of final value {:.1f}: {:.2f}s'.format(final_value_pc, y_t[-1], t_s))
```

    Time to get to 0.95% of final value 1.0: 5.99s

Note that, after $\tau$ seconds, the system gets to the $64\%$ of the final value

To recap:

```
fig = plt.figure(figsize=(10,5))

plt.plot(time, y_t, linewidth=3)
plt.plot(tau, 1 - np.exp(-tau/tau), marker='.', markersize=15)
plt.plot(t_s, final_value_pc*y_t[-1], marker='.', markersize=15)
fig.legend(['Step response', 'Tau seconds', '$t_s$ seconds'], loc='upper left')

plt.title('Step response')
plt.xlabel('time (s)')
plt.grid()
```

### Step response of second order systems

Given a system:
$$G(s) = \frac{1}{\frac{s^2}{w_n^2} + \frac{2\xi}{w_n}s + 1}$$

with $ 0 < \xi < 1 $

The system has two conjugate-complex poles:

 $$s = -\xi w_n \pm j \sqrt{1-\xi^2} $$

As before, we can calculate the step reponse:

- input $u(t)=1(t)$
- output $y(t) = 1 - \frac{1}{\sqrt{1-\xi^2}} e^{-\xi w_n t} sin( w_n\sqrt{1-\xi^2}t + arccos(\xi))$

Let's plot it:

```
xis = [0.1, 0.25, 0.5, 0.7]
wn = 1

fig = plt.figure(figsize=(10,5))
legend_strs = []
for xi in xis:
    y_t = 1 - (1/np.sqrt(1-xi**2)) * np.exp(-xi*wn*time) * np.sin(wn*np.sqrt(1-xi**2)*time+np.arccos(xi))
    plt.plot(time, y_t, linewidth=4)
    legend_strs.append('xi: ' + str(xi))

fig.legend(legend_strs, loc='upper left')

plt.title('Step response')
plt.xlabel('time (s)')
plt.grid()
```

There are a few notable points that we can identify for the response of a second order system:

- Maximum overshoot: $$ S \% = 100 e^{\Large -\frac{\xi\pi}{\sqrt{1-\xi^2}}} $$

- Time of Maximum overshoot: $$t_{max} = \frac{\pi}{w_n\sqrt{1-\xi^2}} $$

- Settling time (within a desired interval, e.g., 5%): $$t_{s} \sim -\frac{1}{\xi w_n}ln(0.05) $$

- Oscillation period: $$T_{P} = \frac{2\pi}{w_n\sqrt{1-\xi^2}} $$

Let's see where they are on the plot:

```
time = np.linspace(0,30,100)
xi = 0.2
wn = 1

# Maximum overshoot
S = np.exp(-xi*3.14/(np.sqrt(1-xi**2)))
# Time of maximum overshoot
t_max = 3.14/(wn*np.sqrt(1-xi**2))
# Settling time within 5%
t_s = -1/(xi*wn)*np.log(0.05)
# Period of the oscillations
Tp = 2*3.14/(wn*np.sqrt(1-xi**2))

fig = plt.figure(figsize=(10,5))
y_t = 1 - (1/np.sqrt(1-xi**2)) * np.exp(-xi*wn*time) * np.sin(wn*np.sqrt(1-xi**2)*time+np.arccos(xi))
plt.plot(time, y_t, linewidth=4)

plt.plot([t_max, t_max], [y_t[-1], y_t[-1]+S], marker='.', markersize=12)
plt.text(t_max+0.2, y_t[-1]+S/2, 'S', fontsize=15)

plt.plot([t_max, t_max], [0, 1], markersize=12, linestyle='--')
plt.text(t_max+0.2, 0.0, 't_max', fontsize=15)

plt.plot([0, time[-1]], [1-0.05, 1-0.05], linestyle='--', color='k')
plt.plot([0, time[-1]], [1+0.05, 1+0.05], linestyle='--', color='k')
plt.plot([t_s, t_s], [0, 1.1], linestyle='--')
plt.text(t_s+0.2, 0.0, 't_s', fontsize=15)

plt.plot([t_max, t_max+Tp], [y_t[-1]+S+0.1, y_t[-1]+S+0.1], marker='.', linestyle='--', color='k', markersize=10)
plt.plot([t_max+Tp, t_max+Tp], [y_t[-1], y_t[-1]+S+0.1], linestyle='--', color='k')
plt.text((t_max+t_max+Tp)/2, y_t[-1]+S, 'Tp', fontsize=15)

fig.legend(['xi:{:.1f}'.format(xi)], loc='upper left')

plt.title('Step response')
plt.xlabel('time (s)')
plt.grid()
```

---------------------