# On Deep Tensor Networks and the Nature of Non-Linearity

__Abstract__: Tensor Networks can be seen as a higher-order generalization of traditional deep neural networks, and yet they lack an explicit non-linearity such as applying the ReLU or sigmoid function as we do with neural nets. In this article, we cover a different perspective on what linearity vs nonlinearity means by treating function arguments as "resources" that can be consumed. Linear functions are those that "conserve resources" (i.e. they don't copy or delete their inputs).

We then cover what tensor networks are, how they work, and how this different notion of linearity gives insight into how tensor networks can learn non-linear transformations. We demonstrate that while tensor networks are locally linear, they can exhibit global non-linear behavior due to the action of copying input data and inpendently transforming each copy. Hence, unlike neural networks, the non-linearity of tensor networks arises soley from the architecture and topology of the network itself. Lastly, we demonstrate how a simple tensor network can perform reasonably well on the FashionMNIST dataset using PyTorch's `einsum` function.

Deep learning algorithms (neural networks) in their simplest form are
just a sequence of two operations composed to some depth: a linear transformation (i.e. a matrix-vector multiplication) followed by the element-wise application of a reasonably well-behaved non-linear function (called the activation function). Together the linear transformation and the non-linear function are called a "layer" of the network, and the composition of many layers forms a deep neural network. Both the depth and the non-linearity of deep neural networks are crucial for their generalizability and learning power.

The most credible explanation for why depth matters in neural networks is that depth models hierarchy, i.e. the idea that some data are at a higher and lower levels of abstraction. "My Jack Terrier dog Spot" is a concrete instance of the more abstract concept of "dog" or even "animal." Hence depth naturally models these kinds of hierarchical relationships in data, and it turns out pretty much all of the data we care about has a hierarchical structure (e.g. letters -> words -> sentences, or musical notes -> chords -> bars -> songs). Another word for hierarchy is composition; complex things are often composed of simpler things, which are composed of yet simpler things. Hence _deep_ neural networks have an (inductive) bias toward learning these compositional/hierarchical relationships, which is exactly what we want them to do.

Okay so depth is important. But why is the non-linear "activation" function so important for deep learning?
An almost tautological answer is that if we want our neural network to be able to model non-linear relationships, it must have some non-linearity itself.

Another reason is that without the non-linear functions, a sequence of just linear transformation layers
can always be "collapsed" into a single linear transformation. This is basic linear algebra; the product
of any sequence of matrices (i.e. matrix composition, hence composition of linear transformations)
can always be reduced to a single matrix/linear transformation. So without the non-linear function in each layer
we forfeit the compositionality (depth) property of deep neural networks, which allow hierarchical abstraction as
you go deeper in the network.

Since we generally use "bias units" in the linear portion of a layer, every layer can actually perform
affine transformations (linear transformation + a translation). We know from linear algebra that
affine transformations can only uniformly stretch and shrink (scale), rotate, sheer and translate vectors in a vector space. If we think of some data of interest as a point cloud in some N-dimensional space, then if it is non-random data, it will have some sort of shape.

For example, some fairly non-complex periodic system might produce data points that lie on a 2D circle or a higher-dimensional loop. The typical goal for a supervised neural network is to map this point cloud onto some other point cloud living in a different space with a different shape. When we one-hot encode the output of a neural network then we're mapping our point cloud onto essentially orthonormal unit basis vectors in a different space. Quite literally, we're transforming some geometric shape (or space) into a different shape. The task of the neural network then is to figure how to construct a new shape (the output/target space) from a starting shape (the input data point cloud) using only the tools of affine transformations and (usually) a single type of non-linear function, such as the rectified linear unit (ReLU) or sigmoid.

On the left is a graph of the sigmoid activation function. Clearly it is a curved function and hence not linear. Sigmoid has mostly fallen out of favor and ReLU has become the de facto standard activation function. The right figure is the graph for ReLU. ReLU is just $max(0,x)$, and has two "pieces," the flat line for when $x \lt 0$ and the sloped line for when $x \geq 0$. So both of ReLU's parts are lines, why does it count as a non-linear function? A standard, but unintuitive, mathematical definition for a linear function is a function that has the following properties:

(Definition 1.1: Linear Function)
$$
f : X \rightarrow Y  \\
\text{$f$ is a function from some domain X to some codomain Y} \\
f(x_1 + x_2) = f(x_1) + f(x_2) \\
f(a \times x) = a \times f(x)
$$

If you check to see if ReLU has these properties you'll find that it fails both properties, and hence it is not a linear function. For example, $relu(-1+1) \neq relu(-1) + relu(1)$ and $relu(-2 * 5) \neq -2*relu(5)$.

But this "linearity checklist" lacks a meaningful interpretation. I can use it to identify functions as linear or not
but it doesn't give me any intuition.

Here's a better (more intuitive after some explanation) definition of a linear function. If we treat the input arguments of a function as resources, then:

(Definition 1.2: Linear Function)
- A linear function is a function whose input data are always used exactly _once_. No more, no less.

This is a better definition from an intuitive point of view, but of course it lacks the mathematical precision of the linearity checklist. Nonetheless, the idea is that if we treat data as a resource that can be consumed
and transformed then a linear function is a function that completely consumes its input data without copying (i.e. duplicating) or discarding it. This notion can be formalized, but we'll wait for later.

A linear function is like a chemical reaction. If you mix oxygen molecules with hydrogen molecules and
add some energy you'll cause them to react to produce water molecules. The reagents are completely used up to produce the water molecule products. It is impossible to clone the oxygen molecules and it's impossible to delete them; all you can do is react them with other molecules. Indeed, the term "linear" as describing a function obviously has its roots in the easily visualizable line, but with definition 1.2, perhaps a better term might be "reactive function".

Consider this simple function: $f(x) = mx$, where $m$ is some constant. It is immediately clear that $x$ only appears once on the right hand side of the equation, thus it is only "used" once. In contrast the non-linear function $f(x) = x^2$ has $x$ appearing twice on the right hand side of the equation (since $x^2 = x \times x$) and it is irreducible. The input argument is being implicitly cloned or copied in the function. It's as if there is a hidden sub-function $\delta: x \rightarrow (x, x)$, such that we can rewrite $f(x) = x^2$ as $f(x) = *\delta(x)$, where the multiplication operator is acting as a prefix notated operator over the 2-tuple that the $\delta$ function produces. This looks nicer in code:

```python
import numpy as np
def clone(x):
    return x,x
def square(x):
    return np.prod(clone(x))
```

```python
square(5)
```

    25

Our slightly more mathematically precise definition of a non-linear function is thus most easily phrased in the language of computation.

(Definition 1.3: Non-linear Function OR the __No-Cloning Rule__)
- A non-linear function is any function that at any point in its computation copies its input data. A linear function never duplicates its input at any point it the computation of its result.

Note that although we can make a function such as $f(x) = x + x$, which appears to similarly clone its input, we can always reduce that expression to a non-cloned version, namely $f(x) = 2x$, hence it is linear. With $f(x) = x^2$ there is no way to reduce that expression to one where $x$ only appears once on the right hand side of the equation. In other words, if you were to write a computer program to calculate or even hand calculate $f(x) = x^2$ you would
necessarily need to work with 2 copies of the value of $x$, whereas with $f(x) = 2x$ you could evaluate the result with just a single copy. This easily extends to multivariate functions, such as $g(x,y) = x + y$. In this case we would say $g(x,y)$ is linear with respect to each of its input arguments, or __multilinear__. Individually, $x$ and $y$ respect the linearity checklist and the no-cloning rule.

The no-cloning rule for non-linearity is reminiscent of the law of conservation of energy from physics (and the no-cloning theorem of quantum mechanics); indeed it can be construed as the information theory equivalent: "information is neither created nor destroyed in an isolated process (function)". This analogy will prove useful later.

With definitions 1.2 and 1.3 we now have a meaningful explanation for why the ReLU activation function is non-linear, it's because it uses the non-linear mechanism of copying. It violates the no-cloning rule. How? Well because it involves a conditional. Remember, $relu(x) = max(0,x)$ and
$$
\begin{align}
max(0,x) & = 0, & x < 0, \\
 & = x, & x >= 0
\end{align}
$$

In order to evaluate a conditional based on some input, you have to first duplicate the input so you can use one copy to evaluate (or measure) it and then use the other copy to actually do what you want with.

Okay, so we have a good intuition for what linear versus non-linear means in terms of the copiability of resources, but how is that intuition useful? Well my eventual goal here is to explore how definitions 1.2 and 2.3 explain how tensor networks work.

One interesting path is information theory. If you have a non-linear function like $f(x) = x^2$, it is non-linear because $x$ is copied. However, consider a function like $f(x,y) = x*y$, it is called multi-linear because it is linear with respect to each of its input variables. Interestingly, if $y = x$ (always) then this function behaves exactly like f(x)=x^2 but it's just in a different form. More generally, if $f(x,y) = x*y$ but y is merely correlated with x, say $y = x + e$ (e = noise) then you'll get f(x,y) = x(x+e) = x^2 + ex, which for a sufficiently small $e$ will behave nearly the same as f(x) = x^2. So just a linear function of two variables that are highly correlated will produce non-linear behavior. Hence, if you have say two separate_linear_ dynamic systems that start off completely un-entangeled with each having their own maximum degrees of freedom, if they start to become correlated/entangled with each other by interaction, then when taken together as a composite system, it may exhibit non-linear dynamics.

### A brief detour into calculus

Let's take a bit of a detour into the world of calculus. You know that the derivative of a function $f(x)$ at a point $a$ is the slope of the tangent line at that point. More generally for multivariate and vector-valued functions, the derivative is thought of as the best linear approximation to that function. You've seen the definition of the derivative in terms of limits or in non-standard analysis as an algebraic operation with an extended real number system (the hyperreals).

You know how to differentiate simple functions such as $f(x) = x^3$ by following some derivative rules. For example, in this case we apply the rule: "take the exponent of $x$ and multiply it by $x$ and then reduce the power of $x$ by 1." So we calculate $\frac{df}{dx} = 3x^2$ (where $\frac{df}{dx}$ is Leibniz notation, read as "the derivative of the function $f$ with respect to variable $x$"). In a mathematically precise way we declare that if $f(x)=x^{r}$, then $df/dx = rx^{r-1}$. And of course we've _memorized_ a bunch of other simple rules, which together form a powerful toolset we can use to compute the derivative of many functions.

Since differentiation is all about linear approximations, what new insights might definitions 1.2 and 1.3
give us about this fundamental concept in mathematics? Well, the limit definition of a derivative and all those differentiation rules can all be seen as special cases of applying definition 1.3 to _non-linear_ functions. That is, what would happen if we took a non-linear function and re-construed it as a linear function?

Take our simple function $f(x) = x^3$ and expand it to make explicit the copying that is happening, $f(x) = x \cdot x \cdot x$. This function makes 3 copies of $x$ and then reacts them together (multiplication) to produce the final result. The copying operation is totally hidden with traditional function notation, but it happened. Where is the copy operation in traditional set-theoretic mathematics? Why is it always hidden in the notation of non-linear functions?

The answer requires back-tracking to linear algebra. Remember how a matrix encodes a linear transformation? A full-rank square matrix will always result in a dimension-preserving transformation, e.g. a full-rank 2x2 matrix will always send a 2D vector to another 2D vector. A non-square matrix such as a 2x3 matrix, on the other hand, will send a 2D vector into a 3-dimensional space.

Thus the copy operation for scalars is the linear algebraic operation of projecting a lower-dimensional vector (even a scalar) into a higher-dimensional space. Thus we can think of $f(x) = x^3$ as a function that takes a scalar, copies its input 3 times, then multiplies them together.

```python
clone = np.array([5]) @ np.array([[1,1,1]])
print(clone)
print(np.prod(clone))
```

    [5 5 5]
    125

$$ f(x) = \prod Ax = x^3$$

Where $A$ is the "copy" matrix $\begin{bmatrix} 1 & 1 & 1 \end{bmatrix}$.

What does this have to do with calculus? Well let's convert $f(x)=x\cdot x \cdot x$ into a multi-linear function, i.e. $f(x,y,z) = x\cdot y \cdot z$, except that we know $x = y = z$ (or that they're highly correlated). It is multi-linear because if we assumed data independence, then the function is linear with respect to each variable. For example, $f(5+3,y,z) = 8yz$ which is the same as $f(5,y,z) + f(3,y,z) = 5yz + 3yz = 8yz$. But since we know our data is not independent, technically this function _isn't_ linear because we can't independently control for each variable. But let's for a momement pretend we didn't know our input data is correlated in this way.

So remember, we already remember from school that the derivative of $f(x)=x^3$ is $\frac{df}{dx} = 3x^2$. We can't take "the derivative" of our new multi-linear function anymore, we can only take partial derivatives with respect to each variable. To take the partial derivative of $x$, we assume the other variables $y,z$ are held as constants, and then it becomes a simple linear equation. We do the same for $y,z$.

$$
\begin{align}
\frac{\partial{f}}{\partial{x}} = yz && \frac{\partial{f}}{\partial{y}} = xz && \frac{\partial{f}}{\partial{z}} = yx
\end{align}
$$

But wait, we can't really hold the other variables constant because we know they're perfectly correlated (actually equal). What we're getting at is a case of the total derivative (see < https://en.wikipedia.org/wiki/Total_derivative >), which is how you take the derivative of a multivariable function with interacting variables. In this particular case, since $x = y = z$, we just need to combine (sum) all these partial derivatives together and make all the variables the same, say $x$.

$$ \frac{df}{dx} = yz + xz + yx = xx + xx + xx = 3x^2 $$

Exactly what we get with through the traditional route. Or consider a slightly more complicated function:
$$
f(x) = 3x^3+x^2 \rightarrow \frac{df}{dx} = 9x^2 + 2x \\
f(a,b,c,d,e) = 3abc+de \rightarrow \frac{df}{d\{abcde\}} = 3bc+3ac+3ab + e + d \rightarrow 9x^2 + 2x
$$

Hence an intuition for copying as non-linearity even makes computing derivatives more intuitive, at least for me.

### Tensor Networks

In the tensor algebra, a scalar is a 0-tensor, a vector is a 1-tensor, a matrix is a 2-tensor, and higher order tensors don't generally have names. But in order for the linear algebra operation we just did, we had to promote a 0-tensor to a 1-tensor. Tensors are often notated by labeled indices, as if they were containers and we find elements of the container by an addressing mechanism. A 0-tensor (scalar) isn't a container, it is the "atomic ingredient" of higher tensors, hence it does not have indices. Once you box together a bunch of scalars, you get a 1-tensor (a vector), and now to locate the individual scalars in the box, we label each one with a positive integer.

If we have a vector $A = \langle{a,b,c}\rangle$, where $a,b,c$ are scalars, then we could label them $1,2,3$ in order. Hence we could refer to the $i'th$ element of $A$ as $A_i$ or $A(i)$. So $A_1 = a$ or $A(3) = c$. Now we can box together a bunch of 1-tensors (vectors) and we'll get a 2-tensor (matrix). A matrix $M(i,j)$ hence would have two indices, so we need to supply two numbers to find a single scalar in the matrix. If we supply a partial address, such as $M(1,j)$ then this would return a 1-tensor, whereas $M(1,3)$ would return a 0-tensor. We can box together a bunch of 2-tensors to get a 3-tensor and so on. Importantly, anytime you box some $k$-tensors together to form a higher order tensor, they must be of the same size.

Tensors are in a sense compositional mathematical objects. Scalars are "made of" nothing. Vectors are "made of" scalars, matrices are made of vectors, 3-tensors are made of 2-tensors. Perhaps this suggests that tensors have even more power to represent compositionality in data than do conventional neural networks, which usually only represent depth-wise hierarchy.

If we have the natural compositionality of individual tensors, we can network them together to form (deep) tensor networks! As we'll soon see, however, there is no explicit non-linear activation function application in a tensor network, everything appears perfectly linear. Yet we Networking in a neural network is nothing more impressive than matrix multiplication. Tensors have a generalization of matrix multiplication called __tensor contraction__.

#### Tensor Contraction

Take a 2-tensor (matrix) and multiply it with a vector (1-tensor).

```python
np.array([[1,0],[0,1]]) @ np.array([-2,5])
```

    array([-2,  5])

The matrix on the left is the identity matrix, so it didn't do anything to our vector, but what's the computation happening and how can it be generalized to tensors?

The answer is tensor (or index) contraction. We'll denote a matrix $A_{rt}$ and a vector $B_t$. Multiplying them together is a simple operation really.

$$ C_r = \sum_{t}A_{rt}\cdot B_t $$

This equation is saying that a tensor contraction between matrix matrix $A_{rt}$ and a vector $B_t$ results in a new vector $C_r$ (because it only has one index), and each element of $C_r$ indexed by $r$ is determined by taking the sum of the product of each subset of $A_{rt}$ with $B_t$ for every value of $t$. Often times the summation symbol is ommitted, so we express a tensor contraction just by juxtaposition, $A_{rt}B_{t}$. This often goes by the name __Einstein summation notation__, so whenever you juxtapose two tensors that have at least one shared index it denotes a tensor contraction.

A simpler example is easier to calculate. The inner product of two vectors returns a scalar.

```python
np.array([1,2,3]) @ np.array([3,2,1])
```

    10

This result is from $1*3+2*2+3*1$. Let's see why using a tensor contraction. We'll define two vectors $F_h = \langle{1,2,3}\rangle, G_i = \langle{3,2,1}\rangle $. We can take the inner product of two vectors when they're of equal length, in which case they are, so we can change the index label of $F_h$ to $F_i$. Now they have the same indices, we can do the tensor summation.

$$ H = \sum_i F_i\cdot G_i $$

It should be clear that we just multiply together corresponding elements then sum them all up, getting the inner product that numpy computed for us. Note that once we've done the sum over products, the matching indices have been fully contracted and the resulting tensor will thus have 2 less indices than the combined tensor. That is, combining the two vectors we get a tensor with two indices (a 2-tensor), but once we contract them, we get a 0-tensor.

There is a convenient graphical representation of tensors that is really helpful in reasoning about tensor contractions and tensor networks. All you do is represent a tensor as some simple geometric shape, we'll use a square, and then for each index the tensor has, you draw that many number of "legs" or __strings__ emanating from the box. For example, here is a matrix and a 3-tensor with 2 and 3 strings, respectively:

Each index has been labeled. Now we can form tensor networks by connecting these tensors if they have compatible indices. For example, here is that inner product we just worked out:

And here is a matrix vector multiplication:

You can see the vector on the left since it only has one string exiting from it whereas the matrix on the right has two strings. Once they share a string (an index), that represents a contraction. Notice that the number of remaining open-ended strings represents the rank or order of the tensor that results after the contraction. The matrix-vector multiplication diagram has 1 open string, so we know this contraction returns a new vector.

Since a simple feedforward neural network is nothing more than a series of matrix multiplications with non-linear activation functions, we can represent this easily diagrammatically:

Where the $\sigma$ (sigma) symbol represents the non-linear function. If we removed that, we would have a tensor network. But remember, this whole post is about non-linearity and tensor networks, so how do we get back the non-linearity in a tensor network?

Copying.

All we need to do is violate the no-cloning rule in the topology of the tensor network, and it will be able to learn non-linear functions. Consider the following two tensor networks. One has two component tensors that are both 3-tensors. Since there are 2 open strings, we know the result is a 2-tensor, however, you can also think of this as a directional network in which we plug in an input vector (or matrix if it's a minibatch) on the left and the network produces an output vector on the right.

The tensor network on the left has a tensor $A$ can can produce two strings from 1 input string, so it has the ability to copying its input, however, both copies get passed to a single tensor $B$, so this network cannot produce non-linear behavior because both copies will be entangled and cannot be transformed independently by tensor $B$. In contrast, the tensor network on the right _can_ produce non-linear behavior (as we'll soon show) and that's because tensor $A$ can produce two copies of its input and each copy gets independently transformed by two different tensors $B,C$, which then pass their result to tensor $D$ which computes the final result.

Ready to see some non-linearity arise from what appears to be a purely linear network? Let's see if we can train the tensor network on the right to learn the ReLU non-linear activation function. That would surely be a sign it can do something non-linear. It turns out numpy, PyTorch and TensorFlow all have functions called __einsum__ that can compute tensor contractions as we've discussed. Let's see.

```python
import torch
import torch.nn.functional as F
import torch.optim as optim
from matplotlib import pyplot as plt
import numpy as np
```

Here we define the tensor network by first setting up our individual tensor components. Then we define a function that will accept some input tensor (in this case a scalar) and will connect up the tensors into a network by defining the tensor contractions that will happen.

The notation used for the _einsum_ function in PyTorch, you write the indices of the involved tensors in a string and then pass the actual tensor objects as the second argument in a tuple. In the string, all the tensor indices for each involved tensor should be listed together (as single characters), then a comma for the next tensor with all of its indices and so on. Then you write '->' to indicate the resultant tensor with its indices.

Take the first string we see, 'sa,abc->sbc'. This means we will contract two tensors, the first has two indices, the second has three. We label the indices in the way we want the indices to contract. In this case, the 's' index represents the batch size and the 'a' index is the actual data. So we want to contract the data with the 2nd tensor, so we label its first index as 'a' as well. The resulting tensor indices will be whatever indices were not contracted.

```python
b1 = torch.rand(1,10,10, requires_grad=True)
b2 = torch.rand(10,10, requires_grad=True)
b3 = torch.rand(10,10, requires_grad=True)
b4 = torch.rand(10,1, requires_grad=True)
def tensorNet1(x):
    r1 = torch.einsum('sa,abc->sbc',(x,b1))
    r2 = torch.einsum('sbc,bd->sd',(r1,b2))
    r3 = torch.einsum('sbc,bd->sd',(r1,b3))
    r4 = torch.einsum('sd,sd,df->sf',(r2,r3,b4))
    return r4
```

If you're familiar with PyTorch, this is just a simple training loop. Normalizing the data seems to be important empirically for tensor contractions. Each tensor in the tensor network is a "trainable object." So if our tensor network contains a 3-tensor with indices of size $10\times 5\times 10$ then this tensor has $10 * 5 * 10 = 500$ total number of parameters. The network we've just defined above has a total of $1*10*10 + 10*10 + 10*10 + 10*1 = 3,000$ parameters.

```python
optimizer = optim.Adam([b1,b2,b3,b4], lr=0.001)
criterion = torch.nn.MSELoss()
losses = []
for epoch in range(5000):
    optimizer.zero_grad()
    x = F.normalize(torch.randn(10,1),dim=0)
    target = F.relu(x)
    out = tensorNet1(x)
    loss = criterion(out, target)
    losses.append(loss)
    if epoch % 500 == 0:
        print("Epoch: {} | Loss: {}".format(epoch, loss))
    loss.backward()
    optimizer.step()
```

    Epoch: 0 | Loss: 286692.1875
    Epoch: 500 | Loss: 949.7288818359375
    Epoch: 1000 | Loss: 0.07024684548377991
    Epoch: 1500 | Loss: 0.025008967146277428
    Epoch: 2000 | Loss: 0.02917313575744629
    Epoch: 2500 | Loss: 0.019280623644590378
    Epoch: 3000 | Loss: 0.03735805302858353
    Epoch: 3500 | Loss: 0.039074432104825974
    Epoch: 4000 | Loss: 0.0267774797976017
    Epoch: 4500 | Loss: 0.026810143142938614

The loss went steadily down, but let's see if it reliably learns the ReLU function. Remember, we're expecting that it will set negative numbers to 0 (or close to 0) and leave positive numbers as their original value (or at least close to their original values).

```python
t1 = F.normalize(torch.randn(4,1),dim=0) #Some test data
print(t1)
tensorNet1(t1)
```

    tensor([[-0.3854],
            [-0.6216],
            [ 0.2522],
            [-0.6337]])

    tensor([[ 0.1582],
            [ 0.4116],
            [ 0.0678],
            [ 0.4277]])

It worked! Well approximately. It's not exactly computing ReLU, but close enough, it's more like it learned how to square the numbers. In any case, the important thing is that it is definitely performing a non-linear function. There is no linear function that can only flip the sign of negative numbers and leave the sign of positive numbers alone. And if you tried this same training task using the "_linear_ topology" from the diagram on the right above, you would be completely unable to learn this non-linear function no matter how you tune the hyperparameters and no matter how big the tensors are. The wiring is everything. The non-linearity arises from the network structure (global) not the individual tensor contractions (locally linear).

To me, this is quite remarkable. That non-linearity can emerge without any explicit non-linear function. Yet it is not that surprising once you understand that no-cloning perspective of linearity. With the right wiring together of tensors, we can allow the tensor computations to return copies of input data, transform each copy independently, then pool together the result. That's how we get non-linearity. Indeed, it has been shown that certain kinds of tensor networks are directly related to convolutional neural networks with max pooling, see < https://arxiv.org/abs/1603.00162 >

The fact that the nonlinearity is due to the network itself suggests that you can dynamically tune how nonlinear the network is, by controlling the degree of data copying that the network can do. I think the generality and flexibility of tensor networks can allow you to design networks with just the right amount of inductive bias given your data. You can design tree networks, grid networks, or any topology you can think of as long as the contractions are possible.

### Are Tensor Networks useful?

Okay, so that's all theoretically very interesting, but are tensor networks useful? Are they better than neural networks at anything? Well, tensor networks arose in the Physics community as a powerful tool to simulate quantum systems, so they're definitely useful in that domain. Unfortunately the jury is still out on whether they're "better" than neural networks. Is there an advantage to imposing non-linearity from network topology alone, or is it more efficient to just include an activation function?

So far, I can't tell either from my own experiments with tensor networks or from the literature. However, because the components of tensor networks, tensors, are just high-order structured data, they have shown to be much more interpretable as parameters than the weights of a neural network that get rammed through activation functions. One small benefit I've noticed (but take it with a big grain of salt) is that I can use a much higher learning rate with a tensor network than a neural network without causing divergence during training.

Tensor networks can be very useful in a method called __tensor decompositions__. The idea is that if you have a huge matrix of data, say 1,000,000 x 1,000,000 entries, you can decompose it into a the contraction of a series of smaller tensors, such that when contracted they will sufficiently approximate the original matrix. It turns out that when you train a tensor decomposition network to approximate your data, it will often learn rather interpretable features.

Below, I've included some code for a linear (i.e. it cannot learn a non-linear function) tensor network that can be trained to classify FashionMNIST clothes. A 2-layer convolutional neural network with about 500,000 parameters can achieve over 92% accuracy, whereas this simple linear tensor network can achieve about 88.5% accuracy (but quite quickly). The reason for using a linear tensor network is mostly because a decently sized tensor network with non-linear topology (e.g. a hierarchical tree like the figure in the beginning of the post) would be too much code, and a bit too confusing to read with raw einsum notation.

I do include a slightly more complex non-linear topology at the very end that does achieve up to 94% train/ 90% test accuracy, which is getting competitive conventional neural networks. Whether more complex topologies of tensor networks can achieve even better results is left as an exercise for the reader.

### Training a (Linear) Tensor Network on Fashion MNIST

#### Setup a training function

```python
def train(model, epochs=50, params=[], lr=0.001, shape=None):
    optimizer = optim.Adam(params, lr=0.001)

    criterion = torch.nn.CrossEntropyLoss()
    train_loss = []
    train_accu = []
    for epoch in range(epochs):
        # trainning
        ave_loss = 0.0
        correct_cnt, ave_loss = 0.0, 0.0
        total_cnt = 0.0
        for batch_idx, (x, target) in enumerate(train_loader):
            optimizer.zero_grad()
            x, target = x, target
            x = torch.squeeze(x)
            if shape:
                x = torch.reshape(x,(x.shape[0],*shape))
            out = model(x)
            loss = criterion(out, target)
            ave_loss = ave_loss * 0.9 + loss.data[0] * 0.1
            _, pred_label = torch.max(out.data, 1)
            total_cnt += x.data.size()[0]
            correct_cnt += float((pred_label == target.data).sum())
            acc = correct_cnt / total_cnt
            train_loss.append(loss)
            train_accu.append(acc)
            loss.backward()
            optimizer.step()
            '''if (batch_idx+1) % 100 == 0 or (batch_idx+1) == len(train_loader):
                print('==>>> epoch: {}, batch index: {}, train loss: {:.6f}, accuracy: {}'.format(
                    epoch, batch_idx+1, ave_loss, acc))'''
        # testing
        correct_cnt, ave_loss = 0.0, 0.0
        total_cnt = 0.0
        for batch_idx, (x, target) in enumerate(test_loader):
            x = torch.squeeze(x)
            if shape:
                x = torch.reshape(x,(x.shape[0],*shape))
            out = model(x)
            loss = criterion(out, target)
            _, pred_label = torch.max(out.data, 1)
            total_cnt += x.data.size()[0]
            correct_cnt += float((pred_label == target.data).sum())
            # smooth average
            ave_loss = ave_loss * 0.9 + loss.data[0] * 0.1
            acc = correct_cnt * 1.0 / total_cnt
            if(batch_idx+1) % 100 == 0 or (batch_idx+1) == len(test_loader):
                print('==>>> epoch: {}, batch index: {}, test loss: {:.6f}, acc: {:.3f}'.format( \
                      epoch, batch_idx+1, ave_loss, acc))
    return train_loss, train_accu
```

#### Load up the FashionMNIST Data

```python
import torchvision
import torchvision.datasets as dset
import torchvision.transforms as transforms

transform = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.5,), (1.0,))])

train_set = torchvision.datasets.FashionMNIST("fashion_mnist", train=True, download=False, transform=transform)
test_set = torchvision.datasets.FashionMNIST("fashion_mnist", train=False, download=False, transform=transform)

batch_size = 100
train_loader = torch.utils.data.DataLoader(
                 dataset=train_set,
                 batch_size=batch_size,
                 shuffle=True,drop_last= True)
test_loader = torch.utils.data.DataLoader(
                dataset=test_set,
                batch_size=batch_size,
                shuffle=False, drop_last=True)

print('==>>> total trainning batch number: {}'.format(len(train_loader)))
print('==>>> total testing batch number: {}'.format(len(test_loader)))
```

    ==>>> total trainning batch number: 600
    ==>>> total testing batch number: 100

#### Define the Tensor Network

##### Total Num. Parameters: 784 \* 25 \* 25 + 25 \* 25 \* 10 = 496,250

In this case I labeled each string with its dimension size, which is often called the __bond dimension__. As you'll see below, the 4 interior strings all have a bond dimension of 25.

```python
A = torch.randn(784,25,25, requires_grad=True).float()
B = torch.randn(25,25,10, requires_grad=True).float()
def tensorNet2(x):
    try:
        C = F.normalize(torch.einsum('sa,abc->sbc',(x,A)))
        D = F.normalize(torch.einsum('sbc,bct->st',(C,B)))
        return F.softmax(D)
    except Exception:
        print("Error: {}".format(x.shape))
```

#### Train

```python
%%time
loss, acc = train(tensorNet2, epochs=50, params=[A,B], lr=0.1, shape=(784,))
```

    /Users/brandonbrown/anaconda3/envs/deeprl/lib/python3.6/site-packages/ipykernel/__main__.py:7: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
    /Users/brandonbrown/anaconda3/envs/deeprl/lib/python3.6/site-packages/ipykernel/__main__.py:20: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number
    /Users/brandonbrown/anaconda3/envs/deeprl/lib/python3.6/site-packages/ipykernel/__main__.py:45: UserWarning: invalid index of a 0-dim tensor. This will be an error in PyTorch 0.5. Use tensor.item() to convert a 0-dim tensor to a Python number

    ==>>> epoch: 0, batch index: 100, test loss: 2.204465, acc: 0.795
    ==>>> epoch: 1, batch index: 100, test loss: 2.199457, acc: 0.812
    ==>>> epoch: 2, batch index: 100, test loss: 2.196803, acc: 0.823
    ==>>> epoch: 3, batch index: 100, test loss: 2.194834, acc: 0.827
    ==>>> epoch: 4, batch index: 100, test loss: 2.193722, acc: 0.837
    ==>>> epoch: 5, batch index: 100, test loss: 2.191949, acc: 0.839
    ==>>> epoch: 6, batch index: 100, test loss: 2.191317, acc: 0.844
    ==>>> epoch: 7, batch index: 100, test loss: 2.190564, acc: 0.847
    ==>>> epoch: 8, batch index: 100, test loss: 2.190050, acc: 0.850
    ==>>> epoch: 9, batch index: 100, test loss: 2.189429, acc: 0.853
    ==>>> epoch: 10, batch index: 100, test loss: 2.188894, acc: 0.856
    ==>>> epoch: 11, batch index: 100, test loss: 2.188893, acc: 0.857
    ==>>> epoch: 12, batch index: 100, test loss: 2.188916, acc: 0.859
    ==>>> epoch: 13, batch index: 100, test loss: 2.187949, acc: 0.859
    ==>>> epoch: 14, batch index: 100, test loss: 2.187448, acc: 0.861
    ==>>> epoch: 15, batch index: 100, test loss: 2.187306, acc: 0.864
    ==>>> epoch: 16, batch index: 100, test loss: 2.187102, acc: 0.863
    ==>>> epoch: 17, batch index: 100, test loss: 2.187094, acc: 0.864
    ==>>> epoch: 18, batch index: 100, test loss: 2.186368, acc: 0.866
    ==>>> epoch: 19, batch index: 100, test loss: 2.186422, acc: 0.865
    ==>>> epoch: 20, batch index: 100, test loss: 2.186429, acc: 0.867
    ==>>> epoch: 21, batch index: 100, test loss: 2.185773, acc: 0.867
    ==>>> epoch: 22, batch index: 100, test loss: 2.186021, acc: 0.867
    ==>>> epoch: 23, batch index: 100, test loss: 2.185484, acc: 0.868
    ==>>> epoch: 24, batch index: 100, test loss: 2.186032, acc: 0.870
    ==>>> epoch: 25, batch index: 100, test loss: 2.185131, acc: 0.871
    ==>>> epoch: 26, batch index: 100, test loss: 2.185170, acc: 0.872
    ==>>> epoch: 27, batch index: 100, test loss: 2.185003, acc: 0.869
    ==>>> epoch: 28, batch index: 100, test loss: 2.184911, acc: 0.873
    ==>>> epoch: 29, batch index: 100, test loss: 2.184867, acc: 0.874
    ==>>> epoch: 30, batch index: 100, test loss: 2.184924, acc: 0.874
    ==>>> epoch: 31, batch index: 100, test loss: 2.184951, acc: 0.874
    ==>>> epoch: 32, batch index: 100, test loss: 2.184638, acc: 0.875
    ==>>> epoch: 33, batch index: 100, test loss: 2.184458, acc: 0.874
    ==>>> epoch: 34, batch index: 100, test loss: 2.184469, acc: 0.874
    ==>>> epoch: 35, batch index: 100, test loss: 2.184160, acc: 0.875
    ==>>> epoch: 36, batch index: 100, test loss: 2.183932, acc: 0.876
    ==>>> epoch: 37, batch index: 100, test loss: 2.184098, acc: 0.875
    ==>>> epoch: 38, batch index: 100, test loss: 2.184113, acc: 0.878
    ==>>> epoch: 39, batch index: 100, test loss: 2.183863, acc: 0.878
    ==>>> epoch: 40, batch index: 100, test loss: 2.183771, acc: 0.878
    ==>>> epoch: 41, batch index: 100, test loss: 2.184383, acc: 0.877
    ==>>> epoch: 42, batch index: 100, test loss: 2.183535, acc: 0.878
    ==>>> epoch: 43, batch index: 100, test loss: 2.183703, acc: 0.878
    ==>>> epoch: 44, batch index: 100, test loss: 2.183593, acc: 0.878
    ==>>> epoch: 45, batch index: 100, test loss: 2.183371, acc: 0.878
    ==>>> epoch: 46, batch index: 100, test loss: 2.183289, acc: 0.880
    ==>>> epoch: 47, batch index: 100, test loss: 2.183360, acc: 0.879
    ==>>> epoch: 48, batch index: 100, test loss: 2.183175, acc: 0.881
    ==>>> epoch: 49, batch index: 100, test loss: 2.183238, acc: 0.880
    CPU times: user 40min 35s, sys: 2min 23s, total: 42min 58s
    Wall time: 16min 17s

Not too bad right?

### Simple Non-Linear Tensor Network
#### Total Params: 784\*20^2+4\*20^2+20^4+20^3 = 479,200

```python
nl1 = torch.randn(784,20,20, requires_grad=True) # a,b,c
nl2 = torch.randn(20,20, requires_grad=True) # c,e
nl3 = torch.randn(20,20, requires_grad=True) # d,f
nl4 = torch.randn(20,20,20,20, requires_grad=True) # e,f,g
nl5 = torch.randn(20,20, requires_grad=True) # c,e
nl6 = torch.randn(20,20, requires_grad=True) # d,f
nl7 = torch.randn(20,20,10, requires_grad=True) # e,f,g
def tensorNet3(x):
    r1 = F.normalize(torch.einsum('sb,bcd->scd',(x,nl1)))
    r2 = F.normalize(torch.einsum('scd,ce->se',(r1,nl2)))
    r3 = F.normalize(torch.einsum('scd,df->sf',(r1,nl3)))
    r4 = F.normalize(torch.einsum('se,sf,efgh->sgh',(r2,r3,nl4)))
    r5 = F.normalize(torch.einsum('scd,ce->se',(r4,nl5)))
    r6 = F.normalize(torch.einsum('scd,df->sf',(r4,nl6)))
    r7 = F.normalize(torch.einsum('se,sf,efg->sg',(r5,r6,nl7)))
    return F.softmax(r7)
```

```python
%%time
loss1b, acc1b = train(model1b, epochs=50, params=[nl1, nl2, nl3, nl4, nl5, nl6, nl7], lr=0.1, shape=(784,))
```

Unfortunately, this non-linear tensor network performs significantly better with the training data (getting over 94% accuracy at 50 epochs) but the test accuracy tops out at around 88%, similar to the linear network. This just demonstrates that with the added non-linear ability, we can much more easily overfit the data. One thing to try is to add trainable bias tensors into the network as we have in neural networks.

### Conclusion

Tensor Networks can be seen as a generalization of neural networks where the non-linearity arises purely out of the network topology as a manifestation of copying data, and not by a particular choice of non-linear activation function. They can be used just about anywhere a neural network can be (albeit with unproven performance) and tensor decompositions offer a powerful way to learn features from data. Tensor Networks are being actively studied and improved and I believe it is likely they will be a great tool in your machine learning toolkit.

### References

- Yau, D. (2015). Operads of Wiring Diagrams. Retrieved from http://arxiv.org/abs/1512.01602
- Maina, S. A. (2017). Graphical Linear Algebra.
- Cichocki, A. (2014). Era of Big Data Processing: A New Approach via Tensor Networks and Tensor Decompositions, 1–30. http://doi.org/abs/1403.2048
- Genovese, F. (2017). The Way of the Infinitesimal. Retrieved from http://arxiv.org/abs/1707.00459
- Mangan, T. (2008). A gentle introduction to tensors. J. Biomedical Science and Engineering, 1, 64–67. http://doi.org/10.1016/S0004-3702(98)00053-8
- Kissinger, A., & Quick, D. (2015). A first-order logic for string diagrams, 171–189. http://doi.org/10.4230/LIPIcs.CALCO.2015.171
- Barber, A. G. (1997). Linear Type Theories , Semantics and Action Calculi. Computing. Retrieved from http://hdl.handle.net/1842/392
- Girard, J.-Y. (1995). Linear Logic: its syntax and semantics. Advances in Linear Logic, 1–42. http://doi.org/10.1017/CBO9780511629150.002
- Wadler, P., Val, A., & Arr, V. A. (1990). Philip Wadler University of Glasgow, (April), 1–21.
- Martin, C. (2004). Tensor Decompositions Workshop Discussion Notes. Palo Alto, CA: American Institute of Mathematics (AIM), 1–27. Retrieved from http://scholar.google.com/scholar?hl=en&btnG=Search&q=intitle:Tensor+Decompositions+Workshop+Discussion+Notes#0
- Kasai, H. (2017). Fast online low-rank tensor subspace tracking by CP decomposition using recursive least squares from incomplete observations, 1–21. Retrieved from http://arxiv.org/abs/1709.10276
- Zhang, Y., Zhou, G., Zhao, Q., Cichocki, A., & Wang, X. (2016). Fast nonnegative tensor factorization based on accelerated proximal gradient and low-rank approximation. Neurocomputing, 198, 148–154. http://doi.org/10.1016/j.neucom.2015.08.122
- Smith, S., Beri, A., & Karypis, G. (2017). Constrained Tensor Factorization with Accelerated AO-ADMM. Proceedings of the International Conference on Parallel Processing, 111–120. http://doi.org/10.1109/ICPP.2017.20
- Stoudenmire, E. M., & Schwab, D. J. (2016). Supervised Learning with Tensor Networks. Advances in Neural Information Processing Systems 29 (NIPS 2016), (Nips), 4799–4807. Retrieved from https://papers.nips.cc/paper/6211-supervised-learning-with-tensor-networks.pdf%0Ahttps://papers.nips.cc/paper/6211-supervised-learning-with-tensor-networksAdvances in Neural Information Processing Systems 29 (NIPS 2016)%0Ahttp://arxiv.org/abs/1605.05775
- Rabanser, S., Shchur, O., & Günnemann, S. (2017). Introduction to Tensor Decompositions and their Applications in Machine Learning, 1–13. Retrieved from http://arxiv.org/abs/1711.10781
- Zafeiriou, S. (2009). Discriminant nonnegative tensor factorization algorithms. IEEE Transactions on Neural Networks, 20(2), 217–235. http://doi.org/10.1109/TNN.2008.2005293
- Cohen, J. E., & Gillis, N. (2018). Dictionary-Based Tensor Canonical Polyadic Decomposition. IEEE Transactions on Signal Processing, 66(7), 1876–1889. http://doi.org/10.1109/TSP.2017.2777393
- Phan, A. H., & Cichocki, A. (2010). Tensor decompositions for feature extraction and classification of high dimensional datasets. Nonlinear Theory and Its Applications, IEICE, 1(1), 37–68. http://doi.org/10.1587/nolta.1.37
- Kossaifi, J., Lipton, Z. C., Khanna, A., Furlanello, T., & Anandkumar, A. (2017). Tensor Contraction & Regression Networks, 1–10. Retrieved from http://arxiv.org/abs/1707.08308
- Ecognition, R., Lipton, Z. C., & Anandkumar, A. (2018). D Eep a Ctive L Earning, 1–15.
- Stoudenmire, E. M. (2017). Learning Relevant Features of Data with Multi-scale Tensor Networks, 1–12. http://doi.org/10.1088/2058-9565/aaba1a


<link rel="stylesheet" href="../../styles/theme_style.css">

<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/font-awesome/4.7.0/css/font-awesome.min.css">

         Generation of Poincaré Plot from ECG Analysis

        Difficulty Level:

                    Tags
                    process|ecg|poincare

The Poincaré Plot defines a geometrical and non-linear approach for studying heart rate variability and in a more generic perspective is also a methodology to quantify the level of chaos in periodic processes.

This structure consists in a bidimensional plot, where each of his points has as coordinates the duration of two consecutive RR intervals (**$RR[i]$, $RR[i + 1]$**), i.e. from tachogram each pair of samples **i** and **i + 1** define a point in the Poincaré Plot.

There are two reference lines mathematically defined by the following equations:

            \begin{equation}
                RR[i + 1] = RR[i]
            \end{equation}

            \begin{equation}
                RR[i + 1] = -RR[i] + 2 avg(RR)
            \end{equation}

If the dispersion of Poincaré points in relation to these lines is high, than it can be concluded that heart rate variability is high.

1 - Importation of the needed packages

```python
# biosignalsnotebooks own package for loading and plotting the acquired data
import biosignalsnotebooks as bsnb

# Scientific packages
import numpy
import math
```

```python
# Base packages used in OpenSignals Tools Notebooks for ploting data
from bokeh.plotting import figure, output_file, show
from bokeh.io import output_notebook
from bokeh.models import Arrow, VeeHead
from bokeh.models.tools import *
#from bokeh.models.glyphs import Ellipse
output_notebook(hide_banner=True)
```

2 - Load of acquired ECG data

```python
# Load of data
data, header = bsnb.load("../../signal_samples/ecg_5_min.h5", get_header=True)
```

3 - Identification of mac address of the device and the channel used during acquisition

```python
channel = list(data.keys())[0]

print ("Mac Address: " + str(header["device name"]) + " Channel: " + str(channel))
```

    Mac Address: b'00:07:80:58:9B:3F' Channel: CH1

4 - Storage of sampling frequency and acquired data inside variables

```python
# Sampling frequency and acquired data
fs = header["sampling rate"]

# Signal Samples
signal = data[channel]
time = numpy.linspace(0, len(signal) / fs, len(signal))
```

5 - Generation of tachogram

```python
tachogram_data, tachogram_time = bsnb.tachogram(signal, fs, signal=True, out_seconds=True)
```

6 - Axes of Poincaré plot

```python
rr_i = tachogram_data[:-1]
rr_i_plus_1 = tachogram_data[1:]
```

7 - Generation of Poincaré Plot

```python
# List that store the figure handler
list_figures_1 = []

# Plotting of Tachogram
list_figures_1.append(figure(x_axis_label='RR\u1D62 (s)', y_axis_label='RR\u1D62\u208A\u2081 (s)', **bsnb.opensignals_kwargs("figure")))
list_figures_1[-1].circle(rr_i, rr_i_plus_1, **bsnb.opensignals_kwargs("line"))
```

GlyphRenderer(

id = '1038', …)

data_source = ColumnDataSource(id='1035', ...),

glyph = Circle(id='1036', ...),

hover_glyph = None,

js_event_callbacks = {},

js_property_callbacks = {},

level = 'glyph',

muted = False,

muted_glyph = None,

name = None,

nonselection_glyph = Circle(id='1037', ...),

selection_glyph = None,

subscribed_events = [],

tags = [],

view = CDSView(id='1039', ...),

visible = True,

x_range_name = 'default',

y_range_name = 'default')

```python
show(list_figures_1[-1])
```

8 - Determination of SD1 and SD2 parameters

*These parameters quantify the dispersion level of the Poincaré samples regarding to each reference line, being proportional to the short and long-term heart rate variability, respectively.*

            \begin{equation}
                SD1 = \sqrt{\frac{SDSD^2}{2}}
            \end{equation}

            \begin{equation}
                SD2 = \sqrt{2 \times SDNN^2 - SD1^2}
            \end{equation}

where:

            \begin{equation}
                SDSD = \sqrt{\frac{\sum_{i=2}^N ((RR[i] - RR[i - 1]) - \overline{RR_i - RR_{i - 1}})^2}{N - 1}}
            \end{equation}

            \begin{equation}
                SDNN = \sqrt{\frac{\sum_{i=1}^N (RR[i] - \overline{RR})^2}{N - 1}}
            \end{equation}

```python
tachogram_diff = numpy.diff(tachogram_data)
SDSD = numpy.std(tachogram_diff)
SDNN = numpy.std(tachogram_data)

SD1 = numpy.sqrt(0.5 * numpy.power(SDSD, 2))
SD2 = numpy.sqrt(2 * numpy.power(SDNN, 2) - numpy.power(SD1, 2))

avg_rr = numpy.average(tachogram_data)
```

9 - Representation of Poincaré Plot with associated parameters

```python
# List that stores the figure handler
list_figures_2 = []

# Plotting of Tachogram
color_1 = "#CF0272"
color_2 = "#F199C1"
list_figures_2.append(figure(x_axis_label='RR\u1D62 (s)', y_axis_label='RR\u1D62\u208A\u2081 (s)', **bsnb.opensignals_kwargs("figure")))
list_figures_2[-1].circle(rr_i, rr_i_plus_1, **bsnb.opensignals_kwargs("line"))
#list_figures_2[-1].ellipse(x=avg_rr, y=avg_rr, width=2 * SD1, height=2 * SD2, angle=-numpy.pi / 4, fill_color="black", fill_alpha=0.1)
list_figures_2[-1].line([numpy.min(rr_i), numpy.max(rr_i)], [numpy.min(rr_i_plus_1), numpy.max(rr_i_plus_1)], line_color="black", line_dash="dashed")
list_figures_2[-1].line([numpy.min(rr_i), numpy.max(rr_i)], [-numpy.min(rr_i) + 2 * avg_rr, -numpy.max(rr_i) + 2 * avg_rr], line_color="black", line_dash="dashed")
list_figures_2[-1].add_layout(Arrow(start=VeeHead(size=15, line_color=color_1, fill_color=color_1), end=VeeHead(size=15, line_color=color_1, fill_color=color_1), x_start=avg_rr - SD2 * math.cos(math.radians(45)), y_start=avg_rr - SD2 * math.sin(math.radians(45)), x_end=avg_rr + SD2 * math.cos(math.radians(45)), y_end=avg_rr + SD2 * math.cos(math.radians(45)), line_color=color_1, line_width=3))
list_figures_2[-1].add_layout(Arrow(start=VeeHead(size=15, line_color=color_2, fill_color=color_2), end=VeeHead(size=15, line_color=color_2, fill_color=color_2), x_start=avg_rr - SD1 * math.sin(math.radians(45)), y_start=avg_rr + SD1 * math.cos(math.radians(45)), x_end=avg_rr + SD1 * math.sin(math.radians(45)), y_end=avg_rr - SD1 * math.cos(math.radians(45)), line_color=color_2, line_width=3))
list_figures_2[-1].line([rr_i[0], rr_i[0]], [rr_i[0], rr_i[0]], legend="2 x SD1", line_color=color_2, line_width=2)
list_figures_2[-1].line([rr_i[0], rr_i[0]], [rr_i[0], rr_i[0]], legend="2 x SD2", line_color=color_1, line_width=2)
```

GlyphRenderer(

id = '1185', …)

data_source = ColumnDataSource(id='1182', ...),

glyph = Line(id='1183', ...),

hover_glyph = None,

js_event_callbacks = {},

js_property_callbacks = {},

level = 'glyph',

muted = False,

muted_glyph = None,

name = None,

nonselection_glyph = Line(id='1184', ...),

selection_glyph = None,

subscribed_events = [],

tags = [],

view = CDSView(id='1186', ...),

visible = True,

x_range_name = 'default',

y_range_name = 'default')

```python
show(list_figures_2[-1])
```

*This procedure can be automatically done by **poincare** function in **process** module of **biosignalsnotebooks** package*

```python
x_axis, y_axis, SD1, SD2 = bsnb.poincare(signal, fs, signal=True, in_seconds=False)
```

**Auxiliary Code Segment (should not be replicated by the user)**

```python
from IPython.display import Javascript
bsnb.opensignals_style(list_figures_1)
bsnb.opensignals_style(list_figures_2)
Javascript("Jupyter.notebook.execute_cells([19, 24])")
```

    <IPython.core.display.Javascript object>

```python
bsnb.css_style_apply()
```

    .................... CSS Style Applied to Jupyter Notebook .........................


This notebook is part of the *orix* documentation https://orix.readthedocs.io. Links to the documentation won’t work from the notebook.

# Crystal directions

This notebook details how to perform operations with and plot directions with
respect to the crystal reference system using Miller indices with the
[orix.vector.Miller](reference.rst#orix.vector.Miller) class.

Many of the initial examples, explaining basic crystallographic computations with
crystal lattice directions $[uvw]$ and crystal lattice planes $(hkl)$, are taken
from the textbook
*Introduction to Conventional Transmission Electron Microscopy*
(De Graef (2003)).
Some of the later examples are also inspired by MTEX' excellent documentation on
[Miller indices](https://mtex-toolbox.github.io/CrystalDirections.html) and
[operations with them](https://mtex-toolbox.github.io/CrystalOperations.html).

```python
%matplotlib inline

from diffpy.structure import Lattice, Structure
import matplotlib.pyplot as plt
import numpy as np
from orix.crystal_map import Phase
from orix.quaternion import Orientation, Rotation, symmetry
from orix.vector import AxAngle, Miller, Vector3d

plt.rcParams.update(
    {
        "figure.figsize": (7, 7),
        "font.size": 20,
        "axes.grid": True,
        "lines.markersize": 10,
        "lines.linewidth": 2,
    }
)
```

To start with, let's create a tetragonal crystal with lattice parameters
$a$ = $b$ = 0.5 nm and $c$ = 1 nm and symmetry elements described by point group
*4*

```python
tetragonal = Phase(
    point_group="4", structure=Structure(lattice=Lattice(0.5, 0.5, 1, 90, 90, 90))
)
print(tetragonal)
print(tetragonal.structure.lattice)
```

Here, the [Phase](reference.rst#orix.crystal_map.Phase) class attaches a point group symmetry,
[Symmetry](reference.rst#orix.quaternion.Symmetry), to a
[Structure](https://www.diffpy.org/diffpy.structure/package.html#diffpy.structure.structure.Structure)
containing a
[Lattice](https://www.diffpy.org/diffpy.structure/mod_lattice.html#diffpy.structure.lattice.Lattice)
(where the crystal axes are defined), and possibly some
[Atom](https://www.diffpy.org/diffpy.structure/mod_atom.html#diffpy.structure.atom.Atom)s.

## Directions $[uvw]$

A crystal lattice direction $\mathbf{m} = u \cdot \mathbf{a} + v \cdot \mathbf{b} + w \cdot \mathbf{c}$
is a vector with coordinates $u, v, w$ with respect to the crystal axes $\mathbf{a}, \mathbf{b}, \mathbf{c}$,
and is denoted $[uvw]$. We can create a set of these Miller indices by passing a list/array/tuple to `uvw` in
[Miller](reference.rst#orix.vector.Miller)

```python
m1 = Miller(uvw=[[1, 2, 0], [3, 1, 1]], phase=tetragonal)
m1.scatter(axes_labels=["x", "y", None], c=["b", "r"])
m1
```

Here, we plotted the lattice directions in the stereographic projection using the
[Vector3d.scatter()](reference.rst#orix.vector.Vector3d.scatter) plotting
method, which also works for `Miller.scatter()` since the `Miller` class
inherits most of the functionality in the `Vector3d` class.

Let's compute the dot product in nanometres and the angle in degrees between the vectors

```python
m1[0].dot(m1[1])
```

```python
np.rad2deg(m1[0].angle_with(m1[1]))
```

The length of a direct lattice vector is available via the
[Miller.length](reference.rst#orix.vector.Miller.length) property, given in
lattice parameter units (nm in this case)

```python
Miller(uvw=[0, -0.5, 0.5], phase=tetragonal).length
```

## Planes $(hkl)$

A crystal lattice plane $(hkl)$ is described by its normal vector
$\mathbf{n} = h\cdot\mathbf{a^*} + k\cdot\mathbf{b^*} + l\cdot\mathbf{c^*}$,
where $\mathbf{a^*}, \mathbf{b^*}, \mathbf{c^*}$ are the reciprocal crystal
axes. As for crystal directions $[uvw]$, we can create a set of these Miller
indices by passing `hkl` instead of `uvw` to
[Miller](reference.rst#orix.vector.Miller)

```python
m2 = Miller(hkl=m1.uvw, phase=tetragonal)
m2.scatter(c=["y", "g"], marker="*")
m2
```

Let's compute the angle in degrees between the lattice plane normals

```python
np.rad2deg(m2[0].angle_with(m2[1]))
```

Note that the lattice plane normals $(hkl)$ are not always parallel to the
lattice directions $[uvw]$, even if the indices are the same

```python
fig = m1.scatter(return_figure=True, c=["b", "r"])
m2.scatter(figure=fig, c=["y", "g"], marker="*")
```

We can get the reciprocal components of the lattice vector [114] (i.e. which
lattice plane the [114] direction is perpendicular to) by accessing
[Miller.hkl](reference.rst#orix.vector.Miller.hkl) for a `Miller` object with
crystal directions (or [Miller.uvw](reference.rst#orix.vector.Miller.uvw) for a
`Miller` object with crystal plane normals)

```python
Miller(uvw=[1, 1, 4], phase=tetragonal).hkl
```

We see that the lattice vector [114] is perpendicular to the lattice plane (1 1 16).

The length of reciprocal lattice vectors can also be accessed via the
`Miller.length` property, and equals
$\left|\mathbf{g}_{\mathrm{hkl}}\right| = \frac{1}{d_{\mathrm{hkl}}}$ in
reciprocal lattice parameter units (nm^-1 in this case), meaning we can obtain the
interplanar spacing $d_{\mathrm{hkl}}$

```python
m2.length
```

```python
1 / m2.length
```

## Zone axes

The cross product of the lattice vectors [110] and [111] in the tetragonal
crystal, in direct space, is described by a vector expressed in reciprocal
space, $(hkl)$

```python
m3 = Miller(uvw=[[1, 1, 0], [1, 1, 1]], phase=tetragonal)
m3perp = m3[0].cross(m3[1])
m3perp
```

The exact "indices" are returned, however, we can get a new `Miller` instance
with the indices rounded down or up to the *closest* smallest integer via the
[Miller.round()](reference.rst#orix.vector.Miller.round) method

```python
m3perp.round()
```

The maximum index that `Miller.round()` might return can be set by passing the
`max_index` parameter.

We can plot these direct lattice vectors $[uvw]$ and the vectors normal to the
cross product vector, i.e. normal to the reciprocal lattice vector $(hkl)$

```python
fig = m3.scatter(return_figure=True, c="r")
m3perp.draw_circle(figure=fig, color="b", linestyle="--")
```

Likewise, the cross product of reciprocal lattice vectors (110) and (111)
results in a direct lattice vector

```python
m4 = Miller(hkl=[[1, 1, 0], [1, 1, 1]], phase=tetragonal)
m4perp = m4[0].cross(m4[1]).round()
m4perp
```

```python
fig = m4.scatter(return_figure=True, c="b")
m4perp.draw_circle(figure=fig, color="r", linestyle="--")
```

## Trigonal and hexagonal index conventions

Crystal lattice vectors and planes in lattices with trigonal and hexagonal
crystal symmetry are typically expressed in Weber symbols $[UVTW]$ and
Miller-Bravais indices $(hkil)$. The definition of $[UVTW]$ used in orix follows
*Introduction to Conventional Transmission Electron Microscopy* (DeGraef, 2003)

\begin{align}
U &= \frac{2u - v}{3},\\
V &= \frac{2v - u}{3},\\
T &= -\frac{u + v}{3},\\
W &= w.
\end{align}

For a plane, the $h$, $k$ and $l$ indices are the same in $(hkl)$ and $(hkil)$,
and $i = -(h + k)$.

The first three Miller indices always sum up to zero, i.e.

\begin{align}
U + V + T &= 0,\\
h + k + i &= 0.
\end{align}

```python
trigonal = Phase(
    point_group="321", structure=Structure(lattice=Lattice(4.9, 4.9, 5.4, 90, 90, 120))
)
trigonal
```

```python
m5 = Miller(UVTW=[2, 1, -3, 1], phase=trigonal)
m5
```

```python
m6 = Miller(hkil=[1, 1, -2, 3], phase=trigonal)
m6
```

```python
fig = m5.scatter(return_figure=True, c="C0", grid_resolution=(30, 30))
m6.scatter(figure=fig, c="C1")
```

One can change the coordinate format of the Miller class, however this does not
change the vector, since all vectors are stored with respect to the cartesian
coordinate system internally

```python
print(m6, "\n\n", m6.data)
```

```python
m6.coordinate_format = "UVTW"
print(m6, "\n\n", m6.data)
```

Getting the closest integer indices, however, changes the vector

```python
m6round = m6.round()
print(m6round, "\n\n", m6round.data)
```

## Symmetrically equivalent directions and planes

The point group symmetry elements of the crystal lattice can be applied to
to describe symmetrically equivalent crystal directions and planes. This applies
to crystals in all seven systems, but we'll use the cubic crystal as an example
because of its high symmetry

```python
cubic = Phase(point_group="m-3m", structure=Structure())
print(cubic, "\n", cubic.structure.lattice.abcABG())
```

The directions parallel to the crystal axes ($\mathbf{a}$, $\mathbf{b}$,
$\mathbf{c}$) given by $[100]$, $[\bar{1}00]$, $[010]$, $[0\bar{1}0]$,
$[001]$, and $[00\bar{1}]$ ($\bar{1}$ means "-1") are symmetrically equivalent,
and can be obtained using
[Miller.symmetrise()](reference.rst#orix.vector.Miller.symmetrise)

```python
m7 = Miller(uvw=[1, 0, 0], phase=cubic)
m7.symmetrise(unique=True)
```

Without passing `unique=True`, since the cubic crystal symmetry is described by
48 symmetry operations (or elements), 48 directions would have been returned.

These six directions, known as a family, may be expressed collectively as
$\left<100\right>$, the brackets implying all six permutations or variants of
1, 0, 0. This particular family is said to have a multiplicity of 6

```python
m7.multiplicity
```

```python
m8 = Miller(uvw=[[1, 0, 0], [1, 1, 0], [1, 1, 1]], phase=cubic)
m8
```

```python
m8.multiplicity
```

Let's plot the symmetrically equivalent directions from the direction families
$\left<100\right>$, $\left<110\right>$, and $\left<111\right>$ impinging on the
upper hemisphere. By also returning the indices of which family each
symmetrically equivalent direction belongs to from `Miller.symmetrise()`, we can
give a unique color per family

```python
m9, idx = m8.symmetrise(unique=True, return_index=True)

fig = m9[idx == 0].scatter(c="C0", return_figure=True)
for i in range(1, m9.size):
    m9[idx == i].scatter(c=f"C{i}", figure=fig)
```

Similarly, symmetrically equivalent planes $(hkl)$ can be collectively
expressed as planes of the form $\{hkl\}$

```python
m10 = Miller(hkl=[[1, 0, 0], [1, 1, 0], [1, 1, 1]], phase=cubic)
m10.multiplicity
```

```python
fig = m10[0].symmetrise(unique=True).scatter(c="C0", return_figure=True)
for i in range(1, m10.size):
    m10[i].symmetrise(unique=True).scatter(c=f"C{i}", figure=fig)
```

We computed the angles between directions and plane normals earlier in this
notebook. In general,
[Miller.angle_with()](reference.rst#orix.vector.Miller.angle_with) does not
consider symmetrically equivalent directions, unless `use_symmetry=True` is
passed. Consider $(100)$ and $(\bar{1}00)$ and $(111)$ and $(\bar{1}11)$ in the stereographic
plot above

```python
m11 = Miller(hkl=[[1, 0, 0], [1, 1, 1]], phase=cubic)
m12 = Miller(hkl=[[-1, 0, 0], [-1, 1, 1]], phase=cubic)
```

```python
np.rad2deg(m11.angle_with(m12))
```

```python
np.rad2deg(m11.angle_with(m12, use_symmetry=True))
```

Thus, passing `use_symmetry=True` ensures that the smallest angles between
`m11` and the symmetrically equivalent directions to `m12` are found.

## Directions and planes in rotated crystals

Let's consider the orientation of a cubic crystal rotated $45^{\circ}$ about the
sample $\mathbf{Z}$ axis. Orientations in `orix` are defined as rotations from
the sample to the crystal (so-called "lab2crystal"), so we have to invert the
orientation to get the vector expressed in the sample reference frame.
Furthermore, as per the discussion by
Rowenhorst et al. (2015), the
axis-angle representation of a $45^{\circ}$ rotation about the sample
$\mathbf{Z}$ is written as $(\mathbf{\hat{n}}, \omega) = ([00\bar{1},90^{\circ})$

```python
o = Orientation.from_axes_angles((0, 0, -1), np.deg2rad(45), symmetry=cubic.point_group)
o
```

We can apply this orientation to a crystal direction $[uvw]$ to find this
direction in this particular crystal with respect to the sample coordinate
system

```python
m12 = Miller(uvw=[1, 1, 1], phase=cubic)
~o * m12
```

```python
# [uvw] in unrotated crystal
fig = m12.scatter(c="r", return_figure=True, axes_labels=["X", "Y", "Z"])

# [uvw] in rotated crystal
(~o * m12).scatter(figure=fig, c="b", marker="x")
```

We see that the $[111]$ vector in the crystal is aligned with the sample
$\mathbf{Y}$ direction.

We can apply the crystal symmetry to obtain the coordinates with respect to the
sample reference frame for all crystallographically equivalent directions

```python
(~o * m12.symmetrise(unique=True)).scatter(c="b", marker="x")
```

```python
o2 = Orientation.from_euler(np.deg2rad([10, 20, 30]), symmetry=trigonal.point_group)
o2
```

```python
m13 = Miller(hkil=[1, -1, 0, 0], phase=trigonal)
m13
```

```python
p = ~o2 * m13.symmetrise(unique=True)
p.scatter(
    hemisphere="both",
    grid_resolution=(30, 30),
    figure_kwargs=dict(figsize=(10, 5)),
    axes_labels=["X", "Y", "Z"],
)
```

The stereographic plots above are essentially the $\{1\bar{1}00\}$ pole figure
representation of the orientation $O_2$.

___

## A diamond [111] pole figure

Let's make a pole figure in the [111] direction of the diamond structure, as
seen in
[this figure from Wikipedia](https://commons.wikimedia.org/wiki/File:DiamondPoleFigure111.png).

The figure caption reads as follows

> *The spots in the stereographic projection show the orientation of lattice planes with the 111 in the center. Only poles for a non-forbidden Bragg reflection are shown between Miller indices -10 <= (h,k,l) <= 10. The green spots contain Miller indices up to 3, for example 111, 113, 133, 200 etc in its fundamental order. Red are those raising to 5, ex. 115, 135, 335 etc, while blue are all remaining until 10, such as 119, 779, 10.10.00 etc.*

```python
diamond = Phase(space_group=227)
md = Miller.from_highest_indices(phase=diamond, uvw=[10, 10, 10])
md
```

Remove duplicates under symmetry using [Miller.unique()](reference.rst#orix.vector.Miller.unique)

```python
md2 = md.unique(use_symmetry=True)
md2.size
```

Symmetrise to get all symmetrically equivalent directions

```python
md3 = md2.symmetrise(unique=True)
md3
```

Remove forbidden reflections in face-centered cubic structures (all hkl must be
all even or all odd)

```python
selection = np.sum(np.mod(md3.hkl, 2), axis=1)
allowed = np.array([i not in [1, 2] for i in selection], dtype=bool)
md4 = md3[allowed]
md4
```

Assign colors to each class of vectors as per the description on Wikipedia

```python
uvw = np.abs(md4.uvw)
green = np.all(uvw <= 3, axis=-1)
red = np.any(uvw > 3, axis=-1) * np.all(uvw <= 5, axis=-1)
blue = np.any(uvw > 5, axis=-1)
rgb_mask = np.column_stack([red, green, blue])

# Sanity check
print(np.count_nonzero(rgb_mask) == md4.size)
```

Rotate directions so that [111] impinges the unit sphere in the north pole
(out of plane direction)

```python
vz = Vector3d.zvector()
v111 = Vector3d([1, 1, 1])
r1 = Rotation.from_neo_euler(
    AxAngle.from_axes_angles(v111.cross(vz), v111.angle_with(vz).data)
)
r2 = Rotation.from_neo_euler(AxAngle.from_axes_angles(vz, np.deg2rad(-15)))
md5 = r2 * r1 * md4
```

Restrict to upper hemisphere and remove duplicates

```python
is_upper = md5.z > 0
md6 = md5[is_upper]
rgb_mask2 = rgb_mask[is_upper]

_, idx = md6.unit.unique(return_index=True)
md7 = md6[idx]
rgb_mask2 = rgb_mask2[idx]
```

Finally, plot the vectors

```python
rgb = np.zeros_like(md7.uvw)
rgb[rgb_mask2] = 1

md7.scatter(c=rgb, s=10, grid=False, figure_kwargs=dict(figsize=(12, 12)))
```


**Brief Honor Code**. Do the homework on your own. You may discuss ideas with your classmates, but DO NOT copy the solutions from someone else or the Internet. If stuck, discuss with TA.

**1**. (20 points)

Consider the linear transformation $f(x)$ on $\mathbb{R}^3$ that takes the standard basis $\left\{e_1,e_2,e_3\right\}$ to $\left\{v_1,v_2,v_3\right\}$ where

$$v_1=\left(\begin{matrix}10\\-10\\16\end{matrix}\right), v_2=\left(\begin{matrix}2\\-5\\20\end{matrix}\right) \textrm {and } v_3=\left(\begin{matrix}1\\-4\\13\end{matrix}\right)$$

1. Write a matrix $A$ that represents the same linear transformation. (4 points)

2. Compute the rank of $A$ using two different methods (do not use `matrix_rank`!). (4 points)

3. Find the eigenvalues and eigenvectors of $A$. (4 points)

4. What is the matrix representation of $f$ with respect to the eigenbasis? (8 points)

```python
import numpy as np

A = np.array([[10,2,1],
            [-10,-5,-4],
            [16,20,13]])
```

```python
#Method 1: SVD
U,S,V = np.linalg.svd(A)
np.count_nonzero(S)
```

    3

```python
#Method 2: LU
import scipy.linalg as la
P, L, U = la.lu(A)
np.count_nonzero(np.diag(U))
```

    3

```python
e, v = la.eig(A)
print('eigenvalues:',e)
print('eigenvectors:',v)
```

    eigenvalues: [9.+0.j 3.+0.j 6.+0.j]
    eigenvectors: [[ 5.77350269e-01  6.31950024e-16 -1.20385853e-01]
     [-5.77350269e-01 -4.47213595e-01 -2.40771706e-01]
     [ 5.77350269e-01  8.94427191e-01  9.63086825e-01]]

```python
la.inv(v) @ A @ v
```

    array([[ 9.00000000e+00,  6.77250238e-15,  6.22152043e-15],
           [ 2.08417700e-14,  3.00000000e+00, -6.87338599e-15],
           [-5.19475027e-15,  3.56964333e-15,  6.00000000e+00]])

**2**. (20 points)

You are given the following x-y coordinates (first column is x, second is y)

```
array([[  0.        ,   4.12306991],
       [  3.        , -15.47355729],
       [  4.        , -11.68725507],
       [  3.        , -20.33756693],
       [  5.        ,  -6.06401989],
       [  6.        ,  32.79353057],
       [  8.        ,  82.48658405],
       [  9.        ,  84.02971858],
       [  4.        ,  -1.30587276],
       [  8.        ,  68.59409878]])
```

- Find the coefficients $(a, b, c)$ of the least-squares fit of a quadratic function $y = a + bx + cx^2$ to the data.
- Plot the data and fitted curve using `matplotlib`.

```python
xs = np.array([
    [  0.        ,   4.12306991],
    [  3.        , -15.47355729],
    [  4.        , -11.68725507],
    [  3.        , -20.33756693],
    [  5.        ,  -6.06401989],
    [  6.        ,  32.79353057],
    [  8.        ,  82.48658405],
    [  9.        ,  84.02971858],
    [  4.        ,  -1.30587276],
    [  8.        ,  68.59409878]])
```

```python
x_sq = xs[:,0]**2
intercept = np.ones(xs.shape[0])
X = np.c_[intercept,xs[:,0],x_sq]
y = xs[:,1]
res = la.lstsq(X,y)
res
```

    (array([ -0.35762896, -11.78531232,   2.53125199]),
     842.0494779002936,
     3,
     array([132.86424774,   4.91317461,   0.97590399]))

```python
import matplotlib.pyplot as plt

x = np.linspace(-0, 9, 10000)
y_fit = res[0][0]+res[0][1]*x+res[0][2]*(x**2)
plt.scatter(xs[:,0],y,c='b',alpha=0.2)
plt.plot(x,y_fit,'r-',lw=2)
pass
```

**3**. (20 points)

Use the `svd` function to solve the least squares problem above, and repeat the same plot. Calculate the residual error $\lvert y - X\beta \rvert$.

```python
U,S,VT = la.svd(X)
m = np.zeros((U.shape[1]-len(S),len(S)))
s = np.r_[np.diag(S),m]
res2 = VT.T@la.pinv(s)@U.T@y
print(res2)

#plot
x = np.linspace(-0, 9, 10000)
y_fit = res2[0]+res2[1]*x+res2[2]*(x**2)
plt.scatter(xs[:,0],y,c='b',alpha=0.2)
plt.plot(x,y_fit,'r-',lw=2)
pass
```

```python
np.sqrt(np.sum((y-X@res2)**2))
```

    29.018088805093502

```python
la.norm(y-X@res2)
```

    29.018088805093505

**4**. (20 points)

Avoiding catastrophic cancellation.

Read the Wikipedia entry on [loss of significance](https://en.wikipedia.org/wiki/Loss_of_significance). Then answer the following problem:

The tail of the standard logistic distribution is given by $1 - F(t) = 1 - (1+e^{-t})^{-1}$.

- Define a function `f1` to calculate the tail probability of the logistic distribution using the formula given above
- Use [`sympy`](http://docs.sympy.org/latest/index.html) to find the exact value of the tail distribution (using the same symbolic formula) to 20 decimal digits
- Calculate the *relative error* of `f1` when $t = 25$ (The relative error is given by `abs(exact - approximate)/exact`)
- Rewrite the expression for the tail of the logistic distribution using simple algebra so that there is no risk of cancellation, and write a function `f2` using this formula. Calculate the *relative error* of `f2` when $t = 25$.
- How much more accurate is `f2` compared with `f1` in terms of the relative error?

```python
def f1(t):
    return 1-1/(1+np.exp(-t))
```

```python
import sympy as sp
def f(t):
    tail = 1-1/(1+sp.exp(-t))
    tail = tail.evalf(20)
    return tail
```

```python
exact = f(25)
appro = f1(25)
res1 = np.abs((exact-appro)/exact)
res1
```

    4.1759147665982646285e-6

```python
def f2(t):
    return np.exp(-t)/(1+np.exp(-t))
appro = f2(25)
res2 = np.abs((exact-appro)/exact)
res2
```

    2.3111252748218700406e-18

```python
print('f2 is',res1/res2,'times more accurate than f1.')
```

    f2 is 1806875123599.7465438 times more accurate than f1.

**5**. (20 points)

Read in `figs/elephant.jpg` as a gray-scale image. The image has $1066 \times 1600$ values. Using SVD, recreate the image with a relative error of less than 0.5%. What is the relative size of the compressed image as a percentage?

```python
from skimage import io
img = io.imread('figs/elephant.jpg', as_grey=True)
plt.imshow(img,cmap='gray')
plt.xticks([])
plt.yticks([])
pass
```

```python
U,S,VT = np.linalg.svd(img)
variance = S**2

#the first variance whose cumsum is larger than 99.5% of sum
idx = np.where(variance.cumsum()>=(variance.sum()*0.995))[0][0]
new_S = S[:idx+1]
r = new_S.shape[0]
m = np.diag(new_S)
new_img = U[:,:r]@m@VT[:r,:]
plt.imshow(new_img,cmap='Greys_r')
plt.xticks([])
plt.yticks([])
pass
```

```python
'''
For the compressed image, we only need to store U, S and VT. S only contains diagonal values so its size is r.
'''
(U.shape[0]*r+r+VT.shape[0]*r)/(1066*1600)
```

    0.032837124765478426


```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
import stanpy as stp

EI = 32000  # kN/m2
P = 5  # kN
q = 4  # kN/m
l = 4  # m

roller_support = {"w": 0, "M": 0, "H": 0}
fixed_support = {"w": 0, "phi": 0}
hinge = {"M": 0}

x_sym = sym.Symbol("x")
E = 3e7  # kN/m2
b = 0.2  # m
ha = hb = 0.3  # m
hc = 0.4  # m
l1 = 4  # m
l2 = 3  # m
hx = ha + (hc - hb) / l2 * x_sym

cs_1 = stp.cs(b=b, h=ha)
cs_2 = stp.cs(b=b, h=hx)

s1 = {"E": E, "cs": cs_1, "l": l, "bc_i": fixed_support, "bc_k": {"w": 0}}
s2 = {"E": E, "cs": cs_1,  "l": l, "bc_k": hinge, "q": q}
s3 = {"E": E, "cs": cs_2,  "l": l, "bc_k": roller_support, "P": (P, l / 2)}

s = [s1, s2, s3]

fig, ax = plt.subplots(figsize=(12,8))
stp.plot_system(ax, *s, render=True, facecolor="gray", alpha=0.2, render_scale=0.5)
stp.plot_load(ax, *s, offset=0.4)
ax.set_ylim(-1.5, 2)
ax.set_aspect("equal")
plt.show()
```

```python
dx = 1e-9
x_annotate = np.array([dx, l-dx, l, 2 * l, 5 * l / 2 - dx, 5 * l / 2])
x = np.sort(np.append(np.linspace(0, 3 * l, 1000), x_annotate))
Zi, Zk = stp.tr_solver(*s)
Fxx = stp.tr(*s, x=x)
Z_x = Fxx.dot(Zi).round(10)

w_x = Z_x[:, 0]
phi_x = Z_x[:, 1]
M_x = Z_x[:, 2]
V_x = Z_x[:, 3]
```

```python

scale = 0.5

fig, ax = plt.subplots(figsize=(12, 5))
stp.plot_system(ax, *s)
stp.plot_solution(
    ax,
    x=x,
    y=M_x,
    annotate_x=[0, l, 2 * l, 5 * l/2],
    fill_p="red",
    fill_n="blue",
    scale=scale,
    alpha=0.2,
    flip_y=True
)

ax.grid(linestyle=":")
ax.set_axisbelow(True)
ax.set_ylim(-1.0, 1)
ax.set_ylabel("M/Mmax*{}".format(scale))
ax.set_title("[M] = kNm")
plt.show()
```

```python
scale = 0.5

fig, ax = plt.subplots(figsize=(12, 5))
stp.plot_system(ax, *s)
stp.plot_solution(
    ax,
    x=x,
    y=V_x,
    annotate_x=[dx, l-dx, l, 2*l, 5*l/2-dx, 5*l/2, 3*l],
    fill_p="red",
    fill_n="blue",
    scale=scale,
    alpha=0.2,
)

ax.grid(linestyle=":")
ax.set_axisbelow(True)
ax.set_ylim(-1.0, 1)
ax.set_ylabel("V/Vmax*{}".format(scale))
ax.set_title("[V] = kN")
plt.show()
```

```python
scale = 0.2
fig, ax = plt.subplots(figsize=(12, 5))
stp.plot_system(ax, *s, lw=1, linestyle=":", c="#111111")
stp.plot_w(ax, x=x, wx=w_x, scale=scale, linestyle="-")
ax.grid(linestyle=":")
ax.set_axisbelow(True)
ax.set_ylim(-1.5, 1.5)
ax.set_ylabel("w/wmax*{}".format(scale))
ax.set_title("[w] = m")
plt.show()
```

```python

```


# Custom Enzyme Catalysed Reactions

This tutorial has been adapted from the tutorials in https://github.com/mic-pan/BGT_BiochemicalNetworkTutorials

```julia
# Since BondGraphs is not yet in the package manager, we will need to include it directly from Github
# NOTE: You will need Julia >= 1.7
using Pkg; Pkg.add(url="https://github.com/jedforrest/BondGraphs.jl")
using BondGraphs
```

```julia
using Plots
using ModelingToolkit
using Catalyst
```

## Enzyme catalysed reactions
We can create custom reaction components which describe enzyme-catalyzed reaction mechanices, such as the Michaelis-Menten rate law.

```julia
# GlobalScope parameters do not have a namespace
@parameters R T
R, T = GlobalScope(R), GlobalScope(T)

@parameters t
D = Differential(t)

# Effort and flow variables
@variables E[1:2](t) F[1:2](t)

@parameters r1 r2 k_c e_T
```

\begin{equation}
\left[
\begin{array}{c}
r1 \\
r2 \\
k_{c} \\
e_{T} \\
\end{array}
\right]
\end{equation}

```julia
# Custom component
ReMM = Dict(
    :description => """
    Michaelis-Menten reaction
    r1: Rate of reaction 1
    r2: Rate of reaction 2
    k_c: Affinity of complex relative to free enzyme
    e_T: Total amount of enzyme
    R: Universal Gas Constant
    T: Temperature
    """,
    :numports => 2,
    :variables => Dict(
        :parameters => Dict(
            r1 => 1,
            r2 => 1,
            k_c => 1,
            e_T => 1
        ),
        :globals => Dict(
            R => 8.314,
            T => 310.0
        )
    ),
    :equations => [
        0 ~ F[1] + F[2],
        0 ~ F[1] - e_T*r1*r2*k_c*(exp(E[1]/R/T) - exp(E[2]/R/T)) / (r1*exp(E[1]/R/T) + r2*exp(E[2]/R/T) + k_c*(r1+r2))
    ]
)
```

    Dict{Symbol, Any} with 4 entries:
      :variables   => Dict{Symbol, Dict{Num}}(:parameters=>Dict{Num, Int64}(e_T=>1,…
      :description => "Michaelis-Menten reaction\nr1: Rate of reaction 1\nr2: Rate …
      :equations   => Equation[0 ~ F[1](t) + F[2](t), 0 ~ (-e_T*k_c*r1*r2*(exp(E[1]…
      :numports    => 2

```julia
ReMM[:equations]
```

\begin{align}
0 =& F_1\left( t \right) + F_2\left( t \right) \\
0 =& \frac{ - e_{T} k_{c} r1 r2 \left(  - e^{\frac{E_2\left( t \right)}{R T}} + e^{\frac{E_1\left( t \right)}{R T}} \right)}{k_{c} \left( r1 + r2 \right) + r1 e^{\frac{E_1\left( t \right)}{R T}} + r2 e^{\frac{E_2\left( t \right)}{R T}}} + F_1\left( t \right)
\end{align}

We can add this component to the BondGraphs component library

```julia
addlibrary!(Dict(:ReMM => ReMM))
haskey(BondGraphs.DEFAULT_LIBRARY, :ReMM)
```

    true

We will create a simple bond graph with this component: $S \rightleftharpoons P$

```julia
bg = BondGraph("Enzyme network")
S = Component(:Ce, "S"; K=1)
P = Component(:Ce, "P"; K=1)
ReMM = Component(:ReMM, "R"; r1=200, r2=200)

add_node!(bg, [S, P, ReMM])

connect!(bg, S, ReMM)
connect!(bg, ReMM, P)

plot(bg)
```

```julia
constitutive_relations(bg)
```

\begin{align}
\frac{dS_{+}q(t)}{dt} =& \frac{R_{+}e_{T} R_{+}k_{c} R_{+}r1 R_{+}r2 \left( P_{+}K P_{+q}\left( t \right) - S_{+}K S_{+q}\left( t \right) \right)}{R_{+}k_{c} R_{+}r1 + R_{+}k_{c} R_{+}r2 + P_{+}K R_{+}r2 P_{+q}\left( t \right) + R_{+}r1 S_{+}K S_{+q}\left( t \right)} \\
\frac{dP_{+}q(t)}{dt} =& \frac{R_{+}e_{T} R_{+}k_{c} R_{+}r1 R_{+}r2 \left( S_{+}K S_{+q}\left( t \right) - P_{+}K P_{+q}\left( t \right) \right)}{R_{+}k_{c} R_{+}r1 + R_{+}k_{c} R_{+}r2 + P_{+}K R_{+}r2 P_{+q}\left( t \right) + R_{+}r1 S_{+}K S_{+q}\left( t \right)}
\end{align}

```julia
constitutive_relations(bg; sub_defaults=true)
```

\begin{align}
\frac{dS_{+}q(t)}{dt} =& \frac{200 P_{+q}\left( t \right) - 200 S_{+q}\left( t \right)}{2 + P_{+q}\left( t \right) + S_{+q}\left( t \right)} \\
\frac{dP_{+}q(t)}{dt} =& \frac{ - 200 P_{+q}\left( t \right) + 200 S_{+q}\left( t \right)}{2 + P_{+q}\left( t \right) + S_{+q}\left( t \right)}
\end{align}

```julia
sol = simulate(bg, (0.,5.); u0=[200,100])
plot(sol)
```

A slightly more complicated example $A \rightleftharpoons B \rightleftharpoons C$

```julia
rn_abc = @reaction_network ABC_Pathway begin
    (1, 1), A <--> B
    (1, 1), B <--> C
end
bg = BondGraph(rn_abc)

# Set K values to integers (for easier simplifying)
bg.A.K, bg.B.K, bg.C.K = 1, 1, 1

# Swap in new reaction types
MM1 = Component(:ReMM, "MM1"; r1=100, r2=100)
MM2 = Component(:ReMM, "MM2"; r1=100, r2=100)

swap!(bg, bg.R1, MM1)
swap!(bg, bg.R2, MM2)

plot(bg)
```

```julia
constitutive_relations(bg; sub_defaults=true)
```

\begin{align}
\frac{dA_{+}q(t)}{dt} =& \frac{ - 100 A_{+q}\left( t \right) + 100 B_{+q}\left( t \right)}{2 + A_{+q}\left( t \right) + B_{+q}\left( t \right)} \\
\frac{dB_{+}q(t)}{dt} =& \frac{ - 200 \left( B_{+q}\left( t \right) \right)^{2} + 200 A_{+q}\left( t \right) - 400 B_{+q}\left( t \right) + 200 C_{+q}\left( t \right) + 200 A_{+q}\left( t \right) C_{+q}\left( t \right)}{\left( 2 + A_{+q}\left( t \right) + B_{+q}\left( t \right) \right) \left( 2 + B_{+q}\left( t \right) + C_{+q}\left( t \right) \right)} \\
\frac{dC_{+}q(t)}{dt} =& \frac{100 B_{+q}\left( t \right) - 100 C_{+q}\left( t \right)}{2 + B_{+q}\left( t \right) + C_{+q}\left( t \right)}
\end{align}

```julia
sol = simulate(bg, (0.,20.); u0=[200,50,100])
plot(sol)
```