Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of **log()**: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-Coursera/labs/Data_files/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2021-02-04 22:14:45 URL:https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-Coursera/labs/Data_files/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

**Did you know?** When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.447527, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice

Can you calculate what is the accuracy of our model?

```python
# write your code here
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

Double-click **here** for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

| Date (YYYY-MM-DD) | Version | Changed By | Change Description    |
| ----------------- | ------- | ---------- | --------------------- |
| 2020-08-04        | 0       | Nayef      | Upload file to Gitlab |
|                   |         |            |                       |

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.

```python

```


# Matrix representation of quantum circuits - notations and gotchas
> A case study by building a tensor network to match qiskit conventions

- toc: true
- badges: true
- comments: true
- categories: [qiskit, tensor networks, quantum concepts]
- image: images/upsidedown.jpg

# Intro
Usually, for experimenting with quantum circuits I use `qiskit`. As any higher level environment it is very convenient for common tasks, but may turn out too inflexible for unusual use cases. A somewhat opposite approach is to use much lower level tools to gain in flexibility at the expense of convenience. Currently I want to use Google's `tensornetwork` [package](https://tensornetwork.readthedocs.io/en/latest/) for simulations and training of quantum circuits, but this requires building many things that are for free in `qiskit` from scratch. It is also necessary to become explicit about conventions for matrix representation of quantum circuits. As long as you stay within a single framework this may not be an issue. However for debugging purposes as well as for comparison between different frameworks this may become unavoidable. Thus, I always anticipated, that a day will come when I need to face my fears and order all terms in a tensor product by hands. Now it seems I'm past the difficult part and I'm better writing this down in case I would need to do something similar in the future.

## Defining the problem
OK, so what is the problem? Consider the following simple circuit built with `qiskit`:

```python
#collapse
import numpy as np
from qiskit import QuantumCircuit
from qiskit.quantum_info import Operator, Statevector

qc = QuantumCircuit(2)
qc.x(0)
qc.y(1)
qc.cx(0,1)
qc.draw(output='mpl')
```

It is not hard or ambiguous to interpret what this circuit does by inspecting the diagram. Say the input state is $q_0=|0\rangle$, $q_1=|1\rangle$. After $X$ acts on $q_0$ it becomes $q_0\to X |0\rangle=|1\rangle$. Similarly, $q_1$ after $Y$ becomes $q_1\to Y|1\rangle=-i |0\rangle$. Since now $q_0$ is "on" the CNOT gate switches the state of $q_1$ further to $q_0 \to -i|1\rangle$. So the end result is that $q_0=|0\rangle, q_1=|1\rangle$ is transformed to $q_0=|1\rangle, q_1=-i|1\rangle$. Or perhaps a picture says it better

Similarly, we can work out what the circuit does for other computational basis states which by linearity fully fixes the action of the circuit. Although quite explicit, this is a clumsy description. This is why the matrix notation is usually used. And indeed, we can obtain the matrix corresponding to our quantum circuit quite easily in `qiskit`:

```python
U_qs = Operator(qc).data
U_qs
```

    array([[0.+0.j, 0.+0.j, 0.+0.j, 0.-1.j],
           [0.+1.j, 0.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+1.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 0.-1.j, 0.+0.j]])

It is important to realize that a number of conventions must be chosen before such explicit matrix representation can be written down. In particular, I will emphasize two points I tripped over while studying this: ordering of the qubit states in the tensor product or "vertical ordering" and ordering of operators or "horizontal ordering".

In the rest of the post I will clarify what are the conventions used in `qiskit` and how to reproduce the circuit with the `tensornetwork` library.

# States: vertical ordering

## Single qubit states
First we need to give matrix representations to two basis states of a single qubit. Here I think it is quite uncontroversial to choose
\begin{align}
|0\rangle = \begin{pmatrix}1\\0\end{pmatrix},\qquad |1\rangle = \begin{pmatrix}0\\1\end{pmatrix} \label{kets}
\end{align}

These are the "ket" vectors. Their "bra" counterparts are
\begin{align}
\langle 0| = \begin{pmatrix}1 & 0\end{pmatrix}, \qquad \langle 1| = \begin{pmatrix}0 & 1\end{pmatrix} \label{bras}
\end{align}

With these, the following operators can be computed
\begin{align}
|0\rangle\langle 0| = \begin{pmatrix}1 & 0 \\ 0 & 0\end{pmatrix},\qquad |0\rangle\langle 1| = \begin{pmatrix}0 & 1 \\ 0 & 0\end{pmatrix} \nonumber\\ |1\rangle\langle 0| = \begin{pmatrix}0 & 0 \\ 1 & 0\end{pmatrix},\qquad |1\rangle\langle 1| = \begin{pmatrix}0 & 0 \\ 0 & 1\end{pmatrix} \label{ketbras}
\end{align}
## Multiple qubit states
When there is more than a single qubit things become a bit more interesting and potentially confusing. For example, the combined Hilbert space of two qubits $\mathcal{H}_2$ is a tensor product of single-qubit Hilbert spaces $\mathcal{H}_2 = \mathcal{H}_1 \otimes \mathcal{H}_1$ but we need to decide which qubit goes first and which goes second. In `qiskit` a convention is adopted that additional qubits join from the *left*, i.e. when we have two qubits as here

```python
#collapse
qc01 = QuantumCircuit(2)
qc01.draw(output='mpl')
```

The state of the system is $|q_1\rangle\otimes |q_0\rangle$ (this is of course only true literally for [non-entangled states](https://idnm.github.io/blog/quantum%20concepts/qiskit/2021/07/12/Entanglement.html) but we can define everything only on the computational basis states ). OK, but how do we translate this into the matrix representation? The states in the tensor product of vector spaces can be represented by the [Kronecker product](https://en.wikipedia.org/wiki/Kronecker_product) which is not symmetric with respect to permutation arguments. Best way to explain how Kronecker product works is, as usual, through examples:

\begin{align}
\begin{pmatrix} 1 \\ 0 \end{pmatrix} \otimes \begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix} a \\ b \\ 0 \\ 0 \end{pmatrix},\qquad \begin{pmatrix} 0 \\ 1 \end{pmatrix} \otimes \begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix} 0\\ 0\\ a \\ b \end{pmatrix}
\end{align}
Result for generic left vector can be obtained by linearity
\begin{align}
\begin{pmatrix} x \\ y \end{pmatrix} \otimes \begin{pmatrix} a \\ b \end{pmatrix} = x \begin{pmatrix} 1 \\ 0 \end{pmatrix} \otimes \begin{pmatrix} a \\ b\end{pmatrix} +y\begin{pmatrix} 0 \\ 1 \end{pmatrix} \otimes \begin{pmatrix} a \\ b \end{pmatrix} = \begin{pmatrix} x a\\ x b\\ y a \\ y b \end{pmatrix}  = \begin{pmatrix} x \begin{pmatrix} a\\ b\end{pmatrix} \\ y  \begin{pmatrix} a \\  b\end{pmatrix} \end{pmatrix}
\end{align}

The last notation here is a bit informal but it shows what happens. One just substitutes the right vector into all elements of the left vector, multiplied by the corresponding components of the left vector. The Kronecker product is defined in the same way for matrices of arbitrary size, not just for two vectors.

So, now we can compute matrix representations of states in the computation basis of two-qubit system

\begin{align}
|00\rangle = \begin{pmatrix}1\\0 \end{pmatrix} \otimes \begin{pmatrix}1\\0 \end{pmatrix} = \begin{pmatrix}1\\0\\0\\0\end{pmatrix},\quad |01\rangle = \begin{pmatrix}1\\0 \end{pmatrix} \otimes \begin{pmatrix}0\\1 \end{pmatrix} = \begin{pmatrix}0\\1\\0\\0\end{pmatrix} \label{01}\\
|10\rangle = \begin{pmatrix}0\\1\end{pmatrix} \otimes \begin{pmatrix}1\\0 \end{pmatrix} = \begin{pmatrix}0\\0\\1\\0\end{pmatrix},\quad |11\rangle = \begin{pmatrix}0\\1\end{pmatrix} \otimes \begin{pmatrix}0\\1 \end{pmatrix} = \begin{pmatrix}0\\0\\0\\1\end{pmatrix}
\end{align}

There is a useful relation between the index of the non-zero element $n$ in the four-dimensional representation and the computational basis bitstring $q_1q_0$, namely $n=2q_1+q_0$. I.e. the bitstring $q_1q_0$ is the binary representation of the index $n$. This extends to arbitrary number of qubits, for example since $101$ is $5$ in binary representation it follows
\begin{align}
|101\rangle = \begin{pmatrix}0\\0\\0\\0\\0\\1\\0\\0 \end{pmatrix} \label{101}
\end{align}
(try to obtain this from the two tensor products!)

Don't believe me? OK, let's check! In `qiskit` there is a convenient function to construct a vector representation from a bit string which we will take advantage of. First start with a two-qubit example:

```python
s01 = Statevector.from_label('01')
s01.data
```

    array([0.+0.j, 1.+0.j, 0.+0.j, 0.+0.j])

 Comparing to \eqref{01} we find agreement. Similarly,

```python
s101 = Statevector.from_label('101')
s101.data
```

    array([0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 0.+0.j, 1.+0.j, 0.+0.j, 0.+0.j])

Again, this is in agreement with \eqref{101}.

However, I am not sure that this relation is sufficient to justify the ordering of the tensor products. To me it is much more natural to read the circuit from top to bottom and construct the Hilbert spaces accordingly, say $\mathcal{H}_0\otimes \mathcal{H}_1 \otimes \mathcal{H}_2 \dots$ instead of $\cdots \mathcal{H}_2\otimes \mathcal{H}_1\otimes \mathcal{H}_0$. Later I will change the ordering of the tensor product to my liking, but for now we stick with the `qiskit` one. Now, with conventions for states in place we can proceed to operators.

# Operators: horizontal ordering
One can say that convention for states representation and ordering of tensor products is a "vertical" convention. There is also a "horizontal" convention which might be potentially confusing. Consider the following circuit

```python
#collapse
qc123 = QuantumCircuit(1)
qc123.rx(1, 0)
qc123.ry(2, 0)
qc123.rz(3, 0)
qc123.draw(output='mpl')
```

Here, the operator $R_x$ is appplied first, the operator $R_y$ second and $R_z$ last. So in mathematical notation the circuit corresponds to $R_z R_y R_x$ and *not* to $R_x R_y R_z$. I think that the circuit notation is actually better. We think and write from left to right, this is also a direction that time flows on paper. When another thing happens, we write it to the right and it would be convenient to apply the corresponding operator also to the right. I heard real mathematicians complain about that issue, but I guess we are stuck with it for now.

# Paper-and-pencil computation
With the set up in place we can compute the circuit of interest by hands. For convenience I plot it here once again:

```python
#collapse
qc.draw(output='mpl')
```

OK, so what is the unitary matrix corresponding to this circuit? It is
\begin{align}
U = CNOT_{01} \cdot (Y\otimes X)
\end{align}
Here
\begin{multline}
CNOT_{01} = \mathbb{1}\otimes |0\rangle\langle 0|+X\otimes |1\rangle\langle 1|=\\\begin{pmatrix}1&0\\0&1\end{pmatrix}\otimes \begin{pmatrix}1&0\\0&0\end{pmatrix}+\begin{pmatrix}0&1\\1&0\end{pmatrix}\otimes \begin{pmatrix}0&0\\0&1\end{pmatrix}=\begin{pmatrix}1&0&0&0\\0&0&0&1\\0&0&1&0\\0&1&0&0\end{pmatrix}
\end{multline}
and
\begin{align}
Y\otimes X = \begin{pmatrix} 0& -i\\i&0\end{pmatrix} \otimes \begin{pmatrix} 0& 1\\1&0\end{pmatrix}=\begin{pmatrix}0&0&0&-i\\0&0&-i&0\\0&i&0&0\\i&0&0&0\end{pmatrix}
\end{align}
Multiplying them together gives
\begin{align}
U = \begin{pmatrix}0 & 0 & 0 & -i \\ i&0&0&0 \\ 0 & i & 0 & 0 \\ 0 & 0 & -i & 0\end{pmatrix}
\end{align}
Alright, so this is indeed the matrix that `qiskit` computes:

```python
U_qs
```

    array([[0.+0.j, 0.+0.j, 0.+0.j, 0.-1.j],
           [0.+1.j, 0.+0.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+1.j, 0.+0.j, 0.+0.j],
           [0.+0.j, 0.+0.j, 0.-1.j, 0.+0.j]])

We can now also check that that the states evolve as we expected. For example recall that we computed that our quantum circuit maps $q_0 =|0\rangle, q_1 =|1\rangle$ to $q_0 =|1\rangle, q_1 =|1\rangle$ with an overall phase $-i$. Agreement with  `qiskit` can be checked as follows:

```python
qs_state = Statevector.from_label('10').evolve(qc).data
our_state = -1j*Statevector.from_label('11').data

np.allclose(qs_state, our_state)
```

    True

# Implementation with `tensornetworks`
I will not give a proper introduction to tensor networks but just make some digressions I think should be helpful as we go along.

First thing we will need are the matrices defining $X, Y$ and $CNOT$ gates. Let us introduce them.

```python
X = np.array([[0, 1], [1, 0]])
Y = np.array([[0, -1j], [1j, 0]])
CNOT = np.array([[1, 0, 0, 0],
                 [0, 0, 0, 1],
                 [0, 0, 1, 0],
                 [0, 1, 0, 0]]).reshape(2,2,2,2)
```

## An aside about reshaping
Note that as usually written, $CNOT$ is a $4\times4$ matrix. Since as a quantum gate it acts on two qubits, so it should rather be a four-legged tensor. This is the purpose of the reshaping operation. At first the reshaping might be a bit tricky, so let me illustrate it with an example. Introduce two $4\times4$ matrices and define their product:

```python
A = np.random.rand(4,4)
B = np.random.rand(4,4)

AB = A @ B
```

Now define the corresponding four-legged tensors.

```python
import tensornetwork as tn

a = tn.Node(A.reshape(2,2,2,2))
b = tn.Node(B.reshape(2,2,2,2))
```

By contracting the legs (or "edges" in terminology of `tensornetworks`) appropriately, we can reproduce the matrix multiplication. First the code:

```python
a[2] ^ b[0]
a[3] ^ b[1]

ab = tn.contractors.greedy([a, b], output_edge_order=[a[0], a[1], b[2], b[3]]).tensor
```

We can check that the contraction performed in this way exactly reproduces the matrix multiplication of original $4\times4$ matrices:

```python
np.allclose(AB, ab.reshape(4,4))
```

    True

This can be interpreted graphically as follows. First, the reshaping procedure can be thought of as splitting each of two four-dimensional legs of the original matrix into two two-dimensional ones

The labels on the legs have nothing to do with qubit states, these are just indices of edges as assigned by `tn.Node` operation on our matrices. The matrix multiplication of the original matrices in terms of four-legged tensors then can be drawn as follows

The index arrangements in the last part explain why we connected the edges in our code the way we did. This is something to watch out for. For example, connecting edges of two identity tensors in the wrong way may produce a $SWAP$ gate.

## Tensor product ordering
The matrix representation of a tensor diagram like this

also comes with a convention for the ordering of tensor products. In `tensornetwork` as well as in my opinion it is natural to order top-down, i.e. the above diagram is $U\otimes \mathbb{1}$ instead of $\mathbb{1}\otimes U$ as is adopted in `qiskit`.

## Circuit from tensor network
Alright, not we are in a position to reproduce the circuit unitary from the tensor network with nodes `x`, `y` and `cnot`:

```python

# Make tensors from matrices
x, y, cnot = list(map(tn.Node, [X, Y, CNOT]))

# Connect edges properly
cnot[2] ^ y[0]
cnot[3] ^ x[0]

# Perform the contraction ~ matrix multiplication
U_tn = tn.contractors.greedy([cnot, x, y], output_edge_order=[cnot[0], cnot[1], y[1], x[1]]).tensor
```

This way of contracting the edges corresponds to the following diagram:

Note that this is basically the original circuit with both the vertical and the horizontal directions reversed. The horizontal reversal is due to mathematical vs circuit notation (circuit is better!) and the vertical reversal is due to the mismatch between `qiskit` and `tensornetwork` ordering of tensor product (`tensornetwork`'s is better!). We can check that the unitary we obtain from this tensor network agrees with `qiskit`'s

```python
np.allclose(U_tn.reshape(4,4), U_qs)
```

    True

## A better way
I find all this misalignment very inconvenient and hard to debug. Ideally I want to look at the quantum circuit and construct the corresponding tensor network just as I read a text: from left to right and from top to bottom. Here I propose a solution which seems much more satisfactory to me. We will deal with horizontal reversal by first defining edges and then applying gates to them. This way we can read the circuit from left to right and simply add new gates, just as in `qiskit`. I will not try to revert the vertical direction directly, because I find it hard to think upside down. Instead, for comparison with `qiskit` I will use a built-in `reverse_bits` method.

So let's start by defining a function that applies a given gate to the collection of qubits (this is a slight modification of an [example](https://tensornetwork.readthedocs.io/en/latest/quantum_circuit.html) from `tensornetwork` docs) :

```python
def apply_gate(qubits, gate_tensor, positions):

    gate = tn.Node(gate_tensor)
    assert len(gate.edges) == 2*len(positions), 'Gate size does not match positions provided.'

    for i, p in enumerate(positions):
        # Connect RIGHT legs of the gate to the active qubits
        gate[i+len(positions)] ^ qubits[p]
        # Reassing active qubits to the corresponding LEFT legs of the gate
        qubits[p] = gate[i]
```

Importantly, here, in contrast to the official docs, we append the gate from the *left*, so that a sequence of application of some $G_1$ followed by $G_2$ is equivalent to the application of $G_2\cdot G_1$. Now there is one more subtlety. Previously we used matrix representation of $CNOT$ assuming that the uppermost qubit comes last in the tensor product. Now that we decided to turn this convention upside down our matrix representation of $CNOT$ must be $CNOT =|0\rangle\langle 0|\otimes \mathbb{1}+|1\rangle\langle 1|\otimes X$ or explicitly

```python
CNOT = np.array([[1, 0, 0, 0],
                 [0, 1, 0, 0],
                 [0, 0, 0, 1],
                 [0, 0, 1, 0]]).reshape(2,2,2,2)
```

With that we are ready to reconstruct our original circuit in a convenient way:

```python

# The context manager  `NodeCollection` is a bit of a magic trick
# which keeps track of all tensors in the network automatically.

all_nodes = []
with tn.NodeCollection(all_nodes):
    # I do not know how to create 'abstract' edges in `tensornetworks`.
    # Instead, I create an identity tensor and use its edges to apply new gates to.
    id0 = tn.Node(np.identity(4).reshape(2,2,2,2))
    qubits0 = id0.edges[2:4]
    qubits = id0.edges[0:2]

    apply_gate(qubits, X, [0])
    apply_gate(qubits, Y, [1])
    apply_gate(qubits, CNOT, [0,1])
```

Now let us check!

```python
U_tn = tn.contractors.greedy(all_nodes, output_edge_order=qubits+qubits0).tensor.reshape(4,4)
U_reversed_qs = Operator(qc.reverse_bits()).data

np.allclose(U_tn, U_reversed_qs)
```

    True

Wohoo, it worked! If that looked simple to you I'm happy. It took me several hours of debugging to finally match the two matrices. Just to make sure, let me conclude with a more complicated example.

```python
qc3 = QuantumCircuit(3)
qc3.x(0)
qc3.cx(0, 1)
qc3.y(1)
qc3.x(2)
qc3.cx(2, 1)
qc3.y(2)

qc3.draw(output='mpl')
```

As you can see, constructing the tensor network analog now works more or less identically:

```python
all_nodes = []
with tn.NodeCollection(all_nodes):
    id0 = tn.Node(np.identity(8).reshape(2,2,2,2,2,2))
    qubits0 = id0.edges[3:6]
    qubits = id0.edges[0:3]

    # The essential part
    apply_gate(qubits, X, [0])
    apply_gate(qubits, CNOT, [0, 1])
    apply_gate(qubits, Y, [1])
    apply_gate(qubits, X, [2])
    apply_gate(qubits, CNOT, [2, 1])
    apply_gate(qubits, Y, [2])
```

And now we compare:

```python
U3_tn = tn.contractors.greedy(all_nodes, output_edge_order=qubits+qubits0).tensor.reshape(8,8)
U3_qs_reversed = Operator(qc3.reverse_bits()).data

np.allclose(U3_tn, U3_qs_reversed)
```

    True

Alright, this resounding **True** is the best way to conclude that comes to mind. I own many thanks to [Ilia Luchnikov](https://github.com/LuchnikovI) for the help with `tensornetwork` library. Any questions are welcome in the comments!

_notebooks/2021-08-18-Matrix representation of quantum circuits.ipynb


# State feedback design for the Apollo moon lander
which has similar dynamics as a quadcopter

```python
import numpy as np
import sympy as sy
import matplotlib.pyplot as plt
import control as cm
sy.init_printing()
%matplotlib notebook
```

```python
k1, k2, h = sy.symbols('k1, k2, h', real=True, positive=True)
s = sy.symbols('s', real=True)
z = sy.symbols('z', real=False)
```

```python
A = sy.Matrix([[0,0,0], [1,0,0], [0,k2,0]])
A*A
```

```python
A*A*A
```

```python
Phi = sy.eye(3) + A*h+ A*A*h**2/2
Phis = sy.eye(3) + A*s+ A*A*s**2/2
Phi
```

```python
Phim = sy.exp(A*h)
Phim
```

```python
B = sy.Matrix([[k1],[0],[0]])
```

```python
Phis*B
```

```python
B
```

```python
Gammas = sy.integrate(Phis*B, s)
Gamma = Gammas.subs({s:h})
Gamma
```

## Discrete-time state space model
\begin{align}
x(k+1) &= \Phi(h) x(k) + \Gamma(h)u(k) = \begin{bmatrix}1 & 0 & 0\\h & 1 & 0\\\frac{h^2k_2}{2} & hk_2 & 1 \end{bmatrix} x(k) + k_1h\begin{bmatrix}1\\\frac{h}{2}\\\frac{k_2h^2}{6} \end{bmatrix} u(k)\\
y &= C x(k) = \begin{bmatrix} 0 & 0 & 1\end{bmatrix} x(k)
\end{align}

In the rest of the development, we will work in normalized time, that is $h=1$ and the sampling period is the unit of time. This gives the model
\begin{align}
x(k+1) &= \Phi(h) x(k) + \Gamma(h)u(k) = \begin{bmatrix}1 & 0 & 0\\1 & 1 & 0\\\frac{k_2}{2} & k_2 & 1 \end{bmatrix} x(k) + k_1\begin{bmatrix}1\\\frac{1}{2}\\\frac{k_2}{6} \end{bmatrix} u(k)\\
y &= C x(k) = \begin{bmatrix} 0 & 0 & 1\end{bmatrix} x(k)
\end{align}

## Check observability and controlability

```python
Wc = sy.Matrix(sy.BlockMatrix([[Gamma, Phi*Gamma, Phi*Phi*Gamma]]))
Wc
```

```python
sy.det(Wc)
```

```python
C = sy.Matrix([[0,0,1]])
Wo = sy.Matrix(sy.BlockMatrix([[C], [C*Phi], [C*Phi*Phi]]))
Wo
```

```python
sy.det(Wo)
```

## Feedback design
Desired poles (3) in z=0.7, z=0.7+0.1i, z = 0.7-0.1i

```python
pd = np.array([])
Phi_n = np.array(Phi.subs({h:1.0, k1:1.0, k2:1.0})).asarray(float)
Gamma_n = np.array(Gamma.subs({h:1.0, k1:1.0, k2:1.0}))
l1,l2,l3 = sy.symbols('l1,l2,l3')
L = sy.Matrix([[l1, l2, l3]])
Phi_min_GL = Phi_n - Gamma_n*L
Phi_min_GL
```

```python
np.asarray([1,1,1], dtype=float)
```

    array([1., 1., 1.])

```python

```


# Modeling and Simulation in Python

Copyright 2017 Allen Downey

License: [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0)

```python
import numpy as np

from sympy import *

init_printing()
```

This notebook uses SymPy to derive the equations of motion for a springy pendulum and a rigid pendulum.

Here are the symbols we need:

```python
x, y, vx, vy, ax, ay = symbols('x, y, vx, vy, ax, ay')
length0, k, m, g, R, t = symbols('length0, k, m, g, R, t')
```

We'll use vectors P, V, and A to represent position, velocity and acceleration.  The Vector class in modsim.py doesn't play nicely with SymPy, so I'll use NumPy arrays instead:

```python
P = np.array([x, y])
P
```

    array([x, y], dtype=object)

```python
V = np.array([vx, vy])
V
```

    array([vx, vy], dtype=object)

```python
A = np.array([ax, ay])
A
```

    array([ax, ay], dtype=object)

The only vector operations we need are `mag` and `hat`:

```python
def mag(P):
    return sqrt(P[0]**2 + P[1]**2)
```

```python
mag(P)
```

```python
def hat(P):
    return P / mag(P)
```

```python
hat(P)
```

    array([x/sqrt(x**2 + y**2), y/sqrt(x**2 + y**2)], dtype=object)

For convenience, I'll define intermediate variables like length:

```python
length = mag(P)
length
```

`f_spring` is the force on the particle due to the spring

```python
f_spring = -k * (length - length0) * hat(P)
f_spring
```

    array([-k*x*(-length0 + sqrt(x**2 + y**2))/sqrt(x**2 + y**2),
           -k*y*(-length0 + sqrt(x**2 + y**2))/sqrt(x**2 + y**2)], dtype=object)

xhat and yhat are unit vectors along the x and y axes:

```python
xhat = np.array([1, 0])
yhat = np.array([0, 1])
```

Now I can write force due to gravity as a Vector

```python
f_grav = -m * g * yhat
f_grav
```

    array([0, -g*m], dtype=object)

To write $\sum F = ma$, I'll define the left-hand side and right-hand sides of the equations separately.

```python
lhs = f_spring + f_grav
```

```python
rhs = m * A
```

Now I can make two equations, one for each component of F and A:

```python
eq1 = Eq(lhs[0], rhs[0])
eq1
```

```python
eq2 = Eq(lhs[1], rhs[1])
eq2
```

Now I want equations that are explicit in ax and ay.  In this case I can get them easily by dividing through by m.  But for the rigid pendulum we will need to use solve, so I want to demonstrate that here.

```python
soln = solve([eq1, eq2], [ax, ay])
```

Now we can extract the expressions for ax and ay

```python
soln[ax]
```

```python
soln[ay]
```

And we can get SymPy to format the result for LaTeX:

```python
print(latex(soln[ax]))
```

    \frac{k length_{0} x}{m \sqrt{x^{2} + y^{2}}} - \frac{k x}{m}

```python
print(latex(soln[ay]))
```

    - g + \frac{k length_{0} y}{m \sqrt{x^{2} + y^{2}}} - \frac{k y}{m}

Or generate Python code we can paste into a slope function:

```python
print(python(soln[ax]))
```

    k = Symbol('k')
    length0 = Symbol('length0')
    x = Symbol('x')
    m = Symbol('m')
    y = Symbol('y')
    e = k*length0*x/(m*sqrt(x**2 + y**2)) - k*x/m

```python
print(python(soln[ay]))
```

    g = Symbol('g')
    k = Symbol('k')
    length0 = Symbol('length0')
    y = Symbol('y')
    m = Symbol('m')
    x = Symbol('x')
    e = -g + k*length0*y/(m*sqrt(x**2 + y**2)) - k*y/m

To see these equations run, see pendulum.ipynb

### Rigid pendulum

Solving the rigid pendulum is almost the same, except we need a third equation to represent the geometric constraint.  The simplest form of the constraint is:

$ x^2 + y ^2 = length$

But this equation doesn't involve vx, vy, ax, and ay, so it's not much help.  However, if we take the time derivative of both sides, we get

$ 2 x vx + 2 y vy = 0 $

And if we take the time derivative one more time (and divide through by 2), we get

$ x ax + y ay + vx^2 + vy^2 = 0 $

And that's just what we need.

```python
eq3 = Eq(x*ax + y*ay + vx**2 + vy**2, 0)
eq3
```

Now we can represent the force due to tension as a vector with unknown magnitude, R, and direction opposite P.

```python
f_tension = -R * hat(P)
```

Again, we can write $\sum F = ma$

```python
lhs = f_grav + f_tension
```

```python
rhs = m * A
```

And make one equation for each dimension:

```python
eq4 = Eq(lhs[0], rhs[0])
eq4
```

```python
eq5 = Eq(lhs[1], rhs[1])
eq5
```

Now we have three equations in three unknowns:

```python
soln = solve([eq3, eq4, eq5], [ax, ay, R])
```

And we can get explicit expressions for ax and ay

```python
soln[ax]
```

```python
soln[ay]
```

Again, we can get the results in LaTeX or Python.

```python
print(latex(soln[ax]))
```

    \frac{x}{x^{2} + y^{2}} \left(g y - vx^{2} - vy^{2}\right)

```python
print(latex(soln[ay]))
```

    - \frac{1}{x^{2} + y^{2}} \left(g x^{2} + y \left(vx^{2} + vy^{2}\right)\right)

To see these equations run, see pendulum2.ipynb

```python
print(python(soln[ax]))
```

    x = Symbol('x')
    g = Symbol('g')
    y = Symbol('y')
    vx = Symbol('vx')
    vy = Symbol('vy')
    e = x*(g*y - vx**2 - vy**2)/(x**2 + y**2)

```python
print(python(soln[ay]))
```

    g = Symbol('g')
    x = Symbol('x')
    y = Symbol('y')
    vx = Symbol('vx')
    vy = Symbol('vy')
    e = -(g*x**2 + y*(vx**2 + vy**2))/(x**2 + y**2)

```python

```


# Tutorial

We are going to illustrate what a kernel does by letting python solve the
following problem:

## Solving the problem

```{admonition} Problem
What is the maximum of $e^{\pi}$ or $\pi ^ e$?
```

We start by creating variables and assigning them to have the value of the two
numbers we want to compare:

```python
import sympy as sym

e_to_the_pi = sym.exp(sym.pi)
pi_to_the_e = sym.pi ** sym.exp(1)
```

We now compute the maximum using Python's `max` tool:

```python
max(e_to_the_pi, pi_to_the_e)
```

$\displaystyle e^{\pi}$

## Making mistakes

For the rest of this tutorial we will make common mistakes when interactively
running code and show how to fix them.

### Undefined variables

Consider the common mistake of not running the first cell:

```python
import sympy as sym

e_to_the_pi = sym.exp(sym.pi)
pi_to_the_e = sym.pi ** sym.exp(1)
```

Running the second cell:

```python
max(e_to_the_pi, pi_to_the_e)
```

Figure {ref}`fig:name_error_because_of_not_running_a_cell` shows the error that
would occur:

```{figure} ./img/name_error_because_of_not_running_a_cell/main.png
---
width: 75%
name: fig:name_error_because_of_not_running_a_cell
---
Getting a name error when not running a cell
```

The error message is specifically telling us that the command `max(e_to_the_pi, pi_to_the_e)` cannot be run because `e_to_the_pi` is not known.

To fix this we return the first cell and run it before running the second cell
again. This is shown in Figure {ref}`fig:rerunning_the_name_error`.

```{figure} ./img/rerunning_the_name_error/main.gif
---
width: 75%
name: fig:rerunning_the_name_error
---
Rerunning the first cell to properly define all needed variables
```

```{attention}
This approach is sometimes going to be necessary even if we run the first cell
but perhaps we had an error (a typo for example) in it at the time. In which
case, we fix the typo and run it again before running the second cell.
```

### Overwriting a tool

Another common mistake is to run some code that might not be the code that
was required.
At times this can be overcome by correcting the mistake but on other occasions
the code that was run might override and essentially delete something necessary.

As an example consider if the first cell was run correctly:

```python
import sympy as sym

e_to_the_pi = sym.exp(sym.pi)
pi_to_the_e = sym.pi ** sym.exp(1)
```

**However** the second cell was run with an error (not `max = (…)` as opposed
to `max(…)`):

```python
max = (e_to_the_pi, pi_to_the_e)
```

Figure {ref}`fig:type_error_because_of_overwriting_max` shows the error that
would occur after running the correct code:

```{figure} ./img/type_error_because_of_overwriting_max/main.png
---
width: 75%
name: fig:type_error_because_of_overwriting_max
---
Getting a type error when the `max` function has been overwritten
```

As Figure {ref}`fig:type_error_because_of_overwriting_max` shows the error is
due to the fact that the `max` tool has been overwritten to be a tuple
containing the two variables.

To fix this we need to _Restart the Kernel_ which allows us to forget the
outcome of all code that has been run and start from again. To restart the
kernel: click on `Kernel` and then `Restart`.

After doing this we can run the correct code as shown in Figure
{ref}`fig:restarting_the_kernel`.

```{figure} ./img/restarting_the_kernel/main.gif
---
width: 75%
name: fig:restarting_the_kernel
---
Restarting the kernel and running the code again.
```

```{important}
In this tutorial we have

- Seen how to rerun a cell if it is needed.
- Seen how to restart a kernel if it is needed.
```


```
%matplotlib inline
from IPython.display import display
from sympy import *
from sympy.abc import x, a, n
k = Symbol("k", positive=True, integer=True)
init_printing()
```

```
n = 6
tj = [2*pi*j/n for j in range(n)]
display(tj)
```

$$\begin{bmatrix}0, & \frac{\pi}{3}, & \frac{2 \pi}{3}, & \pi, & \frac{4 \pi}{3}, & \frac{5 \pi}{3}\end{bmatrix}$$

```
fj = [expand((t - pi)*t*(t-2*pi)) for t in tj]
display(fj)
```

$$\begin{bmatrix}0, & \frac{10 \pi^{3}}{27}, & \frac{8 \pi^{3}}{27}, & 0, & - \frac{8 \pi^{3}}{27}, & - \frac{10 \pi^{3}}{27}\end{bmatrix}$$

```
ak = [2/n * sum([fj[j]*cos(k*tj[j]) for j in range(n)]) for k in range(4)]
display(ak)
```

$$\begin{bmatrix}0, & 0, & 0, & 0\end{bmatrix}$$

```
zk = [[simplify(fj[j]*sin(k*tj[j])) for j in range(n)] for k in range(1, 4)]
display(zk)
bk = [nsimplify(2/n * sum(zk[k])) for k in range(0, 3)]
display(bk)
```

$$\begin{bmatrix}\begin{bmatrix}0, & \frac{5 \sqrt{3}}{27} \pi^{3}, & \frac{4 \sqrt{3}}{27} \pi^{3}, & 0, & \frac{4 \sqrt{3}}{27} \pi^{3}, & \frac{5 \sqrt{3}}{27} \pi^{3}\end{bmatrix}, & \begin{bmatrix}0, & \frac{5 \sqrt{3}}{27} \pi^{3}, & - \frac{4 \sqrt{3}}{27} \pi^{3}, & 0, & - \frac{4 \sqrt{3}}{27} \pi^{3}, & \frac{5 \sqrt{3}}{27} \pi^{3}\end{bmatrix}, & \begin{bmatrix}0, & 0, & 0, & 0, & 0, & 0\end{bmatrix}\end{bmatrix}$$

$$\begin{bmatrix}\frac{2 \sqrt{3}}{9} \pi^{3}, & \frac{2 \sqrt{3}}{81} \pi^{3}, & 0\end{bmatrix}$$

```
plot1 = plot((x-pi)*x*(x-2*pi),(x, 0, 2*pi), show=False)
plot1.extend(plot(bk[0]*sin(x)+bk[1]*sin(2*x), (x, 0, 2*pi), show=False, line_color='r'))
plot1.extend(plot(bk[0]*sin(x), (x, 0, 2*pi), show=False, line_color='g'))
plot1.extend(plot(bk[1]*sin(2*x), (x, 0, 2*pi), show=False, line_color='y'))
plot1.show()
```

```
simplify(integrate(1/pi * x * (x-pi) * (x-2*pi) * cos(k*x), (x, 0, 2*pi)))
```

```
simplify(integrate(1/pi * x * (x-pi) * (x-2*pi) * sin(k*x), (x, 0, 2*pi)))
```

```
[N(bk[j]) for j in range(3)]
```

$$\begin{bmatrix}11.9343214586261, & 1.32603571762512, & 0\end{bmatrix}$$

```
[12/(j**3) for j in range(1, 4)]
```

$$\begin{bmatrix}12.0, & 1.5, & 0.4444444444444444\end{bmatrix}$$


```python
import path_setup
#setup path to include external recourses
nb_dir = path_setup.path_setup()
data_dir = nb_dir + '/Data/'

# The modules used for actually crunching the simulation numbers (nDim_Symbolic_Function, Integrator) were made by me and are
# covered under the MIT free license. Please respect these terms and keep the license file with the code, Thank you!

import nDim_Symbolic_Function as vfunc
import matplotlib.pyplot as plt
import Integrator as numInt
import matplotlib as mpl
import sympy as sym
import numpy as np
import torch
import math

# this is a major player in computational speed using c evaluatable lambda functions
# Without this module, we lose about on the highend, on the order of 100 cpu cycles to return a result fronm python vs c lambdas
from sympy.utilities.lambdify import lambdastr
# set output precision so as to not spam the screen with long numbers
np.set_printoptions(precision=2)
# needs ipympl for realtime interactive displays using vs code
from IPython.display import display, Markdown, Latex
# Shebang line for interactive output in vs_code, comment this out if you have troubles running the notebook
%matplotlib widget

```

# Variables and equations
> Here I will setup the dependant and intependant variables to be used in the simulation of a baseball. I am using sympy to manage the equations in symbolic form and for CAS purposes (algebraic manipulation, reduction, transformations, etc...).
> As a note, I am setting up my axis such that z is in the verticle direction (up to the sky and down to the center of the Earth), with the x-y plane forming the 'ground'
# equations of the kinematics
> I have elected to use just newtons laws for this, as we are not concerned with relativistic effects (and as a note I have had these effects for speeds as low as 1000m/s for a rocket problem I worked on once, this was supprising that the force addition required special relativity to compute properly). The Euler Cromer method that I have used is

```python
t = sym.symbols('t')

# Sympy Functions
x = sym.Function('x')(t)
y = sym.Function('y')(t)
z = sym.Function('z')(t)

dxdt = x.diff(t)
dydt = y.diff(t)
dzdt = z.diff(t)

Cd = 0.35       # unitless coefficent of drag : https://blogs.fangraphs.com/exploring-the-variation-in-the-drag-coefficient-of-the-baseball/
# according to this websight, the values of the Cd for a baseball depending on the season using a typical regulation baseball range from around
# 0.3 to o.375 (unitless), So I will be using a 0.35 for Cd. This differs from the 0.45 of a perfect sphere due to surface imperfections causing
# turbulance similar to a golfball, thereby lowering the Cd.
rho = 1.205     # kg/m^3 <=> 1.205 g/L : this is the density of air at astp
S_a = 0.004145  # m^2 : this is the minimum cross-sectional area of a regulation baseball, it was supprising to me that the top end of this can be
# as much as 5% greater! A bit of a loose regulation if you ask me!
drag_const = Cd*rho*S_a/2 # this is the constant for the drag equation
print(f'The drag constant is equal to {drag_const} kg/m')

# Force components in cartesian coords set equal to the mass times the acceleration and then later integrated numerically
Fx = - drag_const*dxdt**2
Fy = - drag_const*dydt**2
Fz = - 9.81 - drag_const*dzdt**2 # assuming nominal Earth gravity in meters per second

F_ = vfunc.nDim_Symbolic_Function([Fx,Fy,Fz],[dxdt,dydt,dzdt],[[dxdt],[dydt],[dzdt]])
```

    The drag constant is equal to 0.0008740768750000001 kg/m

# Choice of dt
Looking up some numbers on wikipedia, the average MLB fastball is pitched at approximatly 41 meters per second. I chose a dt such that there will be on the order of a few centimeters of movement or less at pitched speed per simulation timeslice. This happens to work out to just about 250microseconds with rounding and gives nice numbers to work for the simulation. After testing however, 2.5E-03 seconds was a reasonable choice with little if any noticible loss in accuracy and brings down the computation time by a whole order of magnitude.

```python
# I set this to 10 seconds for no other reason than to let it run for a while and add
# an order of magnitude to the run time, but you can delete the comment and input whatever you want.
real_time = 10 # int(input('Enter durration of (positive integer) time in seconds to simulate: '))

dt = 2.5E-03 # seconds

# The variable time_slice sets the output frequency so that we are not capturing more data then needed from the
# simulation, IE we dont need to follow the baseball every millimeter, but looking at it in say 5 centimeter
# increments is much less data by an order of magnitude or more, but gives a proper picture of the ball moving overall
# With this in mind, I set the output time step at 1 second after a little bit of experimenting and this seems to be a
# good compromise between granularity and memory for longer running simulations (say, a minute of real time or more).
output_time_step = 1 # second

# initial conditions
# We throw the baseball such that we can see it has proper behavior in 3D, it is thrown at a pi/4 angle from the xy plane and at a pi/4 angle from the xz plane.
V_mag = 44.7 # m/s <=> 100mph
theta = np.pi/4
phi = np.pi/4
Vx = V_mag*np.cos(theta)*np.sin(phi)
Vy = V_mag*np.sin(theta)*np.sin(phi)
Vz = V_mag*np.cos(phi)
x_0 = 0
y_0 = 0
z_0 = 1.8288 # meters. Eew american freedom units make messy numbers lol

print(f'The initial speed is {np.sqrt(Vx**2+Vy**2+Vz**2)} meters per second')
```

    The initial speed is 44.7 meters per second
    the output step size is 400

# Simulation and plotting results
> I reccomend the Euler cromer method here because it is reasonably fast at an average of 1.500132 seconds runtime for a dt of 2.5E-03 seconds for a 10 second total simulation (faster than RK4 which runs at an average of 6.263453 seconds but has a much greater accuracy for lower dt) and has (SLIGHTLLLLLLY pretty much negligibly) more accuracy than the standard Eulers method. To be honest though, there isn't much difference between the two in terms of computational speed or accuract with this sumulation. F.sim_and_plot(arg) is the plotting function and takes an integer 1, 2 or 3. 1 corresponds to Euler, 2 to Euler-Cromer, 3 to Runge Kutta 4.

```python
# This is mostly self contained right now, but needs to be reinstantized each time.
# The __del__ method in python is much more finicky than C++ destructors and a bit of a mystery to me :(

F = numInt.Integrator(F_,0,real_time,output_time_step,dt,x_0,y_0,z_0,Vx,Vy,Vz)
F.sim_and_plot(3)
del F

F = numInt.Integrator(F_,0,real_time,output_time_step,dt,x_0,y_0,z_0,Vx,Vy,Vz)
F.sim_and_plot(2)
del F

F = numInt.Integrator(F_,0,real_time,output_time_step,dt,x_0,y_0,z_0,Vx,Vy,Vz)
F.sim_and_plot(1)
del F
```

    Canvas(toolbar=Toolbar(toolitems=[('Home', 'Reset original view', 'home', 'home'), ('Back', 'Back to previous …

    Euler-Cromer method...
    Completed in 0:00:02.657624!


```python
import numpy as np
import sympy as s
```

```python
from sympy import init_printing
init_printing(use_latex = True)
```

```python
x , nu , t = s.symbols('x nu t')
phi = (s.exp(-(x - 4*t)**2/(4 * nu * (t + 1)))) + (s.exp(-(x - 4*t - 2*s.pi)**2/(4 * nu * (t + 1))))
phi
```

```python
phiprime = phi.diff(x)
phiprime
```

```python
from sympy.utilities.lambdify import lambdify
u = (-2 * nu/phi) * phiprime + 4
u
```

```python
ufunc = lambdify((t , x , nu) , u)
print(ufunc(1 , 4 , 3))
```

    3.49170664206445

```python
from matplotlib import pylab
nx = 200
nt = 200
dx = 2*np.pi/(nx - 1)
nu = 0.07
dt = dx * nu

x = np.linspace(0 , 2 * np.pi , nx)
un = np.empty(nx)
t = 0
u = np.asarray([ufunc(t , x0 , nu) for x0 in x])
u
```

    array([4.        , 4.0315738 , 4.06314759, 4.09472139, 4.12629518,
           4.15786898, 4.18944277, 4.22101657, 4.25259036, 4.28416416,
           4.31573796, 4.34731175, 4.37888555, 4.41045934, 4.44203314,
           4.47360693, 4.50518073, 4.53675452, 4.56832832, 4.59990211,
           4.63147591, 4.66304971, 4.6946235 , 4.7261973 , 4.75777109,
           4.78934489, 4.82091868, 4.85249248, 4.88406627, 4.91564007,
           4.94721387, 4.97878766, 5.01036146, 5.04193525, 5.07350905,
           5.10508284, 5.13665664, 5.16823043, 5.19980423, 5.23137803,
           5.26295182, 5.29452562, 5.32609941, 5.35767321, 5.389247  ,
           5.4208208 , 5.45239459, 5.48396839, 5.51554218, 5.54711598,
           5.57868978, 5.61026357, 5.64183737, 5.67341116, 5.70498496,
           5.73655875, 5.76813255, 5.79970634, 5.83128014, 5.86285394,
           5.89442773, 5.92600153, 5.95757532, 5.98914912, 6.02072291,
           6.05229671, 6.0838705 , 6.1154443 , 6.14701809, 6.17859189,
           6.21016569, 6.24173948, 6.27331328, 6.30488707, 6.33646087,
           6.36803466, 6.39960846, 6.43118225, 6.46275605, 6.49432985,
           6.52590364, 6.55747744, 6.58905123, 6.62062503, 6.65219882,
           6.68377261, 6.71534638, 6.74692008, 6.77849348, 6.81006563,
           6.84163265, 6.87317848, 6.90463692, 6.93573494, 6.96534702,
           6.98884198, 6.98731033, 6.88594413, 6.42420025, 5.05281199,
           2.94718801, 1.57579975, 1.11405587, 1.01268967, 1.01115802,
           1.03465298, 1.06426506, 1.09536308, 1.12682152, 1.15836735,
           1.18993437, 1.22150652, 1.25307992, 1.28465362, 1.31622739,
           1.34780118, 1.37937497, 1.41094877, 1.44252256, 1.47409636,
           1.50567015, 1.53724395, 1.56881775, 1.60039154, 1.63196534,
           1.66353913, 1.69511293, 1.72668672, 1.75826052, 1.78983431,
           1.82140811, 1.85298191, 1.8845557 , 1.9161295 , 1.94770329,
           1.97927709, 2.01085088, 2.04242468, 2.07399847, 2.10557227,
           2.13714606, 2.16871986, 2.20029366, 2.23186745, 2.26344125,
           2.29501504, 2.32658884, 2.35816263, 2.38973643, 2.42131022,
           2.45288402, 2.48445782, 2.51603161, 2.54760541, 2.5791792 ,
           2.610753  , 2.64232679, 2.67390059, 2.70547438, 2.73704818,
           2.76862197, 2.80019577, 2.83176957, 2.86334336, 2.89491716,
           2.92649095, 2.95806475, 2.98963854, 3.02121234, 3.05278613,
           3.08435993, 3.11593373, 3.14750752, 3.17908132, 3.21065511,
           3.24222891, 3.2738027 , 3.3053765 , 3.33695029, 3.36852409,
           3.40009789, 3.43167168, 3.46324548, 3.49481927, 3.52639307,
           3.55796686, 3.58954066, 3.62111445, 3.65268825, 3.68426204,
           3.71583584, 3.74740964, 3.77898343, 3.81055723, 3.84213102,
           3.87370482, 3.90527861, 3.93685241, 3.9684262 , 4.        ])

```python
from matplotlib import pyplot
pyplot.figure(figsize=(11, 7), dpi=100)
pyplot.plot(x, u, marker='o', lw=2)
pyplot.xlim([0, 2 * np.pi])
pyplot.ylim([0, 10]);
```

```python
for n in range(nt):
    un = u.copy()
    for i in range(1, nx-1):
        u[i] = un[i] - un[i] * dt / dx *(un[i] - un[i-1]) + nu * dt / dx**2 *\
                (un[i+1] - 2 * un[i] + un[i-1])
    u[0] = un[0] - un[0] * dt / dx * (un[0] - un[-2]) + nu * dt / dx**2 *\
                (un[1] - 2 * un[0] + un[-2])
    u[-1] = u[0]

u_analytical = np.asarray([ufunc(nt * dt, xi, nu) for xi in x])
```

```python
pyplot.figure(figsize=(11, 7), dpi=100)
pyplot.plot(x,u, marker='o', lw=2, label='Computational')
pyplot.plot(x, u_analytical, label='Analytical')
pyplot.xlim([0, 2 * np.pi])
pyplot.ylim([0, 10])
pyplot.legend();
```


## Generating symbolic expressions
For larger reaction systems it is preferable to generate the system of ordinary differential equations from some serialized format and then generate the callback using code generation.

In this notebook we will define such a serialized format, and use it load a larger set of reactions. We represent a reaction as length 3 tuple of: `(rate_const, coeff_powers, net_effect)`. Representing Robertson's system this way looks like this:

```python
reactions = [
    ('k1', {'A': 1}, {'B': 1, 'A': -1}),
    ('k2', {'B': 1, 'C': 1}, {'A': 1, 'B': -1}),
    ('k3', {'B': 2}, {'B': -1, 'C': 1})
]
names, params = 'A B C'.split(), 'k1 k2 k3'.split()
tex_names = ['[%s]' % n for n in names]
```

the reaction system is still defined as:
$$
A \overset{k_1}{\rightarrow} B \\
B + C \overset{k_2}{\rightarrow} A + C \\
2 B \overset{k_3}{\rightarrow} B + C
$$

We will now write a small convenience function which takes the above representation and creates symbolic expressions for the ODE system:

```python
# %load ../scipy2017codegen/chem.py
from operator import mul
from functools import reduce
import sympy as sym

def prod(seq):
    return reduce(mul, seq) if seq else 1

def mk_exprs_symbs(rxns, names):
    concs = sym.symbols(names, real=True, nonnegative=True)
    c_dict = dict(zip(names, concs))
    f = {n: 0 for n in names}
    for coeff, r_stoich, net_stoich in rxns:
        r = sym.S(coeff)*prod([c_dict[rk]**p for rk, p in r_stoich.items()])
        for nk, nm in net_stoich.items():
            f[nk] += nm*r
    return [f[n] for n in names], concs

def mk_rsys(ODEcls, reactions, names, params=(), **kwargs):
    f, symbs = mk_exprs_symbs(reactions, names)
    return ODEcls(f, symbs, params=map(sym.S, params), **kwargs)

```

```python
sym.init_printing()
f, symbs = mk_exprs_symbs(reactions, names)
f, symbs
```

We create a helper class to represent to ODE system.

```python
# %load ../scipy2017codegen/odesys.py
from itertools import chain  # Py 2.7 does not support func(*args1, *args2)
import sympy as sym
from scipy.integrate import odeint

class ODEsys(object):

    default_integrator = 'odeint'

    def __init__(self, f, y, t=None, params=(), tex_names=None, lambdify=None):
        assert len(f) == len(y), 'f is dy/dt'
        self.f = tuple(f)
        self.y = tuple(y)
        self.t = t
        self.p = tuple(params)
        self.tex_names = tex_names
        self.j = sym.Matrix(self.ny, 1, f).jacobian(y)
        self.lambdify = lambdify or sym.lambdify
        self.setup()

    @property
    def ny(self):
        return len(self.y)

    def setup(self):
        self.lambdified_f = self.lambdify(self.y + self.p, self.f)
        self.lambdified_j = self.lambdify(self.y + self.p, self.j)

    def f_eval(self, y, t, *params):
        return self.lambdified_f(*chain(y, params))

    def j_eval(self, y, t, *params):
        return self.lambdified_j(*chain(y, params))

    def integrate(self, *args, **kwargs):
        integrator = kwargs.pop('integrator', self.default_integrator)
        return getattr(self, 'integrate_%s' % integrator)(*args, **kwargs)

    def integrate_odeint(self, tout, y0, params=(), rtol=1e-8, atol=1e-8, **kwargs):
        return odeint(self.f_eval, y0, tout, args=tuple(params), full_output=True,
                      Dfun=self.j_eval, rtol=rtol, atol=atol, **kwargs)

    def print_info(self, info):
        if info is None:
            return
        nrhs = info.get('num_rhs')
        if not nrhs:
            nrhs = info['nfe'][-1]
        njac = info.get('num_dls_jac_evals')
        if not njac:
            njac = info['nje'][-1]
        print("The rhs was evaluated %d times and the Jacobian %d times" % (nrhs, njac))

    def plot_result(self, tout, yout, info=None, ax=None):
        ax = ax or plt.subplot(1, 1, 1)
        for i, label in enumerate(self.tex_names):
            ax.plot(tout, yout[:, i], label='$%s$' % label)
        ax.set_ylabel('$\mathrm{concentration\ /\ mol \cdot dm^{-3}}$')
        ax.set_xlabel('$\mathrm{time\ /\ s}$')
        ax.legend(loc='best')
        self.print_info(info)

```

```python
odesys = ODEsys(f, symbs, params=params, tex_names=tex_names)
```

```python
import numpy as np
tout = np.logspace(-6, 6)
yout, info = odesys.integrate_odeint(tout, [1, 0, 0], [0.04, 1e4, 3e7], atol=1e-9, rtol=1e-9)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
fig, axes = plt.subplots(1, 2, figsize=(14, 4))
odesys.plot_result(tout, yout, info, ax=axes[0])
odesys.plot_result(tout, yout, ax=axes[1])
axes[1].set_xscale('log')
axes[1].set_yscale('log')
```

The reason for why we went through this trouble is to be able to create a `ODEsys` instance from conveniently serialized data. Here is a much larger set of reactions, describing water radiolysis at 298 K and a doserate of 300 Gy/s (which is a doserate not far from that of a nuclear reactor):

```python
import json
watrad_data = json.load(open('../scipy2017codegen/data/radiolysis_300_Gy_s.json'))
watrad = mk_rsys(ODEsys, **watrad_data)
print(len(watrad.f), watrad.y[0], watrad.f[0])
```

Values correspond to SI units, the concentration of water at 298 K is 55400 mol/m³. Neutral water contains [H+] = [HO-] = 10^-4 mol/m³:

```python
tout = np.logspace(-6, 3, 200)  # close to one hour of operation
c0 = {'H2O': 55.4e3, 'H+': 1e-4, 'OH-': 1e-4}
y0 = [c0.get(symb.name, 0) for symb in watrad.y]
```

```python
%timeit watrad.integrate_odeint(tout, y0)
```

```python
fig, ax = plt.subplots(1, 1, figsize=(14, 6))
watrad.plot_result(tout, *watrad.integrate_odeint(tout, y0), ax=ax)
ax.set_xscale('log')
ax.set_yscale('log')
```


# Variational forms for systems of PDEs

Many mathematical models involve $m+1$ unknown functions
governed by a system of $m+1$ differential equations. In abstract form
we may denote the unknowns by $u^{(0)},\ldots,
u^{(m)}$ and write the governing equations as

$$
\begin{align*}
\mathcal{L}_0(u^{(0)},\ldots,u^{(m)}) &= 0,\\
&\vdots\\
\mathcal{L}_{m}(u^{(0)},\ldots,u^{(m)}) &= 0,
\end{align*}
$$

where $\mathcal{L}_i$ is some differential operator defining differential
equation number $i$.

# Variational forms

There are basically two ways of formulating a variational form
for a system of differential equations. The first method treats
each equation independently as a scalar equation, while the other
method views the total system as a vector equation with a vector function
as unknown.

## Sequence of scalar PDEs formulation

Let us start with the approach that treats one equation at a time.
We multiply equation number $i$ by
some test function $v^{(i)}\in V^{(i)}$ and integrate over the domain:

$$
\begin{equation}
\int_\Omega \mathcal{L}^{(0)}(u^{(0)},\ldots,u^{(m)}) v^{(0)}{\, \mathrm{d}x} = 0,
\label{fem:sys:vform:1by1a} \tag{1}
\end{equation}
$$

$$
\begin{equation}
\vdots
\label{_auto1} \tag{2}
\end{equation}
$$

$$
\begin{equation}
\int_\Omega \mathcal{L}^{(m)}(u^{(0)},\ldots,u^{(m)}) v^{(m)}{\, \mathrm{d}x} = 0
\label{fem:sys:vform:1by1b} \tag{3}
{\thinspace .}
\end{equation}
$$

Terms with second-order derivatives may be integrated by parts, with
Neumann conditions inserted in boundary integrals.
Let

$$
V^{(i)} = \hbox{span}\{{\psi}_0^{(i)},\ldots,{\psi}_{N_i}^{(i)}\},
$$

such that

$$
u^{(i)} = B^{(i)}(\boldsymbol{x}) + \sum_{j=0}^{N_i} c_j^{(i)} {\psi}_j^{(i)}(\boldsymbol{x}),
$$

where $B^{(i)}$ is a boundary function to handle nonzero Dirichlet conditions.
Observe that different unknowns may live in different spaces with different
basis functions and numbers of degrees of freedom.

From the $m$ equations in the variational forms we can derive
$m$ coupled systems of algebraic equations for the
$\Pi_{i=0}^{m} N_i$ unknown coefficients $c_j^{(i)}$, $j=0,\ldots,N_i$,
$i=0,\ldots,m$.

## Vector PDE formulation

The alternative method for deriving a variational form for a system of
differential equations introduces a vector of unknown functions

$$
\boldsymbol{u} = (u^{(0)},\ldots,u^{(m)}),
$$

a vector of test functions

$$
\boldsymbol{v} = (v^{(0)},\ldots,v^{(m)}),
$$

with

$$
\boldsymbol{u}, \boldsymbol{v} \in  \boldsymbol{V} = V^{(0)}\times \cdots \times V^{(m)}
{\thinspace .}
$$

With nonzero Dirichlet conditions, we have a vector
$\boldsymbol{B} = (B^{(0)},\ldots,B^{(m)})$ with boundary functions and then
it is $\boldsymbol{u} - \boldsymbol{B}$ that lies in $\boldsymbol{V}$, not $\boldsymbol{u}$ itself.

The governing system of differential equations is written

$$
\boldsymbol{\mathcal{L}}(\boldsymbol{u} ) = 0,
$$

where

$$
\boldsymbol{\mathcal{L}}(\boldsymbol{u} ) = (\mathcal{L}^{(0)}(\boldsymbol{u}),\ldots, \mathcal{L}^{(m)}(\boldsymbol{u}))
{\thinspace .}
$$

The variational form is derived by taking the inner product of
the vector of equations and the test function vector:

$$
\begin{equation}
\int_\Omega \boldsymbol{\mathcal{L}}(\boldsymbol{u} )\cdot\boldsymbol{v} = 0\quad\forall\boldsymbol{v}\in\boldsymbol{V}{\thinspace .}
\label{fem:sys:vform:inner} \tag{4}
\end{equation}
$$

Observe that ([4](#fem:sys:vform:inner)) is one scalar equation. To derive
systems of algebraic equations for the unknown coefficients in the
expansions of the unknown functions, one chooses $m$ linearly
independent $\boldsymbol{v}$ vectors to generate $m$ independent variational forms
from ([4](#fem:sys:vform:inner)).  The particular choice $\boldsymbol{v} =
(v^{(0)},0,\ldots,0)$ recovers ([1](#fem:sys:vform:1by1a)), $\boldsymbol{v} =
(0,\ldots,0,v^{(m)})$ recovers ([3](#fem:sys:vform:1by1b)), and $\boldsymbol{v} =
(0,\ldots,0,v^{(i)},0,\ldots,0)$ recovers the variational form number
$i$, $\int_\Omega \mathcal{L}^{(i)} v^{(i)}{\, \mathrm{d}x} =0$, in
([1](#fem:sys:vform:1by1a))-([3](#fem:sys:vform:1by1b)).

# A worked example

We now consider a specific system of two partial differential equations
in two space dimensions:

$$
\begin{equation}
\mu \nabla^2 w = -\beta,
\label{fem:sys:wT:ex:weq} \tag{5}
\end{equation}
$$

$$
\begin{equation}
\kappa\nabla^2 T = - \mu ||\nabla w||^2
{\thinspace .}
\label{fem:sys:wT:ex:Teq} \tag{6}
\end{equation}
$$

The unknown functions $w(x,y)$ and $T(x,y)$ are defined in a domain $\Omega$,
while $\mu$, $\beta$,
and $\kappa$ are given constants. The norm in
([6](#fem:sys:wT:ex:Teq)) is the standard Euclidean norm:

$$
||\nabla w||^2 = \nabla w\cdot\nabla w = w_x^2 + w_y^2
{\thinspace .}
$$

The boundary conditions associated with
([5](#fem:sys:wT:ex:weq))-([6](#fem:sys:wT:ex:Teq)) are $w=0$ on
$\partial\Omega$ and $T=T_0$ on $\partial\Omega$.
Each of the equations ([5](#fem:sys:wT:ex:weq)) and ([6](#fem:sys:wT:ex:Teq))
needs one condition at each point on the boundary.

The system ([5](#fem:sys:wT:ex:weq))-([6](#fem:sys:wT:ex:Teq)) arises
from fluid flow in a straight pipe, with the $z$ axis in the direction
of the pipe. The domain $\Omega$ is a cross section of the pipe, $w$
is the velocity in the $z$ direction, $\mu$
is the viscosity of the fluid, $\beta$ is the pressure gradient along
the pipe, $T$ is the temperature,
and $\kappa$ is the heat conduction coefficient of the
fluid. The equation ([5](#fem:sys:wT:ex:weq)) comes from the Navier-Stokes
equations, and ([6](#fem:sys:wT:ex:Teq)) follows from the energy equation.
The term $- \mu ||\nabla w||^2$ models heating of the fluid
due to internal friction.

Observe that the system ([5](#fem:sys:wT:ex:weq))-([6](#fem:sys:wT:ex:Teq)) has
only a one-way coupling: $T$ depends on $w$, but $w$ does not depend on
$T$. Hence, we can solve ([5](#fem:sys:wT:ex:weq)) with respect
to $w$ and then ([6](#fem:sys:wT:ex:Teq)) with respect to $T$.
Some may argue that this is not a real system of PDEs, but just two scalar
PDEs. Nevertheless, the one-way coupling
is convenient when comparing different variational forms
and different implementations.

# Identical function spaces for the unknowns

Let us first apply the same function space $V$ for $w$ and $T$
(or more precisely, $w\in V$ and $T-T_0 \in V$).
With

$$
V = \hbox{span}\{{\psi}_0(x,y),\ldots,{\psi}_N(x,y)\},
$$

we write

$$
\begin{equation}
w = \sum_{j=0}^N c^{(w)}_j {\psi}_j,\quad T = T_0 + \sum_{j=0}^N c^{(T)}_j
{\psi}_j{\thinspace .}
\label{fem:sys:wT:ex:sum} \tag{7}
\end{equation}
$$

Note that $w$ and $T$ in ([5](#fem:sys:wT:ex:weq))-([6](#fem:sys:wT:ex:Teq))
denote the exact solution of the PDEs, while $w$ and $T$
in ([7](#fem:sys:wT:ex:sum)) are the discrete functions that approximate
the exact solution. It should be clear from the context whether a
symbol means the exact or approximate solution, but when we need both
at the same time, we use a subscript e to denote the exact solution.

## Variational form of each individual PDE

Inserting the expansions ([7](#fem:sys:wT:ex:sum))
in the governing PDEs, results in a residual in each equation,

$$
\begin{equation}
R_w = \mu \nabla^2 w + \beta,
\label{fem:sys:wT:ex:weq:R} \tag{8}
\end{equation}
$$

$$
\begin{equation}
R_T = \kappa\nabla^2 T + \mu ||\nabla w||^2
{\thinspace .}
\label{fem:sys:wT:ex:Teq:R} \tag{9}
\end{equation}
$$

A Galerkin method demands $R_w$ and $R_T$ do be orthogonal to $V$:

$$
\begin{align*}
\int_\Omega R_w v {\, \mathrm{d}x} &=0\quad\forall v\in V,\\
\int_\Omega R_T v {\, \mathrm{d}x} &=0\quad\forall v\in V
{\thinspace .}
\end{align*}
$$

Because of the Dirichlet conditions, $v=0$ on $\partial\Omega$.
We integrate the Laplace terms by parts and note that the boundary terms
vanish since $v=0$ on $\partial\Omega$:

$$
\begin{equation}
\int_\Omega \mu \nabla w\cdot\nabla v {\, \mathrm{d}x} = \int_\Omega \beta v{\, \mathrm{d}x}
\quad\forall v\in V,
\label{fem:sys:wT:ex:w:vf1} \tag{10}
\end{equation}
$$

$$
\begin{equation}
\int_\Omega \kappa \nabla T\cdot\nabla v {\, \mathrm{d}x} = \int_\Omega \mu
\nabla w\cdot\nabla w\, v{\, \mathrm{d}x} \quad\forall v\in V
\label{fem:sys:wT:ex:T:vf1} \tag{11}
{\thinspace .}
\end{equation}
$$

The equation $R_w$ in ([8](#fem:sys:wT:ex:weq:R)) is linear
in $w$, while the equation $R_T$ in ([9](#fem:sys:wT:ex:Teq:R))
is linear in $T$ and nonlinear in $w$.

## Compound scalar variational form

The alternative way of deriving the variational from is to
introduce a test vector function $\boldsymbol{v}\in\boldsymbol{V} = V\times V$ and take
the inner product of $\boldsymbol{v}$ and the residuals, integrated over the domain:

$$
\int_{\Omega} (R_w, R_T)\cdot\boldsymbol{v} {\, \mathrm{d}x} = 0\quad\forall\boldsymbol{v}\in\boldsymbol{V}
{\thinspace .}
$$

With $\boldsymbol{v} = (v_0,v_1)$ we get

$$
\int_{\Omega} (R_w v_0 + R_T v_1) {\, \mathrm{d}x} = 0\quad\forall\boldsymbol{v}\in\boldsymbol{V}
{\thinspace .}
$$

Integrating the Laplace terms by parts results in

$$
\begin{equation}
\int_\Omega (\mu\nabla w\cdot\nabla v_0 + \kappa\nabla T\cdot\nabla v_1){\, \mathrm{d}x}
= \int_\Omega (\beta v_0 + \mu\nabla w\cdot\nabla w\, v_1){\, \mathrm{d}x},
\quad\forall \boldsymbol{v}\in\boldsymbol{V}
{\thinspace .}
\label{fem:sys:wT:ex:wT:vf2} \tag{12}
\end{equation}
$$

Choosing $v_0=v$ and $v_1=0$ gives the variational form
([10](#fem:sys:wT:ex:w:vf1)), while $v_0=0$ and $v_1=v$ gives
([11](#fem:sys:wT:ex:T:vf1)).

With the inner product notation, $(p,q) = \int_\Omega pq{\, \mathrm{d}x}$, we
can alternatively write ([10](#fem:sys:wT:ex:w:vf1)) and
([11](#fem:sys:wT:ex:T:vf1)) as

$$
\begin{align*}
 (\mu\nabla w,\nabla v) &= (\beta, v)
\quad\forall v\in V,\\
(\kappa \nabla T,\nabla v) &= (\mu\nabla w\cdot\nabla w, v)\quad\forall v\in V,
\end{align*}
$$

or since $\mu$ and $\kappa$ are considered constant,

$$
\begin{equation}
\mu (\nabla w,\nabla v) = (\beta, v)
\quad\forall v\in V,
\label{fem:sys:wT:ex:w:vf1i} \tag{13}
\end{equation}
$$

$$
\begin{equation}
\kappa(\nabla T,\nabla v) = \mu(\nabla w\cdot\nabla w, v)\quad\forall v\in V
\label{fem:sys:wT:ex:T:vf1i} \tag{14}
{\thinspace .}
\end{equation}
$$

Note that the left-hand side of ([13](#fem:sys:wT:ex:w:vf1i)) is
again linear in $w$, the left-hand side
of ([14](#fem:sys:wT:ex:T:vf1i)) is linear in $T$
and the nonlinearity of $w$ appears in the right-hand side
of  ([14](#fem:sys:wT:ex:T:vf1i))

## Decoupled linear systems

The linear systems governing the coefficients $c_j^{(w)}$ and
$c_j^{(T)}$, $j=0,\ldots,N$, are derived by inserting the
expansions ([7](#fem:sys:wT:ex:sum)) in ([10](#fem:sys:wT:ex:w:vf1))
and ([11](#fem:sys:wT:ex:T:vf1)), and choosing $v={\psi}_i$ for
$i=0,\ldots,N$. The result becomes

$$
\begin{equation}
\sum_{j=0}^N A^{(w)}_{i,j} c^{(w)}_j = b_i^{(w)},\quad i=0,\ldots,N,
\label{fem:sys:wT:ex:linsys:w1} \tag{15}
\end{equation}
$$

$$
\begin{equation}
\sum_{j=0}^N A^{(T)}_{i,j} c^{(T)}_j = b_i^{(T)},\quad i=0,\ldots,N,
\label{fem:sys:wT:ex:linsys:T1} \tag{16}
\end{equation}
$$

$$
\begin{equation}
A^{(w)}_{i,j} = \mu(\nabla {\psi}_j,\nabla {\psi}_i),
\label{_auto2} \tag{17}
\end{equation}
$$

$$
\begin{equation}
b_i^{(w)} = (\beta, {\psi}_i),
\label{_auto3} \tag{18}
\end{equation}
$$

$$
\begin{equation}
A^{(T)}_{i,j} = \kappa(\nabla {\psi}_j,\nabla {\psi}_i),
\label{_auto4} \tag{19}
\end{equation}
$$

$$
\begin{equation}
b_i^{(T)} = \mu((\sum_j c^{(w)}_j\nabla{\psi}_j)\cdot (\sum_k
c^{(w)}_k\nabla{\psi}_k), {\psi}_i)
{\thinspace .}
\label{_auto5} \tag{20}
\end{equation}
$$

It can also be instructive to write the linear systems using matrices
and vectors. Define $K$ as the matrix corresponding to the Laplace
operator $\nabla^2$. That is, $K_{i,j} = (\nabla {\psi}_j,\nabla {\psi}_i)$.
Let us introduce the vectors

$$
\begin{align*}
b^{(w)} &= (b_0^{(w)},\ldots,b_{N}^{(w)}),\\
b^{(T)} &= (b_0^{(T)},\ldots,b_{N}^{(T)}),\\
c^{(w)} &= (c_0^{(w)},\ldots,c_{N}^{(w)}),\\
c^{(T)} &= (c_0^{(T)},\ldots,c_{N}^{(T)}){\thinspace .}
\end{align*}
$$

The system ([15](#fem:sys:wT:ex:linsys:w1))-([16](#fem:sys:wT:ex:linsys:T1))
can now be expressed in matrix-vector form as

$$
\begin{equation}
\mu K c^{(w)} = b^{(w)},
\label{_auto6} \tag{21}
\end{equation}
$$

$$
\begin{equation}
\kappa K c^{(T)} = b^{(T)}{\thinspace .}
\label{_auto7} \tag{22}
\end{equation}
$$

We can solve the first system for $c^{(w)}$, and then
the right-hand side $b^{(T)}$ is known such that we can
solve the second system for $c^{(T)}$. Hence, the
decoupling of the unknowns $w$ and $T$ reduces the
system of nonlinear PDEs to two linear PDEs.

## Coupled linear systems

Despite the fact that $w$ can be computed first, without knowing $T$,
we shall now pretend that $w$ and $T$ enter a two-way coupling such
that we need to derive the
algebraic equations as *one system* for all the unknowns
$c_j^{(w)}$ and $c_j^{(T)}$, $j=0,\ldots,N$. This system is
nonlinear in $c_j^{(w)}$ because of the $\nabla w\cdot\nabla w$ product.
To remove this nonlinearity, imagine that we introduce an iteration
method where we replace $\nabla w\cdot\nabla w$ by
$\nabla w_{-}\cdot\nabla w$, $w_{-}$ being the $w$
computed in the previous iteration. Then the term
$\nabla w_{-}\cdot\nabla w$ is linear in $w$ since $w_{-}$ is
known. The total linear system becomes

$$
\begin{equation}
\sum_{j=0}^N A^{(w,w)}_{i,j} c^{(w)}_j + \sum_{j=0}^N A^{(w,T)}_{i,j} c^{(T)}_j
= b_i^{(w)},\quad i=0,\ldots,N,
\label{fem:sys:wT:ex:linsys:w2} \tag{23}
\end{equation}
$$

$$
\begin{equation}
\sum_{j=0}^N A^{(T,w)}_{i,j} c^{(w)}_j + \sum_{j=0}^N A^{(T,T)}_{i,j} c^{(T)}_j = b_i^{(T)},\quad i=0,\ldots,N,
\label{fem:sys:wT:ex:linsys:T2} \tag{24}
\end{equation}
$$

$$
\begin{equation}
A^{(w,w)}_{i,j} = \mu(\nabla {\psi}_j,\nabla {\psi}_i),
\label{_auto8} \tag{25}
\end{equation}
$$

$$
\begin{equation}
A^{(w,T)}_{i,j} = 0,
\label{_auto9} \tag{26}
\end{equation}
$$

$$
\begin{equation}
b_i^{(w)} = (\beta, {\psi}_i),
\label{_auto10} \tag{27}
\end{equation}
$$

$$
\begin{equation}
A^{(w,T)}_{i,j} = \mu((\nabla w_{-})\cdot\nabla{\psi}_j), {\psi}_i),
\label{_auto11} \tag{28}
\end{equation}
$$

$$
\begin{equation}
A^{(T,T)}_{i,j} = \kappa(\nabla {\psi}_j,\nabla {\psi}_i),
\label{_auto12} \tag{29}
\end{equation}
$$

$$
\begin{equation}
b_i^{(T)} = 0
{\thinspace .}
\label{_auto13} \tag{30}
\end{equation}
$$

This system can alternatively be written in matrix-vector form as

$$
\begin{equation}
\mu K c^{(w)} = b^{(w)},
\label{_auto14} \tag{31}
\end{equation}
$$

$$
\begin{equation}
L c^{(w)} + \kappa K c^{(T)}  =0,
\label{_auto15} \tag{32}
\end{equation}
$$

with $L$ as the matrix from the $\nabla w_{-}\cdot\nabla$ operator:
$L_{i,j} = A^{(w,T)}_{i,j}$. The matrix $K$ is $K_{i,j} =
A^{(w,w)}_{i,j} = A^{(T,T)}_{i,j}$.

The matrix-vector equations are often conveniently written in block form:

$$
\left(\begin{array}{cc}
\mu K & 0\\
L & \kappa K
\end{array}\right)
\left(\begin{array}{c}
c^{(w)}\\
c^{(T)}
\end{array}\right) =
\left(\begin{array}{c}
b^{(w)}\\
0
\end{array}\right),
$$

Note that in the general case where all unknowns enter all equations,
we have to solve the compound system
([23](#fem:sys:wT:ex:linsys:w2))-([24](#fem:sys:wT:ex:linsys:T2)) since
then we cannot utilize the special property that
([15](#fem:sys:wT:ex:linsys:w1)) does not involve $T$ and can be solved
first.

When the viscosity depends on the temperature, the
$\mu\nabla^2w$ term must be replaced by $\nabla\cdot (\mu(T)\nabla w)$,
and then $T$ enters the equation for $w$. Now we have a two-way coupling
since both equations contain $w$ and $T$ and therefore
must be solved simultaneously.
The equation $\nabla\cdot (\mu(T)\nabla w)=-\beta$ is nonlinear,
and if some iteration procedure is invoked, where we use a previously
computed $T_{-}$ in the viscosity ($\mu(T_{-})$), the coefficient is known,
and the equation involves only one unknown, $w$. In that case we are
back to the one-way coupled set of PDEs.

We may also formulate our PDE system as a vector equation. To this end,
we introduce the vector of unknowns $\boldsymbol{u} = (u^{(0)},u^{(1)})$,
where $u^{(0)}=w$ and $u^{(1)}=T$. We then have

$$
\nabla^2 \boldsymbol{u} = \left(\begin{array}{cc}
-{\mu}^{-1}{\beta}\\
-{\kappa}^{-1}\mu \nabla u^{(0)}\cdot\nabla u^{(0)}
\end{array}\right){\thinspace .}
$$

# Different function spaces for the unknowns

It is easy to generalize the previous formulation to the case where
$w\in V^{(w)}$ and $T\in V^{(T)}$, where $V^{(w)}$ and $V^{(T)}$
can be different spaces with different numbers of degrees of freedom.
For example, we may use quadratic basis functions for $w$ and linear
for $T$. Approximation of the unknowns by different finite element
spaces is known as *mixed finite element methods*.

We write

$$
\begin{align*}
V^{(w)} &= \hbox{span}\{{\psi}_0^{(w)},\ldots,{\psi}_{N_w}^{(w)}\},\\
V^{(T)} &= \hbox{span}\{{\psi}_0^{(T)},\ldots,{\psi}_{N_T}^{(T)}\}
{\thinspace .}
\end{align*}
$$

The next step is to
multiply ([5](#fem:sys:wT:ex:weq)) by a test function $v^{(w)}\in V^{(w)}$
and ([6](#fem:sys:wT:ex:Teq)) by a $v^{(T)}\in V^{(T)}$, integrate by
parts and arrive at

$$
\begin{equation}
\int_\Omega \mu \nabla w\cdot\nabla v^{(w)} {\, \mathrm{d}x} = \int_\Omega \beta v^{(w)}{\, \mathrm{d}x}
\quad\forall v^{(w)}\in V^{(w)},
\label{fem:sys:wT:ex:w:vf3} \tag{33}
\end{equation}
$$

$$
\begin{equation}
\int_\Omega \kappa \nabla T\cdot\nabla v^{(T)} {\, \mathrm{d}x} = \int_\Omega \mu
\nabla w\cdot\nabla w\, v^{(T)}{\, \mathrm{d}x} \quad\forall v^{(T)}\in V^{(T)}
\label{fem:sys:wT:ex:T:vf3} \tag{34}
{\thinspace .}
\end{equation}
$$

The compound scalar variational formulation applies a test vector function
$\boldsymbol{v} = (v^{(w)}, v^{(T)})$ and reads

$$
\begin{equation}
\int_\Omega (\mu\nabla w\cdot\nabla v^{(w)} +
\kappa\nabla T\cdot\nabla v^{(T)}){\, \mathrm{d}x}
= \int_\Omega (\beta v^{(w)} + \mu\nabla w\cdot\nabla w\, v^{(T)}){\, \mathrm{d}x},
\label{fem:sys:wT:ex:wT:vf3} \tag{35}
\end{equation}
$$

valid $\forall \boldsymbol{v}\in\boldsymbol{V} = V^{(w)}\times V^{(T)}$.

As earlier, we may decoupled the system in terms
of two linear PDEs as we did with
([15](#fem:sys:wT:ex:linsys:w1))-([16](#fem:sys:wT:ex:linsys:T1))
or linearize the coupled system by introducing the previous
iterate $w_{-}$ as in
([23](#fem:sys:wT:ex:linsys:w2))-([24](#fem:sys:wT:ex:linsys:T2)).
However, we need to distinguish between ${\psi}_i^{(w)}$
and ${\psi}_i^{(T)}$, and the range in the sums over $j$
must match the number of degrees of freedom in the spaces $V^{(w)}$
and $V^{(T)}$. The formulas become

$$
\begin{equation}
\sum_{j=0}^{N_w} A^{(w)}_{i,j} c^{(w)}_j = b_i^{(w)},\quad i=0,\ldots,N_w,
\label{fem:sys:wT:ex:linsys:w1:mixed} \tag{36}
\end{equation}
$$

$$
\begin{equation}
\sum_{j=0}^{N_T} A^{(T)}_{i,j} c^{(T)}_j = b_i^{(T)},\quad i=0,\ldots,N_T,
\label{fem:sys:wT:ex:linsys:T1:mixed} \tag{37}
\end{equation}
$$

$$
\begin{equation}
A^{(w)}_{i,j} = \mu(\nabla {\psi}_j^{(w)},\nabla {\psi}_i^{(w)}),
\label{_auto16} \tag{38}
\end{equation}
$$

$$
\begin{equation}
b_i^{(w)} = (\beta, {\psi}_i^{(w)}),
\label{_auto17} \tag{39}
\end{equation}
$$

$$
\begin{equation}
A^{(T)}_{i,j} = \kappa(\nabla {\psi}_j^{(T)},\nabla {\psi}_i^{(T)}),
\label{_auto18} \tag{40}
\end{equation}
$$

$$
\begin{equation}
b_i^{(T)} = \mu(\sum_{j=0}^{N_w} c^{(w)}_j\nabla{\psi}_j^{(w)})\cdot (\sum_{k=0}^{N_w}
c^{(w)}_k\nabla{\psi}_k^{(w)}) , {\psi}_i^{(T)})
{\thinspace .}
\label{_auto19} \tag{41}
\end{equation}
$$

In the case we formulate one compound linear system involving
both $c^{(w)}_j$, $j=0,\ldots,N_w$, and $c^{(T)}_j$, $j=0,\ldots,N_T$,
([23](#fem:sys:wT:ex:linsys:w2))-([24](#fem:sys:wT:ex:linsys:T2))
becomes

$$
\begin{equation}
\sum_{j=0}^{N_w} A^{(w,w)}_{i,j} c^{(w)}_j + \sum_{j=0}^{N_T} A^{(w,T)}_{i,j} c^{(T)}_j
= b_i^{(w)},\quad i=0,\ldots,N_w,
\label{fem:sys:wT:ex:linsys:w2b} \tag{42}
\end{equation}
$$

$$
\begin{equation}
\sum_{j=0}^{N_w} A^{(T,w)}_{i,j} c^{(w)}_j + \sum_{j=0}^{N_T} A^{(T,T)}_{i,j} c^{(T)}_j = b_i^{(T)},\quad i=0,\ldots,N_T,
\label{fem:sys:wT:ex:linsys:T2b} \tag{43}
\end{equation}
$$

$$
\begin{equation}
A^{(w,w)}_{i,j} = \mu(\nabla {\psi}_j^{(w)},\nabla {\psi}_i^{(w)}),
\label{_auto20} \tag{44}
\end{equation}
$$

$$
\begin{equation}
A^{(w,T)}_{i,j} = 0,
\label{_auto21} \tag{45}
\end{equation}
$$

$$
\begin{equation}
b_i^{(w)} = (\beta, {\psi}_i^{(w)}),
\label{_auto22} \tag{46}
\end{equation}
$$

$$
\begin{equation}
A^{(w,T)}_{i,j} = \mu (\nabla w_{-}\cdot\nabla{\psi}_j^{(w)}), {\psi}_i^{(T)}),
\label{_auto23} \tag{47}
\end{equation}
$$

$$
\begin{equation}
A^{(T,T)}_{i,j} = \kappa(\nabla {\psi}_j^{(T)},\nabla {\psi}_i^{(T)}),
\label{_auto24} \tag{48}
\end{equation}
$$

$$
\begin{equation}
b_i^{(T)} = 0
{\thinspace .}
\label{_auto25} \tag{49}
\end{equation}
$$

Here, we have again performed a linearization by employing a previous iterate $w_{-}$.
The corresponding block form

$$
\left(\begin{array}{cc}
\mu K^{(w)} & 0\\
L & \kappa K^{(T)}
\end{array}\right)
\left(\begin{array}{c}
c^{(w)}\\
c^{(T)}
\end{array}\right) =
\left(\begin{array}{c}
b^{(w)}\\
0
\end{array}\right),
$$

has square and rectangular block matrices: $K^{(w)}$ is $N_w\times N_w$,
$K^{(T)}$ is $N_T\times N_T$, while $L$ is $N_T\times N_w$,


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

### Marginal and Conditional Densities ###
Let random variables $X$ and $Y$ have the joint density defined by
$$
f(x, y) ~ = ~
\begin{cases}
30(y-x)^4, ~~~ 0 < x < y < 1 \\
0 ~~~~~~~~ \text{otherwise}
\end{cases}
$$

```python
def jt_dens(x,y):
    if y < x:
        return 0
    else:
        return 30 * (y-x)**4

Plot_3d(x_limits=(0,1), y_limits=(0,1), f=jt_dens, cstride=4, rstride=4)
```

Then the possible values of $(X, Y)$ are in the upper right hand triangle of the unit square, as in the previous example.

```python
# HIDDEN
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('Possible Values of $(X, Y)$');
```

Here is a quick check that the function given is indeed a joint density.

```python
declare('x', interval=(0, 1))
declare('y', interval=(0, 1))
joint_density = 30*(y-x)**4
```

```python
Integral(joint_density, (y, x, 1), (x, 0, 1)).doit()
```

### Density of $X$ ###
We can use the joint density $f$ to find the density of $X$. Call this density $f_X$. We know that

\begin{align*}
f_X(x)dx &\sim P(X \in dx) \\
&= \int_y P(X \in dx, Y \in dy) \\
&= \int_y f(x, y)dxdy \\
&= \big{(} \int_y f(x, y)dy \big{)}dx
\end{align*}

You can see the reasoning behind this calculation in the graph below. The blue strip shows the event $\{ X \in dx \}$ for a value of $x$ very near 0.25. To find the volume $P(X \in dx)$, we hold $x$ fixed and add over all $y$.

```python
# HIDDEN
plt.plot([0, 0], [0, 1], color='k', lw=2)
plt.plot([0, 1], [0, 1], color='k', lw=2)
plt.plot([0, 1], [1, 1], color='k', lw=2)
plt.plot([0.25, 0.25], [0.25, 1], color='blue', lw=3, alpha=0.3)
plt.xlim(-0.05, 1)
plt.ylim(0, 1.05)
plt.axes().set_aspect('equal')
plt.xticks(np.arange(0, 1.1, 0.25))
plt.yticks(np.arange(0, 1.1, 0.25))
plt.xlabel('$x$')
plt.ylabel('$y$', rotation=0)
plt.title('$X \in dx$');
```

So the density of $X$ is given by
$$
f_X(x) ~ = ~ \int_y f(x, y)dy ~~~~~ \text{for all } x
$$

By analogy with the discrete case, $f_X$ is sometimes called the *marginal density* of $X$.

In our example, the possible values of $(X, Y)$ are the upper left hand triangle as shown above. So for each fixed $x$, the possible values of $Y$ go from $x$ to 1.

Therefore for $0 < x < 1$, the density of $X$ is given by

\begin{align*}
f_X(x) &= \int_{y=x}^1 30(y-x)^4 dy \\
&= 30 \cdot \frac{1}{5} (y-x)^5 \Big{\rvert}_x^1 \\
&= 6(1-x)^5
\end{align*}

Here is the joint density surface again. You can see that $X$ is much more likely to be near 0 than near 1.

```python
Plot_3d(x_limits=(0,1), y_limits=(0,1), f=jt_dens, cstride=4, rstride=4)
```

That can be seen in the shape of the density of $X$.

```python
# HIDDEN
x_vals = np.arange(0, 1.01, 0.01)
f_X = 6*(1-x_vals)**5
plt.plot(x_vals, f_X, color='darkblue', lw=2)
plt.xlabel('$x$')
plt.ylabel('$f_X(x)$', rotation=0)
plt.title('$f_X$: Density of $X$');
```

### Density of $Y$ ###
Correspondingly, the density of $Y$ can be found by fixing $y$ and integrating over $x$ as follows:

$$
f_Y(y) = \int_x f(x, y)dx ~~~~ \text{for all } y
$$

In our example, the joint density surface indicates that $Y$ is more likely to be near 1 than near 0, which is confirmed by calculation. Remember that $y > x$ and therefore for each fixed $y$, the possible values of $x$ are 0 through $y$.

For $0 < y < 1$,
$$
f_Y(y) ~ = ~ \int_0^y 30(y-x)^4dx ~ = ~ 6y^5
$$

```python
# HIDDEN
y_vals = np.arange(0, 1.01, 0.01)
f_Y = 6*y_vals**5
plt.plot(y_vals, f_Y, color='darkblue', lw=2)
plt.xlabel('$y$')
plt.ylabel('$f_Y(y)$', rotation=0)
plt.title('$f_Y$: Density of $Y$');
```

### Conditional Densities ###
Consider the conditional probability $P(Y \in dy \mid X \in dx)$. By the division rule,

$$
P(Y \in dy \mid X \in dx) ~ = ~
\frac{P(X \in dx, Y \in dy)}{P(X \in dx)} ~ = ~
\frac{f(x, y)dxdy}{f_X(x)dx} ~ = ~
\frac{f(x, y)}{f_X(x)} dy
$$

This gives us a division rule for densities. For a fixed value $x$, the *conditional density of $Y$ given $X=x$* is defined by

$$
f_{Y\mid X=x} (y) ~ = ~ \frac{f(x, y)}{f_X(x)} ~~~~ \text{for all } y
$$

Since $X$ has a density, we know that $P(X = x) = 0$ for all $x$. But the ratio above is of densities, not probabilities. It might help your intuition to think of "given $X=x$" to mean "given that $X$ is just around $x$".

Visually, the shape of this conditional density is the vertical cross section at $x$ of the joint density graph above. The numerator determines the shape, and the denominator is part of the constant that makes the density integrate to 1.

Note that $x$ is constant in this formula; it is the given value of $X$. So the denominator $f_X(x)$ is the same for all the possible values of $y$.

To see that the conditional density does integrate to 1, let's do the integral.

$$
\int_y f_{Y\mid X=x} (y)dy ~ = ~ \int_y \frac{f(x, y)}{f_X(x)} dy ~ = ~
\frac{1}{f_X(x)} \int_y f(x, y)dy ~ = ~ \frac{1}{f_X(x)} f_X(x) ~ = ~ 1
$$

In our example, let $x = 0.2$ and consider finding the conditional density of $Y$ given $X = 0.4$. Under that condition, the possible values of $Y$ are in the range 0.4 to 1, and therefore

$$
f_{Y \mid X=0.4} (y) ~ = ~ \frac{30(y - 0.4)^4}{6(1 - 0.4)^5} ~ = ~
\frac{5}{0.6^5} (y - 0.4)^4 ~~~~ y \in (0.4, 1)
$$

This is a density on $(0.4, 1)$:

```python
declare('y', interval=(0.4, 1))
cond_density = (5/(0.6**5)) * (y - 0.4)**4
Integral(cond_density, (y, 0.4, 1)).doit()
```

The figure below shows the overlaid graphs of the density of $Y$ and the conditional density of $Y$ given $X = 0.4$. You can see that the conditional density is more concentrated on large values of $Y$, because under the condition $X = 0.4$ you know that $Y$ can't be small.

```python
# HIDDEN
plt.plot(y_vals, f_Y, color='darkblue', lw=2, label='Density of $Y$')
new_y = np.arange(0.4, 1.01, 0.01)
dens = (5/(0.6**5)) * (new_y - 0.4)**4
plt.plot(new_y, dens, color='gold', lw=2, label='Density of $Y$ given $X=0.4$')
plt.legend()
plt.xlim(0, 1)
plt.xlabel('$y$');
```

#### Using a Conditional Density ####
We can use conditional densities to find probabilities and expectations, just as we would use an ordinary density. Here are some examples of calculations. In each case we will set up the integrals and then use `SymPy`.

$$
P(Y > 0.9 \mid X = 0.4) = \int_{0.9}^1 \frac{5}{0.6^5} (y - 0.4)^4 dy
$$

The answer is about 60%.

```python
declare('y', interval=(0, 1))
cond_density = (5/(0.6**5))*(y - 0.4)**4
Integral(cond_density, (y, 0.9, 1)).doit()
```

Now we will use the conditional density to find a conditional expectation. Remember that in our example, given that $X = 0.4$ the possible values of $Y$ go from $0.4$ to 1.

$$
E(Y \mid X = 0.4) ~ = ~ \int_{0.4}^1 y \frac{5}{0.6^5} (y - 0.4)^4 dy ~ = ~ 0.9
$$

```python
Integral(y*cond_density, (y, 0.4, 1)).doit()
```

#### Endnotes ####
For any fixed value of $y$, the conditional density of $X$ given $Y = y$ is

$$
f_{X \mid Y=y} (x) ~ = ~ \frac{f(x, y)}{f_Y(y)} ~~~~~ \text{for all } x
$$

All the examples in this section and the previous one have started with a joint density function that apparently emerged out of nowhere. In the next section, we will study a context in which they arise.

```python

```


The Knapsack Problem by Stefano Gualandi is licensed under a Creative Commons Attribution 4.0 International License.
Based on a work at https://github.com/mathcoding/opt4ds.

# 3. The Knapsack Problem
In this first notebook, we show how to solve the Knapsack Problem using Integer Linear Programming.

## 3.1 Software Installation
If you are running this notebook in a Colab, you don't need to install anything else on your computer.

Otherwise, if you have installed the recommended Anaconda Python distribution, you have to run the following two commands:

1. To install the [Pyomo](http://www.pyomo.org/) optimization modeling language:

```
conda install -c conda-forge pyomo
```

2. To install the open source [GLPK](https://www.gnu.org/software/glpk/) solver:

```
conda install -c conda-forge glpk
```

3. (Optional) You can install some extra packages of Pyomo using the following command:

```
conda install -c conda-forge pyomo.extras
```

For details about the Pyomo installation, we refer to the official [Pyomo Documentation](https://pyomo.readthedocs.io/en/stable/).

The following lines are for running this notebook in a COLAB:

```python
import shutil
import sys
import os.path

if not shutil.which("pyomo"):
    !pip install -q pyomo
    assert(shutil.which("pyomo"))

if not (shutil.which("glpk") or os.path.isfile("glpk")):
    if "google.colab" in sys.modules:
        !apt-get install -y -qq glpk-utils
    else:
        try:
            !conda install -c conda-forge glpk
        except:
            pass
```

## 3.2 Mixed Integer Programming model
The Knapsack problem can be formulated as follows.

The input data are:

* The index set $I$ referring to the items
* The profit vector $c$
* The weight vector $A$
* The budget value $B$

For each item $i\in I$, we introduce a binary decision variables $x_i \in \{0,1\}$, which is used to define the following **Integer Linear Programming (ILP)**
problem:

\begin{align}\label{eq:1}
\max \;\; & c^T x \\
\mbox{s.t.} \;\; & \sum_{i \in I} A_i x_i \leq B & \\
& x_i \in \{0,1\},& \forall i \in I.
\end{align}

Since $x_i=1$ represents the decision of selecting item the $i$-th item, it is clear that the objective function (1) consists of maximizing the dot product $c^T\,x$. The single constraint (2) limits the number of selected item in such a way that the sum of the weights $A_i$ of the selected items does not exceed the available capacity $B$. The constraints (3) impose the domain on the decision variables $x_i$.

## 3.3 Pyomo Knapsack Model
The ILP model (1)-(3) can be expressed using the Pyomo optimization modeling language as shown next.

As a first step, we need to define the input data using the standard Python data structure. The simplest method to define the data is as follows:

```python
I = range(5)        # Items
C = [2, 3, 1, 4, 3] # Profits
A = [3, 4, 2, 1, 6] # Weights
B = 9               # Budget
```

This snippet of code is defining the following input data:

* The set $I$ as the range of number from 0 to 4, using the [range](https://docs.python.org/3/library/functions.html#func-range) builtin class. You can think of the range as a generator function for the list $[0,1,2,3,4]$.
* The cost vector $C$ and the weight vector $A$ are defined as two lists, using the standard [list()](https://docs.python.org/3/library/functions.html#func-list) builtin class.
* The budget parameter is defined as a given integer.

So far, we have not used any **Pyomo** construct. Before we start, we need to import the Pyomo library:

```python
from pyomo.environ import *
```

```python
# If you want to check all the elements imported from the library uncomment the following line
# who
```

### 3.3.1 Model and Variables
The first step in defining the model consists in choosing the type of Model we want to use. In Pyomo there are two options, the [ConcreteModel](https://pyomo.readthedocs.io/en/stable/library_reference/aml/index.html#pyomo.environ.ConcreteModel) and the [AbstractModel](https://pyomo.readthedocs.io/en/stable/library_reference/aml/index.html#pyomo.environ.AbstractModel). In this first example, we use the simpler ConcreteModel as follows:

```python
# Create concrete model
model = ConcreteModel()
```

The choice of the `model` name is recommended if you are planning to use the Pyomo command line tool.

Once we have defined the model, we can define the binary decision variable by using the [Var](https://pyomo.readthedocs.io/en/stable/library_reference/aml/index.html#pyomo.environ.Var) class. We define an object of type **Var** for each element of the range $I$, with the following command:

```python
# Variables
model.x = Var(I, within=Binary)
```

The `Binary` keyword is part of the `pyomo.environ` setting, and it is used to specify the constraint $x_i \in \{0,1\}$. Other possible values for the optional parameter `within` are: `NonNegativeReals`, `PositiveReals`, `PositiveIntegers`.

Note the the choice of $x$ as a name of the variable is arbitrary.

### 3.3.2 Objective Function
The objective function is defined via the [Objective](https://pyomo.readthedocs.io/en/stable/library_reference/aml/index.html#pyomo.environ.Objective.construct) class, as follows:

```python
# Objective Function: Maximize Profit
model.obj = Objective(expr = sum(C[i]*model.x[i] for i in I),
                      sense = maximize)
```

Again, the `obj` name is arbitrary, and you can select the one you prefer. The parameter `epxr` is mandatory, and is used to define the objective function expression. In the following example, we are using the `list comprehension` syntax to define our linear objective $\sum_{i \in I} c_i x_i$. Note that the python notation is very similar to the mathematical notation.

The parameter `sense` is optional, and it is used to define the type of objective function: `maximize` or `minimize`.

With Pyomo, we are not directly restricted to use only linear objective functions. It is the type solver we use that limits that type of problem that we can solve. As long as we use the GLPK solver, we can only define linear objective function.

### 3.3.3 Constraints
Finally, we need to define the budget constraint.
The constraints are defined using the [Constraint](https://pyomo.readthedocs.io/en/stable/library_reference/aml/index.html#pyomo.environ.Constraint) class. The minimal use of this class requires to define the `expr` input parameter. In the knapsack problem we have to define the budget constraint $\sum_{i \in I} A_i x_i \leq B$ as follows:

```python
# Constraint
model.capacity = Constraint(expr = sum(A[i]*model.x[i] for i in I) <= B)
```

Also the constraints can be named, in this case, we named it `capacity`. The name of the constraint can be used to retrieve information about the status of the constraint in a solution, that is, given a solution $\bar x$ to check whether $\sum_{i \in I} A_i \bar x_i < B$ or $\sum_{i \in I} A_i \bar x_i = B$.

### 3.3.4 Solving the Pyomo model
The complete Pyomo model defined so far is as follows:

```python
# Create concrete model
model = ConcreteModel()
# Variables
model.x = Var(I, within=Binary)
# Objective Function: Maximize Profit
model.obj = Objective(expr = sum(C[i]*model.x[i] for i in I),
                      sense = maximize)
# Constraint
model.capacity = Constraint(expr = sum(A[i]*model.x[i] for i in I) <= B)
```

In order to solve this model, we need to use a **solver**, that is a software that binds the data to the model and solve the corresponding instance of the problem. In this notebook, we use the GLPK solver, using a `SolverFactory` as follows:

```python
# Solve the model
sol = SolverFactory('glpk').solve(model)
```

```python
# Basic info about the solution process
for info in sol['Solver']:
    print(info)
```

Finally, in order to check the values of the decision variables, we can query the solved model by using the variable, objective function, and constraints names:

```python
# Report solution value
print("Optimal solution value: z =", model.obj())
print("Decision variables:")
for i in I:
    print("x_{} = {}".format(i, model.x[i]()))
print("Capacity left in the knapsack:", B-model.capacity())
```

In this case, we have found a solution with value equal to 9, given by selecting the three items $[0, 1, 3]$. We still had a unit of capacity left in the knapsack, but since no left items as a weight equal to 1, and we cannot take fractional items, that capacity stay unused.

## 3.4 Complete Python Script
The whole Python script for solving the Knapsack problem is as follows:

```python
# Import the libraries
from pyomo.environ import ConcreteModel, Var, Objective, Constraint, SolverFactory
from pyomo.environ import maximize, Binary

# CONCRETE MODEL: Data First, then model
I = range(5)        # Items
C = [2, 3, 1, 4, 3] # Profits
A = [3, 4, 2, 1, 6] # Weights
B = 9               # Budget

# Create concrete model
model = ConcreteModel()

# Variables
model.x = Var(I, within=Binary)

# Objective Function: Maximize Profit
model.obj = Objective(expr = sum(C[i]*model.x[i] for i in I),
                      sense = maximize)

# Constraint
model.capacity = Constraint(expr = sum(A[i]*model.x[i] for i in I) <= B)

# Solve the model
sol = SolverFactory('glpk').solve(model)

# Basic info about the solution process
for info in sol['Solver']:
    print(info)

# Report solution value
print("Optimal solution value: z =", model.obj())
print("Decision variables:")
for i in I:
    print("x_{} = {}".format(i, model.x[i]()))
print("Capacity left in the knapsack:", B-model.capacity())
```