# Python and $\LaTeX$

```python
%matplotlib inline

import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
```

## Python uses the $\LaTeX$ language to typeset equations.
Most LaTeX commands are prefixed with a "\". For example \pi is the
command to produce the lower case Greek letter pi.

The characters  # $ % & ~ _ ^ \ { } are special characters in LaTeX. If
you want to typeset them you need to put a \ in front of them. For
example \$ will typeset the symbol $

% - comment. Everything is ignored after a %
$ - Math mode. Start and Stop math mode. $\pi$
^ - Superscript in Math mode. $2^2$
_ - Subscript in Math mode. $2_2$
### Use a single set of `$` to make your $\LaTeX$ inline and a double set `$$` to center
$$ \int \cos(x)\ dx = \sin(x) $$
### This code will produce the output:

### $$ \int \cos(x)\ dx = \sin(x) $$

## Use can use $\LaTeX$ in plots:

```python
plt.style.use('ggplot')

x = np.linspace(0,2*np.pi,100)
y = np.sin(5*x) * np.exp(-x)

plt.plot(x,y)
plt.title("The function $y\ =\ \sin(5x)\ e^{-x}$")
plt.xlabel("This is in units of 2$\pi$")

plt.text(2.0, 0.4, '$\Delta t = \gamma\, \Delta t$', color='green', fontsize=36)
```

## Use can use `sympy` to make $\LaTeX$ equations for you!

```python
z = sp.symbols('z')
```

```python
a =  1/( (z+2)*(z+1) )

print(sp.latex(a))
```

$$ \frac{1}{\left(z + 1\right) \left(z + 2\right)} $$

```python
print(sp.latex(sp.Integral(z**2,z)))
```

$$ \int z^{2}\, dz $$

## `Astropy` can output $\LaTeX$ tables

```python
from astropy.io import ascii
from astropy.table import QTable
```

```python
T = QTable.read('Zodiac.csv', format='ascii.csv')
```

```python
T[0:3]
```

```python
ascii.write(T, format='latex')
```

---

## Some websites to open up for class:

- ## [Special Relativity](https://en.wikipedia.org/wiki/Special_relativity)

---

- ## [ShareLatex](https://www.sharelatex.com/)

- ## [ShareLatex Docs and Help](https://www.sharelatex.com/learn)

- ## [Latex Symbols](https://en.wikipedia.org/wiki/Wikipedia:LaTeX_symbols)

- ## [Latex draw symbols](http://detexify.kirelabs.org/classify.html)

- ## [The SAO/NASA Astrophysics Data System](https://ui.adsabs.harvard.edu/#classic-form)

---

- ## [Latex wikibook](https://en.wikibooks.org/wiki/LaTeX)

---

# Assignment for Week 8
-----------------------------------------------------------------------------
LaTeX homework - Create a LaTeX document with references.
-----------------------------------------------------------------------------

Start with the file: Example.tex

Minimum required elements:

* Between 2 and 4 pages in length (pages > 4 will be ignored).

* At least two paragraphs of text (the text should be coherent).

* At least 5 references from ADS

  * http://adsabs.harvard.edu/abstract_service.html
  * Make sure to \cite{} the references in your paper

* The equation on the back of the handout

* One (or more) equation(s) of your choice.

* One Compulsory plot generated with python (png format)

  * Start with t = np.linspace(0,12*np.pi,2000)
  * generate a butterfly plot: http://en.wikipedia.org/wiki/Butterfly_curve_(transcendental)
  * I have started it for you im the last cell of this notebook

* One other plot/image (do not reuse and old one!)

* One table of at least 4 columns and 4 rows.

* Bonus points given for interesting content!

-----------------------------------------------------------------------------

Create a PDF file:

Save the file as FirstLast.pdf  (i.e. TobySmith.pdf)

Upload the PDF to the class canvas site

-----------------------------------------------------------------------------
Deadline: Tuesday Nov 29 - 5pm (-5 pts for each 30 minutes late)
-----------------------------------------------------------------------------

```python
t = np.linspace(0,12*np.pi,2000)

fig,ax = plt.subplots(1,1)                    # One window
fig.set_size_inches(11,8.5)                   # (width,height) - letter paper landscape

fig.tight_layout()                            # Make better use of space on plot

```


# Monte Carlo Simulations

A Monte Carlo Simulation is any physical simulation making use of random numbers. You've already writen the code for two simple Monte Carlo simulations:

1. The decay of Thallium into lead
2. The Rutherford scattering of alpha particles off a gold atom

A field of physics which makes significant use of Monte Carlo simulations is statistical mechanics (they are also frequently used in astrophysics - I have most often encountered them around the generation of mock catalogues used to simulate astronomical surveys).

Using statistical mechanics as an example application, a common goal is to compute the average of some quantity in a system (e.g., the pressure in a gas) in thermal equilibrium at temperature $T$ (in kelvins).  Statistical mechanics theory tells us that while we cannot say specifically what state the system would be in (what all the particle speeds would be in a gas, for instance), the probability of its being in a state with energy $E_i$ is related to the "Boltzmann factor," $e^{-\beta E_i}$, via

\begin{equation}
P(E_i) = \dfrac{e^{-\beta E_i}}{Z} , (1)
\end{equation}

where $\beta \equiv 1/k_B T$ ($k_B$ is "Boltzmann's constant") and $Z \equiv \sum_i e^{-\beta E_i}$ is the "partition function," involving a sum over all "microstates" of the system.  The average value of some quantity $X$, whose value in the $i^{\mathrm{th}}$ state is $X_i$, then is

\begin{equation}
<X> = \sum_i X_i \, P(E_i) . (2)
\end{equation}

Usually, this sum cannot be determined analytically, so it must be computed numerically.  However, it would be prohibitively time-consuming, if not impossible, to do it directly, as the number of states involved typically is "astronomically" large.  MC techniques can come to the rescue here: the idea is to replace the complete sum by one over a smaller number of terms chosen randomly.  If we choose $N$ of the states, and designate them by the subscript $j$, then we can write

\begin{equation}
<X> \simeq \dfrac{\sum_{j=1}^N X_j P(E_j)}{\sum_{j=1}^N P(E_j)} ,   (3)
\end{equation}

where we have to include the denominator for normalization.  It was not needed in the previous expression because the probability is normalized for the full sum; i.e., $\sum_i P(E_i) = 1$ when *all* states $i$ are included.

It turns out that in this instance the simple approach of using a uniform sampling of states will give poor results: most of the states are associated with Boltzmann factors that are so small (since $E_j \gg k_B T$ for them) that they contribute a negligible amount to the sum.  In fact, most of the sum comes from a very small subset of the terms, most or all of which are likely to be missed in a uniform random sample.  Here's a situation where we should use importance sampling!

### Importance Sampling

The average above is expressible, in terms of weights $w_i$ that we choose, as

\begin{equation}
<X> \simeq \dfrac{1}{N} \sum_{j=1}^N \dfrac{X_j P(E_j)}{w_j} \sum_{i} w_i , (4)
\end{equation}

where the last sum is over *all* of the states, and normally is computed analytically.

The aim now is to choose the weights $w_i$ to pick out the large $P(E_i)$ terms in the sum, and such that the sum of the $w_i$ can be determined analytically.  But this is easy: we simply take $w_i = P(E_i)$ which, by Eq. (1), are normalized, $\sum_i w_i = \sum_i P(E_i) = 1$.  Then, the previous expression becomes

\begin{equation}
<X> \simeq \dfrac{1}{N} \sum_{j=1}^N X_j . (5)
\end{equation}

This is telling us that to average $X$, we just choose $N$ states in proportion to their Boltzmann probabilities; i.e., their likelihood of occurring.  (Of course, the more states we use, the more accurate the approximation.)  This averaging process is a simulation of the actual physical process in that the system will find itself most often in the states having the highest probabilities, so an average of a quantity $X$ in the system will be dominated by the values in the most probable states, just as in the average we've constructed.

### Markov Chains

There's a complication here: the probabilities $P(E_i)$ depend on the partition function $Z$, which we usually don't know *a priori* and which often is difficult to calculate numerically due to the large number of states involved, as mentioned earlier.  It is, however, possible to choose the probabilities without knowing $Z$, using a powerful model for random processes known as a ***Markov chain***, named after Russian mathematician Andrei Markov.

(ASIDE: The Wikipedia page for Markov chain lists numerous applications, including to the modeling of chemical reactions, to speech recognition systems, macroeconomic and population genetics models, and even Google's PageRank algorithm. It is also frequently used in astronomical applications such as statistics of large scale structure, fitting cosmological parameters to data and many others).

A Markov chain represents a random process in which a system transitions through a sequence of connected ***states*** (analogous to links in a chain), the sequence being determined by ***transition probabilities*** $T_{ij}$ from state $i$ to state $j$ which are *independent of the previous states in the chain*.  That is, in a Markov chain $T_{ij}$ depends only on the current state $i$ and the next state $j$.

In the present application, the states in the Markov chain will correspond to actual states of our physical system.  For a long enough chain, the probability of *being in* a particular state will be the same as the probability of *transitioning to* that state.  The idea, then, is to choose the $T_{ij}$ such that the probability of transitioning into any particular state $i$ on any step in the chain is the Boltzmann probability for that state, $P(E_i)$.

In order for the $T_{ij}$ to satisfy this condition, we first note that the $T_{ij}$ will be normalized according to $\sum_j T_{ij} = 1$, since the system must transition to *some* state from the $i^{\mathrm{th}}$ one.  We then choose the $T_{ij}$ so that they satisfy

\begin{equation}
\dfrac{T_{ij}}{T_{ji}} = \dfrac{P(E_j)}{P(E_i)} = \dfrac{e^{-\beta E_j} / Z}{e^{-\beta E_i} / Z} = e^{-\beta (E_j - E_i)}.    \hspace{50pt}  (6)
\end{equation}

Note that the partition function $Z$ has been eliminated from this result, so it will not need to be calculated.

It is not hard to prove (alhtough long, so see Appendix D of Newman's Computational Physics) that using this technique, a system started in any random state will evolve to the Boltzman Distribution (if you wait long enough).

### Metropolis Algorithms

We still need to work out what $T_{ij}$ should be (the transition probabilities we will use). One common method to do this is known as the ***Metropolis algorithm*** (also known as the Metropolis-Hastings algorithm), named after Nicholas Metropolis who was involved in its development.

Suppose that we start with the system in state $i$, and consider making a small change that would put the system in state $j$; e.g., by changing the energy of one gas particle.  (Note that the new state could be the same as the previous state; i.e., $T_{ii} \ne 0$, in general.  This would happen, for instance, if two gas particles collided elastically and interchanged  energies.)  The symbols $i$ and $j$ here designate steps in the Markov chain rather than particular states of the system.)  The Metropolis algorithm instructs us to accept such a change with an ***acceptance probability***

\begin{equation}
  P_a =
  \begin{cases}
    1 & \mathrm{if} \hspace{1mm} E_j \le E_i , \\
    e^{-\beta (E_j - E_i)}  & \mathrm{if} \hspace{1mm} E_j > E_i . (7)
  \end{cases}
\end{equation}

The idea here is that the system always will "choose" to go to a state with equal or lower energy than the current state, but it also may move to a state of higher energy, with some probability dependent on the temperature (through $\beta = 1/k_B T$).  (If this didn't happen, numerical models of thermodynamic systems would always result in them evolving directly to their ground state which, in fact, doesn't happen due to thermal excitations.)  In the second case above, the probability of transitioning to a state of higher energy is inversely related to the change in energy (the exponential factor becomes smaller as $E_j - E_i$ grows), and it's also inversely related to $T$ (smaller $T$ leads to a larger negative exponent and therefore a smaller exponential factor).  This simulated behavior is designed to match what we expect to happen physically: it's more likely for our system to jump to a higher energy if it's in a higher-temperature environment than a lower temperature environment, but the jump is less likely for a larger energy difference.

To convert the acceptance probability into a transition probability, suppose that the number of possible changes to the system is $M$.  Then the probability of transitioning from state $i$ to state $j$ is the probability of choosing a particular change (which would be just $1/M$), times the probability that we accept that change.  This lets us write the transition probabilities as

\begin{equation}
\begin{cases}
T_{ij} = \dfrac{1}{M} \times e^{-\beta (E_j - E_i)} \hspace{3mm} \mathrm{and} \hspace{3mm} T_{ji} = \dfrac{1}{M} \times 1 , & \hspace{5mm} \mathrm{if} \hspace{1mm} E_j > E_i ;  \\[2ex]
T_{ij} = \dfrac{1}{M} \times 1 \hspace{3mm} \mathrm{and} \hspace{3mm} T_{ji} = \dfrac{1}{M} \times e^{-\beta (E_i - E_j)} , & \hspace{5mm} \mathrm{if} \hspace{1mm} E_j \le E_i .
\end{cases} (8)
\end{equation}

then it is true that

\begin{equation}
\frac{T_{ij}}{T_{ij}} = \frac{e^{-\beta (E_j - E_i)}/M}{1/M} =  e^{-\beta (E_j - E_i)} (9)
\end{equation}

which is what we need.

### Markov Chain Monte Carlo Simulation

Now that we have identified all of the pieces, we can spell out the ***Markov Chain Monte Carlo*** (MCMC) simulation method (AKA fancy terms for something quite simple which will make you sound very clever):

1.  Choose a starting state (must be physically allowed, but otherwise this is up to you)

2.  Select a change to the system at random from among the set of possible changes, and compute the change in energy from the previous state.

3.  Compute the corresponding acceptance probability $P_a$ from Equation (7) above using the energy change. Accept the change with probability $P_a$---i.e., with that probability change to the new state; otherwise, reject the change.

4. Compute and store the value of the quantity (or quantities) of interest in the current state.

5. Repeat from step 2.

Step 3 of the sequence can be implemented by using `np.random.random()` to obtain a random number $z$ and comparing it with $e^{-\beta (E_j - E_i)}$; if $z < e^{-\beta (E_j - E_i)}$, then the change to the new state is made; if not, the change is rejected and the system remains in the same state.  (The larger $e^{-\beta (E_j - E_i)}$ is, the more likely $z$ is to be less than it and the more likely the change is to be accepted.)

It's important to emphasize a few points about the MCMC approach:

1. Even if a change of state is rejected and the system remains in its current state, that still counts as a "step" in the Markov chain, and therefore the value of any quantity of interest that's being recorded should be recorded again (and/or again added to any sum of its values).

2. In order for Eq. (9) to hold, the number of changes to the system that are available in going from state $i$ to state $j$ must be the same number that would be available in making the reverse change.  This might require careful selection of the set of possible state changes.

3. It must be possible eventually to get to any state; this property of the system, known as ***ergodicity***, is a requirement for the Metropolis algorithm to work.

4. While the Markov chain process always will converge to the correct Boltzmann probability distribution, there's no way to predict how many steps that will take; i.e., how long it will take for the system to ***equilibrate***.  Basically, one determines when equilibration has been achieved by looking for minimal or no changes in the output values.

### Example: MCMC Simulation of an Ideal Gas

In an ideal gas of N atoms, the total energy of the system is the sum of the energies of the particles

$$ E = \sum^N_{i=1} E(n_x^{(i)},n_y^{(i)},n_z^{(i)}),$$

where in a cubic box of length, $L$ an atom of mass, $m$ has energy given by

$$ E(n_x,n_y,n_z) = \frac{\pi^2 \hbar^2}{2mL^2} (n_x^2 + n_y^2 +n_z^2) $$

For our MCMC we will choose to:

1. Move a single atom into a neighbouring set (i.e. $n_x$, $n_y$ or $n_z$ changes by $\pm1$).

When this happens the change in energy is (for example for $n_x$ changing to $n_x+1$):

$$ \Delta E = \frac{\pi^2 \hbar^2}{2mL^2} ((n_x \pm 1)^2 + n_y^2 +n_z^2) - \frac{\pi^2 \hbar^2}{2mL^2} (n_x^2 + n_y^2 +n_z^2) $$

which simplifies to:

$$ \Delta E = \frac{\pi^2 \hbar^2}{2mL^2} (\pm 2n_x + 1). $$

(and the equivalent for changes in $n_y$ or $n_z$).

2. We will simulated N = 1000 particles, for $k_B T = 1$ and work in theorists units ($m = \hbar = 1$, and $L = 1$).

3. We will start in the "ground state" $N_x = n_y = n_z = 1$.

```python
import numpy as np
import numpy.random as rnd
import matplotlib.pyplot as plt
#This for Notebook display
%matplotlib inline

T = 10.0
N = 1000
steps = 250000

#Array of integers to store quantum numbers

n = np.ones([N,3],int)

#Main loop
eplot=[]
E=3*N*np.pi*np.pi/2

for k in range(steps):
    #Choose the particle and state to move
    i=rnd.randint(N) #Random integer from 0 to N-1
    j=rnd.randint(3) #Random integer from 0 to 2
    #Choose if to go +1 or -1 (50-50)
    if rnd.random()<0.5:
        dn = 1
        dE = (2*n[i,j]+1)*np.pi*np.pi/2
    else:
        dn=-1
        dE=(-2*n[i,j]+1)*np.pi*np.pi/2

    #Decide if move should be accepted.
    #If energy is not going to be reduced (ie. n=n+1 or we're in the ground state) just keep current state
    if n[i,j]>1 or dn==1:
        #Now calculate acceptance probability, and keep new state if random number is less than this
        if rnd.random()<np.exp(-dE/T):
            n[i,j] += dn
            E += dE

    #Save the state
    eplot.append(E)

#Make a graph of energy steps

plt.plot(eplot)
plt.ylabel("Energy")
plt.xlabel("Step number")
plt.show()
```

Comments:

* This code runs quickly, and equilibrates after about 50,000 steps.
* The average of the energy after this point would give an estimate of the energy of the system (in the dimensionless units we chose)
* We could speed this up by accepting all steps where dE is negative (ie. remove the probability calculation)

### Take Away Points about MCMC

Definitions:

* Monte Carlo Simulation - a simulation of a physical system using random numbers
* Markov Chain - a series of steps through states of a system with some probability
* Metropolis Alogorithm  - a way to chose what steps to keep and which not to (based on the change of energy of the states)
* MCMC - doing all of this at once.

To summarize:

* It's not as complicated as it sounds
* You already know how to do it
* So let's have a go at an interesting application

### Simulated Annealing

Annealing is the process of cooling materials slowly enough that their atoms/molecules are able to order themselves before the material cools enough to freeze. This process has been used for centuries by glass and metal works to create stronger materials.

Simulated annealing refers to doing this in your code. It provides a way to identify global minimima in system with complex likelihood surfaces.

The process is as follows:

* Create a Monte Carlo Simulation of your system (or function) with a high temperature
* Gradually decrease the temperature to find the global minima

Many of the applications of this process are not in physics, but real world simulations with many parameters, like optimizing car fuel consumption, traveling salesman problems etc.

So how slow is slow enough? Well there is no correct answer. Often the system is set up with a temperature high enough that it reach equillibrium really quickly, and then the temperature is dropped as

$$ T = T_0 e^{-t/\tau}$$

where $\tau$ is chosen by trial and error. This is another place in computational physics where you have to make a trade off between accuracy/the best answer, and computing power. If $\tau$ is too large the code may take too long to run, but if it's too small you may not get to the global minimum....

### Genetic Algorithms

Another effective optimization method inspired by the real world, which goes by the name ***genetic programming*** (based on ***genetic algorithms***).

This approach adopts the biological concepts of mutation, inheritance, and natural selection, and applies them to possible solutions to an optimization problem.

The idea is to start with a group of candidate solutions, evaluate their "fitness" as solutions to the problem, and then "mutate" them to try to improve their fitness.

### Class Exercise - the Classic "Traveling Salesperson" Problem

Find the shortest route which visits a set of N locations

Example applications:
   * shortest route to fly between N cities
   * shortest route for postal worker to deliver to N houses
   * shortest route to pick up your N friends in a bus
   * shortest route to visit N tourist attractions in a city

This is a classic problem for computer scientists and computational physicists alike.

Set up the problem as follows:

1. Denote the position of location i with a 2D vector, $r_i = (x_i, y_i)$.
2. Randomly select $N$ positions, $r_i$ (start with a small value of $N$)
3. Make a plot of the $N$ locations
4. Start by visiting the locations in the order you set them up, calculating the total distance travelled as

$$ D = \sum^{N-1}_{i=0} | \bf{r}_{i+1} - \bf{r}_i | $$

Run the simulation:

Now use an MCMC with Simulated Annealing to find the shorted route.

In (hopefully) plain English:
  1. Swap two randomly selected cities
  2. If the swap shortens the tour keep it, if not keep it with the Metropolis probability for a "temperature" T and an "energy" $E = D$ (ie. keep if the probability, P is

  $$ P \lt e^{\Delta D/T}$$

for some value of $T$ you have chosen.
  3. Now reduce $T$ to simulate annealing according to your "cooling schedule" $ T = T_0 e^{-t/\tau}$ (you have to pick a value of $\tau$ and $t$ for your time step also).
  4. Repeat
  5. Make a plot showing the final route

Tips/good starting points (but experiment!)

* N = 10 will run in a reasonable time.
* Write functions to calculate the total distance, and the length (magnitude) of a vector.
* ```r=np.empty([N+1,2],float)``` will define an (N+1)x 2 array.
* You can plot this using ```plt.plot(r[:,0],r[:,1]```
* Don't forget to go back to the beginning location at the end
* When swapping locations don't forget to check they are different locations
* You may want to output every some number of steps to check progress. ```if t%5000==0:``` will do something every 5000th step.
* Set a Tmin which is greater than zero (or there will be errors). For example try:

```Tmax = 10.0
Tmin = 1e-3
tau = 2e4```

Once you are sure your code runs, trying timing it for different numbers of locations. You can try to use
```
import time
start_time = time.clock()
main()
print(time.clock() - start_time, "seconds")```

```python

```


# Algorithms Exercise 2

## Imports

```python
%matplotlib inline
from matplotlib import pyplot as plt
import seaborn as sns
import numpy as np
```

## Peak finding

Write a function `find_peaks` that finds and returns the indices of the local maxima in a sequence. Your function should:

* Properly handle local maxima at the endpoints of the input array.
* Return a Numpy array of integer indices.
* Handle any Python iterable as input.

```python
def find_peaks(a):
    """Find the indices of the local maxima in a sequence."""

```

```python
p1 = find_peaks([2,0,1,0,2,0,1])
assert np.allclose(p1, np.array([0,2,4,6]))
p2 = find_peaks(np.array([0,1,2,3]))
assert np.allclose(p2, np.array([3]))
p3 = find_peaks([3,2,1,0])
assert np.allclose(p3, np.array([0]))
```

Here is a string with the first 10000 digits of $\pi$ (after the decimal). Write code to perform the following:

* Convert that string to a Numpy array of integers.
* Find the indices of the local maxima in the digits of $\pi$.
* Use `np.diff` to find the distances between consequtive local maxima.
* Visualize that distribution using an appropriately customized histogram.

```python
from sympy import pi, N
pi_digits_str = str(N(pi, 10001))[2:]
```

```python
w=pi_digits_str
r=[]
for x in w:
    r.append(int(x))
a=np.array(r)
m=find_peaks(a)
s=np.diff(m)
plt.hist(s,bins=50)
s
```

```python
assert True # use this for grading the pi digits histogram
```


This notebook mainly implements the **Structured Low-Rank Matrix Completion (SLRMC)** of the following paper:

Jonathan Gillard, Konstantin Usevich (2018). Structured low-rank matrix completion for forecasting in time series analysis. arXiv:1802.08242.

Technical highlights:

- Embed a time series into a Hankel matrix and the missing data (to be forecasted) are stored in the bottom right-hand corner of this matrix.
- Consider a matrix completion problem for Hankel matrices.
- Choose a proper weighting scheme for the known observations.

```python
import numpy as np
from numpy.linalg import inv as inv
```

The solution of problem
\begin{equation}
\begin{aligned}
&\min_{\boldsymbol{x}\in\mathbb{R}^{(m+n)}}~\|\mathcal{S}(\boldsymbol{x})\|_{*} \\
&\text{s.t.}~\boldsymbol{x}_{[1:n]}=\boldsymbol{y}, \\
\end{aligned}
\end{equation}
is given by

1) Construct the scaled initial data vector $\tilde{\boldsymbol{y}}$ as
$$\tilde{\boldsymbol{y}}=\left(y_1\cdot\operatorname{exp}\left(\frac{\alpha}{2}\right),y_2\cdot\operatorname{exp}\left(\frac{2\alpha}{2}\right),\cdots,y_n\cdot\operatorname{exp}\left(\frac{n\alpha}{2}\right)\right),$$
for a given $\alpha$.

2) Solve the problem
$$\begin{aligned}
&\min_{\boldsymbol{x}\in\mathbb{R}^{(m+n)}}~\|\mathcal{S}(\boldsymbol{x})\|_{*} \\
&\text{s.t.}~\boldsymbol{x}_{[1:n]}=\tilde{\boldsymbol{y}}. \\
\end{aligned}
$$

3) Scale back the weighted values
$$\hat{\boldsymbol{y}}=\left(x_1\cdot\operatorname{exp}\left(-\frac{\alpha}{2}\right),x_2\cdot\operatorname{exp}\left(-\frac{2\alpha}{2}\right),\cdots,x_n\cdot\operatorname{exp}\left(-\frac{n\alpha}{2}\right),\cdots,x_{m+n}\cdot\operatorname{exp}\left(-\frac{(m+n)\alpha}{2}\right)\right).$$

```python
def scale_to_data(vec, alpha):
    dim = vec.shape[0]
    new_vec = np.zeros(dim)
    for i in range(dim):
        new_vec[i] = vec[i] * np.exp((i + 1) * alpha / 2)
    return new_vec

def scale_back_data(vec, alpha):
    dim = vec.shape[0]
    new_vec = np.zeros(dim)
    for i in range(dim):
        new_vec[i] = vec[i] * np.exp(- (i + 1) * alpha / 2)
    return new_vec
```

What is structured low-rank matrix completion?

```python
def hankel(vec, window_length):
    column_num = vec.shape[0] - window_length + 1
    hankel_mat = np.zeros((window_length, column_num))
    for i in range(window_length):
        hankel_mat[i, :] = vec[i : column_num + i]
    return hankel_mat

def hankel2vec(mat):
    dim1, dim2 = mat.shape
    new_mat = np.zeros((dim1, dim1 + dim2 - 1))
    for i in range(dim1):
        new_mat[i, i : dim2 + i] = mat[i, :]
    return np.true_divide(new_mat.sum(0), (new_mat != 0).sum(0))

def svt(mat, tau):
    [m,n] = mat.shape
    if 2 * m < n:
        u, s, v = np.linalg.svd(mat @ mat.T, full_matrices = 0)
        s = np.sqrt(s)
        tol = n * np.finfo(float).eps * np.max(s)
        idx = np.sum(s > max(tau, tol))
        mid = (s[:idx] - tau) / s[:idx]
        return (u[:,:idx] @ np.diag(mid)) @ (u[:,:idx].T @ mat)
    elif m > 2 * n:
        return svt(mat.T, tau).T
    u, s, v = np.linalg.svd(mat, full_matrices = 0)
    idx = np.sum(s > tau)
    return u[:,:idx] @ np.diag(s[:idx]-tau) @ v[:idx,:]

def SLRMC(vec, window_length, forecast_horizon, rho, maxiter):
    mat = hankel(np.append(vec, np.zeros(forecast_horizon)), window_length)
    pos_missing = np.where(mat == 0)
    dim1, dim2 = mat.shape
    T = np.zeros((dim1, dim2))
    Z = mat.copy()
    for it in range(maxiter):
        X = svt(Z - T / rho, 1 / rho)
        Z[pos_missing] = (X + T / rho)[pos_missing]
        T = T + rho * (X - Z)
        vec_hat = hankel2vec(X)
    return vec_hat
```

Toy example: forecast $\boldsymbol{y}=\left(1,2,3,4,5,6,?,?\right)$ where we assume that the last two values are 7 and 8 (i.e., ground truth).

Ready? Let us do it!

```python
y = np.array([1, 2, 3, 4, 5, 6])
alpha = - 1 / 2
window_length = 5
forecast_horizon = 2
rho = 0.1
maxiter = 200
y_tilde = scale_to_data(y, alpha)
x = SLRMC(y_tilde, window_length, forecast_horizon, rho, maxiter)
y_hat = scale_back_data(x, alpha)
print('Original data sequence:')
print(y)
print()
print('Forecasted data sequence:')
print(y_hat)
```

    Original data sequence:
    [1 2 3 4 5 6]

    Forecasted data sequence:
    [1.00016074 1.9998642  2.99993493 4.00004691 5.00041026 5.99939527
     6.99627719 7.9884971 ]

    /Users/xinyuchen/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:13: RuntimeWarning: invalid value encountered in true_divide
      del sys.path[0]

This rather obvious sequence can be easily forecasted. But how about more complicated sequences?

```python
import scipy.io

tensor = scipy.io.loadmat('../datasets/Guangzhou-data-set/tensor.mat')
tensor = tensor['tensor']
data = tensor.reshape([tensor.shape[0], tensor.shape[1] * tensor.shape[2]])[0, - 9 * 144 :]
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
fig = plt.figure(figsize = (14, 2))
plt.plot(data, color = "b")
plt.show()
```

```python
def compute_mape(var, var_hat):
    return np.sum(np.abs(var - var_hat) / var) / var.shape[0]

def compute_rmse(var, var_hat):
    return  np.sqrt(np.sum((var - var_hat) ** 2) / var.shape[0])
```

```python
import time
start = time.time()
steps = 5 * 144 / 12
alpha = - 0.005
window_length = 144
forecast_horizon = 12
rho = 0.05
maxiter = 200

data_hat = np.zeros(np.int(steps * forecast_horizon))
for i in range(np.int(steps)):
    y = data[: 4 * 144 + 12 * i]
    y_tilde = scale_to_data(y, alpha)
    x = SLRMC(y_tilde, window_length, forecast_horizon, rho, maxiter)
    y_hat = scale_back_data(x, alpha)
    data_hat[forecast_horizon * i : forecast_horizon * (i + 1)] = y_hat[- forecast_horizon :]

    if (i + 1) % 60 == 0:
        fig = plt.figure(figsize = (14, 2))
        plt.plot(data, color = "b")
        plt.plot(np.arange(4 * 144, 9 * 144), data_hat, color = 'r')
        plt.show()

mape = compute_mape(data[4 * 144 :], data_hat)
rmse = compute_rmse(data[4 * 144 :], data_hat)
print('Prediction MAPE: {:.6}'.format(mape))
print('Prediction RMSE: {:.6}'.format(rmse))
end = time.time()
print('Running time: %d seconds'%(end - start))
```

This is a case of single time series forecasting. How about multivariate time series forecasting?


# Design of Retaining Wall
http://structengblog.com/retaining-wall-analysis-ipython-sympy-possible-bim-integration/

```python
from sympy import *

init_printing()

ka, q, gs, z = symbols('k_a q gamma_s z') # soil properties and depth
gfq, gfg = symbols('gamma_fq gamma_fg') # partial load factors
pa, va, ma = symbols('p_a v_a m_a') # force effects

pressure = Eq(pa,ka*(gs*z*gfg + q*gfq))
pressure
```

```python
shear = Eq(va, Integral(pressure.rhs, z))
shear
```

```python
shear = shear.doit()
shear
```

```python
moment = Eq(ma, Integral(shear.rhs,z))
moment
```

```python
moment = moment.doit()
moment
```

```python
wallData = ({ka:0.3, gs:19, gfg:1.35, gfq:1.5, q:5, z:3.4})
designSliding = shear.subs(wallData).rhs
print('The ULS sliding force at the base of the wall = {:.2f} kN'.format(designSliding))

```

    The ULS sliding force at the base of the wall = 52.13 kN

```python
designMoment = moment.subs(wallData).rhs
print('The ULS overturning moment at the base of the wall = {:.2f} kNm'.format(designMoment))
```

    The ULS overturning moment at the base of the wall = 63.41 kNm

```python
# geometric properties of the wall
hs, ts, lb, tb, toe = symbols('h_s t_s l_b t_b toe')
gc = symbols('gamma_c') # unit weight of the wall
W, Mst, Ru = symbols('W M_st R_u') # resistances
mu = symbols('mu') # coeff of friction to base
stemWeight = gc*hs*ts
baseWeight = gc*lb*tb
soilLength = lb-(toe+ts)
soilWeight = hs*soilLength*gs
totalWeight = Eq(W,stemWeight+baseWeight+soilWeight)
totalWeight
```

```python
slidingResistance = Eq(Ru, totalWeight.rhs*mu)
slidingResistance
```

```python
# lever arms to the elements
lab = lb/2
las = toe+ts/2
lau = lb - soilLength/2
stabilityMoment = Eq(Mst,stemWeight*las+baseWeight*lab + soilWeight*lau)
stabilityMoment
```

```python
wallGeo = ({gc:24, hs:3.15, ts:0.25, tb:0.25, toe:0.75, gs:19, mu:0.5})
slidingStability = slidingResistance.subs(wallGeo)
slidingStability
```

```python
baseLength1 = solve(slidingStability.rhs - designSliding, lb)[0]
print('The minimum base length to prevent sliding = {:.2f} m'.format(baseLength1))
```

    The minimum base length to prevent sliding = 2.21 m

```python
overturningStability = stabilityMoment.subs(wallGeo)
overturningStability
```

```python
baseLength2 = solve(overturningStability.rhs - designMoment, lb)[1]
print('The minimum base length to resist overturning = {:.2f} m'.format(baseLength2))
```

    The minimum base length to resist overturning = 1.53 m

```python
# by inspection of the above solutions adopt the following base length
baseLength = 2.3
```

```python
# Calculate the eccentricity of load from the centroid of the base
ec = symbols('e_c')
netMoment = stabilityMoment.rhs-moment.rhs
xBar = netMoment / totalWeight.rhs
eccentricity = Eq(ec,lb/2 - xBar)
eccentricity
```

```python
# amend the partial load factors to 1 for the force effectrs
wallData = ({ka:0.3, gs:19, gfg:1.0, gfq:1.0, q:5, z:3.4})
wallGeo = ({gc:24, hs:3.15, ts:0.25, tb:0.25, toe:0.75, gs:19, mu:0.5, lb: baseLength})
wallData.update(wallGeo)
actEc = eccentricity.subs(wallData)
actEc.evalf(3)
```

```python
# The kern of the foundation
kern = lb.subs(wallData)/6
kern.evalf(3)
```

```python
if actEc.rhs > kern:
    print('Tension occurs at the heel')
    qmax = 2*totalWeight.rhs.subs(wallData)/(3*xBar.subs(wallData))
    print('The maximum pressure = {:.2f} kPa'.format(qmax))
else:
    print('No tension occurs under the base')
    qmax = actEc.rhs*totalWeight.rhs.subs(wallData)*6/(lb.subs(wallData)**2)\
    +totalWeight.rhs.subs(wallData)/lb.subs(wallData)
    print('The maximum pressure = {:.2f} kPa'.format(qmax))
```

    No tension occurs under the base
    The maximum pressure = 62.00 kPa

```python
fck, fy = symbols('f_ck f_y')
ec, ecy, ecu = symbols('e_c, e_cy, e_cu')
fc = symbols('f_c')
conc_str = Eq(fc, 4.0/9*fck*(2*(ec/ecy)-(ec/ecy)**2))
conc_str
```

```python
conc_data = ({fck:25, ecy:0.002, ec:0.0025})
conc_stress = conc_str.rhs.subs(conc_data)
conc_stress
```

```python

```


```python
%matplotlib notebook
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import scipy.optimize
```

# Diagnóstico de cancer usando un regresor logístico

Considere el dataset de [diagnóstico de cancer de mama de la Universidad de Wisconsin](https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+(Diagnostic))

```python
df = pd.read_csv('cancer.csv', index_col=0)
df['diagnosis'] = df['diagnosis'].map({'M': 1, 'B': 0})
display(df.head())
```

El objetivo de esta actividad formativa es entrenar un regresor logístico para clasificar las muestras como benignas y malignas (columna *diagnosis*) en base a las demás columnas disponibles. Las columnas corresponden al valor medio (1), desviación estándar (2) y "peor caso" (3) de 10 descriptores de la forma de los nucleos celulares presentes en la biopsia del paciente. Cada fila en la tabla corresponde a un paciente distinto.

Revisando este ejercicio aprenderas a

1. formular una regresión logística
1. estimar los mejores parámetros de un regresor logístico en base a datos
1. implementar un regresor logístico con `numpy` y `scipy`
1. analizar los resultados del regresor logístico

## Modelo de regresión logística

El regresor logístico es la extensión del regresor lineal para problemas de clasificación binaria, es decir donde los datos se agrupan en dos clases

Sean $N$ tuplas $\{\vec x_i, y_i\}$ donde $\vec x_i \in \mathbb{R}^D$ y $y_i \in \{0, 1\}$. El modelo de **regresión logística** para estos datos busca ajustar

$$
y_i \approx \mathcal{S} \left(\theta_0 + \sum_{j=1}^D \theta_j x_{ij}\right),
$$

donde $\mathcal{S}(z) = \frac{1}{1+\exp(-z)}$ se conoce como función logística o sigmoide

```python
z = np.arange(-10, 10, step=0.1)
sigmoide = lambda z : 1/(1+np.exp(-z))

fig, ax = plt.subplots(figsize=(7, 3), tight_layout=True)
ax.plot(z, sigmoide(z));
```

## Función de costo para el regresor logístico

Definamos primero el regresor lineal como

$$
f_\theta(x_i) = \theta_0 + \sum_{j=1}^D \theta_j x_{ij}
$$

en base a esta definición el modelo de regresión logística es entonces $\mathcal{S} (f_\theta(x_i))$

Para optimizarlo usaremos como función de costo el **logaritmo de la verosimilitud** (ver lección estadística inferencial para más detalles)

$$
\hat \theta = \text{arg} \max_\theta \log \mathcal{L}(\theta)
$$

Asumiremos que las observaciones $y_i$ son independientes y que siguen una [distribución de Bernoulli](https://en.wikipedia.org/wiki/Bernoulli_distribution) con parámetro $p=\mathcal{S} (f_\theta(x_i))$. Luego podemos escribir el logaritmo de la verosimilitud como

$$
\begin{align}
\log \mathcal{L}(\theta) &= \sum_{i=1}^N \log p(y_i | \mathcal{S} (f_\theta(x_i)) ) \nonumber \\
&= \sum_{i=1}^N \log \mathcal{S} (f_\theta(x_i))^{y_i} (1-\mathcal{S} (f_\theta(x_i)) )^{1-y_i} \nonumber \\
&= \sum_{i=1}^N y_i \log (\mathcal{S} (f_\theta(x_i)) ) + (1-y_i) \log(1-\mathcal{S} (f_\theta(x_i)) ) \nonumber
\end{align}
$$

donde la primera igualdad viene de reemplazar la función de masa de la distribución de Bernoulli y la segundo de aplicar el logaritmo

Este modelo no admite una solución analítica cerrada, por lo tanto usaremos un método de optimización iterativo que tome el logaritmo de la verosimilitud como función de costo. En particular usaremos un algoritmo basado en gradiente descendente por lo que necesitamos calcular la derivada de la función de costo

La derivada de la log verosimilitud con respecto a $\theta_j$ es

$$
\frac{d}{d\theta_j} \log \mathcal{L}(\theta) = \sum_{i=1}^N \left ( \frac{y_i}{\mathcal{S} (f_\theta(x_i))} - \frac{1-y_i}{1-\mathcal{S} (f_\theta(x_i))} \right) \frac{d \mathcal{S} (f(x_i, \theta))}{d f_\theta(x_i)} \frac{d f_\theta(x_i)}{d \theta_j}
$$

donde

$$
\frac{d \mathcal{S}(z)}{dz} =  \mathcal{S}(z) ( 1 - \mathcal{S}(z))
$$

entonces

$$
\begin{align}
\frac{d}{d\theta_j} \log \mathcal{L}(\theta) &= \sum_{i=1}^N \left[ y_i - y_i\mathcal{S} (f_\theta(x_i)) - \mathcal{S} (f_\theta(x_i))  + y_i \mathcal{S} (f_\theta(x_i)) \right] \frac{d f_\theta(x_i)}{d \theta_j} \nonumber \\
&= \sum_{i=1}^N \left[ y_i  - \mathcal{S} (f_\theta(x_i)) \right] \frac{d f_\theta(x_i)}{d \theta_j} \nonumber
\end{align}
$$

La derivada con respecto a los parámetros es

$$
\frac{d f(x_i, \theta)}{d \theta_0} =  1
$$

y para $j>0$
$$
\frac{d f(x_i, \theta)}{d \theta_j} =  x_{ij}
$$

Con esto tenemos todo lo necesario para implementar el modelo

## Implementación y optimización del modelo

Primero implementamos el logaritmo de la verosimilitud y su gradiente

Lo hacemos de tal forma que sean compatibles con `scipy.optimize.minimize`

```python
# Modelo
def modelo(theta, X):
    f = theta[0] + np.sum(theta[1:]*X, axis=1)
    return sigmoide(f)

# Función de costo
def neglogverosimilitud(theta, *args):
    X, Y = args
    S = modelo(theta, X)
    # Le agregamos un signo menos ya que quiero minimizar en lugar de maximizar
    return -np.sum(Y*np.log(S+1e-10) + (1-Y)*np.log(1-S+1e-10), axis=0)
    # La función logaddexp es más estable numericamente:
    #return -np.sum(-np.logaddexp(0, -f) - (1-y)*f, axis=0)

# Gradiente
def grad_neglogverosimilitud(theta, *args):
    X, Y = args
    N = len(Y)
    S = modelo(theta, X)
    X1 = np.concatenate((np.ones(shape=(N, 1)), X), axis=1)
    e = (Y - S)
    return -np.sum(e[:, np.newaxis]*X1, axis=0)
```

El ajuste del modelo se muestra a continuación.

Para facilitar la convergencia del modelo estándarizamos los atributos restándole su media y dividiendo por su desviación estándar

Aprovechando que tenemos la información del gradiente usaremos BFGS para optimizar. BFGS es un método quasi-Newton que usa información de las primeras derivadas

Para validar que el modelo usaremos un conjunto *hold-out*. Otras técnicas para validar son *k-fold* y *bootstrap*

```python
# Obtener base de datos
Y = df["diagnosis"].values
X = df.drop(columns=["diagnosis"]).values
# Estandarizar
X_std = (X - np.mean(X, axis=0))/np.std(X, axis=0)

N = len(Y)
D = X_std.shape[1]
train_proportion = 0.75
idx = np.random.permutation(len(X))
train_idx, test_idx = idx[:int(N*train_proportion)], idx[int(N*train_proportion):]

# Solución inicial
theta_init = 0.1*np.random.randn(1+D)
# Usaremos un callback para guardar el mejor modelo de validación
best_theta = np.zeros_like(theta_init)
best_logl = np.inf
def eval_model(theta):
    global best_theta, best_logl
    logltrain = neglogverosimilitud(theta, *(X_std[train_idx, :], Y[train_idx]))/len(train_idx)
    logltest = neglogverosimilitud(theta, *(X_std[test_idx, :], Y[test_idx]))/len(test_idx)
    print("Train: %0.4f, Test: %0.4f" %(logltrain, logltest))
    if logltest < best_logl: # Guardar el mejor modelo de test
        best_theta = theta
        best_logl = logltest

# Mejor valor de theta
res = scipy.optimize.minimize(fun=neglogverosimilitud, x0=theta_init,
                              method='BFGS', jac=grad_neglogverosimilitud,
                              args=(X_std[train_idx, :], Y[train_idx]),
                              callback=eval_model, tol=1e-1)

print(res.message)
```

## Evaluación del modelo de clasificación

La salida de este clasificador es un número en el rango $[0, 1]$

```python
p = modelo(best_theta, X_std)

fig, ax = plt.subplots(figsize=(7, 3), tight_layout=True)
ax.hist(p[Y==1], label='Ejemplos con etiqueta maligno', bins=10, range=(0, 1), alpha=0.5, color='r')
ax.hist(p[Y==0], label='Ejemplos con etiqueta benigno', bins=10, range=(0, 1), alpha=0.5, color='b')
ax.set_xlabel('Predicción del modelo')
#ax.set_yscale('log')
ax.legend();
```

Para tomar un decisión se debe seleccionar un umbral $\mathcal{T} \in [0,1]$ tal que

$$
d_i =
\begin{cases}
\text{maligno/1}, & \text{si } p(y_i|\theta, \vec x_i) < \mathcal{T} \\
\text{benigno/0}, & \text{si } p(y_i|\theta, \vec x_i) \geq \mathcal{T}
\end{cases}
$$

Una vez seleccionado el umbral se puede contar la cantidad de
- True positives (TP): Tumores malignos clasificados como malignos
- True negative (TN): Tumores benignos clasificados como benignos
- False positives (FP): Tumores benignos clasificados como malignos
- False negative (FN): Tumores malignos clasificados como benignos

Estas métricas son la base para construir una "tabla o matriz de confusión" para el clasificador

|Clasificado como/En realidad era|Positivo/1:|Negativo/0:|
|---|---|---|
|Positivo/1:|TP | FP |
|Negativo/0:| FN | TN |

Por ejemplo si usamos $\mathcal{T} = 0.5$ tendríamos

```python
def matriz_confusion(Y_real, Y_pred):
    TP = sum(Y_real & Y_pred)
    FP = sum(~Y_real & Y_pred)
    FN = sum(Y_real & ~Y_pred)
    TN = sum(~Y_real & ~Y_pred)
    return np.array([[TP, FP], [FN, TN]])

C = matriz_confusion(Y[test_idx], modelo(res.x, X_std[test_idx]) > 0.5)
display(C)
```

En la próxima unidad veremos formas más sofisticadas para evaluar modelos de clasifición

```python

```


# SymPyにおける並列回路演算子"//"のオーバーロード
© 2020 @RR_Inyo

電気回路において，2つのインピーダンス$Z_{1}$，$Z_{2}$の並列回路のインピーダンス$Z_{p}$は
$$
Z_{p} = \frac{Z_{1} Z_{2}}{Z_{1} + Z_{2}}
$$
と書ける。このような「『和』分の『積』」の形は煩雑であるため，電気工学の分野ではこれを簡略化して，
$$
Z_{p} = Z_{1} \: // \: Z_{2}
$$
と略記する。この演算子"//"をSymPyでも使えるようにオーバーロードしてみよう。
Pythonの特殊メソッド \_\_floordiv()\_\_ をSymbolのアトリビュートに追加すれば可能である。

```python
# 準備
import sympy as sp
```

```python
# 2つのインピーダンスのシンボルを定義
Z1, Z2 = sp.symbols('Z_{1} Z_{2}')

# オーバーロード前の挙動
print('Z1 // Z2 =')
display(Z1 // Z2)
print('切り捨て除算(floor division)となっている。')
print('ぶっちゃけ，切り捨て除算はあまり使うことはないと思われる。')
```

    Z1 // Z2 =

$\displaystyle \left\lfloor{\frac{Z_{1}}{Z_{2}}}\right\rfloor$

    切り捨て除算(floor division)となっている。
    ぶっちゃけ，切り捨て除算はあまり使うことはないと思われる。

```python
# 演算子のオーバーロード
## オーバーロードするための関数を準備する
def __floordiv__(self, other):
    return self * other / (self + other)

## Symbolのアトリビュートに追加
sp.Symbol.__floordiv__ = __floordiv__

# オーバーロード後の挙動
print('Z1 // Z2 =')
display(Z1 // Z2)
print('並列回路の「『和』分の『積』」となっている！')
```

    Z1 // Z2 =

$\displaystyle \frac{Z_{1} Z_{2}}{Z_{1} + Z_{2}}$

    並列回路の「『和』分の『積』」となっている！

```python

```


```python
#Libraries
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.collections import LineCollection
from matplotlib.colors import ListedColormap, BoundaryNorm
from scipy.integrate import odeint
from scipy import signal

from mpmath import *
from sympy import *
from sympy.solvers import solve, nonlinsolve
```

# Example 1

$ \dot{x} = \sin{x}$

```python
X, t = symbols("X t")

# Define EDO to solve: x' = sin(x)
eq = X(t).diff(t)- sin(X(t))

# Solve Equation
sol=dsolve(eq, X(t))

print(sol)
```

    [Eq(X(t), -acos((-exp(2*C1 + 2*t) - 1)/(exp(2*C1 + 2*t) - 1)) + 2*pi), Eq(X(t), acos((-exp(2*C1 + 2*t) - 1)/(exp(2*C1 + 2*t) - 1)))]

```python
# x limits
xi = -10
xf = 10
Nx = 1000

x, xstep = np.linspace(xi, xf , Nx, retstep=True)

dx = np.sin(x)

plt.close()
plt.title(r'$\dot{x}$ vs. x ')
plt.plot(x, dx,"-")
plt.xlabel(r'$x$')
plt.ylabel(r'$\dot{x}$')
plt.grid()
plt.show()
```

```python
X = symbols('X', real=True)

eq = sin(X)

sol1 = solve([X >= -10, X <= 10, eq ], X)

print(sol1)
```

    (-10 <= X) & (X <= 10) & (Eq(X, 0) | Eq(X, pi) | Eq(X, 2*pi))

```python
# x limits
xi = -10
xf = 10
Nx = 1000

x, xstep = np.linspace(xi, xf , Nx, retstep=True)
dx = np.sin(x)

x_stable = np.array([-3*np.pi, -np.pi, np.pi, 3*np.pi])
dx_stable = np.zeros(len(x_stable))

x_unstable = np.array([-2*np.pi, 0, 2*np.pi])
dx_unstable = np.zeros(len(x_unstable))

plt.close()
plt.title(r'$\dot{x}$ vs. x ')
plt.plot(x, dx,"-", x_stable, dx_stable, 'bo', x_unstable, dx_unstable, 'ro')
plt.xlabel(r'$x$')
plt.ylabel(r'$\dot{x}$')
plt.grid()
plt.show()
```

```python
plt.close()

# x limits
xi = -10
xf = 10
Nx = 1000

x, xstep = np.linspace(xi, xf , Nx, retstep=True)

dx = np.sin(x)

# Create a set of line segments so that we can color them individually
# This creates the points as a N x 1 x 2 array so that we can stack points
# together easily to get the segments. The segments array for line collection
# needs to be (numlines) x (points per line) x 2 (for x and y)
points = np.array([x, dx]).T.reshape(-1, 1, 2)
segments = np.concatenate([points[:-1], points[1:]], axis=1)

fig = plt.figure(figsize=(8,5))
axes = plt.gca()

# Use a boundary norm instead
cmap = ListedColormap(['r', 'b'])
norm = BoundaryNorm([-1, 0.0, 1], cmap.N)
lc = LineCollection(segments, cmap=cmap, norm=norm)
lc.set_array(dx)
lc.set_linewidth(2)
line = axes.add_collection(lc)
fig.colorbar(line, ax=axes)

axes.set_xlim(x.min(), x.max())
axes.set_ylim(-1.1, 1.1)
plt.show()
```

```python
plt.close()

# x limits
xi = -10
xf = 10
Nx = 1000

# Data
x, xstep = np.linspace(xi, xf , Nx, retstep=True)
dx = np.sin(x)

# Stable fixed points
x_stable = np.array([-3*np.pi, -np.pi, np.pi, 3*np.pi])
dx_stable = np.zeros(len(x_stable))

# Unstable fixed points
x_unstable = np.array([-2*np.pi, 0, 2*np.pi])
dx_unstable = np.zeros(len(x_unstable))

# Segments
points = np.array([x, dx]).T.reshape(-1, 1, 2)
segments = np.concatenate([points[:-1], points[1:]], axis=1)

fig = plt.figure(figsize=(8,5))
axes = plt.gca()

# Use a boundary norm instead
cmap = ListedColormap(['gray', 'black'])
norm = BoundaryNorm([-1.0, 0.0, 1.0], cmap.N)
lc = LineCollection(segments, cmap=cmap, norm=norm)
lc.set_array(dx)
lc.set_linewidth(2)
line = axes.add_collection(lc)

axes.plot(x_stable, dx_stable, 'bo', x_unstable, dx_unstable, 'ro')

Dx = x_stable[1] - x_unstable[0]

for i in range(len(x_unstable)):
    axes.arrow(x_unstable[i]+0.5, 0.0, Dx -1.0 , 0.0, width=0.05)

for i in range(len(x_unstable)):
    axes.arrow(x_unstable[i]- 0.5, 0.0, - Dx +1.0 , 0.0, width=0.05)

axes.set_title(r'$\dot{x}$ vs. x ')
axes.set_xlabel(r'$x$')
axes.set_ylabel(r'$\dot{x}$')
axes.set_xlim(x.min(), x.max())
axes.set_ylim(-1.1, 1.1)
axes.grid()

plt.show()
```

```python

```


For all numerical experiments, we will be using the Chambolle-Pock primal-dual algorithm - details can be found on:
1. [A First-order Primal-dual Algorithm for Convex Problems with Applications to Imaging](https://link.springer.com/article/10.1007/s10851-010-0251-1), A. Chambolle, T. Pock, Journal of Mathematical Imaging and Vision (2011). [PDF](https://hal.archives-ouvertes.fr/hal-00490826/document)
2. [Recovering Piecewise Smooth Multichannel Images by Minimization of Convex Functionals with Total Generalized Variation Penalty](https://link.springer.com/chapter/10.1007/978-3-642-54774-4_3), K. Bredies, Efficient algorithms for global optimization methods in computer vision (2014). [PDF](https://imsc.uni-graz.at/mobis/publications/SFB-Report-2012-006.pdf)
3. [Second Order Total Generalized Variation (TGV) for MRI](https://onlinelibrary.wiley.com/doi/full/10.1002/mrm.22595), F. Knoll, K. Bredies, T. Pock, R. Stollberger (2010). [PDF](https://onlinelibrary.wiley.com/doi/epdf/10.1002/mrm.22595)

In order to compute the spatia dependent regularization weights we follow:

4. [Dualization and Automatic Distributed Parameter Selection of Total Generalized Variation via Bilevel Optimization](https://arxiv.org/pdf/2002.05614.pdf), M. Hintermüller, K. Papafitsoros, C.N. Rautenberg, H. Sun, arXiv preprint, (2020)

# Huber Total Variation Denoising

We are solving the discretized version of the following minimization problem
\begin{equation}\label{L2-TV}
\min_{u} \int_{\Omega} (u-f)^{2}dx + \alpha \int_{\Omega}  \varphi_{\gamma}(\nabla u)dx
\end{equation}
were $\phi_{\gamma}:\mathbb{R}^{d}\to \mathbb{R}^{+}$ with
\begin{equation}
\phi_{\gamma}(v)=
\begin{cases}
|v|-\frac{1}{2}\gamma & \text{ if } |v|\ge \gamma,\\
\frac{1}{2\gamma}|v(x)|^{2}& \text{ if } |v|< \gamma.\\
\end{cases}
\end{equation}

## Import data...

```python
import scipy.io as sio
import matplotlib.pyplot as plt
import numpy as np

mat_contents = sio.loadmat('tutorial1_classical_reg_methods/parrot')
clean=mat_contents['parrot']
f=mat_contents['parrot_noisy_01']

plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(clean)
imgplot2.set_cmap('gray')

plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(f)
imgplot2.set_cmap('gray')

from tutorial1_classical_reg_methods.Tutorial_Codes import psnr, reproject, dxm, dym, dxp, dyp, function_TGV_denoising_CP, P_a_Huber, function_HuberTV_denoising_CP

```

## Task 1

Choose different values for $\alpha$ and $\gamma$ and interprent your results:
- Fix $\gamma$ small, e.g. $\gamma=0.01$ and play with the values of $\alpha$. What do you observe for large $\alpha$? What for small?
- Fix $\alpha$ and play with the values of $\gamma$. What do you observe for large $\gamma$? What for small?

```python
alpha=0.085
gamma=0.001
uTV =  function_HuberTV_denoising_CP(f,clean, alpha, gamma,1000)
```

```python
uTikhonov =  function_HuberTV_denoising_CP(f,clean, 5, 2,1000)
```

# Total Generalized Variation Denoising

We are solving the discretized version of the following minimization problem
\begin{equation}\label{L2-TGV}
\min_{u} \int_{\Omega} (u-f)^{2}dx + TGV_{\alpha,\beta}(u)
\end{equation}

where
\begin{equation}
TGV_{\alpha,\beta}(u)=\min_{w} \alpha \int_{\Omega}  |\nabla u-w|dx + \beta \int_{\Omega} |Ew|dx
\end{equation}

## Task 2a

Choose different values for $\alpha$ and $\beta$ and solve the TGV denoising minimization problem.

- What happens for small $\alpha$ and large $\beta$?
- What happens for large $\alpha$ and small $\beta$?
- What happens where both parameters are small/large?
- Try to find the combination of parameters that gives the highest PSNR value.

```python
#alpha=0.085
#beta=0.15

alpha=0.085
beta=0.15

uTGV =  function_TGV_denoising_CP(f,clean, alpha, beta, 500)
```

## Task 2b

Import the following spatial dependent regularization weights, which are taken from this work:

- [Dualization and Automatic Distributed Parameter Selection of Total Generalized Variation via Bilevel Optimization](https://arxiv.org/pdf/2002.05614.pdf), M. Hintermüller, K. Papafitsoros, C.N. Rautenberg, H. Sun, arXiv preprint, (2020)

```python
weight_contents = sio.loadmat('tutorial1_classical_reg_methods/spatial_dependent_weights')
alpha_spatial=weight_contents['TGV_alpha_spatial']
beta_spatial=weight_contents['TGV_beta_spatial']

#plt.figure(figsize = (7,7))
#imgplot2 = plt.imshow(alpha_spatial)
#imgplot2.set_cmap('gray')

#plt.figure(figsize = (7,7))
#imgplot2 = plt.imshow(beta_spatial)
#imgplot2.set_cmap('gray')

from mpl_toolkits.mplot3d import Axes3D

(n,m)=alpha_spatial.shape
x=range(n)
y=range(m)
X, Y = np.meshgrid(x, y)
halpha = plt.figure(figsize = (7,7))
h_alpha = halpha.add_subplot(111, projection='3d')
h_alpha.plot_surface(X, Y, alpha_spatial)

hbeta = plt.figure(figsize = (7,7))
h_beta = hbeta.add_subplot(111, projection='3d')
h_beta.plot_surface(X, Y, beta_spatial)

hclean = plt.figure(figsize = (7,7))
h_clean = hclean.add_subplot(111, projection='3d')
h_clean.plot_surface(X, Y, clean)
```

And run again the algorithm with this weight:

```python
uTGVspatial =  function_TGV_denoising_CP(f,clean, alpha_spatial, beta_spatial, 500)
```

Now you can see all the reconstructions together:

```python
plt.rcParams['figure.figsize'] = np.array([4, 3])*3
plt.rcParams['figure.dpi'] = 120
fig, axs = plt.subplots(ncols=3, nrows=2)

# remove ticks from plot
for ax in axs.flat:
    ax.set(xticks=[], yticks=[])

axs[0,0].imshow(clean, cmap='gray')
axs[0,0].set(xlabel='Clean')

axs[0,1].imshow(f, cmap='gray')
axs[0,1].set(xlabel='Noisy, PSNR = ' + str(np.around(psnr(f, clean),decimals=2)))

axs[0,2].imshow(uTikhonov, cmap='gray')
axs[0,2].set(xlabel='Tikhonov, PSNR = ' + str(np.around(psnr(uTikhonov, clean),decimals=2)))

axs[1,0].imshow(uTV, cmap='gray')
axs[1,0].set(xlabel='TV, PSNR = ' + str(np.around(psnr(uTV, clean),decimals=2)))

axs[1,1].imshow(uTGV, cmap='gray')
axs[1,1].set(xlabel = 'TGV, PSNR = ' + str(np.around(psnr(uTGV, clean),decimals=2)))

axs[1,2].imshow(uTGVspatial, cmap='gray')
axs[1,2].set(xlabel = 'TGV spatial, PSNR = ' + str(np.around(psnr(uTGVspatial, clean),decimals=2)))
```

# TV and TGV MRI reconstruction

Here we will be solving the discretized version of the following minimization problem
\begin{equation}
\min_{u} \int_{\Omega} (S \circ F u-g)^{2}dx + \alpha TV(u)
\end{equation}
and
\begin{equation}
\min_{u} \int_{\Omega} (S \circ F u-g)^{2}dx + TGV_{\alpha,\beta}(u)
\end{equation}

The code for the examples below was kindly provided by Clemens Sirotenko.

## Import data

```python
from tutorial1_classical_reg_methods.Tutorial_Codes import normalize, subsampling, subsampling_transposed, compute_differential_operators, function_TV_MRI_CP, function_TGV_MRI_CP
from scipy import sparse
import scipy.sparse.linalg

image=np.load('tutorial1_classical_reg_methods/img_example.npy')
image=np.abs(image[:,:,3])
image = normalize(image)

plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(image)
imgplot2.set_cmap('gray')

```

## Simulate noisy data and subsampled data

Create noisy data $ S \circ F x  + \varepsilon = y^{\delta}$ where $x$ is th clean image and $ \varepsilon \sim \mathcal{N}(0,\sigma^2)$ normal distributed centered complex noise

```python
mask = np.ones(np.shape(image))
mask[:,1:-1:3] = 0

Fx = np.fft.fft2(image,norm='ortho') #ortho means that the fft2 is unitary
(M,N) = image.shape
rate = 0.039 ##noise rate
noise = np.random.randn(M,N) + (1j)*np.random.randn(M,N) #cmplx noise
distorted_full = Fx + rate*noise
distorted = subsampling(distorted_full, mask)
zero_filling = np.real(np.fft.ifft2(subsampling_transposed(distorted, mask), norm = 'ortho'))

plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(mask)
imgplot2.set_cmap('gray')

plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(zero_filling)
imgplot2.set_cmap('gray')
```

## TV MRI reconstruction

```python
x_0 = zero_filling
data = distorted
alpha = 0.025
tau = 1/np.sqrt(12)
sigma = tau
h = 1
max_it = 3000
tol = 1e-4 # algorithm stops if |x_k - x_{k+1}| < tol
x_TV = function_TV_MRI_CP(data,image,mask,x_0,tau,sigma,h,max_it,tol,alpha)
```

```python
plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(x_TV)
imgplot2.set_cmap('gray')
```

## TGV MRI reconstruction

```python
alpha = 0.02
beta = 0.035

x_0 = zero_filling
data = distorted
tau = 1/np.sqrt(12)
sigma = tau
lambda_prox = 1
h = 1
tol = 1e-4
max_it = 2500
x_TGV = function_TGV_MRI_CP(data,image, mask,x_0,tau,sigma,lambda_prox,h,max_it,tol,beta,alpha)
```

```python
plt.figure(figsize = (7,7))
imgplot2 = plt.imshow(x_TGV)
imgplot2.set_cmap('gray')
```

Now you can see all the reconstructions together:

```python
plt.rcParams['figure.figsize'] = np.array([2, 2])*5
plt.rcParams['figure.dpi'] = 120
fig, axs = plt.subplots(ncols=2, nrows=2)

# remove ticks from plot
for ax in axs.flat:
    ax.set(xticks=[], yticks=[])

axs[0,0].imshow(normalize(image), cmap='gray')
axs[0,0].set(xlabel='Clean Image')

axs[1,0].imshow(normalize(x_TV), cmap='gray')
axs[1,0].set(xlabel='TV Reconstruction, PSNR = ' + str(np.around(psnr(x_TV, image),decimals=2)))

axs[0,1].imshow(normalize(x_0), cmap='gray')
axs[0,1].set(xlabel = 'Zero Filling Solution , PSNR = ' + str(np.around(psnr(x_0, image),decimals=2)))

axs[1,1].imshow(normalize(x_TGV), cmap='gray')
axs[1,1].set(xlabel='TGV Reconstruction , PSNR = ' + str(np.around(psnr(x_TGV, image),decimals=2)))

```

```python

```


# Exercises to Lecture 2: Introduction to Python 3.6
By Dr. Anders S. Christensen
`anders.christensen @ unibas.ch`

## Exercise 2.1: define a function

One of the fundamental things we learned was how to define functions.

For example, the function defined below `square(x)` calculates the square, $x^2$:

```
def square(x):
  y = 2 * x**2
  return y

print(square(2))
```

### Question 2.1.1:

Similarly to the above code, make a function called. for example, `poly(x)` to calculate the following polynomial:
\begin{equation}
p\left( x \right) = 5 x^2 - 4x + 1
\end{equation}

Lastly, print the value of $p\left( 5 \right)$

```
def poly(x):

  # Fill out the rest of the function yourself!

# Print the value for x=5
print(poly(5))
```

## Exercise 2.2: Loop within function

The code below implements the product of all numbers up to n, i.e. the factorial function "$!$"

\begin{equation}
f(n) = n! = \prod_{i=1}^n i = 1 \cdot 2 \cdot \quad  ... \quad \cdot (n -1) \cdot n
\end{equation}

As an example, the code to calculate $5!$ is shown here:

```
n = 5

f = 1
for i in range(1,n+1):
  f =  f * i

print("Factorial", n, "is", f)
```

### Question 2.2.1

Unfortunately the above code is not very practical and re-usable, and will only work for $n=5$. Instead we would like a function named `factorial(n)`. In this Exercise, write your own function which calculates the factorial.

As output print `factorial(10)`.

```
def factorial(n):

  # Write the rest of the function yourself

print(factorial(10))
```

### Question 2.2.2:

Using the `factorial(n)` function you wrote in the previous question, print all $n!$ for all n from 1 to 20.

Hint: Use another for loop!

```
# Below, write the code to print all n! from n=1 to n=20

```

## Exercise 2.3: `if` / `else` statements

`if` and `else` statments can are used to make decisions based on defined criteria.

```
n = 10

if n < 10:
  print("n is less than 10")
else:
  print("n is greater than 10")

```

### Question 2.3.1:

One example of mathematical functions that contain such statements are the so-called "rectifier linear unit" (ReLU) functions which are often used in Neural Networks.

An example is given here:
\begin{equation}
f\left(x\right) =
\begin{cases}
      0 & x\leq 0 \\
      x & x \gt 0
   \end{cases}
\end{equation}

In this question, write the above ReLU function, which returns $0$ if $x \leq 0$ and returns $x$ otherwise.

Lastly, verify the correctness of your code your code using the 5 print statments in the code block below.

```
def relu(x):

  # Implement the content of the ReLU function yourself!

print(relu(-10))  # should return 0
print(relu(-0.1)) # should return 0
print(relu(0))    # should return 0
print(relu(0.1))  # should return 0.1
print(relu(10))   # should return 10
```

## Exercise 2.4: Plotting

In the below example, the value of
 $$y= x^2$$
 is calculated for six values of $x$, and appended to a list.

Next, the points are plottet using the `plt.plot()` function from the `pyplot` library.

The function `plt.plot()` tells matplotlib to draw a lineplot that connects all the pairs of points in two lists.

```
import matplotlib.pyplot as plt

# Some x-values
x = [0.0, 1.0, 2.0, 3.0, 4.0, 5.0]

# Initialize an empty list for the y-values we wish to plot
y = []

for i in range(len(x)):

  # Calculate the square of each x in the list
  value = x[i]**2

  # Add the calculated value to the list "y"
  y.append(value)

# Print the two lists
print(x)
print(y)

# Make a line plot/figure
plt.plot(x, y)

# Actually draw the figure below
plt.show()
```

### Question 2.4.1:

Instead of plotting a line through all points, it is also possible to plot datapoints using a so-called [scatterplot](https://en.wikipedia.org/wiki/Scatter_plot).

For this behavior, you can replace the function `plt.plot()` in the above example with the function `plt.scatter()`.

In this question you are given two lists of numbers below, `a` and `b`.

Use pyplot to draw a *scatterplot* of the two lists.

```
a = [10.0, 8.0, 13.0, 9.0, 11.0, 14.0, 6.0, 4.0, 12.0, 7.0, 5.0]
b = [8.04, 6.95, 7.58, 8.81, 8.33, 9.96, 7.24, 4.26, 10.84, 4.82, 5.68]

# Here, write the code to show a scatterplot for the lists a, b
```

### Question 2.4.2:

In order to give the plot a title and label the axes, insert the three functions in the code in Question 2.4.1:

*    `plt.xlabel()`
*    `plt.ylabel()`
*    `plt.title()`

Make the plot title "Python 2020", label the x-axis "Apples" and the y-axis "Oranges"

```

```

## Exercise 2.5: More plotting (harder)

### Question 2.5.1:
In the example in Problem 2.4, the code for a line plot for $y = x^2$ is shown.

Write your own code to plot $y = \cos(x)$ from 0 to 10.

**Hints:**

*   Import the `np.cos()` function using `import numpy as np`
*   If the figure does not look smooth, how can you make the points on the x-axis closer than 1.0?

```
import numpy as np

## Create the numpy arrays or lists of x and y values

# Print the two lists to verify that you have the right numbers
# print(x)
# print(y)

# Make the figure
# plt.plot(x, y)

# Actually draw the figure below
# plt.show()
```


# Time-varying frame
Renato Naville Watanabe

Consider that we have the position vector of a particle, moving in the path described by the parametric curve $s(t)$, described in a fixed reference frame as:

$${\bf\hat{r}}(t) = {x}{\bf\hat{i}}+{y}{\bf\hat{j}} + {z}{\bf\hat{k}}$$

Often we describe all the kinematic variables in this fixed reference frame. However, it is often useful to define a time-varying basis, attached to some point of interest. In this case, what is usually done is to choose as one of the basis vector a unitary vector in the direction of the velocity of the particle. Defining this vector as:

$${\bf\hat{e}_t} = \frac{{\bf\vec{v}}}{\Vert{\bf\vec{v}}\Vert}$$

For the second vector of the basis, we define first a vector of curvature of the path (the meaning of this curvature vector will be seeing in another notebook):

$$ {\bf\vec{C}} = \frac{d{\bf\hat{e}_t}}{ds}$$

Note that $\bf\hat{e}_t$ is a function of the path $s(t)$. So, by the chain rule:

$$ \frac{d{\bf\hat{e}_t}}{dt} = \frac{d{\bf\hat{e}_t}}{ds}\frac{ds}{dt} \longrightarrow \frac{d{\bf\hat{e}_t}}{ds} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\frac{ds}{dt}} \longrightarrow {\bf\vec{C}} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\frac{ds}{dt}}\longrightarrow {\bf\vec{C}} = \frac{\frac{d{\bf\hat{e}_t}}{dt}}{\Vert{\bf\vec{v}}\Vert}$$

Now we can define the second vector of the basis, ${\bf\hat{e}_n}$:

$${\bf\hat{e}_n} = \frac{{\bf\vec{C}}}{\Vert{\bf\vec{C}}\Vert}$$

The third vector of the basis is obtained by the cross product between ${\bf\hat{e}_n}$ and ${\bf\hat{e}_t}$.

$${\bf\hat{e}_b} = {\bf\hat{e}_t} \times {\bf\hat{e}_n} $$

Note that the vectors ${\bf\hat{e}_t}$, ${\bf\hat{e}_n}$ and ${\bf\hat{e}_b}$ vary together with the particle movement.

## Example
For example, consider that a particle follows the path described by the parametric curve below:

$$\vec{r}(t) = (10t+100){\bf{\hat{i}}} + (-\frac{9,81}{2}t^2+50t+100){\bf{\hat{j}}}$$

This curve could be, for example, from a projectile motion. See http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/ProjectileMotion.ipynb for an explanation on projectile motion.

```python
import numpy as np
import sympy as sym
from sympy.vector import CoordSys3D
import matplotlib.pyplot as plt
sym.init_printing()
from sympy.plotting import plot_parametric
from sympy.physics.mechanics import ReferenceFrame, Vector, dot
```

The computation here will be performed symbolically, with the symbolic math package of Python, Sympy. Below,a reference frame, called O,  and a varible for time (t) are defined.

```python
O = sym.vector.CoordSys3D(' ')
t = sym.symbols('t')
```

Below the vector $r(t)$ is defined symbolically.

```python
r = (10*t+100)*O.i + (-9.81/2*t**2+50*t+100)*O.j+0*O.k
r
```

```python
plot_parametric(r.dot(O.i),r.dot(O.j), (t,0,10))
```

```python
v = sym.diff(r)
v
```

```python
et = v/sym.sqrt(v.dot(v))
et
```

```python
C = sym.diff(et)/sym.sqrt(v.dot(v))
C
```

```python
en = C/(sym.sqrt(C.dot(C)))
sym.simplify(en)
```

```python
from matplotlib.patches import FancyArrowPatch
plt.rcParams['figure.figsize'] = 10, 10
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
ax.axis("on")
time = np.linspace(0,10,30)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+10*float(et.dot(O.i).subs(t,instant)), float(r.dot(O.j).subs(t, instant))+10*float(et.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${\hat{e_t}}$')
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+10*float(en.dot(O.i).subs(t, instant)), float(r.dot(O.j).subs(t, instant))+10*float(en.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${\hat{e_n}}$')
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((90,250))
plt.ylim((90,250))
plt.xlabel('x')
plt.legend(handles=[vt,vn],fontsize=20)
plt.show()
```

## Velocity and Acceleration in a time-varying frame

Given the expression of $r(t)$ in a fixed frame (see on the the top of the page) we can write the velocity
${\bf\vec{v}(t)}$ as a function of the fixed frame of reference ${\bf\hat{i}}$, ${\bf\hat{j}}$ and ${\bf\hat{k}}$ (see (http://nbviewer.jupyter.org/github/BMClab/bmc/blob/master/notebooks/KinematicsParticle.ipynb)).

$${\bf\vec{v}}(t) = \dot{x}{\bf\hat{i}}+\dot{y}{\bf\hat{j}}+\dot{z}{\bf\hat{k}}$$

However, this can lead to very complex functions. So it is useful to use the basis find previously ${\bf\hat{e}_t}$, ${\bf\hat{e}_n}$ and ${\bf\hat{e}_b}$.

The velocity ${\bf\vec{v}}$ of the particle is, by the definition of ${\bf\hat{e}_t}$, in the direction of ${\bf\hat{e}_t}$:

$${\bf\vec{v}}={\Vert\bf\vec{v}\Vert}.{\bf\hat{e}_t}=v{\bf\hat{e}_t}$$

The acceleration can be written in the fixed frame of reference as:

$${\bf\vec{a}}(t) = \ddot{x}{\bf\hat{i}}+\ddot{y}{\bf\hat{j}}+\ddot{z}{\bf\hat{k}}$$

But for the same reasons of the velocity vector, it is useful to describe the acceleration vector in the time varying basis. We know that the acceleration is the time derivative of the velocity:

$${\bf\vec{a}} = \frac{{d\bf\vec{v}}}{dt}=\frac{{d({\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}})}{dt}=\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}\dot{{\bf\hat{e}_t}}= \dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}\frac{d{\bf\hat{e}_t}}{ds}\frac{ds}{dt}=\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\frac{d{\bf\hat{e}_t}}{ds}=\dot{\Vert\bf\vec{v}\Vert}{\bf\hat{e}_t}+{\Vert\bf\vec{v}\Vert}^2\Vert{\bf\vec{C}} \Vert{\bf\hat{e}_n}$$

From the example above, we can define the vectors ${\bf\vec{v}}$ and ${\bf\vec{a}}$ in the time varying frame.

```python
v = sym.sqrt(v.dot(v))*et
v
```

```python
a = sym.diff(sym.sqrt(v.dot(v)))*et+v.dot(v)*sym.sqrt(C.dot(C))*en
sym.simplify(sym.simplify(a))
```

```python
from matplotlib.patches import FancyArrowPatch
fig = plt.figure()
ax = fig.add_axes([0, 0, 1, 1])
ax.axis("on")
time = np.linspace(0,10,10)
for instant in time:
    vt = FancyArrowPatch([float(r.dot(O.i).subs(t,instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t,instant))+float(v.dot(O.i).subs(t,instant)), float(r.dot(O.j).subs(t, instant))+float(v.dot(O.j).subs(t,instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="r",label='${{v}}$')
    vn = FancyArrowPatch([float(r.dot(O.i).subs(t, instant)),float(r.dot(O.j).subs(t,instant))],
                         [float(r.dot(O.i).subs(t, instant))+float(a.dot(O.i).subs(t, instant)), float(r.dot(O.j).subs(t, instant))+float(a.dot(O.j).subs(t, instant))],
                         mutation_scale=20,
                         arrowstyle="->",color="g",label='${{a}}$')
    ax.add_artist(vn)
    ax.add_artist(vt)
plt.xlim((60,250))
plt.ylim((60,250))
plt.legend(handles=[vt,vn],fontsize=20)
plt.show()
```

## Solving numerically

Now we will obtain the time-varying basis numerically, instead of symbolically. This method is useful when it is not available a mathematical expression of the path. This often happens when you have available data collected experimentally. First, data will be obtained from the expression of $r(t)$.

```python
t = np.linspace(0,10,30)
r = np.transpose(np.array([10*t +100,-9.81/2*t**2+50*t+100]))

```

```python
Ts = t[1]
v = np.diff(r,1,0)/Ts
vNorm  = np.sqrt(v[:,0]**2+v[:,1]**2)
```

```python
et = np.transpose(np.array([v[:,0]/vNorm, v[:,1]/vNorm]))

```

```python
C = np.diff(et,1,0)/Ts
C = np.transpose(np.array([C[:,0]/vNorm[1:],C[:,1]/vNorm[1:]]))
```

```python
CNorm = np.sqrt(C[:,0]**2+C[:,1]**2)
en = np.transpose(np.array([C[:,0]/CNorm,C[:,1]/CNorm]))

```

```python
from matplotlib.patches import FancyArrowPatch
fig = plt.figure()
plt.plot(r[:,0],r[:,1],'.')
ax = fig.add_axes([0,0,1,1])
time = np.linspace(0,10,10)
for i in np.arange(len(t)-2):
    vec1 = FancyArrowPatch(r[i,:],r[i,:]+20*et[i,:],mutation_scale=20,color='r')
    vec2 = FancyArrowPatch(r[i,:],r[i,:]+20*en[i,:],mutation_scale=20,color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((80,250))
plt.ylim((80,250))
plt.show()
```

```python
v  = np.transpose(np.array([vNorm*et[:,0], vNorm*et[:,1]]))
vNormDot  = np.diff(vNorm,1,0)/Ts
a = np.transpose(np.array([vNormDot*et[1:,0]+vNorm[1:]**2*CNorm*en[:,0],
                           vNormDot*et[1:,1]+vNorm[1:]**2*CNorm*en[:,1]]))
```

```python
from matplotlib.patches import FancyArrowPatch
%matplotlib inline
plt.rcParams['figure.figsize']=10,10
fig = plt.figure()
plt.plot(r[:,0],r[:,1],'.')
ax = fig.add_axes([0,0,1,1])
for i in np.arange(len(t)-2):
    vec1 = FancyArrowPatch(r[i,:],r[i,:]+v[i,:],mutation_scale=20,color='r')
    vec2 = FancyArrowPatch(r[i,:],r[i,:]+a[i,:],mutation_scale=20,color='g')
    ax.add_artist(vec1)
    ax.add_artist(vec2)
plt.xlim((80,250))
plt.ylim((80,250))
plt.show()

```

## Problems

1. Obtain the vectors $\hat{e_n}$ and $\hat{e_t}$ for the problem 17.1.1 from Ruina and Rudra's book.
2. Solve the problem 17.1.9 from Ruina and Rudra's book.
3. Write a Python program to solve the problem 17.1.10 (only the part of $\hat{e_n}$ and $\hat{e_t}$).

## References

+ Ruina A, Rudra P (2015) Introduction to Statics and Dynamics. Oxford University Press. http://ruina.tam.cornell.edu/Book/RuinaPratap-Jan-20-2015.pdf

```python

```


+ This notebook is part of lecture 12 *Graphs, netwroks, and incidence matrices* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, symbols, Matrix
from warnings import filterwarnings
from IPython.display import Image
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Graphs and networks
# Incidence matrices
# Kirchhoff's laws

* This lecture is about the application of matrices

## Graphs and networks

* In this instance we refer to nodes and there connections called edges
* Consider the graph below:

```python
Image(filename = 'Graph1.png')
```

* We will call the nodes *n* (columns), in this case *n* = 4
* The edges (connections) will be called *m* (rows), with *m* = 5 in this case
* This will give us a *m*×*n* = 5×4 matrix
* We will have to give a direction to every edge

## The incidence matrix

* This corresponds to the graph above

```python
A = Matrix([[-1, 1, 0, 0], [0, -1, 1, 0], [-1, 0, 1, 0], [-1, 0, 0, 1], [0, 0, -1, 1]])
A
# For each row (edge) look only at that edge (line)
# In the case of row (edge, line) 1, the arrow point away from node 1, hence the first -1 in the matrix
# The arrow point towards node 2, hence the 1
# It does not point to nodes 3 and 4, hence the 0's
```

* Edges 1, 2, and 3 form a loop
* Notice for the first loop (edges 1, 2, and 3) the corresponding third row is a linear combination of rows 1 and 2
* Intuitively, you can see that you can reach node 3 from node 1 by a combination of edges (rows) 1 and 2

```python
A.rref()
```

* We note that we have three pivot columns, hence a rank, *r* = 3
* We have one column without a pivot and will thus have one in the nullspace (*n* - *r* = 4 - 3 = 1)

```python
A.nullspace()
```

*  The basis for this subspace is one dimensional and includes all scalar multiplications of this vector
* The meaning in our example is that nothing will happen when the solutions fall on this line in 4-dimensional space, i.e. no current will flow

* If you think of the solution **x** and every component of **x** being a potential at a node, the matrix multiplication A**x** gives you the potential differences along the edges
* The nullspace would then be the solution where all the potential differences are 0

```python
x1, x2, x3, x4 = symbols('x1, x2, x3, x4')
```

```python
x_vect = Matrix([x1, x2, x3, x4])
x_vect
```

```python
A * x_vect
```

* For the nullspace, each row now equals 0 (the potential difference between two nodes)

* Let's look at the row space and the nullspace of the row picture
* We now to get the rowspace by transposing the row that contain pivots

```python
A_row = Matrix([[1, 0, 0, -1], [0, 1, 0, -1], [0, 0, 1, -1]]).transpose()
A_row
```

```python
A
```

```python
A.transpose()
```

```python
A.transpose().rref()
```

* Note how the pivot columns are columns 1, 2, and 4
* These represent edges 1, 2, 4
* Note (form the graph above) that thye are independent as they are not a part of a loop
* A graph without a loop (with 1 less edge than nodes) is called a *tree*
* It has a nullspace of

```python
A.transpose().nullspace()
```

* The dimension of the nullspace of AT is *m* - *r* = number of edges minus (number of nodes - 1)
* ∴ number of nodes - number of edges + number of loops = 1
* This is Euler's formula and works for all graphs
* It tells you how many independent loops there are

* There is a connection between potentials and currents
* With 5 edges we will have 5 currents, which we can represent as a vector **y**
$$ \overline { y } =\begin{bmatrix} { y }_{ 1 } & { y }_{ 2 } & { y }_{ 3 } & { y }_{ 4 } & { y }_{ 5 } \end{bmatrix} $$
* This relationship is Ohm's law

## Kirchhoff's law

* By the way, Kirchhoff's current law is: AT**y** = **0**
* We can look at it in the following way

```python
A.transpose()
```

```python
y1, y2, y3, y4, y5 = symbols('y1, y2, y3, y4, y5')
```

```python
y_vect = Matrix([y1, y2, y3, y4, y5])
y_vect
```

```python
A.transpose() * y_vect
```

* For row 1 (setting it equal to 0 and looking at graph above tells us that current flows out from node 1 on all these 3 edges
* For row 2 (doing the same as above) we note that for node 2 current flow towards it on edge *y*1 and away from it along edge *y*2
* For row 3 we note that current flows from node three along edges 2 (edge *y*2) and 3 (edge *y*3) and away from it along edge 5 (edge *y*5)
* For row 4 we note that current flows towards it along edges 4 (edge *y*4) and 5 (edge *y*5)

* Look back at the nullspace of AT
* The two basis vectors show the flow in current that will allow for NO current to accumulate at a node
* In this example, current flowed along the loop of edges 1, 2, and 3 (with nothing along 4 and 5
* The other solution would be current flowing all along the periphery, with nothing along 3
* These are the basis vectors of the nullspace
* Another valid basis would include flow along the upper loop
* Notice that the basis is two dimensional as (between the 3 flows explained above) one is a linear combination of the other two

## Putting it all together

* All of the above can be stated as follows
$$ \overline {e} = {A} \overline {x} $$
$$ \overline {y} = {C} \overline {e} $$
$$ A^{ T }\overline { y } =\overline { f } $$
* Where
    * **e** is the potential differences
    * **f** is an external current in Kirchhoff's law
* This gives us the fundamental equation for applications as stated here
$$ {A}^{T}{C}{A} \overline{x}=\overline{f} $$
* These equations are for equilibrium (no Newton's law, no time)

* Remember that ATA is always symmetric

## Example problem

### Example problem 1

```python
Image(filename = 'Graph2.png')
```

* Calculate the incidence matrix A
* Calculate the nullspaces of A and AT
* Calculate the trace of ATA

#### Solution

```python
A = Matrix([[-1, 1, 0, 0, 0], [0, -1, 1, 0, 0], [-1, 0, 1, 0, 0], [0, -1, 0, 1, 0], [0, 0, 0, -1, 1], [0, 0, 1, 0, -1]])
A
```

```python
A.rref()
```

* We note that we have 4 independent columns
* The dimension of the nullspace will be *n* - *r* = 5 - 4 = 1
* We will let *x*5 = *s*, then from the row-reduced echelon form abobe we have
$$ { x }_{ 1 }-{ x }_{ 5 }=0\\ { x }_{ 2 }-{ x }_{ 5 }=0\\ { x }_{ 3 }-{ x }_{ 5 }=0\\ { x }_{ 4 }-{ x }_{ 5 }=0\\ \begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ { x }_{ 3 } \\ { x }_{ 4 } \\ { x }_{ 5 } \end{bmatrix}=s\begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \\ 1 \end{bmatrix} $$

```python
A.nullspace()
```

* It represents a potential difference between all nodes t be zero: A**x** = **0**
* This means that the potential at all nodes must be a constant

```python
A.transpose().nullspace()
```

* It is of dimension 2, as there are two independent loops
* As per Euler's formula
    * nodes - edges + loops = 1
    * 5 - 6 + 2 = 1
* This tells us about current that needs to flow so as not to accumulate current at a node
* It therefor indicates the independent loops
* It works out beautifully
    * Look at the two loops and assign flow as per the two vector columns for each edge and you will see perfect flow along either of the two independent loops with no current accumulating at any node

* We could calculate it from the row-reduced echelon for of AT

```python
A.transpose().rref()
```

* This gives us 4 independent columns, with dependent *y*3 and *y*6
$$ y_{ 6 }=s\\ { y }_{ 3 }=t\\ { y }_{ 1 }+{ y }_{ 3 }={ y }_{ 1 }+t=0\\ \therefore \quad { y }_{ 1 }=-t\\ { y }_{ 2 }+{ y }_{ 3 }+{ y }_{ 6 }=\quad 0\\ \therefore \quad { y }_{ 2 }=-s-t\\ { y }_{ 4 }-{ y }_{ 6 }={ y }_{ 4 }-s=0\\ \therefore \quad { y }_{ 4 }=s\\ { y }_{ 5 }-{ y }_{ 6 }={ y }_{ 5 }-s=0\\ \therefore \quad { y }_{ 5 }=s\\ \begin{bmatrix} { y }_{ 1 } \\ { y }_{ 2 } \\ { y }_{ 3 } \\ { y }_{ 4 } \\ { y }_{ 5 } \\ { y }_{ 6 } \end{bmatrix}=\begin{bmatrix} -t \\ -s-t \\ t \\ s \\ s \\ s \end{bmatrix}=\begin{bmatrix} 0 \\ -s \\ 0 \\ s \\ s \\ s \end{bmatrix}+\begin{bmatrix} -t \\ -t \\ t \\ 0 \\ 0 \\ 0 \end{bmatrix}=s\begin{bmatrix} 0 \\ -1 \\ 0 \\ 1 \\ 1 \\ 1 \end{bmatrix}+t\begin{bmatrix} -1 \\ -1 \\ 1 \\ 0 \\ 0 \\ 0 \end{bmatrix} $$

```python
A.transpose() * A
```

```python
(A.transpose() * A).trace()
```

* The degree of the node is the number of edges it has
* Look at the columns of the incidence matrix A
* Every non-trivial (non-zero) entry represents an edge
* Note that there are 2 in column 1
    * This gives us a degree of 2, which will also be the first entry on the diagonal of ATA
* Column 2 has 3 entries representing 3 edges from node 2 and an entry of 3 on the diagonal of ATA
* ... and so on
* The trace is therefor just the sum of the degree of all the nodes

```python

```


```python

%matplotlib inline
import matplotlib.pyplot as plt
import sympy as sym
from sympy import oo
sym.init_printing()

def fourier_transform(x):
    return sym.transforms._fourier_transform(x, t, w, 1, -1, 'Fourier')
    #return sym.integrate(x*sym.exp(-1j*w*t), (t,0,oo)).evalf()
```

## Definición

La [transformada de Fourier](https://en.wikipedia.org/wiki/Fourier_transform) está definida por

\begin{equation}
X(j \omega) = \int_{-\infty}^{\infty} x(t) \, e^{-j \omega t} \; dt
\end{equation}

donde $X(j \omega) = \mathcal{F} \{ x(t) \}$ se usa como notación de la Transformada de Fourier de la señal $x(t)$. $X(j \omega)$ es el espectro de la señal $x(t)$. El argumento $j \omega$, como exponente de la exponencial, encierra el comportamiento de todas las señales oscilatorias $cos(\omega t)$.

Observe que la forma de la transformada de Fourier corresponde a la forma de la correlación. De esta manera, podría interpretarse como el "parecido" entre la señal $x(t)$ y  $e^{j \omega t}$, es decir entre $x(t)$ y $cos(\omega t)$

La transformada inversa de Fourier $x(t) = \mathcal{F}^{-1} \{ X(j \omega) \}$ se define como

\begin{equation}
x(t) = \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t} \; d\omega
\end{equation}

## Propiedades

### Invertible

\begin{equation}
x(t) = \mathcal{F}^{-1} \left\{ \mathcal{F} \{ x(t) \} \right\}
\end{equation}

Tomando las expresiones de la Transformada de Fourier y la Transformada Inversa de Fourier, se obtiene:

\begin{equation}
\begin{split}
x(t) &= \frac{1}{2 \pi} \int_{-\infty}^{\infty} \underbrace{\int_{-\infty}^{\infty} x(\tau) e^{-j \omega \tau} d\tau}_{X(j \omega)} \; e^{j \omega t} d\omega \\
&= \int_{-\infty}^{\infty} x(\tau) \left( \frac{1}{2 \pi} \int_{-\infty}^{\infty} e^{-j \omega \tau} e^{j \omega t}  d\omega \right) d\tau \\
&= \int_{-\infty}^{\infty} x(\tau) \delta(t - \tau) d\tau = x(t)
\end{split}
\end{equation}

### Linealidad

\begin{equation}
\mathcal{F} \{ A \cdot x_1(t) + B \cdot x_2(t) \} = A \cdot X_1(j \omega) + B \cdot X_2(j \omega)
\end{equation}

Tomando la expresión de la Transformada de Fourier se obtiene:

\begin{equation}
\begin{split}
&= \int_{-\infty}^{\infty} (A \cdot x_1(t) + B \cdot x_2(t)) \, e^{-j \omega t} \; dt \\
&= \int_{-\infty}^{\infty} A \cdot x_1(t) \, e^{-j \omega t} \; dt + \int_{-\infty}^{\infty} B \cdot x_2(t) \, e^{-j \omega t} \; dt \\
&= A \cdot \int_{-\infty}^{\infty} x_1(t) \, e^{-j \omega t} \; dt + B \cdot\int_{-\infty}^{\infty} x_2(t) \, e^{-j \omega t} \; dt
\end{split}
\end{equation}

**Ejemplo - Transformada de Fourier de una señal exponencial causal**

\begin{equation}
x(t) = e^{- \alpha t} \cdot \epsilon(t)
\end{equation}

con $\alpha \in \mathbb{R}^+$

```python
t,w = sym.symbols('t omega', real=True)
a = 4
x = sym.exp(-a * t)*sym.Heaviside(t)
x
```

```python
X = fourier_transform(x)
X
```

```python
plt.rcParams['figure.figsize'] = 5, 2
sym.plot(x, (t,-1,10), ylabel=r'Amp',line_color='blue',legend=True, label = 'x(t)')
sym.plot(sym.re(X), (w,-20,20), ylabel=r'Real',line_color='blue',legend=True, label = 'X(w)')
sym.plot(sym.im(X), (w,-20,20), ylabel=r'Imag',line_color='blue',legend=True, label = 'X(w)')

sym.plot(sym.sqrt(
         (sym.im(X)*sym.im(X)) + (sym.re(X)*sym.re(X))),
         (w,-20,20), ylabel=r'Mag',line_color='blue',legend=True, label = 'X(w)')

```

    <sympy.plotting.plot.Plot at 0x20726b965c8>

Observe que:

- $X(\omega)$ es una función definida para todos los valores de $\omega$ y no solamente para los múltiplos enteros de un valor determinado $\omega_0$.

- $X(\omega)$ es una función compleja, es decir que tiene parte imaginaria y parte real. Así, puede expresarse de forma cartesiana ($real + j \cdot imaginario$) o de forma polar ($magnitud \angle ángulo$). El "parecido" entre la señal $x(t)$ con $sin(\omega t)$ se puede apreciar en la magnitud de $X(\omega)$.

- $|X(\omega)|$ tiene un valor máximo en $x=0$ y un decaimiento a medida que aumenta $\omega$

Analizando la magnitud de $X(\omega)$

```python
X_real  = sym.re(X)
X_real
```

```python
X_imag  = sym.im(X)
X_imag
```

```python
X_magn = sym.sqrt(X_real*X_real + X_imag*X_imag).simplify()
X_magn
```

La magnitud de $X(\omega)$ es simétrica respecto a $\omega = 0$. Así, será suficiente analizar solamente un lado del espectro de una señal de tiempo continuo.

**Ejemplo - Transformada de Fourier de una señal exponencial por una senoidal**

\begin{equation}
x(t) = sin(\omega_0 t) \cdot e^{- \alpha t} \cdot \epsilon(t)
\end{equation}

con $\omega \in \mathbb{R}^+$

```python
t,w = sym.symbols('t omega', real=True)
w0 = 10
x1 = sym.sin(w0 * t)*sym.exp(-a*t)*sym.Heaviside(t)
x1
```

```python
X1 = fourier_transform(x1)
X1
```

```python
plt.rcParams['figure.figsize'] = 5, 2
sym.plot(x1, (t,-1,10), ylabel=r'Amp',line_color='blue',legend=True, label = 'x1(t)')
#sym.plot(sym.re(X), (w,-20,20), ylabel=r'Real',line_color='blue',legend=True, label = 'X(w)')
#sym.plot(sym.im(X), (w,-20,20), ylabel=r'Imag',line_color='blue',legend=True, label = 'X(w)')

sym.plot(sym.sqrt(
         (sym.im(X1)*sym.im(X1)) + (sym.re(X1)*sym.re(X1))),
         (w,-20,20), ylabel=r'Mag',line_color='blue',legend=True, label = 'X1(w)')

```

    <sympy.plotting.plot.Plot at 0x20726dd3488>

Observe que:
- $x1(t)$ corresponde a $x(t)$ multiplicada con una función senoidal de frecuencia angular de $10$ rad/seg.

- Al igual que con la magnitud de $X(\omega)$, la magnitud de $X1(\omega)$ decae con $\omega$, sin embargo, hay un pico en $\omega=10$ que se relaciona con la senoidal.

**Ejemplo - Transformada de Fourier de senoidales y combinaciones causales**

\begin{equation}
x(t) = sin(\omega_0 t) \cdot \epsilon(t)
\end{equation}

con $\omega \in \mathbb{R}^+$

```python
w1 = 10
w2 = 5
x2_1 = sym.sin(w1 * t)*sym.Heaviside(t)

x2_2 = sym.sin(w2 * t)*sym.Heaviside(t)

x2 = x2_1 + x2_2
x2
```

```python
X2_1 = fourier_transform(x2_1)
X2_1

```

```python
X2_2 = fourier_transform(x2_2)
X2_2
```

```python
(X2_1+X2_2).simplify()
```

```python
X2 = fourier_transform(x2)
X2
```

```python
plt.rcParams['figure.figsize'] = 5, 2
gt2_1 = sym.plot(x2_1, (t,-1,5), ylabel=r'Amp',line_color='blue',legend=True, label = 'x2_1(t)', show = False)
gt2_2 = sym.plot(x2_2, (t,-1,5), ylabel=r'Amp',line_color='green',legend=True, label = 'x2_2(t)', show = False)
gt2 = sym.plot(x2, (t,-1,5), ylabel=r'Amp',line_color='red',legend=True, label = 'x2(t)', show = False)
gt2.extend(gt2_1)
gt2.extend(gt2_2)
gt2.show()

```

```python

plt.rcParams['figure.figsize'] = 7, 5

gw2_1 = sym.plot(sym.sqrt(
         (sym.im(X2_1)*sym.im(X2_1)) + (sym.re(X2_1)*sym.re(X2_1))),
         (w,0,15), ylabel=r'Mag',line_color='blue',legend=True, label = 'X2_1(w)',show = False)
gw2_2 = sym.plot(sym.sqrt(
         (sym.im(X2_2)*sym.im(X2_2)) + (sym.re(X2_2)*sym.re(X2_2))),
         (w,0,15), ylabel=r'Mag',line_color='green',legend=True, label = 'X2_2(w)',show = False)
gw2 = sym.plot(sym.sqrt(
         (sym.im(X2)*sym.im(X2)) + (sym.re(X2)*sym.re(X2))),
         (w,0,15), ylabel=r'Mag',line_color='red',legend=True, label = 'X2(w)',show = False)

gw2.extend(gw2_1)
gw2.extend(gw2_2)
gw2.show()
```

En la gráfica anterior se observa el efecto de la superposición lineal de los espectros de las dos señales senoidales.

**Ejercicio**

Analice el espectro de

\begin{equation}
x(t) = (sin(\omega_0 t) + e^{2t}) \cdot \epsilon(t)
\end{equation}

con $\omega_0 \in \mathbb{R}^+$

```python

```

### Dualidad

Observe que la **Transformada de Fourier** y la **Transformada Inversa de Fourier** tienen formas parecidas.

\begin{align}
X(\omega) &= \int_{-\infty}^{\infty} x(t) \, e^{-j \omega t} \; dt \\
x(t) &= \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t} \; d\omega
\end{align}

La principal diferencia está en el factor de normalización $2 \pi$ y el signo de la exponencial.

Suponga que:

\begin{equation}
x_2(\omega) = \mathcal{F} \{ x_1(t) \}
\end{equation}

Puede pensarse que:

\begin{equation}
x_2(t) = x_2(\omega) \big\vert_{\omega=t}
\end{equation}

Entonces

\begin{equation}
\mathcal{F} \{ x_2(t) \} = \int_{-\infty}^{\infty} x_2(\omega) \big\vert_{\omega=t} \, e^{-j \omega t} \; dt
\end{equation}

Esta tiene la forma de **Transformada de Fourier**, pero la función interna tiene a $\omega$ como variable, esto indica que la integral se trata como una **Transformada inversa**.  Así, para volver a la **Transformada de Fourier**, se debe multiplicar por $2\pi$ y quitar el signo de la exponencial del kernel de transformación.

\begin{equation}
\mathcal{F} \{ x_2(t) \} = 2 \pi \cdot x_1(- \omega)
\end{equation}

Esta propiedad permite llevar los análisis desde el dominio de las frecuencias al dominio del tiempo y viceversa.

## Teoremas

### Derivadas

Dadas una señal $x(t)$ y su derivada respecto al tiempo $\frac{d x(t)}{dt}$, y conocida la **Transformada de Fourier** $X(\omega)$ :

\begin{equation}
\mathcal{F} \left\{ \frac{d x(t)}{dt} \right\}  = \int_{-\infty}^{\infty} \frac{d x(t)}{dt} \, e^{-j \omega t} \; dt
\end{equation}

La integral se puede resolver por partes:

\begin{equation}
\begin{split}
\mathcal{F} \left\{ \frac{d x(t)}{dt} \right\} &= x(t) \cdot e^{-j \omega t} \big\vert_{-\infty}^{\infty} - \int_{-\infty}^{\infty} x(t) (-j \omega) e^{-j \omega t} \; dt \\
&= j \omega \int_{-\infty}^{\infty} x(t) e^{-j \omega t} \; dt \\
&= j \omega X(\omega) \
\end{split}
\end{equation}

\begin{equation}
\frac{d x(t)}{dt} = \frac{d \delta(t)}{dt} * x(t)
\end{equation}

La principal aplicación está en la transformación de ecuaciones diferenciales.

** Ejemplo**

\begin{equation}
2y(t) + 2 \frac{dy}{dt} - x(t) = 0
\end{equation}

Aplicando la **Transformada de Fourier** con sus propiedades se obtiene:
\begin{equation}
2Y(\omega) + 2 j \omega Y(\omega) - X(\omega) = 0
\end{equation}

Observe que en el modelo en el dominio del tiempo (ecuación diferencial) no es posible despejar una expresión equivalente a $\frac{x(t)}{y(t)}$. Por su parte, usando el modelo en el dominio de la frecuencia, se obtiene:

\begin{equation}
Y(\omega)(2 + 2 j \omega ) = X(\omega)
\end{equation}

\begin{equation}
\frac{Y(\omega)}{X(\omega)} = \frac{1}{2+2j\omega} = F(\omega)
\end{equation}

Esta relación es conocida como **Función de transferencia** y representa el efecto que tiene el sistema sobre una señal de entrada que en el caso de la transformada de Fourier es senoidal.

```python
# La función de transferencia
F = 1 / (2 +1j*2*w  )

F
```

```python
plt.rcParams['figure.figsize'] = 5, 2
sym.plot(sym.Abs(F), (w,-10,10), ylabel=r'Mag',line_color='blue',legend=True, label = 'F(w)', show = True)

```

    <sympy.plotting.plot.Plot at 0x20726d6ff88>

La respuesta de un sistema ante una entrada senoidal de frecuencia específica $\omega$ se determina por el valor complejo que toma $F(\omega)$. Por ejemplo, en la frecuencia $\omega = 1$, $F(1) = \frac{1}{2+2j}$

```python
F1 = F.subs(w,1)
magF1 = sym.Abs(F1)
magF1
```

Así, si el sistema es excitado con un seno de frecuencia 1, con amplitud 1, la salida será un seno de amplitud 0.35 y tendrá un desfase definido por el ángulo de la función de transferencia.

\begin{equation}
0.35 sin(1t + ang)
\end{equation}

### Ejercicio

Realice una búsqueda o deduzca cómo se calcula el ángulo de fase que introduce el sistema.


# Clustering of univariate data via Dirichlet Process Mixture

this is a continuation of 'estimate_univ_density'. Make sure to check it before going through this tutorial!

```python
import numpy as np
import matplotlib.pyplot as plt

from pybmix.core.mixing import DirichletProcessMixing, StickBreakMixing
from pybmix.core.hierarchy import UnivariateNormal
from pybmix.core.mixture_model import MixtureModel

np.random.seed(2021)
```

## DP and clustering

Recall that $\tilde p \sim DP(\alpha, G_0)$ means that $\tilde p = \sum_{h=1}^\infty w_h \delta_{\tau_h}$ with $\{w_h\}_h \sim GEM(\alpha)$ and $\{\tau_h\}_h \sim G_0$. Hence, realizations from a DP are almost surely discrete probability measures.

Hence, sampling

\begin{equation}
\begin{aligned}
\theta_1, \ldots, \theta_n | \tilde{p} & \sim \tilde{p} \\
\tilde{p} &\sim DP(\alpha, G_0)
\end{aligned}
\end{equation}

entails that with positive probability $\theta_i = \theta_j$ (with $i \neq j$). In a sample of size $n$ there will be $k \geq n$ unique values $\theta^*_1, \ldots, \theta^*_k$ among the $\theta_i$'s and clusters are defined as $C_j = \{i : \theta_i = \theta^*_j \}$.

When considering a mixture model, the $\theta_i$'s are not observations but latent variables. In the case of a univariate normal mizture models, $\theta_i = (\mu_i, \sigma^2_i)$ and the model can be written as

\begin{equation}
\begin{aligned}
    y_i | \theta_i = (\mu_i, \sigma^2_i) &\sim \mathcal N(\mu_i, \sigma^2_i) \\
    \theta_1, \ldots, \theta_n | \tilde{p} &\sim \tilde{p} \\
    \tilde{p} &\sim DP(\alpha, G_0)
\end{aligned}
\end{equation}

and the clustering among the observations $y_i$'s is inherited by the clustering among the $\theta_i$'s.

Let's go back to the previous example

```python
def sample_from_mixture(weigths, means, sds, n_data):
    n_comp = len(weigths)
    clus_alloc = np.random.choice(np.arange(n_comp), p=[0.5, 0.5], size=n_data)
    return np.random.normal(loc=means[clus_alloc], scale=sds[clus_alloc])

y = sample_from_mixture(
    np.array([0.5, 0.5]), np.array([-3, 3]), np.array([1, 1]), 200)

mixing = DirichletProcessMixing(total_mass=5)
hierarchy = UnivariateNormal()
hierarchy.make_default_fixed_params(y, 2)
mixture = MixtureModel(mixing, hierarchy)

mixture.run_mcmc(y, algorithm="Neal2", niter=2000, nburn=1000)
```

We can extract the cluster allocation MCMC chain very easily

```python
mcmc_chain = mixture.get_chain()
cluster_alloc_chain = mcmc_chain.extract("cluster_allocs")
print(cluster_alloc_chain.shape)
```

cluster_alloc_chain is a matrix of shape [niter - nburn, ndata].

To get the posterior distribution of the number of clusters, we count in each row the number of unique values

```python
n_clust_chain = np.apply_along_axis(lambda x: len(np.unique(x)), 1,
                                    cluster_alloc_chain)

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))
axes[0].vlines(np.arange(len(n_clust_chain)), n_clust_chain - 0.3, n_clust_chain + 0.3)
axes[0].set_title("Traceplot")

clusgrid = np.arange(1, 10)
probas = np.zeros_like(clusgrid)
for i, c in enumerate(clusgrid):
    probas[i] = np.sum(n_clust_chain == c)

probas = probas / np.sum(probas)
axes[1].bar(clusgrid, probas)
axes[1].set_xticks(clusgrid)
axes[1].set_title("Posterior number of clusters")

plt.show()
```

Let's inspect two iterations: the first one and the last one, and look at the cluster allocations of the first 5 observations

```python
print("First iteration: ", cluster_alloc_chain[0][:5])
print("Last iteration: ", cluster_alloc_chain[-1][:5])
```

Observe that the clustering are identicals: the one is made of observations $\{1, 2, 5\}$ and the other cluster of observations $\{3, 4\}$. However the labels associated to each cluster are differend depending on the iterations: in the first iteration, $\{1, 2, 5\}$ are the first cluster (0th cluster) and  $\{3, 4\}$ are the second cluster, while in the last iteration the opposite happens.

This is due to the so-called "label-switching". Usually to interpret the clustering result, a suitable point-estimate is chosen to minimize a loss function.

```python
from pybmix.estimators.cluster_estimator import ClusterEstimator

clus_est = ClusterEstimator(mixture)
best_clust = clus_est.get_point_estimate()

plt.hist(y, density=True, alpha=0.3)
for cluster_idx in clus_est.group_by_cluster(best_clust):
    data = y[cluster_idx]
    plt.scatter(data, np.zeros_like(data) + 5e-3)

plt.show()
```

Note how the posterior mode of the number of clusters is 3, but the point estimate for the best clustering consists of 2 clusters