# The Efficient Frontier of Optimal Portfolio Transactions

### Introduction

[Almgren and Chriss](https://cims.nyu.edu/~almgren/papers/optliq.pdf) showed that for each value of risk aversion there is a unique optimal execution strategy. The optimal strategy is obtained by minimizing the **Utility Function** $U(x)$:

\begin{equation}
U(x) = E(x) + \lambda V(x)
\end{equation}

where $E(x)$ is the **Expected Shortfall**, $V(x)$ is the **Variance of the Shortfall**, and $\lambda$ corresponds to the trader’s risk aversion. The expected shortfall and variance of the optimal trading strategy are given by:

In this notebook, we will learn how to visualize and interpret these equations.

# The Expected Shortfall

As we saw in the previous notebook, even if we use the same trading list, we are not guaranteed to always get the same implementation shortfall due to the random fluctuations in the stock price. This is why we had to reframe the problem of finding the optimal strategy in terms of the average implementation shortfall and the variance of the implementation shortfall. We call the average implementation shortfall, the expected shortfall $E(x)$, and the variance of the implementation shortfall $V(x)$. So, whenever we talk about the expected shortfall we are really talking about the average implementation shortfall. Therefore, we can think of the expected shortfall as follows. Given a single trading list, the expected shortfall will be the value of the average implementation shortfall if we were to implement this trade list in the stock market many times.

To see this, in the code below we implement the same trade list on 50,000 trading simulations. We call each trading simulation an episode. Each episode will consist of different random fluctuations in stock price. For each episode we will compute the corresponding implemented shortfall. After all the 50,000 trading simulations have been carried out we calculate the average implementation shortfall and the variance of the implemented shortfalls. We can then compare these values with the values given by the equations for $E(x)$ and $V(x)$ from the Almgren and Chriss model.

```python
%matplotlib inline

import matplotlib.pyplot as plt

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Set the liquidation time
l_time = 60

# Set the number of trades
n_trades = 60

# Set trader's risk aversion
t_risk = 1e-6

# Set the number of episodes to run the simulation
episodes = 10

utils.get_av_std(lq_time = l_time, nm_trades = n_trades, tr_risk = t_risk, trs = episodes)

# Get the AC Optimal strategy for the given parameters
ac_strategy = utils.get_optimal_vals(lq_time = l_time, nm_trades = n_trades, tr_risk = t_risk)
ac_strategy
```

# Extreme Trading Strategies

Because some investors may be willing to take more risk than others, when looking for the optimal strategy we have to consider a wide range of risk values, ranging from those traders that want to take zero risk to those who want to take as much risk as possible. Let's take a look at these two extreme cases. We will define the **Minimum Variance** strategy as that one followed by a trader that wants to take zero risk and the **Minimum Impact** strategy at that one followed by a trader that wants to take as much risk as possible. Let's take a look at the values of $E(x)$ and $V(x)$ for these extreme trading strategies. The `utils.get_min_param()` uses the above equations for $E(x)$ and $V(x)$, along with the parameters from the trading environment to calculate the expected shortfall and standard deviation (the square root of the variance) for these strategies. We'll start by looking at the Minimum Impact strategy.

```python
import utils

# Get the minimum impact and minimum variance strategies
minimum_impact, minimum_variance = utils.get_min_param()
```

### Minimum Impact Strategy

This trading strategy will be taken by trader that has no regard for risk. In the Almgren and Chriss model this will correspond to having the trader's risk aversion set to $\lambda = 0$. In this case the trader will sell the shares at a constant rate over a long period of time. By doing so, he will minimize market impact, but will be at risk of losing a lot of money due to the large variance. Hence, this strategy will yield the lowest possible expected shortfall and the highest possible variance, for a given set of parameters. We can see that for the given parameters, this strategy yields an expected shortfall of \$197,000 dollars but has a very big standard deviation of over 3 million dollars.

```python
minimum_impact
```

AC Optimal Strategy for Minimum Impact

  Number of Days to Sell All the Shares:	     250	       Initial Portfolio Value:	         $50,000,000.00

  Half-Life of The Trade:	                1,284,394.9	   Expected Shortfall:	                $197,000.00

  Utility:	                               $197,000.00	   Standard Deviation of Shortfall:	  $3,453,707.55

### Minimum Variance Strategy

This trading strategy will be taken by trader that wants to take zero risk, regardless of transaction costs. In the Almgren and Chriss model this will correspond to having a variance of $V(x) = 0$. In this case, the trader would prefer to sell the all his shares immediately, causing a known price impact, rather than risk trading in small increments at successively adverse prices. This strategy will yield the smallest possible variance, $V(x) = 0$, and the highest possible expected shortfall, for a given set of parameters. We can see that for the given parameters, this strategy yields an expected shortfall of over 2.5 million dollars but has a standard deviation equal of zero.

```python
minimum_variance
```

AC Optimal Strategy for Minimum Variance

  Number of Days to Sell All the Shares:	       1	         Initial Portfolio Value:	         $50,000,000.00

  Half-Life of The Trade:	                     0.2	        Expected Shortfall:	               $2,562,500.00

  Utility:	                               $2,562,500.00	   Standard Deviation of Shortfall:	      $0.00

# The Efficient Frontier

The goal of Almgren and Chriss was to find the optimal strategies that lie between these two extremes. In their paper, they showed how to compute the trade list that minimizes the expected shortfall for a wide range of risk values. In their model, Almgren and Chriss used the parameter $\lambda$ to measure a trader's risk-aversion. The value of $\lambda$ tells us how much a trader is willing to penalize the variance of the shortfall, $V(X)$, relative to expected shortfall, $E(X)$. They showed that for each value of $\lambda$ there is a uniquely determined optimal execution strategy. We define the **Efficient Frontier** to be the set of all these optimal trading strategies. That is, the efficient frontier is the set that contains the optimal trading strategy for each value of $\lambda$.

The efficient frontier is often visualized by plotting $(x,y)$ pairs for a wide range of $\lambda$ values, where the $x$-coordinate is given by the equation of the expected shortfall, $E(X)$, and the $y$-coordinate is given by the equation of the variance of the shortfall, $V(X)$. Therefore, for a given a set of parameters, the curve defined by the efficient frontier represents the set of optimal trading strategies that give the lowest expected shortfall for a defined level of risk.

In the code below, we plot the efficient frontier for $\lambda$ values in the range $(10^{-7}, 10^{-4})$, using the default parameters in our trading environment. Each point of the frontier represents a distinct strategy for optimally liquidating the same number of stocks. A risk-averse trader, who wishes to sell quickly to reduce exposure to stock price volatility, despite the trading costs incurred in doing so, will likely choose a value of $\lambda = 10^{-4}$. On the other hand, a trader
who likes risk, who wishes to postpones selling, will likely choose a value of $\lambda = 10^{-7}$. In the code, you can choose a particular value of $\lambda$ to see the expected shortfall and level of variance corresponding to that particular value of trader's risk aversion.

```python
%matplotlib inline

import matplotlib.pyplot as plt

# Add-on : Hide Matplotlib deprecate warnings
import warnings
warnings.filterwarnings("ignore")
# High resolution plot outputs for retina display
%config InlineBackend.figure_format = 'retina'

import utils

# Set the default figure size
plt.rcParams['figure.figsize'] = [17.0, 7.0]

# Plot the efficient frontier for the default values. The plot points out the expected shortfall and variance of the
# optimal strategy for the given the trader's risk aversion. Valid range for the trader's risk aversion (1e-7, 1e-4).
utils.plot_efficient_frontier(tr_risk = 1e-6)
```

```python

```


# QISKit hands on lab addition : Grover with circuits and Aqua

### Jean-Michel Torres, IBM Q Hub France, torresjm@fr.ibm.com

# https://github.com/jmit34/

### Agenda :

What is Grover search algorithm

Tiny sample coding Grover operator

Using QISKit AQUA for 3SAT with Grover

#  A. Grover search algorithm

Grover search alogrithm finds an element from a non ordered set, faster than any classical algorithm.

In other words Grover search algorithm will find an element matching a criteria, out of an unordered list.

 Trusted Notebook" width="600 px" align="center">

Let's apply to the very tiny example of finding one element out of four, and to relate to an example : let's find where is the Queen of Hearts out of four Queen.
This is equivalent to find what index in the list is set to 1 (when others are set to 0).

Classical process requires an average of 2.25 tries to find the index value of one element out of four (i.e: find the Queen of hearts hidden amongst four Queen).

| draw     | Probability p of finding QH | Quantity q of draw(s) | p x q  |
| ----------- | ------------- |-----------------------------------------------| --- |
|First | 0.25| 1 | 0.25 |
|Second | 0.25  | 2 | 0.5 |
|Third (1)| 0.5  | 3 | 1.5 |
|TOTAL   | - |- |2.25|

(1) At this point, wether we draw the Queen of Hearts or not, we know where it is, so this is the final draw, no need for the 4rth draw.

On the other hand, in this case, Grover search algorithm can find the Queen of hearts in a single attempt.

More generally Grover search algorithm will require a number of attempts proportionnal to square root of N, where N is the set size:

\begin{equation}
\frac{\pi}{4}{\sqrt{N}}
\end{equation}

```python
# Classical way: hiding one card at a random position, and then count the attempts before the card is found.
from random import randint

def hide(n):
    ''' this creates a list of n '0', with one '1' at a random position'''
    cards = [0 for i in range(n)]  # only 0'
    position = randint(0, n-1)     # draw a random position
    cards[position] = 1            # this position now has a 1 (others keep 0)
    return cards

def find(cards):
    n = 0 # initialize attempts needed to find the hidden position
    while len(cards) > 1: # when there is one card left, we know it is the hidden position
        trial = randint(0,len(cards)-1) # try one position at random
        if cards[trial]:
            n+=1
            break
        else:
            n+=1
            # remove that card from the list
            cards = cards[0:trial] + cards[trial+1:len(cards)]
    return n

# params : lentgh = number of cards face down in the game,
#          could be 4, 32, or any
# params : draws = quantity of times the experience will
#          be made to be able to calculate the average draws
#          needed to find the hidden position
length = 4
draws = 1000

# initalize count of shots needed to find hidden position
shots = 0
for d in range(draws):
    cards = hide(length)
    shots += find(cards)

print(f"With {length} cards and {draws} experiments:")
print(f"    the hidden card was found in a average of {shots/draws} draws")
```

So now, let's go with the quantum computing demo with N=4 (4 cards, 4 positions)

#  B. Grover search algorithm with QISKit circuits

## 1. SETUP

### General setup

The cell below imports the needed function from qiskit and other libraries:

```python
from IPython.display import Image, display
from random import randint
from qiskit import QuantumRegister, ClassicalRegister, QuantumCircuit, execute

print("The required objects have been imported from QISKit and other Python libraries")
```

### Defining the sub functions for setting the index as data entry

We are defining 4 custom gates (need QISKit version 0.9 and above)

```python
sub_qr = QuantumRegister(2)

# let's define a list of custom gates, there will be four of them, each one to "hide"
# the card in of of the four possible positions
hide = []
# this dummy circuit is just to fill position 0 in the list, so we do not get confused
# later witn an ofsset between position and index.
dummy = QuantumCircuit(1)
hide.append(dummy)

# first position
sub_circ = QuantumCircuit(sub_qr, name='Hide')
for i in range(2):
    sub_circ.h(sub_qr[i])
    sub_circ.s(sub_qr[i])
sub_circ.h(sub_qr[1])
sub_circ.cx(sub_qr[0], sub_qr[1])
sub_circ.h(sub_qr[1])
for i in range(2):
    sub_circ.s(sub_qr[i])
# Convert to a gate
hide.append(sub_circ.to_instruction())

# second position
sub_circ = QuantumCircuit(sub_qr, name='Hide')
for i in range(2):
    sub_circ.h(sub_qr[i])
sub_circ.s(sub_qr[1])
sub_circ.h(sub_qr[1])
sub_circ.cx(sub_qr[0], sub_qr[1])
sub_circ.h(sub_qr[1])
sub_circ.s(sub_qr[1])
# Convert to a gate
hide.append(sub_circ.to_instruction())

# third position
sub_circ = QuantumCircuit(sub_qr, name='Hide')
sub_circ.h(sub_qr[0])
sub_circ.h(sub_qr[1])
sub_circ.s(sub_qr[0])
sub_circ.h(sub_qr[1])
sub_circ.cx(sub_qr[0], sub_qr[1])
sub_circ.h(sub_qr[1])
sub_circ.s(sub_qr[0])
# Convert to a gate
hide.append(sub_circ.to_instruction())

# fourth position
sub_circ = QuantumCircuit(sub_qr, name='Hide')
sub_circ.h(sub_qr[0])
sub_circ.h(sub_qr[1])
sub_circ.h(sub_qr[1])
sub_circ.cx(sub_qr[0], sub_qr[1])
sub_circ.h(sub_qr[1])
# Convert to a gate
hide.append(sub_circ.to_instruction())

display(Image(filename='./images/cards/4FacesDown.jpeg', width=800))

print("We have created our own quantum gates,")
print("to prepare the state of our QH hidden in any of the four positions")

# uncomment next lines to visualize
#q = QuantumRegister(2, 'q')
#circ = QuantumCircuit(q)
#for i in range(4):
#circ.append(hide[i], [q[0], q[1]])
#circ.draw(output='mpl')

```

### Building the Grover operator, this sub function will be used to reveal the hidden index in one shot

```python
# Grover operator
sub_circ = QuantumCircuit(sub_qr, name='Grover-Op')
for i in range(2):
    sub_circ.h(sub_qr[i])
    sub_circ.x(sub_qr[i])
sub_circ.h(sub_qr[1])
sub_circ.cx(sub_qr[0], sub_qr[1])
sub_circ.h(sub_qr[1])
for i in range(2):
    sub_circ.x(sub_qr[i])
    sub_circ.h(sub_qr[i])

# Convert to a gate
grover = sub_circ.to_instruction()

print("We have created our Grover operator on 2 qubits")
print(" ")
print("We are ready to go !")

# uncomment next lines to visualize
q = QuantumRegister(2, 'q')
circ = QuantumCircuit(q)
circ.append(grover, [q[0], q[1]])
circ.draw(output='mpl')
sub_circ.draw(output="mpl")
```

## 2. DRAW A RANDOM POSITION

### ... and keep it secret.

```python
secret = randint(1,4)

print("Now, the secret position has been chosen, no one knows what it is,")
print(" ")
print("Let's first use Grover to find it !")
```

## 3. Now build the circuit:

### adding the chosen sub circuit (at random) and the grover operator, then measurement step.

```python
# define needed registers, and our quantum circuit.
qr = QuantumRegister(2)
cr = ClassicalRegister(2)

circ = QuantumCircuit(qr,cr)

# use the selected hide gate (determined by random value "secret")
circ.append(hide[secret], [qr[0],qr[1]])
# add grover operator
circ.append(grover, [qr[0],qr[1]])
# add measuring stage
circ.measure(qr,cr)
# that's it !
circ.draw(output='mpl')
```

```python

```

## 4. Execute the circuit (on the local simulator) :

- choose backend

- define job

- get result

```python
from qiskit import Aer
#print(Aer.backends())
backend = Aer.get_backend('qasm_simulator')
# define job, get results
job = execute(circ,backend,shots=100)
my_result = job.result()
d = (my_result.get_counts(circ))

max = 0
for x in d:
    if d[x] > max:
        val = x

if val == '00':
    position = "first"
if val == '10':
    position = "second"
if val == '01':
    position = "third"
if val == '11':
    position = "fourth"

print(" ")
print("The position found by Grover search algorithm is:", position)

```

```python
from qiskit.tools.visualization import plot_histogram
# define job, get results
job = execute(circ,backend,shots=100)
my_result = job.result()
d = (my_result.get_counts(circ))

plot_histogram(my_result.get_counts(circ))
```

## 5. Reveal what was the hidden position (the index of the data element searched)

```python
print(f"The hiding position was: {secret}")
print("Grover search needed to turn only one card to discover the queen of Hearts: ")

filename = './images/cards/QHFaceUPPos'+str(secret)+'.jpeg'

display(Image(filename=filename, width=800))

```

## You can also run this on a real backend using IBMQ.

#  C.  QISKit AQUA for 3SAT with Grover

```python
import pylab
import numpy as np
from qiskit import Aer
from qiskit.tools.visualization import plot_histogram
from qiskit.aqua import QuantumInstance
from qiskit.aqua.algorithms import Grover
from qiskit.aqua.components.oracles import LogicalExpressionOracle, TruthTableOracle
```

```python

```

Again, let's search for a card in a deck.
This time, within a 32 cards deck (7,8,9,10,J,Q,K,A,Spades,Hearts,Diamonds,Clubs).
Let's use these booleans to catagorize the cards:

$x_1$ : the card is a number (7,8,9,10)

$x_2$ : the card is red (hearts, diamonds)

$x_3$ : the suit has a sharp edge on top (spades, diamonds)

$x_4$ : the card is at odd position (7,9,J,K)

$x_5$ : the card is in the middle values (9,10,J,Q)

With this we can form a logical expression, in the 3SAT form:

$F = ( x_1 \lor x_2 \lor x_5) \land ( x_1 \lor  x_2 \lor \neg x_5) \land ( x_1 \lor \neg x_2 \lor x_5) \land ( \neg x_1 \lor  x_2 \lor  x_5) \land ( \neg x_1 \lor  x_2 \lor \neg x_5) ...$

$... ( \neg x_1 \lor \neg x_2 \lor x_4) \land (\neg x_2 \lor \neg x_3 \lor x_4) \land ( \neg x_2 \lor \neg x_3 \lor \neg  x_4) \land ( \neg x_2 \lor x_3 \lor \neg  x_4)
$

... and use the Grover algorithm to find if card (and which) satisfies the expression.

```python
# Create the expression above in DIMACS-CNF format
# Logical expression :(𝑣1∨¬𝑣2∨𝑣3)∧(𝑣1∨𝑣2∨¬𝑣3)∧(¬𝑣1∨𝑣2∨¬𝑣3)∧(𝑣1∨¬𝑣2∨¬𝑣3)∧(¬𝑣1∨¬𝑣2∨¬𝑣3)
input_3sat='''
c example DIMACS-CNF 3-SAT
p cnf 5 8
1 2 5 0
1 2 -5 0
1 -2 5 0
-1 2 5 0
-1 2 -5 0
-1 -2 4 0
-2 -3 4 0
-2 -3 -4 0
-2 3 -4 0
'''
```

```python
# Construct the Oracle Circuit with the LogicalExpressionOracle and set the optimisation parameter to True
oracle = LogicalExpressionOracle(input_3sat, optimization=True)

# Draw the circuit of oracle with the method .circuit.draw()
oracle.circuit.draw(output='mpl', scale=0.6)
```

```python
# Create a grover instance with the Grover function
# use the num_iterations by default = 1
grover = Grover(oracle, num_iterations=4)
```

```python
# Load the qasm_simulator backend from Aer module
backend = Aer.get_backend('qasm_simulator')

# Create a quantum_instance with QuantumInstance
quantum_instance = QuantumInstance(backend, shots=8192)

# run the Grover algorithm on the quantum_instance
result = grover.run(quantum_instance)

# print the results of the experiment result['result'] or result['measurement']
print('The corresponding result is: ',result['result'], 'or',  result['measurement'])

```

```python
# plot the histogram with the plot_histogram function on result['measurement']
plot_histogram(result['measurement'])
```

```python

```

So we find 01001 a card with a letter index, red, with no sharp edge on top of the suit symbol, at even position, and in the middle values. Did you guess what card it is ?

## Thank you for your attention


## [Problem 21](https://projecteuler.net/problem=21)
Amicable numbers

```python
import numpy as np
import sympy
from tqdm import tqdm
```

```python
def d(n):
    return sum(sympy.divisors(n)[:-1])
```

```python
ans=[x for x in range(1,10001) if x!=d(x) and x==d(d(x))]
print(sum(ans))
```

    31626

## [Problem 22](https://projecteuler.net/problem=22)
Names scores

```python
import string
ABC_=[letter for letter in string.ascii_uppercase]
```

```python
ABC={}
ABC.update({ABC_[i]:i+1 for i in range(len(ABC_))})
```

```python
def s(name):
    num=0
    for alphabet in name:
        num+=ABC[alphabet]
    return num
```

```python
with open("./data/p022.txt") as f:
    names=f.read()
names=names.split(',')
names=sorted([name[1:-1] for name in names])
```

```python
scores=[s(name)*(names.index(name)+1) for name in names]
score=sum(scores)
print(score)
```

    871198282

## [Problem 23](https://projecteuler.net/problem=23)
Non-abundant sums

```python
import itertools
```

```python
abundant=[x for x in range(1,28123) if x<d(x)]
sum_abundant=[i+j for i,j in itertools.product(abundant,repeat=2) if i+j<28123]
```

```python
ans=set(range(1,28123))-set(sum_abundant)
ans=list(ans)
print(sum(ans))
```

    4179871

## [Problem 24](https://projecteuler.net/problem=24)
Lexicographic permutations

```python
digits=[letter for letter in string.digits]
```

```python
def number(n):
    return [digits[i] for i in n]
```

```python
order=[number(n) for n in itertools.permutations(range(10))]
```

```python
ans_list=order[10**6-1]
ans_str=""
for x in ans_list:
    ans_str+=x
ans=int(ans_str)
print(ans)
```

    2783915460

## [Problem 25](https://projecteuler.net/problem=25)
1000-digit Fibonacci number

```python
from math import sqrt
from math import log10
```

```python
phi=log10((1+sqrt(5))/2)
psi=0.5*log10(5)
```

```python
digit=0
num=0
while digit<999:
    num+=1
    digit=num*phi-psi
print(num)
```

    4782

## [Problem 26](https://projecteuler.net/problem=26)
Reciprocal cycles

```python
divisor=list(sympy.primerange(0,1000))
divisor.remove(2)
divisor.remove(5)
```

```python
def cycle(n):
    y=9
    ans=1
    while y%n!=0:
        ans+=1
        y=y*10+9
    return ans
```

```python
ans=0
for x in divisor:
    tmp=cycle(x)
    if tmp>ans:
        ans=tmp
        prime=x
print(prime)
```

    983

## [Problem 27](https://projecteuler.net/problem=27)
Quadratic primes

```python
A=list(range(-999,1000,2))
B=list(sympy.primerange(0,1000))
B_=[-x for x in B]
B.extend(B_)
```

```python
def quadra(a,b,n):
    return n**2+a*n+b

def num_quadratic_prime(a,b):
    n=0
    tmp=quadra(a,b,n)
    while sympy.isprime(tmp):
        n+=1
        tmp=quadra(a,b,n)
    return n
```

```python
max_num=0
for a,b in itertools.product(A,B):
    tmp=num_quadratic_prime(a,b)
    if tmp>max_num:
        max_num=tmp
        ans=a*b
print(ans)
```

    -59231

## [Problem 28](https://projecteuler.net/problem=28)
Number spiral diagonals

```python
def spiral(n):
    ans=1
    diag=[1]
    step=0
    for i in range(n//2):
        step+=2
        tmp=diag[-1]+step
        diag=[tmp+x*step for x in range(4)]
        ans+=sum(diag)
    return ans
```

```python
print(spiral(1001))
```

    669171001

## [Problem 29](https://projecteuler.net/problem=29)
Distinct powers

```python
combi=itertools.product(range(2,101),repeat=2)
```

```python
ans=[0 for _ in range(99**2)]
k=0
for a,b in combi:
    ans[k]=a**b
    k+=1
ans=list(set(ans))
print(len(ans))
```

    9183

## [Problem 30](https://projecteuler.net/problem=30)
Digit fifth powers

```python
def digit_power_sum(n,d):
    ans=0
    num=map(int,str(n))
    for x in num:
        ans+=x**d
    return ans
```

```python
d=5
ans=0
for i in tqdm(range(2,10**7)):
    if digit_power_sum(i,d)==i:
        ans+=i
print(ans)
```

    100%|██████████| 9999998/9999998 [00:47<00:00, 211754.62it/s]

    443839

```python

```


$\newcommand{\Re}{\mathbb{R}} \newcommand{\EE}{\mathbb{E}}$
# Polynomial Optimization and the Generalized Moment Problem

## Polynomial Optimization

The polynomial optimization problem is to

$
\begin{align}
\text{minimize} \quad &f(x)\\
    \text{subject to} \quad &g_i(x) \geq 0,\quad i=1,\ldots,N
\end{align}
$

where $x \in \Re^d$ and $f(x), g_i(x) \in \Re[x]$ are polynomials. We implement the techniques developed by Parrilo et al. and Lasserre et al. where this problem is relaxed to a semidefinite program.

In the first part of this worksheet, we show how to solve polynomial optimization problems using mompy where several example problems in the [GloptiPoly paper](http://homepages.laas.fr/henrion/papers/gloptipoly3.pdf) and the [solution extraction paper](http://homepages.laas.fr/henrion/papers/extract.pdf) are reproduced.

In the second part, we give the simplest example of estimating a mixture model by solving the Generalized Moment Problem (GMP). Some extensions can be found in [the extra examples worksheet](extra_examples.ipynb). This software was used for our paper [Estimating mixture models via mixtures of polynomials](http://papers.nips.cc/paper/5702-estimating-mixture-models-via-mixtures-of-polynomials.pdf) where we look at some more elaborate settings for the mixture model problem.

Other implementation for solving the polynomial optimization problem / and Generalized Moment Problem are [GloptiPoly](http://homepages.laas.fr/henrion/software/gloptipoly/) and [SOSTOOLS](http://www.cds.caltech.edu/sostools/) in MATLAB, and [ncpol2sdpa](https://pypi.python.org/pypi/ncpol2sdpa/) in python described [here](http://arxiv.org/abs/1308.6029).

```python
# we are dependent on numpy, sympy and cvxopt.
import numpy as np
import cvxopt
import mompy as mp

# just some basic settings and setup
mp.cvxsolvers.options['show_progress'] = False
from IPython.display import display, Markdown, Math, display_markdown
sp.init_printing()

def print_problem(obj, constraints = None, moment_constraints = None):
    display_markdown(mp.problem_to_str(obj,constraints,moment_constraints, False), raw=True)
```

### 1) First example

```python
x,y = sp.symbols('x,y')
f =  x**2 + y**2
gs = [x+y>=4, x+y<=4]
print_problem(f, gs)
sol = mp.solvers.solve_GMP(f, gs)
mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 1)
```

Minimizing $\mathcal{L}(x^{2} + y^{2})$

subject to 	 $x + y \geq 4$, 	 $x + y \leq 4$,

    the maximum degree appearing in the problem is 2
    slacks
    1e-06
    status: optimal
    round=1,	 rank=1,	 size=3,	 obj=8.000
    lost 0.0000001

    {x: array([ 1.99999971]), y: array([ 1.99999971])}

### 2) Unconstrained optimization: the six hump camel back function

A plot of this function can be found at [library of simutations](http://www.sfu.ca/~ssurjano/camel6.html). MATLAB got the solution $x^* = [0.0898   -0.7127]$ and corresponding optimal values of $f(x^*) = -1.0316$

```python
x1,x2 = sp.symbols('x1:3')
f = 4*x1**2+x1*x2-4*x2**2-2.1*x1**4+4*x2**4+x1**6/3
print_problem(f)
sol = mp.solvers.solve_GMP(f, rounds=1)
mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 2)

```

Minimizing $\mathcal{L}(\frac{x_{1}^{6}}{3} - 2.1 x_{1}^{4} + 4 x_{1}^{2} + x_{1} x_{2} + 4 x_{2}^{4} - 4 x_{2}^{2})$	 subject to no constraints

    the maximum degree appearing in the problem is 6
    slacks
    1e-06
    status: optimal
    round=1,	 rank=3,	 size=10,	 obj=-1.032

    {x1: array([ 0.08984255, -0.08984255]), x2: array([-0.71265247,  0.71265247])}

### 3) Multiple rounds

Generally more rounds are needed to get the correct solutions.

```python
x1,x2,x3 = sp.symbols('x1:4')
f = -(x1 - 1)**2 - (x1 - x2)**2 - (x2 - 3)**2
gs = [1 - (x1 - 1)**2 >= 0, 1 - (x1 - x2)**2 >= 0, 1 - (x2 - 3)**2 >= 0]
print_problem(f, gs)
sol = mp.solvers.solve_GMP(f, gs, rounds=4)
mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 3)
```

Minimizing $\mathcal{L}(- \left(x_{1} - 1\right)^{2} - \left(x_{1} - x_{2}\right)^{2} - \left(x_{2} - 3\right)^{2})$

subject to 	 $- \left(x_{1} - 1\right)^{2} + 1 \geq 0$, 	 $- \left(x_{1} - x_{2}\right)^{2} + 1 \geq 0$, 	 $- \left(x_{2} - 3\right)^{2} + 1 \geq 0$,

    the maximum degree appearing in the problem is 2
    slacks
    1e-06
    status: optimal
    round=1,	 rank=3,	 size=3,	 obj=-3.000
    slacks
    1e-06
    status: optimal
    round=2,	 rank=6,	 size=6,	 obj=-3.000
    slacks
    1e-06
    status: optimal
    round=3,	 rank=7,	 size=10,	 obj=-2.000
    slacks
    1e-06
    status: optimal
    round=4,	 rank=8,	 size=15,	 obj=-2.000

    {x1: array([ 2.00000811,  2.00000032,  0.99999978]),
     x2: array([ 3.00000234,  2.00000006,  2.00000007])}

Yet another example

```python
x1,x2,x3 = sp.symbols('x1:4')
f = -2*x1 + x2 - x3
gs = [0<=x1, x1<=2, x2>=0, x3>=0, x3<=3, x1+x2+x3<=4, 3*x2+x3<=6,\
      24-20*x1+9*x2-13*x3+4*x1**2 - 4*x1*x2+4*x1*x3+2*x2**2-2*x2*x3+2*x3**2>=0]; hs = [];
print_problem(f, gs)
sol = mp.solvers.solve_GMP(f, gs, hs, rounds=4)
print mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 2)
```

Minimizing $\mathcal{L}(- 2 x_{1} + x_{2} - x_{3})$

subject to 	 $x_{1} \geq 0$, 	 $x_{1} \leq 2$, 	 $x_{2} \geq 0$, 	 $x_{3} \geq 0$, 	 $x_{3} \leq 3$, 	 $x_{1} + x_{2} + x_{3} \leq 4$, 	 $3 x_{2} + x_{3} \leq 6$, 	 $4 x_{1}^{2} - 4 x_{1} x_{2} + 4 x_{1} x_{3} - 20 x_{1} + 2 x_{2}^{2} - 2 x_{2} x_{3} + 9 x_{2} + 2 x_{3}^{2} - 13 x_{3} + 24 \geq 0$,

    the maximum degree appearing in the problem is 2
    slacks
    1e-06
    status: optimal
    round=1,	 rank=4,	 size=4,	 obj=-6.000
    slacks
    1e-06
    status: optimal
    round=2,	 rank=8,	 size=10,	 obj=-5.692
    slacks
    1e-06
    status: optimal
    round=3,	 rank=15,	 size=20,	 obj=-4.068
    slacks
    1e-06
    status: optimal
    round=4,	 rank=17,	 size=35,	 obj=-4.000
    {x3: array([  3.48007643e-07,   2.99999775e+00]), x1: array([ 1.99999986,  0.50000288]), x2: array([ -1.10400714e-08,   1.46357718e-05])}

### 4) Motzkin polynomial
The Motzkin polynomial is non-negative, but cannot be expressed in sum of squares. It attains global minimum of 0 at $|x_1| = |x_2| = \sqrt{3}/3$ (4 points). The first few relaxations are unbounded and might take a while for cvxopt to realize this.

```python
x1,x2 = sp.symbols('x1:3')
f = x1**2 * x2**2 * (x1**2 + x2**2 - 1) + 1./27
print_problem(f)
sol = mp.solvers.solve_GMP(f, rounds=7)

```

Minimizing $\mathcal{L}(x_{1}^{2} x_{2}^{2} \left(x_{1}^{2} + x_{2}^{2} - 1\right) + 0.037037037037037)$	 subject to no constraints

    the maximum degree appearing in the problem is 6
    slacks
    1e-06
    status: unknown
    round=1,	 rank=10,	 size=10,	 obj=-1119264077.823
    slacks
    1e-06
    status: unknown
    round=2,	 rank=15,	 size=15,	 obj=-408995.200
    slacks
    1e-06
    status: unknown
    round=3,	 rank=21,	 size=21,	 obj=-5690.864
    slacks
    1e-06
    status: unknown
    round=4,	 rank=28,	 size=28,	 obj=-918.009
    slacks
    1e-06
    status: unknown
    round=5,	 rank=36,	 size=36,	 obj=-67.576
    slacks
    1e-06
    status: optimal
    round=6,	 rank=13,	 size=45,	 obj=0.000
    slacks
    1e-06
    status: optimal
    round=7,	 rank=16,	 size=55,	 obj=0.000

```python
print mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 4, maxdeg=3)
```

    {x1: array([-0.57735018, -0.5773504 ,  0.57735014,  0.57735044]), x2: array([ 0.57735087, -0.57735087, -0.57735112,  0.57735112])}

## Generalized Moment Problem (GMP)
The GMP is to

$
\begin{align}
\text{minimize} \quad &f(x)\\
    \text{subject to} \quad &g_i(x) \geq 0,\quad i=1,\ldots,N\\
    \quad &\mathcal{L}(h_j(x)) \geq 0,\quad j=1,\ldots,M
\end{align}
$

where $x \in \Re^d$ and $f(x), g_i(x), h_j(x) \in \Re[x]$ are polynomials, and $\mathcal{L}(\cdot): \Re[x] \mapsto \Re$ is a linear functional. Loosely speaking, we would like a measure $\mu$ to exist so that $\mathcal{L}(h) = \int h\ d\mu$.

For details see:

Jean B. Lasserre, 2011, Moments, Positive Polynomials and Their Applications

Jean B. Lasserre, 2008, A semidefinite programming approach to the generalized problem of moments

### Estimating mixtures of exponential distributions
We are interested in estimating the parameters of a mixture model from some observed moments, possibly with constraints on the parameters. Suppose that we have a mixture of 2 exponential distributions with density function $p(x; \beta) \sim \exp(-x/\beta)/\beta$ in for $x \in [0,\infty)$. Suppose that $\pi_1 = \pi_2 = 0.5$ and $\beta_1 = 1, \beta_2 = 2$. We are allowed to observe moments of the data $\EE[X^n] = {n!}(\pi_1 \beta_1^n + \pi_2 \beta_2^n)$ for $n=1,\ldots,6$. For more details on estimating mixture model using this method, see our paper [Estimating mixture models via mixtures of polynomials](http://papers.nips.cc/paper/5702-estimating-mixture-models-via-mixtures-of-polynomials.pdf) and see [the extra examples worksheet](extra_examples.ipynb) for another easy example on mixture of Gaussians.

```python
beta = sp.symbols('beta')
beta0 = [1,2]; pi0 = [0.5,0.5]
hs = [beta**m - (pi0[0]*beta0[0]**m+pi0[1]*beta0[1]**m) for m in range(1,5)]
f = sum([beta**(2*i) for i in range(3)])
# note that hs are the LHS of h==0, whereas gs are sympy inequalities
print_problem(f, None, hs)

sol = mp.solvers.solve_GMP(f, None, hs)
print mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 2, tol = 1e-3)
```

Minimizing $\mathcal{L}(\beta^{4} + \beta^{2} + 1)$

subject to 	$\mathcal{L}(\beta - 1.5) = 0$, 	$\mathcal{L}(\beta^{2} - 2.5) = 0$, 	$\mathcal{L}(\beta^{3} - 4.5) = 0$, 	$\mathcal{L}(\beta^{4} - 8.5) = 0$,

    the maximum degree appearing in the problem is 4
    slacks
    1e-06
    status: optimal
    round=1,	 rank=2,	 size=3,	 obj=12.000
    lost 0.0000000
    {beta: array([ 1.,  2.])}

Now we try to solve the problem with insufficient moment conditions, but extra constraints on the parameters themselves.

```python
gs=[beta>=1]
f = sum([beta**(2*i) for i in range(3)])
hs_sub = hs[0:2]
print_problem(f, gs, hs_sub)

sol = mp.solvers.solve_GMP(f, gs, hs_sub)
print mp.extractors.extract_solutions_lasserre(sol['MM'], sol['x'], 2, tol = 1e-3)
```

Minimizing $\mathcal{L}(\beta^{4} + \beta^{2} + 1)$

subject to 	 $\beta \geq 1$,
$\mathcal{L}(\beta - 1.5) = 0$, 	$\mathcal{L}(\beta^{2} - 2.5) = 0$,

    the maximum degree appearing in the problem is 4
    slacks
    1e-06
    status: optimal
    round=1,	 rank=2,	 size=3,	 obj=12.000
    lost 0.0000001
    {beta: array([ 0.99999876,  1.99999896])}


# Neuromatch Academy: Week 1, Day 3, Tutorial 4
# Model Fitting: Multiple linear regression

#Tutorial Objectives

This is Tutorial 4 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of linear models by  generalizing to multiple linear regression (Tutorial 4). We then move on to polynomial regression (Tutorial 5). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 6) and two common methods for model selection, AIC and Cross Validation (Tutorial 7).

In this tutorial, we will generalize our linear model to incorporate multiple linear features.
- Learn how to structure our inputs for multiple linear regression using the 'Design Matrix'
- Generalize the MSE for multiple features using the ordinary least squares estimator
- Visualize our data and model fit in multiple dimensions

# Setup

```python
# @title  Imports
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

```python
#@title Figure Settings
%matplotlib inline
fig_w, fig_h = (8, 6)
plt.rcParams.update({'figure.figsize': (fig_w, fig_h)})
%config InlineBackend.figure_format = 'retina'
```

# Multiple Linear Regression

```python
#@title Video: Multiple Linear Regression
from IPython.display import YouTubeVideo
video = YouTubeVideo(id="uQjKnlhGEVY", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)
video
```

    Video available at https://youtube.com/watch?v=uQjKnlhGEVY

Now that we have considered the univariate case and how to produce confidence intervals for our estimator, we turn now to the general linear regression case, where we can have more than one regressor, or feature, in our input.

Recall that our original univariate linear model was given as

\begin{align}
y = \theta_0 + \theta_1 x
\end{align}

where $\theta_0$ is the intercept, $\theta_1$ is the slope. We can easily extend this to the multivariate scenario by adding another parameter for each additional feature

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_1 x_2 + ... +\theta_d x_d
\end{align}

where $d$ is the dimensionality (number of features) in our input.

We can condense this succinctly using vector notation for a single data point

\begin{align}
y_i = \boldsymbol{\theta}^{\top}\mathbf{x}_i
\end{align}

and fully in matrix form

\begin{align}
\mathbf{y} = \mathbf{X}\boldsymbol{\theta}
\end{align}

where $\mathbf{y}$ is a vector of measurements, $\mathbf{X}$ is a matrix containing the feature values (columns) for each input sample (rows), and $\boldsymbol{\theta}$ is our parameter vector.

This matrix $\mathbf{X}$ is often referred to as the "[design matrix](https://en.wikipedia.org/wiki/Design_matrix)".

For this tutorial we will focus on the two-dimensional case ($d=2$), which allows us to fully explore the multivariate case while still easily visualizing our results. As an example, think of a situation where a scientist records the spiking response of a retinal ganglion cell to patterns of light signals that vary in contrast and in orientation.

In this case our model can be writen as

\begin{align}
y = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \epsilon
\end{align}

or in matrix form where

\begin{align}
\mathbf{X} =
\begin{bmatrix}
1 & x_{1,1} & x_{1,2} \\
1 & x_{2,1} & x_{2,2} \\
\vdots & \vdots & \vdots \\
1 & x_{n,1} & x_{n,2}
\end{bmatrix},
\boldsymbol{\theta} =
\begin{bmatrix}
\theta_0 \\
\theta_1 \\
\theta_2 \\
\end{bmatrix}
\end{align}

For our actual exploration dataset we shall set $\boldsymbol{\theta}=[0, -2, -3]$ and draw $N=40$ noisy samples from $x \in [-2,2)$. Note that setting the value of $\theta_0 = 0$ effectively ignores the offset term.

```python
np.random.seed(121)
theta = [0, -2, -3]
n_samples = 40
n_regressors = len(theta)
x = np.random.uniform(-2, 2, (n_samples, n_regressors))
noise = np.random.randn(n_samples)
y = x @ theta + noise
```

Now that we have our dataset, we want to find an optimal vector of paramters $\boldsymbol{\hat\theta}$. Recall our analytic solution to minimizing MSE for a single regressor:

\begin{align}
\hat\theta = \sum_i \frac{x_i y_i}{x_i^2}.
\end{align}

The same holds true for the multiple regressor case, only now expressed in matrix form

\begin{align}
\boldsymbol{\hat\theta} = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}.
\end{align}

This is called the [ordinary least squares](https://en.wikipedia.org/wiki/Ordinary_least_squares) (OLS) estimator.

### Exercise: Ordinary Least Squares Estimator

In this exercise you will implement the OLS approach to estimating $\boldsymbol{\hat\theta}$ from the design matrix $\mathbf{X}$ and measurement vector $\mathbf{y}$. You can the `@` symbol for matrix multiplication, `.T` for transpose, and `np.linalg.inv` for matrix inversion.

```python
def ordinary_least_squares(x, y):
  """Ordinary least squares estimator for linear regression.

  Args:
    x (ndarray): design matrix of shape (n_samples, n_regressors)
    y (ndarray): vector of measurements of shape (n_samples)

  Returns:
    ndarray: estimated parameter values of shape (n_regressors)
  """
  ######################################################################
  ## TODO for students: solve for the optimal parameter vector using OLS
  ######################################################################
  # comment this out when you've filled
  raise NotImplementedError("Student excercise: solve for theta_hat vector using OLS")
  return theta_hat
```

```python
theta_hat = ordinary_least_squares(x, y)
theta_hat
```

    array([ 0.27846561, -2.01651235, -3.14249005])

Now that we have our $\mathbf{\hat\theta}$, we can obtain $\mathbf{\hat y}$ and thus our mean squared error.

```python
y_hat = x @ theta_hat
print(f"MSE = {np.mean((y - y_hat)**2):.2f}")
```

    MSE = 0.57

Finally, the following code will plot a geometric visualization of the data points (blue) and fitted plane. You can see that the residuals (green bars) are orthogonal to the hyperplane.

```python
xx, yy = np.mgrid[-2:2:50j, -2:2:50j]
y_hat_grid = np.array([xx.flatten(), yy.flatten()]).T @ theta_hat[1:]
y_hat_grid = y_hat_grid.reshape((50, 50))

ax = plt.subplot(projection='3d')
ax.plot(x[:,1], x[:,2], y, '.')
ax.plot_surface(xx, yy, y_hat_grid, linewidth=0, alpha=0.5, color='C1',
                cmap=plt.get_cmap('coolwarm'))

for i in range(len(x)):
  ax.plot((x[i, 1], x[i, 1]),
          (x[i, 2], x[i, 2]),
          (y[i], y_hat[i]),
          'g-', alpha=.5)

ax.set(
    xlabel='$x_1$',
    ylabel='$x_2$',
    zlabel='y'
)
plt.tight_layout()

```

# Summary
- linear regression generalizes naturally to multiple dimensions
- linear algebra affords us the mathematical tools to reason and solve such problems beyond the two dimensional case

**NOTE** in practice, multidimensional least squares problems can be solve very efficiently (thanks to numerical routines such as LAPACK).

```python

```


# Spectral Analysis of Deterministic Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Introduction

The analysis of the spectral properties of a signal plays an important role in digital signal processing. Some application examples are the

* [Spectrum analyzer](https://en.wikipedia.org/wiki/Spectrum_analyzer)
* Detection of (harmonic) signals
* [Estimation of fundamental frequency and harmonics](https://en.wikipedia.org/wiki/Modal_analysis)
* Spectral suppression: acoustic echo suppression, noise reduction, ...

In the practical realization of spectral analysis techniques, the [discrete Fourier transformation](https://en.wikipedia.org/wiki/Discrete_Fourier_transform) (DFT) is applied to discrete finite-length signals in order to gain insights into their spectral composition. A basic task in spectral analysis is to determine the amplitude (and phase) of dominant harmonic contributions in a signal mixture. The properties of the DFT with respect to the analysis of an harmonic exponential signal are discussed in the following.

## The Leakage Effect

[Spectral leakage](https://en.wikipedia.org/wiki/Spectral_leakage) is a fundamental effect of the DFT. It limits the ability to detect harmonic signals in signal mixtures and hence the performance of spectral analysis. In order to discuss this effect, the DFT of a discrete exponential signal is derived from the Fourier transform of the continuous exponential signal. The connections between the Fourier transform, the [discrete-time Fourier transform](https://en.wikipedia.org/wiki/Discrete-time_Fourier_transform) (DTFT) and the DFT for a uniformly sampled signal are illustrated below.

Consequently, the leakage effect is discussed in the remainder of this section by considering the following four steps:

1. Fourier transform of an harmonic exponential signal,
2. discrete-time Fourier transform (DTFT) of a discrete harmonic exponential signal, and
3. DTFT of a finite-length discrete harmonic exponential signal
4. sampling of the DTFT.

### Fourier Transformation of an Exponential Signal

The harmonic exponential signal is defined as

\begin{equation}
x(t) = \mathrm{e}^{\,\mathrm{j}\, \omega_0 \, t}
\end{equation}

where $\omega_0 = 2 \pi f$ denotes the angular frequency of the signal. The Fourier transform of the exponential signal is

\begin{equation}
X(\mathrm{j}\, \omega) = \int\limits_{-\infty}^{\infty} x(t) \,\mathrm{e}^{\,- \mathrm{j}\, \omega \,t} \mathrm{d}t = 2\pi \; \delta(\omega - \omega_0)
\end{equation}

The spectrum consists of a single shifted Dirac impulse located at the angular frequency $\omega_0$ of the exponential signal. Hence the spectrum $X(\mathrm{j}\, \omega)$ consists of a clearly isolated and distinguishable event. In practice, it is not possible to compute the Fourier transformation of a continuous signal by means of digital signal processing.

### Discrete-Time Fourier Transformation of a Discrete Exponential Signal

Now lets consider sampled signals. The discrete exponential signal $x[k]$ is derived from its continuous counterpart $x(t)$ above by equidistant sampling $x[k] := x(k T)$ with the sampling interval $T$

\begin{equation}
x[k] = \mathrm{e}^{\,\mathrm{j}\, \Omega_0 \,k}
\end{equation}

where $\Omega_0 = \omega_0 T$ denotes the normalized angular frequency. The DTFT is the Fourier transformation of a sampled signal. For the exponential signal it is given as

\begin{equation}
X(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \sum_{k = -\infty}^{\infty} x[k]\, \mathrm{e}^{\,-\mathrm{j}\, \Omega \,k} = 2\pi \sum_{n = -\infty}^{\infty} \delta((\Omega-\Omega_0) - 2\,\pi\,n)
\end{equation}

The spectrum of the DTFT is $2\pi$-periodic due to sampling. As a consequence, the transformation of the discrete exponential signal consists of a series Dirac impulses. For the region of interest $-\pi < \Omega \leq \pi$ the spectrum consists of a clearly isolated and distinguishable event, as for the continuous case.

The DTFT cannot be realized in practice, since is requires the knowledge of the signal $x[k]$ for all time instants $k$. In general, a measured signal is only known within a finite time-interval. The DFT of a signal of finite length can be derived from the DTFT in two steps:

1. truncation (windowing) of the signal and
2. sampling of the DTFT spectrum of the windowed signal.

The consequences of these two steps are investigated in the following.

### Discrete-Time Fourier Transformation of a Truncated Discrete Exponential Signal

In general, truncation of a signal $x[k]$ to a length of $N$ samples is modeled by multiplying the signal with a window function $w[k]$ of length $N$

\begin{equation}
x_N[k] = x[k] \cdot w[k]
\end{equation}

where $x_N[k]$ denotes the truncated signal and $w[k] = 0$ for $\{k: k < 0 \wedge k \geq N \}$. The spectrum $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ can be derived from the multiplication theorem of the DTFT as

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \frac{1}{2 \pi} X(\mathrm{e}^{\,\mathrm{j}\, \Omega}) \circledast_N W(\mathrm{e}^{\,\mathrm{j}\, \Omega})
\end{equation}

where $\circledast$ denotes the cyclic/[circular convolution](https://en.wikipedia.org/wiki/Circular_convolution) of length $N$. A hard truncation of the signal to $N$ samples is modeled by the rectangular signal

\begin{equation}
w[k] = \text{rect}_N[k] = \begin{cases}
1 & \mathrm{for} \; 0\leq k<N \\
0 & \mathrm{otherwise}
\end{cases}
\end{equation}

Its spectrum is given as

\begin{equation}
W(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \mathrm{e}^{\,-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})}
\end{equation}

The DTFT $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ of the truncated exponential signal is derived by introducing the DTFT of the exponential signal and the window function into above cyclic convolution. Since, both the DTFT of the exponential signal and the window function are periodic with a period of $2 \pi$, the cyclic convolution with period $2 \pi$ is given by linear convolution of both spectra within $-\pi < \Omega \leq \pi$

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega}) = \delta(\Omega-\Omega_0) * \mathrm{e}^{\,-\mathrm{j} \, \Omega \,\frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,\Omega}{2})}{\sin(\frac{\Omega}{2})} =
\mathrm{e}^{\,-\mathrm{j}\, (\Omega-\Omega_0) \, \frac{N-1}{2}} \cdot \frac{\sin(\frac{N\, (\Omega-\Omega_0)}{2})}{\sin(\frac{(\Omega-\Omega_0)}{2})}
\end{equation}

Clearly the DTFT of the truncated harmonic exponential signal $x_N[k]$ is not given by a series of Dirac impulses. Above equation is evaluated numerically in order to illustrate the properties of $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

Om0 = 1  # frequency of exponential signal
N = 32  # length of signal

# DTFT of finite length exponential signal (analytic)
Om = np.linspace(-np.pi, np.pi, num=1024)
XN = np.exp(-1j * (Om-Om0) * (N-1) / 2) * (np.sin(N * (Om-Om0) / 2)) / (np.sin((Om-Om0) / 2))

# plot spectrum
plt.figure(figsize = (10, 8))
plt.plot(Om, abs(XN), 'r')
plt.title(r'Absolute value of the DTFT of a truncated exponential signal $e^{j \Omega_0 k}$ with $\Omega_0=$%2.2f' %Om0)
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|X_N(e^{j \Omega})|$')
plt.axis([-np.pi, np.pi, -0.5, N+5])
plt.grid()
```

**Excercise**

* Change the frequency `Om0` of the signal and rerun the example. How does the magnitude spectrum change?
* Change the length `N` of the signal and rerun the example. How does the magnitude spectrum change?

The maximum of the absolute value of the spectrum is located at the frequency $\Omega_0$. It should become clear that truncation of the exponential signal leads to a broadening of the spectrum. The shorter the signal, the wider the mainlobe becomes.

### The Leakage Effect of the Discrete Fourier Transformation

The DFT is derived from the DTFT $X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})$ of the truncated signal $x_N[k]$ by sampling the DTFT equidistantly at $\Omega = \mu \frac{2 \pi}{N}$

\begin{equation}
X[\mu] = X_N(\mathrm{e}^{\,\mathrm{j}\, \Omega})\big\vert_{\Omega = \mu \frac{2 \pi}{N}}
\end{equation}

For the DFT of the exponential signal we finally get

\begin{equation}
X[\mu] = \mathrm{e}^{\,\mathrm{j}\, (\Omega_0 - \mu \frac{2 \pi}{N}) \frac{N-1}{2}} \cdot \frac{\sin(\frac{N \,(\Omega_0 - \mu \frac{2 \pi}{N})}{2})}{\sin(\frac{\Omega_0 - \mu \frac{2 \pi}{N}}{2})}
\end{equation}

The sampling of the DTFT is illustrated in the following example. Note that the normalized angular frequency $\Omega_0$ has been expressed in terms of the periodicity $P$ of the exponential signal $\Omega_0 = P \; \frac{2\pi}{N}$.

```python
N = 32  # length of the signal
P = 10.33  # periodicity of the exponential signal
Om0 = P * (2*np.pi/N)  # frequency of exponential signal

# truncated exponential signal
k = np.arange(N)
x = np.exp(1j*Om0*k)

# DTFT of finite length exponential signal (analytic)
Om = np.linspace(0, 2*np.pi, num=1024)
Xw = np.exp(-1j*(Om-Om0)*(N-1)/2)*(np.sin(N*(Om-Om0)/2))/(np.sin((Om-Om0)/2))

# DFT of the exponential signal by FFT
X = np.fft.fft(x)
mu = np.arange(N) * 2*np.pi/N

# plot spectra
plt.figure(figsize = (10, 8))
ax1 = plt.gca()

plt.plot(Om, abs(Xw), 'r', label=r'$|X_N(e^{j \Omega})|$')
plt.stem(mu, abs(X), label=r'$|X_N[\mu]|$', basefmt=' ')
plt.ylim([-0.5, N+5]);
plt.title(r'Absolute value of the DTFT/DFT of a truncated exponential signal $e^{j \Omega_0 k}$ with $\Omega_0=$%2.2f' %Om0, y=1.08)
plt.legend()

ax1.set_xlabel(r'$\Omega$')
ax1.set_xlim([Om[0], Om[-1]])
ax1.grid()

ax2 = ax1.twiny()
ax2.set_xlim([0, N])
ax2.set_xlabel(r'$\mu$', color='C0')
ax2.tick_params('x', colors='C0')
```

**Exercise**

* Change the periodicity `P` of the exponential signal and rerun the example. What happens if the periodicity is an integer? Why?
* Change the length `N` of the DFT? How does the spectrum change?
* What conclusions can be drawn for the analysis of a single exponential signal by the DFT?

You should have noticed that for an exponential signal whose periodicity is an integer $P \in \mathbb{N}$, the DFT consists of a discrete Dirac pulse $X[\mu] = N \cdot \delta[\mu - P]$. In this case, the sampling points coincide with the maximum of the main lobe or the zeros of the DTFT. For non-integer $P$, hence non-periodic exponential signals with respect to the signal length $N$, the DFT has additional contributions. The shorter the length $N$, the wider these contributions are spread in the spectrum. This smearing effect is known as *leakage effect* of the DFT. It limits the achievable frequency resolution of the DFT when analyzing signal mixtures consisting of more than one exponential signal. This is illustrated by the following numerical examples.

### Analysis of Signal Mixtures by the Discrete Fourier Transformation

In order to discuss the implications of the leakage effect when analyzing signal mixtures, the superposition of two exponential signals with different amplitudes and frequencies is considered

\begin{equation}
x_N[k] = A_1 \cdot e^{\mathrm{j} \Omega_1 k} + A_2 \cdot e^{\mathrm{j} \Omega_2 k}
\end{equation}

For convenience, a function is defined that calculates and plots the magnitude spectrum of $x_N[k]$.

```python
def dft_signal_mixture(N, A1, P1, A2, P2):
    # N: length of signal/DFT
    # A1, P1, A2, P2: amplitude and periodicity of 1st/2nd complex exponential

    # generate the signal mixture
    Om0_1 = P1 * (2*np.pi/N)  # frequency of 1st exponential signal
    Om0_2 = P2 * (2*np.pi/N)  # frequency of 2nd exponential signal
    k = np.arange(N)
    x = A1 * np.exp(1j*Om0_1*k) + A2 * np.exp(1j*Om0_2*k)

    # DFT of the signal mixture
    mu = np.arange(N)
    X = np.fft.fft(x)

    # plot spectrum
    plt.figure(figsize = (10, 8))
    plt.stem(mu, abs(X), basefmt=' ')
    plt.title(r'Absolute value of the DFT of a signal mixture')
    plt.xlabel(r'$\mu$')
    plt.ylabel(r'$|X[\mu]|$')
    plt.axis([0, N, -0.5, N+5]);
    plt.grid()
```

Lets first consider the case that the frequencies of the two exponentials are rather far apart

```python
dft_signal_mixture(32, 1, 10.3, 1, 15.2)
```

Investigating the magnitude spectrum one could conclude that the signal consists of two major contributions at the frequencies $\mu_1 = 10$ and $\mu_2 = 15$. Now lets take a look at a situation where the frequencies are closer together

```python
dft_signal_mixture(32, 1, 10.3, 1, 10.9)
```

From visual inspection of the spectrum it is rather unclear if the mixture consists of one or two exponential signals. So far the levels of both signals where chosen equal.

Lets consider the case where the second signal has a much lower level that the first one. The frequencies have been chosen equal to the first example

```python
dft_signal_mixture(32, 1, 10.3, 0.1, 15.2)
```

Now the contribution of the second exponential is almost hidden in the spread spectrum of the first exponential. From these examples it should have become clear that the leakage effect limits the spectral resolution of the DFT.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


### Implementation of inversion of real valued Laplace transforms computable at any point.

References:
1. Kryzhniy V. V. On regularization method for numerical inversion of the Laplace transforms computable at any point on the real  axis. Journal of Inverse and Ill-Posed Problems,18(4), 2010
2. Kryzhniy V. V. Numerical inversion of the Laplace transform: Analysis via regularized analytic continuation. Inverse Problems 22, 2006
3. H. Bateman, A. Erdelyi, Higher Transcendental Functions, Volume 2
4. H. Bateman, A. Erdelyi, Tables of Integral Transforms, Volume 1
5.  W.H.Press at al, Numerical Recipes, any edition

As shown in [1, 2], the regularized inverse Laplace transform can be obtained by computing the following integral:

$f_R(t)=\frac{1}{t}\int_0^\infty F\left(\frac{v+\alpha}{t}\right)\Pi (R,v)\mathrm{d}v$,   (1)

where

$ \Pi(R,x) =\frac{1}{\pi}\mathcal{L}^{-1}\left[\frac{\sin(R\ln{p})}{p-1}w(p)\right]\enspace$ (2)

Multiple kernels can be obtained by selecting an approriate weigh function $w(p)$.

Appropriate values of free parametes can be found by minimizing the discrepancy between two inverses [1], which correspond to the following weght functions

$w_1(p)= p^a\mathrm{e}^{\alpha(1 - p)}, $ and
$w_2(p)= p^a\mathrm{e}^{\alpha(1- p)} \frac{2}{p+1},\quad \alpha \geq 0\enspace. $

The corresponding kernels $\Pi_1(R,x)$ and $\Pi_2(R,x)$, computed with the help of formula (5.4.8) in [4], involve the confluent hypergeometric functions.

For numerical implementation, we replace the hypergeometric functions with incomplete gamma functions using equality [3]:

$ _1F_1(1;1+a;z)z^a = a\mathrm{e}^z \gamma(a,z)$.

Then, it is straightforward to obtain:

$\Pi_1(R,x) = \frac{\mathrm{e}^\alpha}{\pi}\mathrm{Im}\left[\frac{\mathrm{e}^{x}\gamma(-a - iR,x)}{\Gamma(-a-iR)}\right]$,  and      $\Pi_2(R,x) = \Pi_1(R,x) -
\frac{\mathrm{e}^{\alpha}}{\pi}\mathrm{Im}\left[\frac{\mathrm{e}^{-x}(-1)^{-a - iR)}\gamma(-a - iR,-x)}{\Gamma(-a-iR)}\right]$

These formulae can be used for relatively small $x$. For large $x$ we use an equality:

$\gamma(-a-iR,x) = \Gamma(-a-iR) - \Gamma(-a -iR,x)$.

Then, after dropping a large real term we get:

$\Pi_1(R,x) = -\frac{\mathrm{e}^\alpha}{\pi}\mathrm{Im}\left[\frac{\mathrm{e}^{x}\Gamma(-a - iR,x)}{\Gamma(-a-iR)}\right]$

Functions $\gamma(-a-ir,x)$ and $\Gamma(-a-iR,x)$ are computed using standard formulae [3, 5].

For large $x$, the function $\gamma(-a-iR,-x)$ in $\Pi_2(R,x)$ is computed using asympotic expansion [3]

### Limitations of inversion of real-valued Laplace transforms
Limitations are illustrated by examples from [1].
For a limited precision of imput data, the inversion of real-valued Laplace transforms yields satisfactory results where the original function $ f(t)$ rapidly tends to a smooth and monotonic function as  $t \rightarrow \infty $.

Two consecutive extrema of the exact inverse transfirm located at $t_1$ and $t_2 > t_1$ can be resolved
only when

$ t_2/t_1 > exp(\pi/r)$ [2]  (**).

For double precision of input  data $ r \approx 20 $.

It follows from (**)
that oscillating functions of frequency $\omega $ can be inverted only for small $  t < t_{max} = r/\omega $

These limitations are common for all real-valued methods [2], and are inherent in the problem itself.

```python
import matplotlib.pyplot as plt
import scipy.special
import numpy as np

import InvertLT
ilt = InvertLT.InvertLT()

```

```python
def F1(x):
    return np.log(x)/x
def f1(t):
    return -np.log(t) - 0.57721566490153
t = np.linspace(0.01,100,endpoint=True)
ret = ilt.invert(F1,t,digits = 15, pw = 1)
ret = ilt.invert(F1,t,params = (-0.70021853, 0, 13.85404401))
ilt_num = ret[0]
ilt_exact = f1(t)
plt.plot(t,ilt_exact,'b',label = 'ilt_exact')
plt.plot(t,ilt_num,'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_1(t)$')
plt.title(r'$f_1(t) = log(t) - \gamma$')
plt.grid(True)
plt.legend()
plt.show()

```

```python
#reuse computed parameters
#computing the inverse at different set of points
#we get quite good results because $f_1(t)$ is a monotonic function
t = np.linspace(0.01,200,num=200,endpoint=True)
ret = ilt.invert(F1,t,params = ret[1])
plt.plot(t,f1(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_1(t)$')
plt.title(r'$f_1(t) = log(t) - \gamma$')
plt.grid(True)
plt.legend()
plt.show()
```

```python
def F2(x):
    return np.exp(1/x)/x
def f2(t):
    return scipy.special.iv(0,2*np.sqrt(t))
t = np.linspace(0.01,15)
ret = ilt.invert(F2,t,15,np.inf)
plt.plot(t,f2(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_2(t)$')
plt.title(r'$f_2(t) = I_0(2\sqrt{t})$')
plt.grid(True)
plt.legend()
plt.show()
```

```python
#reuse parameters, expand t
t = np.linspace(0.01,30,num=200)
ret = ilt.invert(F2,t,params = ret[1])
plt.plot(t,f2(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_2(t)$')
plt.title(r'$f_2(t) = I_0(2\sqrt{t})$')
plt.grid(True)
plt.legend()
plt.show()
```

```python
def F3(x):
    return  256/np.power((4+x),4)
def f3(t):
    return 128*np.power(t,3)*np.exp(-4*t)/3
t = np.logspace(-2,1,num = 100)
ret = ilt.invert(F3,t,15,0)
plt.semilogx(t,f3(t),'b',label = 'ilt_exact')
plt.semilogx(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_3(t)$')
plt.title(r'$f_3(t) = \frac{128}{3}t^{3}\mathrm{e}^{-4t}$')
plt.grid(True)
plt.legend()
plt.show()
```

```python
def F4(x):
    return np.exp(-np.sqrt(x/2))
def f4(t):
    return np.exp(-0.125/t)/(2*t*np.sqrt(2*np.pi*t))
t = np.logspace(-2,1,num = 100)
ret = ilt.invert(F4,t,15,0)
plt.semilogx(t,f4(t),'b',label = 'ilt_exact')
plt.semilogx(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_4(t)$')
plt.title(r'$f_4(t) = \frac{exp(-1/8t)}{2t\sqrt{2\pi t}}$')
plt.grid(True)
plt.legend()
plt.show()
```

```python
def F4a(x):
    return F3(x) + F4(x)
def f4a(t):
    return f3(t) + f4(t)
t = np.logspace(-2,1,num = 100)
ret = ilt.invert(F4a,t,15,0)
plt.semilogx(t,f4a(t),'b',label = 'ilt_exact')
plt.semilogx(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_{4a}(t)$')
plt.title('Example 4a')
plt.grid(True)
plt.legend()
plt.show()

```

For monotonic functions we can invert Laplace transforms computed numerically.

Consider the heat equation:
\begin{equation}
 \frac{ \partial U(x,t)}{\partial t} = \frac{\partial^2 U(x,t)}{\partial x^2},\quad 0 \leq x \leq 1,\quad t\geq 0 \enspace,
\end{equation}
with the initial and boundary conditions:
\begin{equation}
 U(x,0) = 0, \quad U(0,t) = 0, \quad \frac{ \partial U(1,t)}{\partial x} = 1 \enspace.
\end{equation}
Applying Laplace transform we obtain
\begin{equation}
 \frac{\mathrm{d}^2 U(x,p)}{\mathrm{d}x^2} -p U(x,p) = 0, \quad U(0,p) = 0, \quad \frac{ \mathrm{d}U(1,p)}{\mathrm{d} x} = \frac{1}{p} \enspace (*)
 \end{equation}

 Analytical solution yields the Laplace transform $\frac{\sinh(x\sqrt{p})}{p\sqrt{p}\cosh{\sqrt{p}}} $. For testing, we solve the equation (*) numerically using FDM with step $h=1/32$. Then, numerically computed Laplace transform is inverted for $x = 0.5$. We assume that such computed Laplace transform has 7 accurate digits.

```python
def F5(x):
    '''
        Finite differences method for
        Y''(x) -pY=0; Y(0)=0; Y'(1)=1;
        n = 32 knots
        output Y(0.5)
    '''
    if x == 0:
        return  x
    n = 32
    h = 1.0 / n
    a = np.zeros(n + 2)
    b = np.zeros(n + 2)
    c = np.zeros(n + 2)
    d = np.zeros(n + 2)
    ksi = np.zeros(n + 2)
    eta = np.zeros(n + 2)
    y = np.zeros(n + 2)
    b[0] = 1.0
    for i in range(1, n+1, 1):
        a[i] = 1.0
        b[i] = 2 + x * h * h
        c[i] = 1.0
        d[i] = 0.0
        b[n] /=  2
        c[n] = 0.0
        d[n] = -h
        ksi[0] = 0.0
        eta[0] = 0.0
    for i in range(0, n + 1):
        zn = b[i] - a[i] * ksi[i]
        ksi[i + 1] = c[i] / zn
        eta[i + 1] = (a[i] * eta[i] - d[i]) / zn
        y[n] = 0.0
    for i in range(n, -1, -1):
        y[i] = ksi[i + 1] * y[i +1 ] + eta[i + 1]
    return y[16] / x

def f5(t):
    x = 0.5
    res = x
    ad = 1.0
    n = 0
    fct = -2
    while 1:
        d = (n + 0.5) * np.pi
        ad = fct * np.exp(-d * d * t) / (d * d)
        res += ad * np.sin(d * x)
        n += 1
        fct *= -1
        if max(abs(ad)) < 1.0E-12:
            break
    return res

t = np.linspace(0.01,2);
ret = ilt.invert(F5,t,7,1)
plt.plot(t,f5(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_5(t)$')
plt.title(r'Example 5')
plt.grid(True)
plt.legend()
plt.show()
```

```python
#Oscillating function. Can be inverted accurately only for relatively small $t$
def F6(x):
    return np.exp(-x/np.sqrt(x*x+1))/x
def f6(t):
    '''see Duffy, Dean G. Transform methods for solving partial differential equations.
        Chapman & Hall/CRC, 2004
    '''
    nn = 1000
    x = 1.0
    b = 1.0
    sm = 0
    eta = 0
    d_eta = b / nn
    for n in range(0, nn -1, 2):
        if n == 0:
            sm = np.cos(eta * t) * x / b
        else:
            sm += np.cos(eta * t) * np.sin( x* eta / np.sqrt(b * b - eta * eta)) / eta
        eta += d_eta
        sm += 4 * np.cos(eta * t) * np.sin( x * eta / np.sqrt(b * b - eta * eta)) / eta
        eta += d_eta
        if eta < b:
            sm += np.cos(eta * t) * np.sin(x * eta / np.sqrt(b * b - eta * eta)) / eta
    return  1 - 2 * d_eta * sm.real / (3 * np.pi)
t = np.linspace(0.1,20,num = 100)
ret = ilt.invert(F6,t,15,1)
plt.plot(t,f6(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_5(t)$')
plt.title(r'Example 6')
plt.grid(True)
plt.legend()
plt.show()
```

```python
#for large $t$, obtained inverse tends to a constant
t = np.linspace(0.1,40,num = 100)
ret = ilt.invert(F6,t,params = ret[1])
plt.plot(t,f6(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_6(t)$')
plt.title(r'Example 6a')
plt.grid(True)
plt.legend()
plt.show()
```

```python
def F7(x):
    return 1/np.sqrt(x*x+1)
def f7(t):
    return scipy.special.jv(0,t)
t = np.linspace(0.01,20)
ret = ilt.invert(F7,t,15,0)
plt.plot(t,f7(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_7(t)$')
plt.title(r'Example 7')
plt.grid(True)
plt.legend()
plt.show()
```

```python
#Oscillating function.
#Results are not accurate for large $t$
t = np.linspace(0.01,40,num = 100)
ret = ilt.invert(F7,t,params = ret[1])
plt.plot(t,f7(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_7(t)$')
plt.title(r'Example 7a')
plt.grid(True)
plt.legend()
plt.show()
```

```python
#sum of delta functions
peaks = [(1,0.5), (2,0.8), (4,2), (100,20)]
def F8(x):
    ret = 0
    for ampl,t0 in peaks:
        ret += ampl*np.exp(-t0*x)
    return ret
def sum_delta_r(t):
    def delta_r(tau):
        global a, alpha, r
        dlt_r = np.exp(-alpha*tau/t)*np.sin(r*np.log(tau/t))*np.power(tau/t, a )/(tau - t)
        return dlt_r*np.exp(alpha)/np.pi
    ret = np.zeros_like(t)
    for ampl,t0 in peaks:
        ret += ampl*delta_r(t0)
    return ret
t = np.logspace(-1,2, num = 500)
ret = ilt.invert(F8,t,params = (- 0.5, 0, 20))
a = ret[1][0]
alpha = ret[1][1]
r = ret[1][2]
plt.semilogx(t,sum_delta_r(t),'b',label = 'ilt_theoretical')
plt.semilogx(t,ret[0],'r--',label='ilt_num')
plt.xlabel('$t$')
plt.ylabel('inverse')
plt.title('')
plt.grid(True)
plt.legend()
plt.show()
```

```python
#pulse function
def F9(x):
    return (np.exp(-x) - np.exp(-2*x))/x
def f9(t):
    return (t >= 1)*(t <= 2)
t = np.logspace(-1,1, num = 100)
ret = ilt.invert(F9,t,15,1)
plt.semilogx(t,f9(t),'b',label = 'ilt_exact')
plt.semilogx(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_=9(t)$')
plt.title('Example 9')
plt.grid(True)
plt.legend()
plt.show()
```

```python
def F10(x):
    return x**2/((x**2 +0.25)**3)
def f10(t):
    t1 = t/2
    return (1 +t1**2)*np.sin(t1) - t1*np.cos(t1)
t = np.linspace(0.01,30,endpoint=True)
ret = ilt.invert(F10,t,15,1)
plt.plot(t,f10(t),'b',label = 'ilt_exact')
plt.plot(t,ret[0],'r--',label='ilt_num')
plt.xlabel(r'$t$')
plt.ylabel(r'$f_{10}(t)$')
plt.title('Example 10')
plt.grid(True)
plt.legend()
plt.show()
```

```python

```