 # **Model project**
 ***

# The standard static labour supply model

In this model project, we present and solve a standard two-good static labour supply model, where agents value leisure and consumption. In the model presented an individual faces exogenously determined wage and non-labour income. The model is solved in three ways: theoretically, analytically and numerically. For the numerical optimisation, the chosen algorithm is a sequential (least-squares) quadratic programming (SQP) algorithm (SLSQP). Besides the simple model, we extend the model in two ways by including a proportional income tax and a progressive income tax. The size of the tax payment affects the income and therefore also hours worked and consumption level.

In part 2, we introduce the standard model and solve it theoretically, analytically and numerically. In part 3 we add a proportional tax to the standard model and in part 4 the model is solved with a progressive tax.

## The model

### The general assumptions

The agent only gets utility from enjoying leisure and consumption.

The utility function is a standard quasiconcave and monotone utility function: $U(C,L)$ with $U'_C>0$ and $U'_L>0$

The individual earn W per hour and the price of the consumption good is P. P is set as numeraire.

The real wage can be denoted by $w = \frac{W}{P}$

The individual has non-labour income $\tilde{R}$, which in real terms will be expressed by $R = \frac{\tilde{R}}{P}$

### The utility function

For simplicity we assume that the utility function is a Cobb Douglas function: $$U(C,L)=C^{\alpha} L^{1-\alpha},$$ where $0 <\alpha < 1$. The agent gets utility from consumption, $C$, and leisure, $L$.

### The budget constraint

The agent has $T$ hours available for consumption, $C$, and leisure, $L$. Hours spend on leisure is hours spend not working, i.e. hours worked is given by $h = T - L$.

The budget constraint is given by: $$C \leq wh+R \implies C \leq w(T - L) + R$$

### The agent's maximisation problem

The agent wishes to maximise:$$max_{C,L} U(C,L)=C^{\alpha} L^{1-\alpha}$$ subject to: $$C \leq w(T - L) + R$$

Since we are only interested in an interior solution we can replace the greater and equal sign with an equality: $$C=w(T-L) + R$$

This means that we only look at the intensive labour supply i.e. hours worked given employment.

## Solving the model using Python

Before solving the model using Python, we import the relevant packages for this project and a package used for pretty printing of our results.

```python
import scipy as sp
from scipy import optimize
import scipy.integrate as integrate
import numpy as np
import sympy as sm
import matplotlib.pyplot as plt

# for pretty printing
sm.init_printing(use_unicode=True)
```

### Analytically

We start by solving the model analytically.

Defining symbols:

```python
C = sm.symbols('C')
L = sm.symbols('L')
R = sm.symbols('R')
W = sm.symbols('W')
T = sm.symbols('T')
alpha = sm.symbols('alpha')
Lmin = sm.symbols('Lmin')
```

Defining the objective function (utility function) and budget constraint:

```python
objective = C**(alpha)*L**(1-alpha)
objective
```

```python
budget_constraint= sm.Eq(W*(T-L) + R,C)
budget_constraint
```

Solving in **four steps**:

1. **Isolate** $L$ from the budget constraint
2. **Substitute** in $L$
3. **Take the derivative** wrt. $C$
4. **Solve the FOC** for $C$

**Step 1: Isolate**

```python
L_from_con = sm.solve(budget_constraint,L)
L_from_con
```

**Step 2: Substitute**

```python
objective_subs = objective.subs(L,L_from_con[0])
objective_subs
```

**Step 3: Take the derivative**

```python
foc_C = sm.diff(objective_subs,C)
foc_C
```

**Step 3: Solve the FOC**

```python
sol_C = sm.solve(sm.Eq(foc_C,0),C)
sol_C
```

Substituting $C^*$ into the expression for L obtained in step 1 yields $L^*$:

```python
L_star = sm.simplify(L_from_con[0].subs(C,sol_C[0]))
L_star
```

This means that the analytical solution to the problem is given by: $$(C^*,L^*) = \left(\alpha(R+Tw),\frac{R + Tw - \alpha(R+Tw)}{w}\right)$$

### Numerically

Most estimators require the optimisation (maximisation or minimisation) of some objective function. However, in most cases, optimisation problems are too complicated to be solved analytically and therefore we can use numerical optimisation.

Numerical optimisation requires the use of an algorithm. The definition of an algorithm is an iterative process that generates a series $x_0,x_1,...,x_n$ that converges to the local minimum (or maximum) $\hat{x}$. It is defined by a starting point $x_0$ (initial guess), an iterative formula (how to move from $x_n$ to $x_n+1$) and a stopping rule.

The iterative formula is the centrepiece of an algorithm as it states how the iterative method updates the current estimate of x.

In this project, the chosen algorithm for the numerical optimisation is called a sequential (least-squares) quadratic programming (SQP) algorithm (SLSQP). The algorithm works well for constrained nonlinear optimisation. The drawback is that it is not as efficient as e.g. the BFGS optimisation method. The reason why we use the SLSQP anyway is due to the extension later on in the project where we have a non-linear budget constraint, and for consistency, we have chosen this algorithm for all parts of this project.

We start by defining the utility function and the budget constraint as python variables. Further, we also specify a value for T, R and W.

```python
# a. define utility function
def CD_utility(x, parameters):
    c=x[0]
    l=x[1]
    alpha=parameters

    u= (c**alpha)*l**(1-alpha)
    return -u

# b. define budget constraint
def eq_constraint(x):
    return w*(T-x[1]) + R - x[0]

# c. setting values for T, R and w
T=14 # hours available
R=200 # non-labour income
w=150 # wage rate
```

We have added a minus in front of the utility function above since we are using `scipy.optimize` to maximise utility subject to the budget constraint. However, `scipy.optimize` does only have a minimisation attribute, which is the reason we have added a minus in front of the utility function. Further, x[0] corresponds to consumption and x[1] corresponds to leisure.

We have set hours available equal to 14 ($T=14$) meaning the agent can maximum choose to work 14 hours. The non-labour income is equal to 200 ($R=200$). Lastly, the wage rate is set equal to 150 ($w=150$).

```python
# a. setup
eq_con = {'type': 'eq', 'fun': eq_constraint}

# b. call optimizer
x0 = (5,5) # initial guess (a starting point)
alpha1=0.5 # setting a value for alpha
result = optimize.minimize(CD_utility,x0,args=alpha1,
                             method='SLSQP',
                             bounds=None,
                             constraints=eq_con,
                             options={'disp':True})

# c. storing the results
result_C1 = result.x[0]
result_L1 = result.x[1]

# d. printing the results
print('\n(C,L) = ',result.x)
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -93.8971068015958
                Iterations: 13
                Function evaluations: 53
                Gradient evaluations: 13

    (C,L) =  [1149.98802263    7.66674652]

The results are that the agent consumes 1150 and has 7.67 hours of leisure (and works 14 - 7.67 = 6.33 hours).

Inserting T=14, R=200, w=150 and $\alpha=0.5$ in the analytical solution yields:

$$(C,L) = \left(\alpha(R+Tw),\frac{R + Tw - \alpha(R+Tw)}{w}\right) = \left(0.5(200 + 14*150),\frac{200 + 14*150 - 0.5(200 + 14*150)}{150}\right) = (1150 , 7.67)$$

This shows that the analytical and numerical solution is the same (the deviations can be attributed to rounding and numerical precision). This means that the optimiser works and we can continue.

**The effect of $\alpha$ on consumption and leisure**

$\alpha$ is the budget share spent on consumption, and can also be a measure of the relative preference for consumption to leisure. The optimal bundle of consumption and leisure is, therefore, sensitive to the value chosen for $\alpha$. In the following, we show how the results for $C^*$ and $L^*$ changes when $\alpha$ changes (non-labour income and wage rate are held constant).

```python
# a. specifying different values for alpha
alphas = [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]

# b. setup
eq_con = {'type': 'eq', 'fun': eq_constraint}

x0 = (5,5) # initial guess (starting point)
result_C = [] # create empty vector for result of consumption
result_L = [] # create empty vector for result of hours of leisure

# c. call optimizer
for alpha in alphas: # for every value in 'alphas' optimise
    result = optimize.minimize(CD_utility,x0,args=alpha,
                             method='SLSQP',
                             bounds=None,
                             constraints=eq_con,
                             options={'disp':False})
    result_C.append(round((result.x[0]),1)) # saving the results for consumption
    result_L.append(round((result.x[1]),1)) # saving the results for leisure

# d. printing results
print('\nalpha = ', alphas,'\nC = ',result_C, '\nL = ', result_L)
```

    alpha =  [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8]
    C =  [460.0, 689.9, 920.0, 1150.0, 1380.0, 1610.0, 1840.0]
    L =  [12.3, 10.7, 9.2, 7.7, 6.1, 4.6, 3.1]

The results are plotted in the two figures below.

```python
# a. the figure size
fig = plt.figure(figsize=(8,4),dpi=100)

# b. left plot - choice of L given w
ax_left = fig.add_subplot(1,2,1)

ax_left.plot(result_L,alphas)

ax_left.set_title('1A. Hours of leisure chosen')
ax_left.set_xlabel('$L$')
ax_left.set_ylabel('alpha')
ax_left.grid(True)

# c. right plot - choice of C given w
ax_right = fig.add_subplot(1,2,2)

ax_right.plot(result_C,alphas)

ax_right.set_title('1B. Consumption level chosen')
ax_right.set_xlabel('$C$')
ax_right.set_ylabel('$alpha$')
plt.xlim(400,2000)
ax_right.grid(True)
```

The above results show that a smaller value of $\alpha$ leads to a larger amount of leisure at the expense of a lower consumption level. This makes intuitive sense since an individual with a low value for $\alpha$ values consumption less to leisure than an individual with a high value for $\alpha$. Hence, an individual with a low value for $\alpha$ will spent a lower share of their budget on consumption.

In what follows we will set $\alpha = 0.5$.

**The effect of w on consumption and leisure**

In the following we show how the results for $C^*$ and $L^*$ changes when the wage rate changes ($\alpha$ and non-labour income are held constant).

```python
# a. specifying different values for wages (ws)
ws = [25, 50, 75, 100, 125, 150, 175, 200, 225, 250, 275, 300]

# b. setup
eq_con = {'type': 'eq', 'fun': eq_constraint}
x0 = (5,5) # initial guess (starting point)
alpha=0.5
result_C = [] # create empty vector for result of consumption
result_L = [] # create empty vector for result of hours of leisure

# c. call optimizer
for w in ws: # for every value in 'ws' optimise
    result = optimize.minimize(CD_utility,x0,args=alpha1,
                             method='SLSQP',
                             bounds=None,
                             constraints=eq_con,
                             options={'disp':False})
    result_C.append(result.x[0]) #saving the results for consumption
    result_L.append(result.x[1]) #saving the results for leisure
```

```python
# plotting the results

# a. the figure size
fig = plt.figure(figsize=(8,4),dpi=100)

# b. left plot - choice of L given w
ax_left = fig.add_subplot(1,2,1)

ax_left.plot(result_L,ws)

ax_left.set_title('2A. Hours of leisure chosen')
ax_left.set_xlabel('$L$')
ax_left.set_ylabel('w')
ax_left.grid(True)

# c. right plot - choice of C given w
ax_right = fig.add_subplot(1,2,2)

ax_right.plot(result_C,ws)

ax_right.set_title('2B. Consumption level chosen')
ax_right.set_xlabel('$C$')
ax_right.set_ylabel('$w$')
ax_right.grid(True)
```

The left figure shows that an individual will enjoy more leisure when the wage rate decreases, which makes intuitive sense since the wage rate is the opportunity cost of leisure. However, the wage rate has to decrease quite a lot (from 300 to below 100) before the agent starts to enjoy substantially more leisure. The right figure shows that consumption increases linearly with the wage rate.

# The standard model with proportional income taxation

## The model

Inserting a proportional tax in the standard model means that all workers have to pay a tax for each dollar earned. This can both affect the hours worked and the labour force participation. However, we will continue to only look at labour supply on the intensive margin. It is the same utility function as in the standard model.

### The budget constraint

The budget constraint is now given by:
$$C=(1-\tau)w(T-L) + R,$$
where $\tau$ is the tax rate, $0< \tau \leq 1$. Since the tax is proportional to the wage you have to pay more in taxes the more you work (and hence earn).

## Solving the model numerically in Python

We use the same utility function defined in the standard model without taxation. The new budget constraint is defined below.

We set the tax rate equal to 30% ($\tau=0.30$).

```python
# a. defining the new budget constraint with a proportional tax
def eq_constraint2(x):
    return (1-tau)*w*(T-x[1]) + R - x[0]

# b. setting values for T, R, w and tau
T=14 # hours available
R=200 # non-labour income
w=150 # wage rate
tau=0.3 # tax rate
```

```python
# a. setup
eq_con2 = {'type': 'eq', 'fun': eq_constraint2}

# b. call optimizer
x0 = (5,5) # initial guess (starting point)
alpha1 = 0.5 # as defined in the standard model
result = optimize.minimize(CD_utility,x0,args=alpha1,
                             method='SLSQP',
                             bounds=None,
                             constraints=eq_con2,
                             options={'disp':True})

# c. storing the results
result_C2 = result.x[0]
result_L2 = result.x[1]

# d. printing the results
print('\n(C,L) = ',result.x)
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -81.48765608386502
                Iterations: 13
                Function evaluations: 52
                Gradient evaluations: 13

    (C,L) =  [834.98879454   7.95248767]

The results are that the agent consumes 835 and has 7.95 hours of leisure (and work 14 - 7.95 = 6.05 hours).

The model with proportional tax results in a lower level of consumption and a higher level of leisure compared to the model without tax. The proportional tax decreases the effective wage rate, which means that the effective opportunity cost of leisure decreases. This will increase the level of leisure when leisure is a normal good. Consumption decreases because total income falls when the effective wage rate decreases.

# The standard model with progressive income taxation

## The model

Inserting a progressive tax in the standard model gives us a non-linear budget constraint with varying tax rates depending on how much you work. In this case, you pay a tax rate of zero up to a threshold of certain hours worked and thereafter you pay a positive tax rate on hours worked above that threshold. It is the same utility function as in the standard model.

### The budget constraint

The budget constraint is now given by:
\\[
\begin{eqnarray*}
C&=&\begin{cases}
(1-\tau) w (T-L) + R + w \tau (T - L_{min}) & \text{for }L \leq L_{min}\\
w (T - L) +R & \text{for }L \geq L_{min}
\end{cases}\\
\end{eqnarray*}
\\]

This budget constraint states that if you work more hours (and spent fewer hours on leisure) than a certain minimum (maximum) the wage earned in those hours of work is taxed. If you work below the certain minimum your budget constraint is as in the standard model.

## Solving the model numerically using Python

We use the same utility function defined in the standard model without taxation. The new budget constraint is defined below.

We set the tax rate equal to 70% ($\tau_{top}=0.70$) and the threshold equal to 8 hours ($L_{min}=8$).

```python
# a. define the new budget constraint with a progressive tax
def eq_constraint3(x):
    if x[1] <= Lmin:
        eq_constraint3 = (1-tau)*w*(T-x[1]) + R + w*tau*(T-Lmin) - x[0]
    else:
        eq_constraint3 = w*(T-x[1]) + R - x[0]
    return eq_constraint3

# b. setting values for T, R, w, tau_top and Lmin
T=14 # hours available
R=200 # non-labour income
w=150 # wage rate
tau_top=0.7 # tax rate
Lmin=8 # after 8 hours of work, you will be taxed
```

```python
# a. setup
eq_con3 = {'type': 'eq', 'fun': eq_constraint3}

# b. call optimizer
x0 = (5,5) # initial guess (starting point)
alpha1 = 0.5 # as defined in the standard model
result = optimize.minimize(CD_utility,x0,args=alpha1,
                             method='SLSQP',
                             bounds=None,
                             constraints=eq_con3,
                             options={'disp':True})

# c. storing the results
result_C3 = result.x[0]
result_L3 = result.x[1]

# d. print the results
print('\n(C,L) = ',result.x)
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: -93.8083146672497
                Iterations: 20
                Function evaluations: 98
                Gradient evaluations: 20

    (C,L) =  [1099.99985106    8.00000099]

The results are that the agent consumes 1100 and has 8 hours of leisure (and works 14 - 8 = 6 hours).

The model with progressive tax results in a lower level of consumption and about half an hour more leisure compared to the model without tax. Compared to the model with proportional tax, the agent consumes more and spent a little bit more time on leisure.

Most developed countries use a progressive tax system meaning high-income agents have a higher marginal tax rate and hence pay more in taxes. In our model, an agent gets a higher income by working more hours as the wage rate is constant. We note that the optional hours of leisure is equal to the threshold. This indicates that the agents are unwilling to work at the high marginal tax rate (even though they would have in absence of the tax). In this case, the agents bunch at the kink earning the highest possible income without having to pay the high marginal tax rate.

# Summing up

In this project we have studied the two-good standard static labour supply model without tax, with a proportional tax and with a progressive tax. In the following we sum up the results by plotting the solutions for the three versions of the model.

The exogenous variables are the same as they have been throughout the project:

```python
T=14
R=200
w=150
alpha1=0.5
x0=(5,5)
tau = 0.3 #the constant tax rate in the model with proportional tax
tau_top = 0.7 #the top tax rate in the model with progressive taxation
Lmin=8
```

We start by defining the wage functions for the model without a tax, with a proportional income tax and a progressive income tax.

```python
def wage(L):
    return w

def wage_lump(L):
    return w*(1-tau)

def wage_prog(L):
    if L <= Lmin:
        return w*(1-tau_top)
    else:
        return w
```

We define the leisure expenditure function for the model without a tax, with a proportional income tax and a progressive income tax. Note that the leisure expenditure function for the model without a tax and with a proportional income tax only differ because of the different effective wage rate (i.e. we can use the same leisure expenditure function and inserting the different wage rates).

```python
def leisureexp(L,wage):
    return sp.integrate.quad(wage,0,L)[0]

def leisureexp_prog(L,wage):
    if L <= Lmin:
        return w*(1-tau_top)*L
    else:
        return w*(L-Lmin)+w*(1-tau_top)*Lmin
```

Taken into account the different functional forms for the wage function and leisure expenditure function the budget constraint function, utility for optimal consumption bundle and indifference curve can be defined as follows for both the model without a tax, with a proportional income tax and a progressive income tax.

```python
# a. defining budget constraint
def budget(x, wage, maxlabinc, leisureexp):
    C=x[0]
    L=x[1]
    maximum = maxlabinc + R
    budget= maximum - leisureexp(L, wage) - C
    return  budget

def budget_func(wage,maxlabinc,leisureexp):
    return {'type':'eq', 'fun':budget, 'args': [wage, maxlabinc,leisureexp]}

# b. defining the utility for optimal consumption bundle
def max_u(wage, maxlabinc, leisureexp, parameters):
    temp = optimize.minimize(CD_utility, x0, args=parameters, method='SLSQP',
                            constraints=[budget_func(wage, maxlabinc, leisureexp)], options={'disp':False})
    return -temp.fun

# c. defining the indifference curve
def indif_opt(max_u, parameters):
    c_indif=lambda L: max_u**(1/parameters)*L**((parameters-1)/parameters)
    return [c_indif(x) for x in np.linspace(0.01,14,100)]
```

The following code plots the model without a tax, with a proportional income tax and a progressive income tax, plus the respective solutions.

```python
# Specifying the figure size
fig = plt.figure(figsize=(8,4),dpi=100)

# A. FIGURE 1 - MODEL WITH NO TAX
# i. defining the maximum labour income with no tax
maxlabinc=leisureexp(T,wage)

# ii. defining the income potential in the model with no tax:
maximum = maxlabinc + R

# iii. defining the function for the budget line
goods = lambda L: maximum -leisureexp(L,wage)
cons = [goods(x) for x in np.linspace(0,14,100)]
leisure1 = np.linspace(0,14,100)

ax1 = fig.add_subplot(2,2,1)

# iv. plot the budget line with axis names
ax1.plot(leisure1,cons)

# v. plotting the indiffence curve
ax1.plot(leisure1,indif_opt(max_u=max_u(wage=wage,maxlabinc=maxlabinc,leisureexp=leisureexp,parameters=alpha1),parameters=alpha1))
ax1.set_xlabel("Leisure")
ax1.set_ylabel("Consumption")
ax1.set_title('3A. Labour supply model with no tax')
ax1.set_ylim(0,2500)
ax1.set_xlim(0,15)
plt.plot(result_L1, result_C1, '-ok')

# B. FIGURE 2 - MODEL WITH LUMSUMP TAX
# i. defining the maximum labour income with no tax
maxlabinc_lump=leisureexp(T,wage_lump)

# ii. defining the income potential in the model with no tax:
maximum_lump = maxlabinc_lump + R

# iii. defining the function for the budget line
goods_lump = lambda L: maximum_lump -leisureexp(L,wage_lump)
cons_lump = [goods_lump(x) for x in np.linspace(0,14,100)]
leisure_lump = np.linspace(0,14,100)

ax2 = fig.add_subplot(2,2,2)

# iv. plot the budget line with axis names
ax2.plot(leisure_lump,cons_lump)

# v. plotting the indiffence curve
ax2.plot(leisure_lump,indif_opt(max_u=max_u(wage=wage_lump,maxlabinc=maxlabinc_lump,leisureexp=leisureexp,parameters=alpha1),parameters=alpha1))
ax2.set_xlabel("Leisure")
ax2.set_ylabel("Consumption")
ax2.set_title('3B. Labour supply model with lumpsum tax')
ax2.set_ylim(0,2500)
ax2.set_xlim(0,15)
plt.plot(result_L2, result_C2, '-ok')

# C. FIGURE 3 - MODEL WITH PROGRESSIVE TAX

# i. defining the maximum labour income with progressive tax
maxlabinc_prog=leisureexp_prog(T,wage_prog)

# ii. defining the income potential in the model with progressive tax
maximum_prog=maxlabinc_prog + R

# iii. define the function for the budget line
goods_prog = lambda L: maximum_prog -leisureexp_prog(L,wage_prog)
cons_prog = [goods_prog(x) for x in np.linspace(0,14,100)]
leisure_prog = np.linspace(0,14,100)

ax3 = fig.add_subplot(2,2,3)

# iv. plot the budget line with axis names
ax3.plot(leisure_prog,cons_prog)

# v. plotting the indiffence curve
ax3.plot(leisure_prog,indif_opt(max_u=max_u(wage=wage_prog,maxlabinc=maxlabinc_prog,leisureexp=leisureexp_prog,
                                            parameters=alpha1),parameters=alpha1))
ax3.set_xlabel("Leisure")
ax3.set_ylabel("Consumption")
ax3.set_title('3C. Labour supply model with progressive tax')
ax3.set_ylim(0,2500)
ax3.set_xlim(0,15)
plt.plot(result_L3, result_C3, '-ok')
plt.plot((Lmin,Lmin),(0,goods(Lmin)),'k--')

plt.subplots_adjust(top=2.5,right=2)
```

Figure 3A shows the optimal bundle of consumption and leisure in the model without tax. Figure 3B shows the optimal bundle in the model with a proportional tax rate of 30%. This changes the slope of the budget constraint and limits the possible bundles of consumption and leisure for the agent. The proportional tax also decreases the effective wage rate, which means the opportunity cost of leisure decreases. This will increase the level of leisure when leisure is a normal good. From figure 3A and 3B we see that introducing a proportional tax increases hours of leisure with 0.28 hours. Consumption decreases by 315 because total income falls when the effective wage rate decreases.

Figure 3C shows the optimal bundle of consumption and leisure in the model with progressive taxation, where the wage for the hours of work above 6 hours (less than 8 hours of leisure) is taxed with 70%. This creates a kink in the budget line at L = 8, which is illustrated by the dashed line in the figure. The progressive tax increases hours of leisure with 0.33 hours compared to the model without tax and 0.05 hours compared to the model with proportional taxation. Consumption is increased by 265 compared to the model with proportional tax, and decrease by 50 compared to the model without taxation. This result indicates that agents are willing to give up a small amount of consumption to not work at the high marginal tax rate (i.e. they bunch at the threshold). According to the literature of taxation effects, this result is a general phenomenon. Agents with $L^*$ just above the threshold tend to bunch at the threshold. Only agents that value consumption a lot more than leisure will continue to consume leisure below the threshold even though they have to pay a higher marginal tax rate. Agents that consume more leisure than the threshold will not be affected by the introduction of a progressive tax system.


# Bayesian Linear Regression

```python
import sys
# Add the symgp folder path to the sys.path list
module_path = r'/Users/jaduol/Documents/Uni (original)/Part II/IIB/MEng Project/'
if module_path not in sys.path:
    sys.path.append(module_path)

from symgp import SuperMatSymbol, utils, MVG, Variable
from sympy import symbols, ZeroMatrix, Identity
from IPython.display import display, Math, Latex
```

```python
# Define some symbols
D, N, Ns = symbols('D N Ns')
sig_y = symbols('\u03c3_y')
```

## 1. Prior

```python
# Prior
w = Variable('w',D,1)
p_w = MVG([w],mean=ZeroMatrix(D,1),cov=Identity(D))

print("p_w:")
display(Latex(utils.matLatex(p_w)))
```

    p_w:

\begin{align*}
p\left(\mathbf{w}\right)&= \mathcal{N}\left(\mathbf{w};\mathbf{m}_{\mathbf{w}},\mathbf{\Sigma}_{\mathbf{w}}\right)\\
\mathbf{m}_{\mathbf{w}} &= \mathbf{0}\\
\mathbf{\Sigma}_{\mathbf{w}} &= \mathbf{I}\\
\end{align*}

# 2. Likelihood

```python
# Likelihood of w given X
X, y = utils.variables('X y',[(D,N), N])
p_y = MVG([y], mean=X.T*w,
               cov=sig_y**2*Identity(N),
               cond_vars=[w,X])

print("p_y:")
display(Latex(utils.matLatex(p_y)))
```

    p_y:

\begin{align*}
p\left(\mathbf{y}|\mathbf{w},\mathbf{X}\right)&= \mathcal{N}\left(\mathbf{y};\mathbf{m}_{\mathbf{y}|\mathbf{w},\mathbf{X}},\mathbf{\Sigma}_{\mathbf{y}|\mathbf{w},\mathbf{X}}\right)\\
\mathbf{m}_{\mathbf{y}|\mathbf{w},\mathbf{X}} &= \mathbf{X}^T \mathbf{w}\\
\mathbf{\Sigma}_{\mathbf{y}|\mathbf{w},\mathbf{X}} &= \sigma_y^{2} \mathbf{I}\\
\end{align*}

# 3. Posterior

```python
# Joint of w and y
p_w_y = p_w*p_y

print("p_w_y:")
display(Latex(utils.matLatex(p_w_y)))
```

    cond_vars:  {w}
    conditional_cond_vars:  {w, X}
    new_conditioned_vars:  [X]
    p_w_y:

\begin{align*}
p\left(\mathbf{y},\mathbf{w}|\mathbf{X}\right)&= \mathcal{N}\left(\left[\begin{smallmatrix}\mathbf{y}\\\mathbf{w}\end{smallmatrix}\right];\mathbf{m}_{\mathbf{y},\mathbf{w}|\mathbf{X}},\mathbf{\Sigma}_{\mathbf{y},\mathbf{w}|\mathbf{X}}\right)\\
\mathbf{m}_{\mathbf{y},\mathbf{w}|\mathbf{X}} &= \left[\begin{smallmatrix}\mathbf{0}\\\mathbf{0}\end{smallmatrix}\right]\\
\mathbf{\Sigma}_{\mathbf{y},\mathbf{w}|\mathbf{X}} &= \left[\begin{smallmatrix}\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}&\mathbf{X}^T\\\mathbf{X}&\mathbf{I}\end{smallmatrix}\right]\\
\end{align*}

```python
# Inference: posterior over w
p_w_post = p_w_y.condition([y])

print("p_w_post:")
display(Latex(utils.matLatex(p_w_post)))
```

    p_w_post:

\begin{align*}
p\left(\mathbf{w}|\mathbf{X},\mathbf{y}\right)&= \mathcal{N}\left(\mathbf{w};\mathbf{m}_{\mathbf{w}|\mathbf{X},\mathbf{y}},\mathbf{\Sigma}_{\mathbf{w}|\mathbf{X},\mathbf{y}}\right)\\
\mathbf{m}_{\mathbf{w}|\mathbf{X},\mathbf{y}} &= \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\\
\mathbf{\Sigma}_{\mathbf{w}|\mathbf{X},\mathbf{y}} &= \mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\\
\end{align*}

# 4. Prediction

```python
#Prediction

# Likelihood of w given Xs
Xs, ys = utils.variables('X_{*} y_{*}',[(D,Ns), Ns])
p_ys = MVG([ys], mean=Xs.T*w,
                 cov=sig_y**2*Identity(Ns),
                 cond_vars=[w,Xs])

print("p_ys:")
display(Latex(utils.matLatex(p_ys)))
```

    p_ys:

\begin{align*}
p\left(\mathbf{y_{*}}|\mathbf{w},\mathbf{X_{*}}\right)&= \mathcal{N}\left(\mathbf{y_{*}};\mathbf{m}_{\mathbf{y_{*}}|\mathbf{w},\mathbf{X_{*}}},\mathbf{\Sigma}_{\mathbf{y_{*}}|\mathbf{w},\mathbf{X_{*}}}\right)\\
\mathbf{m}_{\mathbf{y_{*}}|\mathbf{w},\mathbf{X_{*}}} &= \mathbf{X_{*}}^T \mathbf{w}\\
\mathbf{\Sigma}_{\mathbf{y_{*}}|\mathbf{w},\mathbf{X_{*}}} &= \sigma_y^{2} \mathbf{I}\\
\end{align*}

```python
# Joint of w and ys
p_w_ys = p_w_post*p_ys

print("p_w_ys:")
display(Latex(utils.matLatex(p_w_ys)))
```

    cond_vars:  {w}
    conditional_cond_vars:  {X_{*}, w}
    new_conditioned_vars:  [X_{*}, X, y]
    p_w_ys:

\begin{align*}
p\left(\mathbf{y_{*}},\mathbf{w}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}\right)&= \mathcal{N}\left(\left[\begin{smallmatrix}\mathbf{y_{*}}\\\mathbf{w}\end{smallmatrix}\right];\mathbf{m}_{\mathbf{y_{*}},\mathbf{w}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}},\mathbf{\Sigma}_{\mathbf{y_{*}},\mathbf{w}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}}\right)\\
\mathbf{m}_{\mathbf{y_{*}},\mathbf{w}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}} &= \left[\begin{smallmatrix}\mathbf{X_{*}}^T \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\\\mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\end{smallmatrix}\right]\\
\mathbf{\Sigma}_{\mathbf{y_{*}},\mathbf{w}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}} &= \left[\begin{smallmatrix}\sigma_y^{2} \mathbf{I} + \mathbf{X_{*}}^T \left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right) \mathbf{X_{*}}&\mathbf{X_{*}}^T \left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right)\\\left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right) \mathbf{X_{*}}&\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\end{smallmatrix}\right]\\
\end{align*}

```python
# Predictive distribution of ys
p_ys_post = p_w_ys.marginalise([w])

print("p_ys_post:")
display(Latex(utils.matLatex(p_ys_post)))
```

    self.name:  S_{y_{*},y_{*}|X_{*},X,y}
    name:
    p_ys_post:

\begin{align*}
p\left(\mathbf{y_{*}}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}\right)&= \mathcal{N}\left(\mathbf{y_{*}};\mathbf{m}_{\mathbf{y_{*}}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}},\mathbf{\Sigma}_{\mathbf{y_{*}}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}}\right)\\
\mathbf{m}_{\mathbf{y_{*}}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}} &= \mathbf{X_{*}}^T \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{y}\\
\mathbf{\Sigma}_{\mathbf{y_{*}}|\mathbf{X_{*}},\mathbf{X},\mathbf{y}} &= \sigma_y^{2} \mathbf{I} + \mathbf{X_{*}}^T \left(\mathbf{I} - \mathbf{X} \left(\sigma_y^{2} \mathbf{I} + \mathbf{X}^T \mathbf{X}\right)^{-1} \mathbf{X}^T\right) \mathbf{X_{*}}\\
\end{align*}

```python

```


+ This notebook is part of lecture 16 *Projection matrices and least squares* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

```python
from sympy import init_printing, Matrix, symbols
from IPython.display import Image
from warnings import filterwarnings
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Projection matrices and least squares

```python
Image(filename = 'Line.png')
```

## Least squares

* Consider from the previous lecture the three data point in the plain
$$ ({t}_{i},{y}_{i}) =(1,1), (2,2),(3,2) $$
* From this we need to construct a straight line
* This could be helpful say in, statistics (remember, though in statistics we might have to get rid of statistical outliers)
* Nonetheless (view image above) we note that we have a straight line in slope-intercept form
$$ {y}={C}+{Dt} $$
* On the line at *t* values of 1, 2, and 3 we will have
$$ {y}_{1}={C}+{D}=1 \\ {y}_{2}={C}+{2D}=2 \\ {y}_{3}={C}+{3D}=2 $$
* The actual *y* values at these *t* values are 1, 2, and 2, though
* We are thus including an error of
$$ \delta{y} \\ { \left( { e }_{ 1 } \right)  }^{ 2 }={ \left[ \left( C+D \right) -1 \right]  }^{ 2 }\\ { \left( { e }_{ 2 } \right)  }^{ 2 }={ \left[ \left( C+2D \right) -2 \right]  }^{ 2 }\\ { \left( { e }_{ 3 } \right)  }^{ 2 }={ \left[ \left( C+3D \right) -2 \right]  }^{ 2 } $$
* Since some are positive and some are negative (actual values below or above the line), we simply determine the square (which will always be positive)
* Adding the (three in our example here) squares we have the sum total of the error (which is actuall just the sqautre of the distance between the line and actual *y* values)
* The line will be the best fit when this error sum is at a minimum (hence *least squares*)
* We can do this with calculus or with linear algebra
* For calculus we take the partial derivatives of both unknowns and set to zero
* For linear algebra we project orthogonally onto the columnspace (hence minimizing the error)
    * Note that the solution **b** does not exist in the columnspace (it is not a linear combination of the columns)

### Calculus method

* We'll create a function *f*(C,D) and successively take the partial derivatives of both variables and set it to zero
* We fill then have two equation with two unknowns to solve (which is easy enough to do manually or by simple linear algebra and row reduction)

```python
C, D = symbols('C D')
```

```python
e1_squared = ((C + D) - 1) ** 2
e2_squared = ((C + 2 * D) - 2) ** 2
e3_squared = ((C + 3 * D) - 2) ** 2
f = e1_squared + e2_squared + e3_squared
f
```

```python
f.expand() # Expanding the expression
```

* Doing the partial derivatives will be
$$ f\left( C,D \right) =3{ C }^{ 2 }+12CD-10C+14{ D }^{ 2 }-22D+9\\ \frac { \partial f }{ \partial C } =6C+12D-10=0\\ \frac { \partial f }{ \partial D } =12C+28D-22=0 $$

```python
f.diff(C) # Taking the partial derivative with respect to C
```

```python
f.diff(D) # Taking the partial derivative with respect to D
```

* Setting both equal to zero (and creating a simple augmented matrix) we get
$$ 6C+12D-10=0\\ 12C+28D-22=0\\ \therefore \quad 6C+12D=10\\ \therefore \quad 12C+28D=22 $$

```python
A_augm = Matrix([[6, 12, 10], [12, 28, 22]])
A_augm
```

```python
A_augm.rref() # Doing a Gauss-Jordan elimination to reduced row echelon form
```

* We now have a solution
$$ {y}=\frac{2}{3} + \frac{1}{2}{t}$$

### Linear algebra

* We note that we can construct the following
$$ {C}+{1D}={1} \\ {C}+{2D}={2} \\ {C}+{3D}={2} \\ {C}\begin{bmatrix} 1 \\ 1\\ 1 \end{bmatrix}+{D}\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}=\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} \\ A\underline { x } =\underline { b } \\ \begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix}\begin{bmatrix} C \\ D \end{bmatrix}=\begin{bmatrix} 1 \\ 2 \\ 2 \end{bmatrix} $$
* **b** is not in the columnspace of A and we have to do orthogonal projection
$$ { A }^{ T }A\hat { x } ={ A }^{ T }\underline { b } \\ \hat { x } ={ \left( { A }^{ T }A \right)  }^{ -1 }{ A }^{ T }\underline { b }  $$

```python
A = Matrix([[1, 1], [1, 2], [1, 3]])
b = Matrix([1, 2, 2])
A, b # Showing the two matrices
```

```python
x_hat = (A.transpose() * A).inv() * A.transpose() * b
x_hat
```

* Again, we get the same values for C and D

* Remember the following
$$ \underline{b} = \underline{p}+\underline{e} $$
* **p** and **e** are perpendicular
* Indeed **p** is in the columnspace of A and **e** is perpendicular to the columspace (or any vector in the columnspace)

## Example problem

### Example problem 1

* Find the quadratic (second order polynomial) equation through the origin, with the following data points: (1,1), (2,5) and (-1,-2)

#### Solution

* Let's just think about a quadratic equation in *y* and *t*
$$ {y}={c}_{1} +{C}{t}+{D}{t}^{2} $$
* Through the origin (0,0) means *y* = 0 and *t* = 0, thus we have
$$ {0}={c}_{1} +{C}{0}+{D}{0}^{2} \\ {c}_{1}=0 \\ {y}={C}{t}+{D}{t}^{2} $$

* This gives us three equation for our three data points
$$ C\left( 1 \right) +D{ \left( 1 \right)  }^{ 2 }=1\\ C\left( 2 \right) +D{ \left( 2 \right)  }^{ 2 }=5\\ C\left( -1 \right) +D{ \left( -1 \right)  }^{ 2 }=-2\\ C\begin{bmatrix} 1 \\ 2 \\ -1 \end{bmatrix}+D\begin{bmatrix} 1 \\ 4 \\ 1 \end{bmatrix}=\begin{bmatrix} 1 \\ 5 \\ -2 \end{bmatrix}\\ A=\begin{bmatrix} 1 & 1 \\ 2 & 4 \\ -1 & 1 \end{bmatrix}\\ \underline { x } =\begin{bmatrix} C \\ D \end{bmatrix}\\ \underline { b } =\begin{bmatrix} 1 \\ 5 \\ -2 \end{bmatrix} $$

* Clearly **b** is not in the columnspace of A and we have to project orthogonally onto the columnspace using
$$ \hat { x } ={ \left( { A }^{ T }A \right)  }^{ -1 }{ A }^{ T }\underline { b } $$

```python
A = Matrix([[1, 1], [2, 4], [-1, 1]])
b = Matrix([1, 5, -2])
x_hat = (A.transpose() * A).inv() * A.transpose() * b
x_hat
```

* Here's a simple plot of the equation

```python
import matplotlib.pyplot as plt # The graph plotting module
import numpy as np # The numerical mathematics module
%matplotlib inline
```

```python
x = np.linspace(-2, 3, 100) # Creating 100 x-values
y = (41 / 22) * x + (5 / 22) * x ** 2 # From the equation above
plt.figure(figsize = (8, 6)) # Creating a plot of the indicated size
plt.plot(x, y, 'b-') # Plot the equation above , in essence 100 little plots using small segmnets of blue lines
plt.plot(1, 1, 'ro') # Plot the point in a red dot
plt.plot(2, 5, 'ro')
plt.plot(-1, -2, 'ro')
plt.plot(0, 0, 'gs') # Plot the origin as a green square
plt.show(); # Create the plot
```

```python

```


# Matplotlib: Hydrogen Wave Function
---

The normalized hydrogen wave function

\begin{equation}
    \label{eq:hydro_wfc}
    \psi_{nlm}(r,\theta,\phi)
    =
    \sqrt{
        \left(\frac{2}{na_0}\right)^3\, \frac{(n-l-1)!}{2n[(n+l)!]}
    }
    e^{-r / na_0}
    \left(\frac{2r}{na_0}\right)^l
    L_{n-l-1}^{2l+1} \cdot Y_l^m(\theta, \phi)
\end{equation}

https://en.wikipedia.org/wiki/Hydrogen_atom

```python
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.ticker import AutoMinorLocator

from sympy import lambdify, integrate, oo
from sympy.abc import r, theta, phi
from sympy.physics.hydrogen import R_nl, Psi_nlm

plt.style.use('dark_background')
mpl.rcParams['mathtext.fontset'] = 'cm'

def cart2sphe(x, y, z):
    '''
    3D Cartesian coordinates to spherical coordinates.

    input:
        x, y, z : numpy arrays
    '''
    xy2     = x**2 + y**2
    r       = np.sqrt(xy2 + z**2)
    theta   = np.arctan2(np.sqrt(xy2), z) # the polar angle in radian angles
    phi     = np.arctan2(y, x)            # the azimuth angle in radian angles
    phi[phi < 0] += np.pi * 2             # np.arctan2 returns the angle in the range [-pi, pi]

    return r, theta, phi

################################################################################
if __name__ == "__main__":
    Nmax = 4
    nlms  = [
        (n, l, m)
        for n in range(1, Nmax+1)
        for l in range(n)
        for m in range(l+1)
    ]
    PSI  = [
        lambdify((r, phi, theta), Psi_nlm(n, l, m, r, phi, theta), 'numpy')
        for n, l, m in nlms
    ]

    ################################################################################
    # print(plt.figaspect(2 / Nmax),)
    fig = plt.figure(
        figsize=(9.6, 4.8),
        # figsize=plt.figaspect(2 / Nmax),
        dpi=300,
        constrained_layout=True
    )

    axes_array = np.zeros((Nmax, (Nmax*(Nmax+1))//2), dtype=int)
    axes_id    = 0
    for ii in range(Nmax):
        lm_sum = ((ii+1) * (ii+2)) // 2
        for jj in range(0, lm_sum):
            axes_id += 1
            axes_array[ii, -(jj+1)] = axes_id

    axes = fig.subplot_mosaic(
        axes_array,
        empty_sentinel=0,
    )
    axes = [axes[ii+1] for ii in range(len(axes))]

    cut_planes = np.ones(len(axes), dtype=int) * 2
    ################################################################################

    for ii in range(len(axes)):
        n, l, m = nlms[ii]

        # L       = np.ceil(integrate(R_nl(n, l, r, Z=1)**2 * r**3, (r, 0, oo)) / 10) * 10
        # L       = integrate(R_nl(n, l, r, Z=1)**2 * r**3, (r, 0, oo)) * 2
        # N       = 100
        # print(n, l, L)

        if n == 1:
            L, N = 1.5*n**2 + 1.0, 100
        else:
            L, N = 1.5*n**2 + 5.0, 100

        a0, b0 = np.mgrid[
            -L:L:N*1j,
            -L:L:N*1j
        ]
        nx, ny = a0.shape
        xyz_planes_sphe = [
            cart2sphe(a0, b0, 0),
            cart2sphe(0, a0, b0),
            cart2sphe(a0, 0, b0),
        ]

        ax = axes[ii]
        ax.set_aspect(1.0)

        r0, theta, phi = xyz_planes_sphe[cut_planes[ii]]
        wfc = PSI[ii](r0, phi, theta).reshape(nx, ny)
        ax.pcolormesh(a0, b0, np.abs(wfc), cmap='magma')

        ax.set_xlim(-L, L)
        ax.set_ylim(-L, L)

        STEP = 5
        if n == 1:
            ax.set_xticks([-2.5, 2.5])
            ax.set_yticks([-2.5, 2.5])
            ax.xaxis.set_minor_locator(AutoMinorLocator(n=2))
            ax.yaxis.set_minor_locator(AutoMinorLocator(n=2))
        else:
            ax.set_xticks([-x for x in np.arange(0, L, STEP)] + [x for x in np.arange(0, L, STEP)])
            ax.set_yticks([-x for x in np.arange(0, L, STEP)] + [x for x in np.arange(0, L, STEP)])
        ax.set_xticklabels([])
        ax.set_yticklabels([])

        ax.text(0.05, 0.05, "({}, {}, {})".format(n, l, m),
                fontsize='xx-small',
                ha='left', va='bottom', transform=ax.transAxes, color='w')

    fig.text(0.0, 1.00,  r'Hydrogen Wave Function',
            fontsize=28,
            ha='left', va='top', transform=fig.transFigure, color='w')

    fig.text(0.0, 0.90,  r'Probability Density',
            fontsize=20,
            ha='left', va='top', transform=fig.transFigure, color='w')

    fig.text(0.0, 0.80,  r'$ \psi_{nlm}(r,\theta,\phi) = \sqrt{ \left(\frac{2}{na_0}\right)^3\, \frac{(n-l-1)!}{2n[(n+l)!]} } e^{-r / na_0} \left(\frac{2r}{na_0}\right)^l L_{n-l-1}^{2l+1}\cdot Y_l^m(\theta, \phi)$',
            fontsize='xx-large',
            ha='left', va='top', transform=fig.transFigure, color='w')

    # fig.text(0.00, 0.64,  "NOTE:\n\t1. The distance between two major ticks in the plot is $5\,a_0$,\n\t2. The number in the braces are the three quantum numbers $(n, l, m)$",
    #         fontsize='small',
    #         linespacing=1.8,
    #         ha='left', va='top', transform=fig.transFigure, color='w')

    fig.text(0.00, 0.64,  "NOTE:",
            fontsize='small',
            ha='left', va='top', transform=fig.transFigure, color='w')

    fig.text(0.00, 0.59,  "    1. The distance between two major ticks in the plot is $5\,a_0$.\n    2. The numbers in the brace are the three quantum numbers $(n,\,l,\,m).$",
            fontsize='small',
            linespacing=1.5,
            ha='left', va='top', transform=fig.transFigure, color='w')

    plt.savefig('hydro_wfc.png')
```

**Figure 1.** Norm of hydrogen wave function $\psi_{nlm}(r,\theta,\phi)$ projected on the $xy$-plane, i.e. $z=0$. Note that the size of the box in each plot is different, where the distance between two major ticks (long thick sticks) is $5a_0$, i.e. 5 times the Bohr radius. The three quantum numbers for each wave function are show in braces. Also, the reason that norm of the wave function was shown is that the nodes, i.e. zeros of the wave functions, can be barely discerned if norm-squared of the wave functions were used.

```python

```


# The Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Properties

The Fourier transform has a number of specific properties. They can be concluded from its definition. The most important ones in the context of signals and systems are reviewed in the following.

### Invertibility

According to the [Fourier inversion theorem](https://en.wikipedia.org/wiki/Fourier_inversion_theorem), for many types of signals it is possible to recover the signal $x(t)$ from its Fourier transformation $X(j \omega) = \mathcal{F} \{ x(t) \}$

\begin{equation}
x(t) = \mathcal{F}^{-1} \left\{ \mathcal{F} \{ x(t) \} \right\}
\end{equation}

A sufficient condition for the theorem to hold is that both the signal $x(t)$ and its Fourier transformation are absolutely integrable and $x(t)$ is continuous at the considered time $t$. For this type of signals, above relation can be proven by applying the definition of the Fourier transform and its inverse and rearranging terms. However, the invertibility of the Fourier transformation holds also for more general signals $x(t)$, composed for instance from Dirac delta distributions.

**Example**

The invertibility of the Fourier transform is illustrated at the example of the [rectangular signal](../continuous_signals/standard_signals.ipynb#Rectangular-Signal) $x(t) = \text{rect}(t)$. The inverse of [its Fourier transform](definition.ipynb#Transformation-of-the-Rectangular-Signal) $X(j \omega) = \text{sinc} \left( \frac{\omega}{2} \right)$ is computed to show that the rectangular signal, although it has discontinuities, can be recovered by inverse Fourier transformation.

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

def fourier_transform(x):
    return sym.transforms._fourier_transform(x, t, w, 1, -1, 'Fourier')

def inverse_fourier_transform(X):
    return sym.transforms._fourier_transform(X, w, t, 1/(2*sym.pi), 1, 'Inverse Fourier')
```

```python
t, w = sym.symbols('t omega')

X = sym.sinc(w/2)
x = inverse_fourier_transform(X)
x
```

```python
sym.plot(x, (t,-1,1), ylabel=r'$x(t)$');
```

### Duality

Comparing the [definition of the Fourier transform](definition.ipynb) with its inverse

\begin{align}
X(j \omega) &= \int_{-\infty}^{\infty} x(t) \, e^{-j \omega t} \; dt \\
x(t) &= \frac{1}{2 \pi} \int_{-\infty}^{\infty} X(j \omega) \, e^{j \omega t} \; d\omega
\end{align}

reveals that both are very similar in their structure. They differ only with respect to the normalization factor $2 \pi$ and the sign of the exponential function. The duality principle of the Fourier transform can be deduced from this observation. Let's assume that we know the Fourier transformation $x_2(j \omega)$ of a signal $x_1(t)$

\begin{equation}
x_2(j \omega) = \mathcal{F} \{ x_1(t) \}
\end{equation}

It follows that the Fourier transformation of the signal

\begin{equation}
x_2(j t) = x_2(j \omega) \big\vert_{\omega=t}
\end{equation}

is given as

\begin{equation}
\mathcal{F} \{ x_2(j t) \} = 2 \pi \cdot x_1(- \omega)
\end{equation}

The duality principle of the Fourier transformation allows to carry over results from the time-domain to the spectral-domain and vice-versa. It can be used to derive new transforms from known transforms. This is illustrated at an example. Note, that the Laplace transformation shows no duality. This is due to the mapping of a complex signal $x(t)$ with real valued independent variable $t \in \mathbb{R}$ to its complex transform $X(s) \in \mathbb{C}$ with complex valued independent variable $s \in \mathbb{C}$.

#### Transformation of the exponential signal

The Fourier transform of a shifted Dirac impulse $\delta(t - \tau)$ is derived by introducing it into the definition of the Fourier transform and exploiting the sifting property of the Dirac delta function

\begin{equation}
\mathcal{F} \{ \delta(t - \tau) \} = \int_{-\infty}^{\infty} \delta(t - \tau) \, e^{-j \omega t} \; dt =  e^{-j \omega \tau}
\end{equation}

Using the duality principle, the Fourier transform of $e^{-j \omega_0 t}$ can be derived from this result by

1. substituting $\omega$ with $t$ and $\tau$ with $\omega_0$ on the right-hand side to yield the time-domain signal $e^{-j \omega_0 t}$
2. substituting $t$ by $- \omega$, $\tau$ with $\omega_0$ and multiplying the result by $2 \pi$ on the left-hand side

\begin{equation}
\mathcal{F} \{ e^{-j \omega_0 t} \} = 2 \pi \cdot \delta(\omega + \omega_0)
\end{equation}

### Linearity

The Fourier transform is a linear operation. For two signals $x_1(t)$ and $x_2(t)$ with Fourier transforms $X_1(j \omega) = \mathcal{F} \{ x_1(t) \}$ and $X_2(j \omega) = \mathcal{F} \{ x_2(t) \}$ the following holds

\begin{equation}
\mathcal{F} \{ A \cdot x_1(t) + B \cdot x_2(t) \} = A \cdot X_1(j \omega) + B \cdot X_2(j \omega)
\end{equation}

with $A, B \in \mathbb{C}$. The Fourier transform of a weighted superposition of signals is equal to the weighted superposition of the individual Fourier transforms. This property is useful to derive the Fourier transform of signals that can be expressed as superposition of other signals for which the Fourier transform is known or can be calculated easier. Linearity holds also for the inverse Fourier transform.

#### Transformation of the cosine and sine signal

The Fourier transform of $\cos(\omega_0 t)$ and $\sin(\omega_0 t)$ is derived by expressing both as harmonic exponential signals using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula)

\begin{align}
\cos(\omega_0 t) &= \frac{1}{2} \left( e^{j \omega_0 t} + e^{-j \omega_0 t} \right) \\
\sin(\omega_0 t) &= \frac{1}{2j} \left( e^{j \omega_0 t} - e^{-j \omega_0 t} \right)
\end{align}

together with the Fourier transform $\mathcal{F} \{ e^{-j \omega_0 t} \} = 2 \pi \cdot \delta(\omega - \omega_0)$ from above yields

\begin{align}
\mathcal{F} \{ \cos(\omega_0 t) \} &= \pi \left( \delta(\omega + \omega_0) + \delta(\omega - \omega_0)  \right) \\
\mathcal{F} \{ \sin(\omega_0 t) \} &= j \pi \left( \delta(\omega + \omega_0) - \delta(\omega - \omega_0) \right)
\end{align}

### Symmetries

In order to investigate the symmetries of the Fourier transform $X(j \omega) = \mathcal{F} \{ x(t) \}$ of a signal $x(t)$, first the case of a real valued signal $x(t) \in \mathbb{R}$ is considered. The results are then generalized to complex signals $x(t) \in \mathbb{C}$.

#### Real valued signals

Decomposing a real valued signal $x(t) \in \mathbb{R}$ into its even and odd part $x(t) = x_\text{e}(t) + x_\text{o}(t)$ and introducing these into the definition of the Fourier transform yields

\begin{align}
X(j \omega) &= \int_{-\infty}^{\infty} \left[ x_\text{e}(t) + x_\text{o}(t) \right] e^{-j \omega t} \; dt \\
&= \int_{-\infty}^{\infty} \left[ x_\text{e}(t) + x_\text{o}(t) \right] \cdot \left[ \cos(\omega t) - j \sin(\omega t) \right] \; dt \\
&= \underbrace{\int_{-\infty}^{\infty} x_\text{e}(t) \cos(\omega t) \; dt}_{X_\text{e}(j \omega)} +
j \underbrace{\int_{-\infty}^{\infty} - x_\text{o}(t) \sin(\omega t) \; dt}_{X_\text{o}(j \omega)}
\end{align}

For the last equality the fact was exploited that an integral with symmetric limits is zero for odd functions. Note that the multiplication of an odd function with an even/odd function results in an even/odd function. In order to conclude on the symmetry of $X(j \omega)$ its behavior for a reverse of the sign of $\omega$ has to be investigated. Due to the symmetry properties of $\cos(\omega t)$ and $\sin(\omega t)$, it follows that the Fourier transform of the

* even part $x_\text{e}(t)$ is real valued with even symmetry $X_\text{e}(j \omega) = X_\text{e}(-j \omega)$
* odd part $x_\text{o}(t)$ is imaginary valued with odd symmetry $X_\text{o}(j \omega) = - X_\text{o}(-j \omega)$

Combining this, it can be concluded that the Fourier transform $X(j \omega)$ of a real-valued signal $x(t) \in \mathbb{R}$ shows complex conjugate symmetry

\begin{equation}
X(j \omega) = X^*(- j \omega)
\end{equation}

It follows that the magnitude spectrum $|X(j \omega)|$ of a real-valued signal shows even symmetry

\begin{equation}
|X(j \omega)| = |X(- j \omega)|
\end{equation}

and the phase $\varphi(j \omega) = \arg \{ X(j \omega) \}$ odd symmetry

\begin{equation}
\varphi(j \omega) = - \varphi(- j \omega)
\end{equation}

Due to these symmetries, both are often plotted only for positive frequencies $\omega \geq 0$. However, without the information that the signal is real-valued it is not possible to conclude on the magnitude spectrum and phase for the negative frequencies $\omega < 0$.

#### Complex Signals

By following the same procedure as above for an imaginary signal, the symmetries of the Fourier transform of the even and odd part of an imaginary signal can be derived. The results can be combined, by decomposing a complex signal $x(t) \in \mathbb{C}$ and its Fourier transform into its even and odd part for both the real and imaginary part. This results in the following symmetry relations of the Fourier transform

**Example**

The Fourier transform $X(j \omega)$ of the signal $x(t) = \text{sgn}(t) \cdot \text{rect}(t)$ is computed. The signal is real valued with odd symmetry due to the sign function. It follows from the symmetry realations of the Fourier transform, that $X(j \omega)$ is imaginary with odd symmetry.

```python
class rect(sym.Function):

    @classmethod
    def eval(cls, arg):
        return sym.Heaviside(arg + sym.S.Half) - sym.Heaviside(arg - sym.S.Half)
```

```python
x = sym.sign(t)*rect(t)
sym.plot(x, (t, -2, 2), xlabel=r'$t$', ylabel=r'$x(t)$');
```

```python
X = fourier_transform(x)
X = X.rewrite(sym.cos).simplify()
X
```

```python
sym.plot(sym.im(X), (w, -30, 30), xlabel=r'$\omega$', ylabel=r'$\Im \{ X(j \omega) \}$');
```

**Exercise**

* What symmetry do you expect for the Fourier transform of the signal $x(t) = j \cdot \text{sgn}(t) \cdot \text{rect}(t)$? Check your results by modifying above example.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.