# Python Reader's Digest

In order to be able to interact with the figures, it is necessary (only when you use jupyter notebook) to invoke the magic command %matplotlib notebook.

```python
%matplotlib notebook
```

## 0. Preamble: outputting info in the console
Getting info in the console. When doing operations or writing complex scripts, it is important to monitor what's happening. Complex programs would use logging while simpler one would just output in the console messages or results of operations. To do so : one function **print()**

```python
print('this is a message')  # strings are written within quotes
print("another one")  # simple or double quote works
print("This wouldn't work without both")  # mis the two if you need to ouput one in the message
```

    this is a message
    another one
    This wouldn't work without both

when printing other variables than strings, there is automatic conversion (to strings)

```python
print(24)
print(2.6)
print([1.2, 'dhd', 4,2+1j])
```

    24
    2.6
    [1.2, 'dhd', 4, (2+1j)]

For log message we often mix strings and data using formatter:

```python
print('message using the format method to output this value:{}'.format(21.54657)) # the formatted value is placed within the braces {}
#using the format method of the string objects and adding formatters. Floating point notation with 2 digits
print('the value of you calculation is {:.2f}'.format(21.54657))  #
#exponential notation with 3 digits after the dot
print('the value of you calculation is {:.3e}'.format(21.54657))  #
print('many data: {} and this one {}'.format(12, 56.12))
```

    message using the format method to output this value:21.54657
    the value of you calculation is 21.55
    the value of you calculation is 2.155e+01
    many data: 12 and this one 56.12

and from Python 3.6 f-strings 'same as strings with a f just before the quotes) have been introduced to simplify readability:

```python
print(f'message using the format method to output this value:{21.54657}') # the formatted value is placed within the braces {}
print(f'the value of you calculation is {21.54657:.2f}')  # using f-stringss and adding formatters. Floating point notation with 2 digits
print(f'the value of you calculation is {21.54657:.3e}')  # exponential notation with 3 digits after the dot
print(f'many data: {12} and this one { 56.12}')
```

    message using the format method to output this value:21.54657
    the value of you calculation is 21.55
    the value of you calculation is 2.155e+01
    many data: 12 and this one 56.12

### Try by yourself

```python
# Try to print few sentences including data
a=[12.63112202,45,4852,784]

print("I want to write two numbers : {} and {}".format(25.6,a))
print(f"I want to write two numbers : {25.6} and {a}")

```

    I want to write two numbers : 25.6 and [12.63112202, 45, 4852, 784]
    I want to write two numbers : 25.6 and [12.63112202, 45, 4852, 784]

***

## 1. Libraries

A lot of functions have been already developed by the python's community.

Two librairies are inescapable in python for science :
- the mathematical librairy  numpy that contains the usual mathematical functions
- the plotting library pyplot from the matplotlib librairy that allows you to plot different types of graphs.

In order to have a better control on the used functions, a good pratice consists in importing these librairies using an alias :

```python
import numpy as np
import matplotlib.pyplot as plt
```

NB : In the following, we are going to use other specific librairies that will be described at that time.

Remark : In python there are several ways to import libraries :

```python
# Total import with alias
import numpy as np
print(f"PI number is equal to {np.pi}")
print(f"cos(PI) number is equal to {np.cos(np.pi)}")
```

    PI number is equal to 3.141592653589793
    cos(PI) number is equal to -1.0

```python
#Total import with full name
import numpy
print(f"PI number is equal to {numpy.pi}")
```

    PI number is equal to 3.141592653589793

```python
#Total import without name nor alias
from numpy import *
print(f"PI number is equal to {pi}")
```

    PI number is equal to 3.141592653589793

Partial import is also possible when you plan to use just some specific functions :

```python
#Partial import with alias
from numpy import pi as nombrepi
print("PI number is equal to {nombrepi}")
print(f"cos(PI) number is equal to {cos(nombrepi)}")
```

    PI number is equal to {nombrepi}
    cos(PI) number is equal to -1.0

```python
#Partial import without alias
from numpy import pi
print(f"PI number is equal to {pi}")
```

    PI number is equal to 3.141592653589793

In these two cases, all the other functions from numpy are ignored.

***

## 2. Types of data

It is worth noting that in python, typing of data (that is attributing a type to a variable) is not mandatory, the tpe is attributed at execution depending on the context. A same variable can change its type along the same script.

### 2.1 Numerical data

There are 3 types of numerical data in python : integers, floats and complex numbers. To know the type of a variable, you can use the command type().

```python
#integer
a = 3
print(f"a is an integer equal to {a}")
print(f"Type of a is {type(a)}")

print("------------------------")

# Float
a=3.256985
print(f"a is now a float equal to {a}")
print(f"Type of a is {type(a)}")
print("------------------------")
#Complex
a=3+0j
print(f"a is finally a complex equal to {a}")
print(f"Its real part is {a.real} and its imaginary part is {a.imag}")
print(f"Type of a is {type(a)}")

```

    a is an integer equal to 3
    Type of a is <class 'int'>
    ------------------------
    a is now a float equal to 3.256985
    Type of a is <class 'float'>
    ------------------------
    a is finally a complex equal to (3+0j)
    Its real part is 3.0 and its imaginary part is 0.0
    Type of a is <class 'complex'>

### 2.2 Strings

A string is an ensemble of alphanumerical characters in between " " or ' '.

Two character strings can be concatenate using the +.

```python
A='Hello'
B="How are you ?"

print(A+',today, '+B)
```

    Hello,today, How are you ?

### 2.3 Converting data from one type to another

#### Example : converting a float to a string

```python
a=3.1416

print(f'The type of a is {type(a)}')

a=str(a)
print(f'After conversion, the type is now : {type(a)}')
```

    The type of a is <class 'float'>
    After conversion, the type is now : <class 'str'>

#### Conversion of a float to a string with formatting (see preamble)

```python
a=3.1416

b=f"{a:4.6f}"

print(b)
print(f"The type of b is {type(b)}")
```

    3.141600
    The type of b is <class 'str'>

#### Conversion of a string to a float

```python
a="2.25635"
print(f"Before conversion, type is {type(a)} and its value is '{a}'")

a=float(a)
print(f"After conversion, type is {type(a)} and its value is {a}")
```

    Before conversion, type is <class 'str'> and its value is '2.25635'
    After conversion, type is <class 'float'> and its value is 2.25635

#### Conversion from float to integer

We use for that the function int() which keeps the whole part of the float. It is different from the function round() which returns the closest integer.

```python
a=2.55
print(f"Before conversion, type is {type(a)} and its value is {a}")

b= int(a)
print(f"After conversion, type is {type(b)} and its value is {b}")

# Use of round() function
c=round(a)

print(f"The closest integer of {a} is {c}")

d=floor(a)
print(f"The closest float of {a} is {d}")
```

    Before conversion, type is <class 'float'> and its value is 2.55
    After conversion, type is <class 'int'> and its value is 2
    The closest integer of 2.55 is 3
    The closest float of 2.55 is 2.0

### Try by yourself

```python
# Define different types of data and try to convert them to an other type
```

### 2.4 Array vs Lists vs tuples
**Lists** and **tuples** are a standard type in python, which is not the case for arrays. To manipulate arrays it is then necessary to import numpy library.

* Lists and tuples can contain inhomegeneous data: e.g. float, string, int, ...
* List are mutable iterator and represented/created using square brackets [..., ..., ...]
* tuples are imutable iterator and represented/created using parenthesis (..., ..., ...) or even without : ..., ..., ...
* numpy arrays are mutable and contains only homogeneous data: all floats or all ints or all strings ....

IMPORTANT REMARK : Index of the first element of python iterators is 0 in python (as in most languages but matlab ;-)!

#### Tuples
Once intialized, tuples cannot be modified (imutable). They are often used to return results from functions

```python
t = (1,2,3)
print(t)
print(t[1])
t[1] = 12  # will trigger an error
```

#### Lists

- two lists can be concatenate using the + symbol. Operations -, * and / do not exist for lists
- to add an element A at the end of a list L, one can use the list object method ``append()`` as : L.append(A)
- an empty list (for initialisation for example) can be constructed using L=[] or ``L=list()``

```python
L1=[1,2,3,4,5]
L2=[6,7,8,8]

# Concatenate totally or partially several lists
L=L1+L2+L1[0:2]
print("-------------------------------")
print(f"Concatenate {L1} and {L2}")
print(f"Result L is {L}")

```

    -------------------------------
    Concatenate [1, 2, 3, 4, 5] and [6, 7, 8, 8]
    Result L is [1, 2, 3, 4, 5, 6, 7, 8, 8, 1, 2]

```python
#Append one element
A=234
print("")
print(f"Append the element {A}")
L.append(A)
print(f"Result L is then {L}")
```

    Append the element 234
    Result L is then [1, 2, 3, 4, 5, 6, 7, 8, 8, 1, 2, 234]

```python
#create an empty list
L3=[]
print("-------------------------------")
print(f"L3 is an empty list : {L3}")
#add an element to L3
B=45
L3.append(B)
print(f"Append the element {B} to the empty list L3 :")
print(f"Result L3 is then {L3}")
```

    -------------------------------
    L3 is an empty list : []
    Append the element 45 to the empty list L3 :
    Result L3 is then [45]

```python
#create a list with all numbers identical
L4=[2]*6
print("-------------------------------")
print(f"L4 is a list composed of identical elements : {L4}")
```

    -------------------------------
    L4 is a list composed of identical elements : [2, 2, 2, 2, 2, 2]

```python
#Select a part of a list
L5=[0,1,2,3,4,5,6,7,8,9]
print("-------------------------------")
print(f"L5 is equal to {L5}")
print(f"Part of L5 : {L5[2:5]}")  # part of L5 comprised between the index 2 and the index 5-1
```

    -------------------------------
    L5 is equal to [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
    Part of L5 : [2, 3, 4]

```python
# get last element or up to the last element
print(L5[5:])  # print from the fifth to the end
print(L5[-1])  # get the last element
print(L5[-2])  # get element before the last
```

    [5, 6, 7, 8, 9]
    9
    8

#### Use of the range command

It is possible to quickly construct a generator using range command. But be carefull, it is not a list. We will see in details later (when we will take a look at for loops the use of this range function to iterate).

```python
# Generator of N element starting at 0
N=10
L=range(N)
print(f"Create a generator with {N} elements starting at 0")
print(L)
print(f"The type is {type(L)}. It is not a list")

print('')

#Generator from A to B-P with step P
A=5
B=16
P=2
L=range(A,B,P)
print(f'Generator from {A} to {B}-{P} with step {P}') # the last value is excluded hence B-P
print(L)
print(f"The type is {type(L)}. It is not a list")
```

    Create a generator with 10 elements starting at 0
    range(0, 10)
    The type is <class 'range'>. It is not a list

    Generator from 5 to 16-2 with step 2
    range(5, 16, 2)
    The type is <class 'range'>. It is not a list

### Try by yourself

```python
# Define and manipulate lists

```

#### Arrays

Arrays are parts of numpy library.

Operations on arrays :
- Numerical operations (+, -, * and /) between 2 arrays, but also the power operation ** are done element by element (the 2 arrays must have the same dimension). It exists specialized methods for algebra calculation (e.g. matrix multiplication)
- It is possible to create N dimension arrays
- As for the lists, first index is 0

```python
import numpy as np

# creation using the array method and conversion of an iterable (list or tuple) to arrays
# 1D array
A = np.array((1,2,3))  # using the tuple (1,2,3)
Aprim = np.array([2,3,4])  # using a list
print(f"1D array : {A}")
print(f"1D array : {Aprim}")

print(f"The result is :{A+Aprim}")
```

    1D array : [1 2 3]
    1D array : [2 3 4]
    The result is :[3 5 7]

```python
#2D array using nested lists of nested tuples
B=np.array(((1,2,3),(4,5,6)))  # using the nested tuple ((1,2,3),(4,5,6))
print(f"2D array : {B}")
print('')
```

    2D array : [[1 2 3]
     [4 5 6]]

```python
# init an array with zero values
czero = np.zeros((10, 3))  # create an array with shape (dimensions (10, 3))
print(czero)
print(f'the shape of this array is {czero.shape}')
```

    [[0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]
     [0. 0. 0.]]
    the shape of this array is (10, 3)

```python
#Create an 1D array from a to b-p with step p
a=2
b=19
p=3
print(f"1D array from {a} to {b}-{p} with step {p}")
C=np.arange(a,b,p)
print(C)
```

    1D array from 2 to 19-3 with step 3
    [ 2  5  8 11 14 17]

```python
#1D linearly spaced array from a to b with N points
a=0.23
b=0.89
N=13
print('')
print(f"1D linearly spaced array from {a} to {b} with {N} points")
D=np.linspace(a,b,N)
print(D)
print('')

theta=np.linspace(-2*np.pi,2*np.pi,100)
y=np.cos(theta)
print(y)
```

    1D linearly spaced array from 0.23 to 0.89 with 13 points
    [0.23  0.285 0.34  0.395 0.45  0.505 0.56  0.615 0.67  0.725 0.78  0.835
     0.89 ]

    [ 1.          0.99195481  0.9679487   0.92836793  0.87384938  0.80527026
      0.72373404  0.63055267  0.52722547  0.41541501  0.29692038  0.17364818
      0.04758192 -0.07924996 -0.20480667 -0.32706796 -0.44406661 -0.55392006
     -0.65486073 -0.74526445 -0.82367658 -0.88883545 -0.93969262 -0.97542979
     -0.99547192 -0.99949654 -0.98743889 -0.95949297 -0.91610846 -0.85798341
     -0.78605309 -0.70147489 -0.60560969 -0.5        -0.38634513 -0.26647381
     -0.14231484 -0.01586596  0.1108382   0.23575894  0.35688622  0.47227107
      0.58005691  0.67850941  0.76604444  0.84125353  0.90292654  0.95007112
      0.9819287   0.99798668  0.99798668  0.9819287   0.95007112  0.90292654
      0.84125353  0.76604444  0.67850941  0.58005691  0.47227107  0.35688622
      0.23575894  0.1108382  -0.01586596 -0.14231484 -0.26647381 -0.38634513
     -0.5        -0.60560969 -0.70147489 -0.78605309 -0.85798341 -0.91610846
     -0.95949297 -0.98743889 -0.99949654 -0.99547192 -0.97542979 -0.93969262
     -0.88883545 -0.82367658 -0.74526445 -0.65486073 -0.55392006 -0.44406661
     -0.32706796 -0.20480667 -0.07924996  0.04758192  0.17364818  0.29692038
      0.41541501  0.52722547  0.63055267  0.72373404  0.80527026  0.87384938
      0.92836793  0.9679487   0.99195481  1.        ]

```python
#1D log-spaced array from 10**a to 10**b with N points
a=1
b=10
N=10
print('')
print(f"1D log-spaced array from 10**{a} to 10**{b} with {N} points")
E=np.logspace(a,b,N)
print(E)
print('')
```

    1D log-spaced array from 10**1 to 10**10 with 10 points
    [1.e+01 1.e+02 1.e+03 1.e+04 1.e+05 1.e+06 1.e+07 1.e+08 1.e+09 1.e+10]

### Try by yourself

```python
# Define two arrays with the same dimension and calculate the scalar product
#(you can use for that the function np.sum)

a=np.array((1,2,3))

b=np.array((6,8,9))

print(f"The scalar product of a.b is {np.sum(a*b)}")
```

    The scalar product of a.b is 49

***

## 3. Functions

Syntax :
- Function definition always starts with the keyword def followed by the function name and the variables between parenthesis. This first line finishes with the symbol :
- The lines that follow constitute the body of the function and must be indented (in general 4 spaces).
- if the function should give values (numbers, strings,etc...), returned values are preceded by the keyword return
- Definition of the function must appear in the script before its use

#### Function that returns a numerical value
Let's implement the following function :

$U(t,\tau) = e^{-\frac{t}{\tau}} \sin(\omega t)$

```python
#library import
import numpy as np
omega=2*np.pi #global variable

tau=10
#function
def U(t,tau):
    """This is a documented string
    Parameter
    ---------
    t: (float) elapsed time
    tau: (float) characteristic time

    Return
    ------
    float: the value of the function
    """
    f=np.exp(-t/tau)*np.sin(omega*t)
    return f,t,tau

t=np.linspace(0,10,10)
tau2=1
function,time,tautau=U(t,tau2)
print(function)

#print(f"Function for t={t} and tau={tau} is equal to {U(t,tau)}")
```

    [ 0.00000000e+00  2.11601174e-01  1.06721669e-01  3.08945845e-02
      4.01655749e-03 -1.32222256e-03 -1.10213320e-03 -4.12577451e-04
     -8.86486451e-05 -1.11197757e-19]

#### Function that does an action without returning a value (in fact it returns None by default)

```python
def FemtoUp(x):
    print(("FemtoUp is a "+x+" school !\n")*10)  # \n is used to return to the line

FemtoUp('difficult')
```

    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !
    FemtoUp is a difficult school !

### Try by yourself

```python
# Define a function that return the value of an Gaussian function
def gauss(X,SIGMA,AMPLITUDE):
    return AMPLITUDE*np.exp(-(X/SIGMA)**2.0)

# Define an array of x from xmin to xmax and calculate the value of the Gaussian for all x
xmin,xmax=-10.,10.
N=100
x=np.linspace(xmin,xmax,N)

sigma=1
amplitude=1
y=gauss(x,sigma,amplitude)

print(f'The result of my gaussian is {y}')
```

    The result of my gaussian is [3.72007598e-44 2.03026390e-42 1.02118442e-40 4.73376699e-39
     2.02236984e-37 7.96279046e-36 2.88948872e-34 9.66335032e-33
     2.97841666e-31 8.46046409e-30 2.21489900e-28 5.34397817e-27
     1.18830091e-25 2.43522554e-24 4.59941880e-23 8.00603952e-22
     1.28435035e-20 1.89889213e-19 2.58742690e-18 3.24927813e-17
     3.76059612e-16 4.01122945e-15 3.94320471e-14 3.57249927e-13
     2.98295005e-12 2.29546576e-11 1.62797100e-10 1.06407812e-09
     6.40990174e-09 3.55860870e-08 1.82079103e-07 8.58600196e-07
     3.73140910e-06 1.49453385e-05 5.51683065e-05 1.87682835e-04
     5.88451217e-04 1.70038545e-03 4.52830117e-03 1.11140983e-02
     2.51399360e-02 5.24089078e-02 1.00692485e-01 1.78295348e-01
     2.90960459e-01 4.37601640e-01 6.06561603e-01 7.74857597e-01
     9.12262625e-01 9.89848834e-01 9.89848834e-01 9.12262625e-01
     7.74857597e-01 6.06561603e-01 4.37601640e-01 2.90960459e-01
     1.78295348e-01 1.00692485e-01 5.24089078e-02 2.51399360e-02
     1.11140983e-02 4.52830117e-03 1.70038545e-03 5.88451217e-04
     1.87682835e-04 5.51683065e-05 1.49453385e-05 3.73140910e-06
     8.58600196e-07 1.82079103e-07 3.55860870e-08 6.40990174e-09
     1.06407812e-09 1.62797100e-10 2.29546576e-11 2.98295005e-12
     3.57249927e-13 3.94320471e-14 4.01122945e-15 3.76059612e-16
     3.24927813e-17 2.58742690e-18 1.89889213e-19 1.28435035e-20
     8.00603952e-22 4.59941880e-23 2.43522554e-24 1.18830091e-25
     5.34397817e-27 2.21489900e-28 8.46046409e-30 2.97841666e-31
     9.66335032e-33 2.88948872e-34 7.96279046e-36 2.02236984e-37
     4.73376699e-39 1.02118442e-40 2.03026390e-42 3.72007598e-44]

***

## 4. Graphs

### 4.1 1D graphs

Some features :
- the pyplot library has been developped to be quite similar with the matlab one
- with matplotlib.pyplot it is possible to include some Latex code using the key symbols $...$
- To add a legend, use the keyword label="..." in plt.plot and call it with plt.legend()
- plt.show() is used to show the figure. It is not always necessary (depends on the editor)

```python
#libraries import
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# Close all opened figures
#plt.close('all')

# Function we want to plot
def sinus(x):
    return np.sin(x)

# Variables
theta=np.linspace(-4*np.pi,4*np.pi,1000)
y=sinus(2*np.pi*theta)

plt.figure() # Open an empty figure
plt.plot(theta,y,label='The function') # label will be print if plt.legend() is used
plt.title('Sinus function',fontsize=16,color='red') #title
plt.xlabel("Angle $\\theta$ (rad)")
plt.ylabel("$\sin(t)$")
plt.legend()

# You can also add some text at a specific place
xpos=-10.0
ypos=0.5
plt.text(xpos,ypos,"Here is some text",fontsize=13,color='blue',backgroundcolor='yellow')
#plt.show()
```

### Try by yourself

```python
# Plot the Gaussian function that you have define in the previous section

plt.figure()
x=np.linspace(-10,10,100)
sigma=1
amplitude=10
y1=gauss(x,sigma,amplitude)
y2=gauss(x,2*sigma,amplitude/1.2)
plt.plot(x,y1,color='green',label='y1')
plt.title('My figure 1')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.legend()

plt.figure()
plt.plot(x,y2,'r',label='y2')

plt.title('My figure 2')
plt.xlabel('$x$')
plt.ylabel('$y$')
plt.legend()

```

### 4.2 2D graphs

This example is inspired from the lecture of Rodolphe Boisgard. Let's try to plot the 2D representation of a temperature field represented by :

$T(x,y)=T_b+T_h e^{- \frac{(x-x_h)^2+(y-y_h)^2}{\sigma^2}}$
$+ T_c e^{- \frac{(x-x_c)^2+(y-y_c)^2}{\sigma^2} }$

with :
- $T_b=20^\circ$C the base temperature
- $T_h=100^\circ$C the temperature of the hot spot
- $T_c=-30^\circ$C the temperature of the cold spot
- $(x_h,y_h)$ and $(x_c,y_c)$ the loaclizations of the hot and cold spots

Important Remark : we will use the function meshgrid to generate coordinates in 2D.

```python
# Libraries import
import numpy as np
import matplotlib.pyplot as plt

#plt.close('all')

# Function definition
def temp(x,y):
    return Tb+Th*np.exp(-((x-xh)**2.0+(y-yh)**2.0)/sigma**2.0)+Tc*np.exp(-((x-xc)**2.0+(y-yc)**2.0)/sigma**2.0)

# Parameters
xh,yh,xc,yc=-2,2,2,-2 #localization of hot and cold spots
sigma = 2.0
Tb,Th,Tc=20,100,-30  #Temperatures (base,hot,cold)
xmin,xmax,ymin,ymax=-5,5,-5,5 # 2D Space
NbX,NbY = 50,60 # Number of points along x and y

# creation of 1D vector for space
x=np.linspace(xmin,xmax,NbX)
y=np.linspace(ymin,ymax,NbY)

# creation of 2D matrices for x and y
X,Y = np.meshgrid(x,y)
"""
print("The value of X is {}".format(X))
print("The value of Y is {}".format(Y))
# Get the temperature field

print(T)
"""
T=temp(X,Y)
# 2D plot
plt.figure()
plt.imshow(T,origin='lower',extent=(xmin,xmax,ymin,ymax),cmap='gray',interpolation='bilinear')
```

```python
plt.figure()
plt.imshow(T,origin='lower',extent=(xmin,xmax,ymin,ymax),cmap='jet',interpolation='bilinear')
# Let's add some contour (isovalues)
CS=plt.contour(X,Y,T,16)
plt.clabel(CS,fontsize=10,fmt='%1.1f')
plt.axis('equal')

# 2D plot change colormap
plt.figure()
plt.imshow(T,origin='lower',extent=(xmin,xmax,ymin,ymax),cmap='hsv',interpolation='bilinear')
CS=plt.contour(X,Y,T,16,colors='blue')
plt.clabel(CS,fontsize=10,fmt='%1.1f')
plt.axis('equal')
```

### 4.3 Application - Densities of probability
For an electron in hydrogen atom, one can show that stationnary state in spherical coordinates are given by :

\begin{align}
\psi(r,\theta,\varphi) = R_{n,\ell}(r) Y_{\ell}^m (\theta,\varphi)
\end{align}

Probability density to find electron at a distance $r$ from the atom is written:

\begin{align}
\rho (\mathbf{r},t) = |\psi(\mathbf{r},t)|^2
\end{align}

\begin{array}
 OOrbital & Angular Part & Radial Part \\
\hline
1s & \frac{1}{2\sqrt{\pi}} & R_{10}(r)=\frac{2}{a_0^{3/2}}e^{-\frac{r}{a_0}} \\
2s & \frac{1}{2\sqrt{\pi}} & R_{20}(r)=\frac{1}{a_0^{3/2}} \frac{1}{\sqrt{2}} \left( 1-\frac{r}{2a_0} \right) e^{-\frac{r}{2a_0}} \\
3s & \frac{1}{2\sqrt{\pi}} & R_{30}(r)=\frac{1}{a_0^{3/2}} \frac{2}{\sqrt{3}} \left( 1-\frac{2r}{a_0}+ \frac{4}{27} \left( \frac{r}{a_0} \right)^2 \right) e^{-\frac{r}{3a_0}} \\
2p_x & \frac{1}{2}\left( \frac{3}{\pi}\right)^{1/2} \frac{x}{r} & R_{21}(r)=\frac{1}{a_0^{3/2}} \frac{1}{2\sqrt{6}}\frac{r}{a_0} e^{-\frac{r}{2a_0}} \\
2p_z & \frac{1}{2}\left( \frac{3}{\pi}\right)^{1/2} \frac{z}{r} & R_{21}(r) \\
3d_{xz} & \frac{1}{2}\left( \frac{15}{\pi}\right)^{1/2} \frac{xz}{r^2} & R_{32}(r)=\frac{1}{a_0^{3/2}} \frac{4}{81\sqrt{30}}\left(\frac{r}{a_0}\right)^2 e^{-\frac{r}{3a_0}} \\
3d_{z^2} & \frac{1}{4}\left( \frac{5}{\pi}\right)^{1/2} \frac{2z^2-x^2-y^2}{r^2} & R_{32}(r) \\
\end{array}

1. Plot the radial density of the 3 first orbitals
2. Plot the probability density of orbitals in the $x0z$ plane

```python
a0=1
# radial unctions
def R10(r):
    return 2.0/a0**(3/2)*np.exp(-r/a0)

def R20(r):
    return 1/a0**(3/2)*1/np.sqrt(2)*(1-r/(2*a0))*np.exp(-r/(2*a0))

def R30(r):
    return 1/a0**(3/2)*2/np.sqrt(3)*(1-2*r/a0+4/27*(r/a0)**2)*np.exp(-r/(3*a0))

def R21(r):
    return 1/a0**(3/2)*1/(2*np.sqrt(6))*r/a0*np.exp(-r/(2*a0))

def R32(r):
    return 1/a0**(3/2)*4/(81*np.sqrt(30))*(r/a0)**2*np.exp(-r/(3*a0))

# Angular function
def s():
    return 1/(2*np.sqrt(np.pi))

def px(x,r):
    return 1/2*(3/np.pi)**(1/2)*x/r

def pz(z,r):
    return 1/2*(3/np.pi)**(1/2)*z/r

def dxz(x,z,r):
    return 1/2*(3/np.pi)**(1/2)*x*z/r**2

def dzz(x,z,r):
    # y=0 for x0z plane
    return 1/4*(5/np.pi)**(1/2)*(2*z**2-x**2)/r**2

# 1D radial space

r=np.linspace(0,20,100)

#Plot of the radial density
plt.figure()
plt.plot(r,R10(r),label='$R_{10}(r)$')
plt.plot(r,R20(r),label='$R_{20}(r)$')
plt.plot(r,R30(r),label='$R_{30}(r)$')
plt.plot(r,R21(r),label='$R_{21}(r)$')
plt.plot(r,R32(r),label='$R_{32}(r)$')
plt.xlabel('$r$')
plt.title('Radial functions')
plt.legend()

#2D space
NbX=100
NbZ=100
xmin,xmax,zmin,zmax=-20,20,-20,20

x=np.linspace(xmin,xmax,NbX)
z=np.linspace(zmin,zmax,NbZ)

X,Z=np.meshgrid(x,z)
R=np.sqrt(X**2+Z**2)

# Probability densities
#s orbital
plt.figure()
plt.subplot(1,3,1)
plt.imshow(abs(s()*R10(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('1s orbital')
plt.subplot(1,3,2)
plt.imshow(abs(s()*R20(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('2s orbital')
plt.subplot(1,3,3)
plt.imshow(abs(s()*R30(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('3s orbital')

#p orbitals
plt.figure()
plt.subplot(1,2,1)
plt.imshow(abs(px(X,R)*R21(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('$2p_x$ orbital')
plt.subplot(1,2,2)
plt.imshow(abs(pz(Z,R)*R21(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('$2p_z$ orbital')

#d orbitals
plt.figure()
plt.subplot(1,2,1)
plt.imshow(abs(dxz(X,Z,R)*R32(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('$3d_{xz}$ orbital')
plt.subplot(1,2,2)
plt.imshow(abs(dzz(X,Z,R)*R32(R))**2.0,\
           origin='lower',extent=(xmin,xmax,zmin,zmax),cmap='gray',interpolation='bilinear')
plt.title('$3d_{z^2}$ orbital')
```

***

## 5. Structure
### 5.1 For loops

```python
#import libaries
import numpy as np

#loop using a range generator
N=10
print("Loop over {} elements starting at 0".format(N))
for i in range(N):
    print(i)

print('Here you leave the for loop')
```

    Loop over 10 elements starting at 0
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    Here you leave the for loop

```python
L=['milli','micro','nano','pico','femto','atto','zepto']
print('')
print('Loop over the element of a list')
for i in L:
    print(i)

```

    Loop over the element of a list
    milli
    micro
    nano
    pico
    femto
    atto
    zepto

```python
print('')
print('Loop and iteraror')
for expo, prefix in zip(range(3,22,3),L):
    print(f"Prefix {prefix} corresponds to 1e-{expo}")
```

    Loop and iteraror
    Prefix milli corresponds to 1e-3
    Prefix micro corresponds to 1e-6
    Prefix nano corresponds to 1e-9
    Prefix pico corresponds to 1e-12
    Prefix femto corresponds to 1e-15
    Prefix atto corresponds to 1e-18
    Prefix zepto corresponds to 1e-21

```python
# get current index in a loop
for ind, prefix in enumerate(L):
    print(f'Index of {prefix} in the list is {ind}')
```

    Index of milli in the list is 0
    Index of micro in the list is 1
    Index of nano in the list is 2
    Index of pico in the list is 3
    Index of femto in the list is 4
    Index of atto in the list is 5
    Index of zepto in the list is 6

### Try by yourself

```python
# Append a list using a for-loop

```

### 5.2 If structure

```python
N=10
for ind in range(N):
    if (np.mod(ind, 2) == 0):
        print(f"{ind} is even")
        print('something else')
    elif (np.mod(ind, 2) == 1):
        print(f"{ind} is odd")
```

    0 is even
    something else
    1 is odd
    2 is even
    something else
    3 is odd
    4 is even
    something else
    5 is odd
    6 is even
    something else
    7 is odd
    8 is even
    something else
    9 is odd

```python
print('Same result :')
for ind in range(N):
    if (np.mod(ind, 2) == 0):
        print(f"{ind} is even")
    else:
        print(f"{ind} is odd")

```

    Same result :
    0 is even
    1 is odd
    2 is even
    3 is odd
    4 is even
    5 is odd
    6 is even
    7 is odd
    8 is even
    9 is odd

### Try by yourself

```python
# Define a for then else structure

```

***

## 6. Import or save data from or to a text file
### 6.1. - Save data

There is a lot of specific libraries specialized in data handling (see for example Pandas, ``h5py``, ...), but in our case we are going to focus on the very simple function savetxt from numpy.

As an example, we will start by generating a set of data (sinusoidal signal) to which we will add some random noise(using the random function). This signal will be finally saved in a text file as a two column set:
* angle in radians
* noisy signal

```python
# if it is not already done, import the two following libraries
#import numpy as np
#import matplotlib.pyplot as plt

# random function import. Generate a random number between 0 et 1
from numpy.random import normal
#PI import (lazy thing)
from numpy import pi

Nbpoints = 1000
theta=np.linspace(-3.*pi, 3.0*pi, Nbpoints)
noise = normal(0, 0.2, Nbpoints)
y = np.sin(theta) + noise

plt.figure()
plt.plot(theta,y)

# Save data as 'data.txt'

np.savetxt('data.txt', np.array([theta, y]).T, fmt="%.6f", header="theta (rad)   y")
```

In this function, parameters are:
- 'data.dat' : name of the file to save
- np.array([theta, y]).T : an array that contains values of $\theta$ and $y$. In order to put these data on a column form, it is necessary to transpose the matrix. This is done with the method .T
- fmt="%.6f" : format for the numbers : floats with 6 decimal numbers (you could try ``%.3e``)
- header="theta   y" : header for the columns

### 6.2. - Import data from a text file

Use of the function loadtxt from numpy library.

```python
# If it is not already done, import libraries
#import matplotlib.pyplot as plt
#import numpy as np

theta, y = np.loadtxt('data.txt', unpack=True)

# Check taht each varaiable contains the right number of points (i.e. 1000)
print(f"Number of points : {theta.size}")
print(f"Number of points : {theta.shape}")
print(f"Number of points : {len(theta)}")
```

    Number of points : 1000
    Number of points : (1000,)
    Number of points : 1000

The unpack=True option is used to directly separate the two column data is distinct variables.

### 6.3. - Plot these data in 2 dimensions

In order to simulate an experiment in which a trace is recorded for different value of a parameter, we can start by generating a list of datafiles that contain noisy sinus with varying period.

```python
# Uncomment the following line if the library has not been imported before
#import numpy as np
from numpy import pi

def noisy_sinus(theta, periode, Nbpoints):
    noise = normal(0, 0.05, Nbpoints)
    return np.sin(2.0 * pi / periode * theta) + noise

Nbpoints=1000
theta=np.linspace(-3.0*pi,3.0*pi,Nbpoints)

periode=np.linspace(0.1,10.,101)

#print(periode)

for T in periode:
    y = noisy_sinus(theta,T,Nbpoints)
    np.savetxt('data_' + str(T) + '.txt', np.array([theta, y]).T, fmt="%.6f", header="theta   y")

```

Thus, we have generated  101 files that contain for each a noisy sinus with a different period.

Let's pack these traces to generate a 2D surface that includes all the experimental measurements.

```python
# Uncomment the following line if the library has not been imported before
#import matplotlib.pyplot as plt
#from numpy import pi
#import numpy as np

periode = np.linspace(0.1,10.,101)
Nbpoints = 1000
Nbtraces = np.shape(periode)[0]

#generate an empty 2D matrix (Nbpoints,Nbtraces)
data = np.zeros((Nbpoints,Nbtraces))

# Data imported and saved in data variable
for i in range(Nbtraces):
    theta,data[:,i] = np.loadtxt('data_'+str(periode[i])+'.txt',unpack=True)

thetamin = min(theta)
thetamax = max(theta)
Tmin = min(periode)
Tmax = max(periode)

# Plot of the 2D surface with imshow
plt.figure()
plt.imshow(data,extent=(Tmin,Tmax,thetamin,thetamax),interpolation="bicubic",
       origin="lower",cmap='gray')
plt.title('Sinus as a function of the period')
plt.ylabel('$\\theta$ (radians)')
plt.xlabel('Period')
plt.axis('tight')

plt.savefig("sinus_vs_periode.png")
```

***

## 7. Curve fitting

### 7.1. - Without taking into account the experimental errors
Let's take for that the noisy sinus defined just before and we define a function sinus that will be used for fitting.

```python
def sinus(x, Amp, w0):
    """Definition of the fitting function
    Parameters
    ----------
    x: (array of float or float) function's variable
    Amp : (float) amplitude
    w0 : (float) angular frequency
    """
    return Amp * np.sin(2.0 * pi * w0 * x)
```

We import the library from scipy.optimize to make the fitting

```python
from scipy.optimize import curve_fit
```

Fitting:

```python
T = 5
ydata = noisy_sinus(theta, T, Nbpoints)
fitted_coeff, pcov = curve_fit(sinus, theta, ydata, p0=([1.0,0.2]))  # p0 is the initial guess
print("Amplitude is equal to {:2.6e} and the angular frequency {:2.6e} rad/s".format(*fitted_coeff))
```

    Amplitude is equal to 9.976572e-01 and the angular frequency 1.999856e-01 rad/s

One can plot on the same graph the experimental points and the fit:

```python
plt.figure()
plt.plot(theta,ydata,'*',label='Experimental points')
plt.plot(theta,sinus(theta,fitted_coeff[0],fitted_coeff[1]),'r',label='fit')
plt.legend()
```

### 7.2. - Fitting by taking into account the experimental errors

One solution (but it exists a lot of other libraries that are designed to do the same job) could be to use the odr function from scipy.

```python
from scipy.odr import Model, RealData, ODR

# We define the fitting function, here a linear function
def linear(p, x):
    a, b = p
    return a * x + b

#Set up the fitting model
linear_model = Model(linear)

```

We simulate an experimental data set (with noise) and we add the errors associated to measures.

```python
#Simulation of experimental data (example : measurement of the speed of sound with x=time and y=distance)
x = np.linspace(1, 10, 5)  # time measurement
x_err = np.ones(np.shape(x)) * 1.  # errors on the time measurement
y = linear([350, 0], x) + normal(0, 100., np.shape(x))  # associated distance with some noise
y_err = np.ones(np.shape(y)) * 100  # errors on the distance measurement

```

Fit of the experimental data with error bars :

```python
# Creation of a realdata object from our measurement
data = RealData(x, y, sx=x_err, sy=y_err)

# Set up of the ODR (orthogonal distance regression) with the model and data
odr = ODR(data, linear_model, beta0=[350., 0.])

# Fitting procedure
out = odr.run()

# Results
#out.pprint()

print(f"The speed of sound is equal to ({out.beta[0]:.2e} +/- {2.*out.sd_beta[0]:.2e}) m/s")
```

    The speed of sound is equal to (3.48e+02 +/- 3.08e+01) m/s

Plot of data and fit :

```python
plt.figure()
plt.plot(x,y,'*',label='Experimental points',zorder=2)
plt.errorbar(x, y, xerr = x_err, yerr = y_err,
  fmt = 'none', capsize = 10, ecolor = 'red', zorder = 1)
plt.plot(x, linear([out.beta[0], out.beta[1]], x), 'b', label='Fit')
plt.legend()
```

***
## 8. Fourier Transform

### 8.1. - 1 dimension

As an example, we will use data as a function of time, but it can also be used with all other type of signal related by Fourier Transform.

Let's start by generating a sinusoidal signal.

```python
t = np.linspace(-10., 10., 1000)  #time
nu0 = 0.5 # fréquence
ydata = noisy_sinus(t, 1/nu0, np.shape(t)[0])

plt.figure()
plt.plot(t, ydata, label="Sinus")
plt.legend()
```

Fourier transform of this signal can be computed using fft and fftfreq functions from scipy.fftpack library.

```python
from scipy.fftpack import fft,fftfreq

pas_t = t[1] - t[0] # timestep
fft_sin = fft(ydata,2048) # Fast Fourier Transform
freq = fftfreq(np.shape(fft_sin)[0], d=pas_t) # Associated frequencies
```

Real Part

```python
plt.figure()
plt.plot(freq,fft_sin.real,label="Real part")
plt.xlim([-2, 2])
plt.legend()
```

Imaginary part

```python
plt.figure()
plt.plot(freq,fft_sin.imag,'*-',label="Imaginary part")
plt.xlim([-2, 2])
plt.legend()
```

Squared modulus

```python
plt.figure()
plt.plot(freq, abs(fft_sin)**2., label="Squared modulus")
plt.xlim([-2, 2])
plt.legend()
```

We retrieve indeed the frequency used to generate the signal.

### Zero padding

In order to increase the frequency resoltion, it is possible to use zero-padding adding zeros at the end of the signal. To do that, use fft function with the additional parameter n which is the number of points (initial points + zeros).

In this case code looks like : fft(function,n=2048) with 2048 the total number of points.

#### Using the previous sinusoidal signal, compute the FFT with 2048 points:

```python
#FFT of the previous signal with 2048 points

```

### Try by yourself

1) Generate a temporal signal as the sum of two sines :

$f(t) = A_1 \sin (2\pi \nu_1 t + \varphi_1 ) + A_2 \sin (2\pi \nu_2 t + \varphi_2 )$

With
- $A_1 = 1.$
- $A_2 = 0.5$
- $\varphi_1 = \pi/6$
- $\varphi_2 = 0.$
- $\nu_1 = 0.25$
- $\nu_2 = 0.33$

```python

# Temporal signal
def f(t,A1,A2,nu1,nu2,phi1,phi2):
    return A1*np.sin(2*np.pi*nu1*t+phi1)+A2*np.sin(2*np.pi*nu2*t+phi2)

# Parameters
A1,A2=1,0.5
nu1,nu2=0.25,0.33
phi1,phi2=np.pi/6,0

tmin=0.
tmax=1000.0
dt=0.1
t=np.arange(tmin,tmax,dt)
Nt=size(t)

plt.figure()
plt.plot(t,f(t,A1,A2,nu1,nu2,phi1,phi2))
plt.title('Signal')
```

2) Compute the fft of this signal. Plot on a same graph the real, imaginary part, and the amplitude.

```python
from scipy.fftpack import fft,fftfreq,fftshift,ifft,ifftshift

freq=fftshift(fftfreq(Nt,dt))

TF1=dt*fftshift(fft(f(t,A1,A2,nu1,nu2,phi1,phi2)))

#FIGURE
plt.figure()
plt.plot(freq,TF1.real,label='real')
plt.legend()
plt.title('real part of fft of the signal')
plt.xlabel('frequency')
plt.figure()
plt.plot(freq,TF1.imag,label='imaginary')
plt.legend()
plt.title('fft of the signal')
plt.xlabel('frequency')
plt.figure()
plt.plot(freq,abs(TF1),label='amplitude')
plt.xlim((0.2,0.4)) #zoom
plt.legend()
plt.title('fft of the signal')
plt.xlabel('frequency')
```

3) Add random noise using the function random.random_sample from numpy.random library.

```python
#%% Ajout de bruit
from numpy.random import random_sample
noise=2*random_sample(Nt)-1

signal=f(t,A1,A2,nu1,nu2,phi1,phi2)+0.3*noise

plt.figure()
plt.plot(t,signal)
plt.title('Noisy signal')

plt.figure()
plt.plot(t,f(t,A1,A2,nu1,nu2,phi1,phi2),label='Original signal')
plt.plot(t,signal,'r',label='Noisy signal')
plt.xlim((20,40))
```

4) Compute the fft of this noisy signal. Plot on a same graph the real, imaginary part, and the amplitude. Compare to previous result.

```python
TF2=dt*fftshift(fft(signal))

#FIGURE
plt.figure()
plt.plot(freq,TF2.real,label='real')
plt.legend()
plt.title('real part of fft of the noisy signal')
plt.xlabel('frequency')
plt.figure()
plt.plot(freq,TF2.imag,label='imaginary')
plt.legend()
plt.title('fft of the noisy signal')
plt.xlabel('frequency')
plt.figure()
plt.plot(freq,abs(TF2),label='amplitude')
plt.xlim((0.2,0.4)) #zoom
plt.legend()
plt.title('fft of the noisy signal')
plt.xlabel('frequency')
```

5) Filter this fft using a step function as :

$\Pi \left( \frac{x-x_0}{b} \right)$

Warning: you need to filter both real and imaginary part

```python
def step(x,x0,b):
    if (abs(x-x0)<=b/2.):
        return 1
    else:
        return 0

vstep=np.vectorize(step)

#%% FILTERING
# Here I put only one step function but you can put 2 of course !
TF3=TF2*(vstep(freq,0.29,0.15)+vstep(freq,-0.29,0.15))
TF3=ifftshift(TF3)
plt.figure()
plt.plot(freq,abs(TF3))

```

6) Compute the inverse fft on this filtered signal and compare to the initial raw signal. Did you manage to remove the noise ?

Warning: ifft function must be applied on the non recentered fft around the zero frequency. If needed, you may use the ifftshift function

```python
filtre=1/dt*(ifft((TF3)))
plt.figure()
plt.plot(t,real(filtre),'r')

#To check
plt.figure()
plt.plot(t,f(t,A1,A2,nu1,nu2,phi1,phi2),label='Original signal')
plt.plot(t,real(filtre),'r',label='Filtered signal')
plt.xlim((20,40))
```

### 8.2. - 2 dimensions

As an example, we will compute the Fourier transform of the image of a grid, as it can be done optically when using strioscopy.

```python
from PIL import Image # an other method to import images

# Image is imported as grayscale
grille = Image.open("grille_position_1.png").convert('F')

# 1px=5.2 microns
xmax = np.shape(grille)[1]/2*5.2e-6
xmin = -xmax
ymax = np.shape(grille)[0]/2*5.2e-6
ymin = -ymax

plt.figure()
plt.imshow(grille,cmap="gray",extent=(xmin,xmax,ymin,ymax))
plt.xlabel('$x$ (m)')
plt.ylabel('$y$ (m)')
```

```python
# FFT libraries
from scipy.fftpack import fft2,rfft,ifft2
from scipy.fftpack import fftshift
from scipy.fftpack import fftfreq
from numpy import amax
from numpy import shape

#gamma=abs(OAprime/OA) # magnification of the system
gamma = 1.0
#Correction by the transversal magnification of the system where 1px=5.2 microns
pas=5.2e-6 / gamma

TF_grille = fft2(grille)
freqx = fftfreq(shape(TF_grille)[1], d=pas)
freqy = fftfreq(shape(TF_grille)[0], d=pas)

print(np.shape(freqx))

magnitude=abs(fftshift(TF_grille))**2.0
plt.figure()
plt.imshow(np.log10(magnitude), cmap='gray', vmin=10, extent=[min(freqx), max(freqx), min(freqy), max(freqy)])
plt.savefig('TF_calculated_position_1.png')
plt.xlim([-50000, 50000])
plt.ylim([-50000, 50000])
plt.show()

```

Applying the Inverse Fourier transform it is then possible to retrieve the initial image.

```python
TF_TF_grille=ifft2(TF_grille)
spacex=fftfreq(shape(TF_TF_grille)[1],d=freqx[1]-freqx[0])
spacey=fftfreq(shape(TF_TF_grille)[0],d=freqy[1]-freqy[0])

new_grid=abs(TF_TF_grille)**2.0

plt.figure()
plt.imshow(new_grid,cmap='gray',vmin=0,extent=[min(spacex),max(spacex),min(spacey),max(spacey)],aspect=1)
plt.savefig('rerieved_grid.png')
plt.show()
```

We can also introduce Fourier filtering between the Fourier transform and inverse Fourier transform. This simulates an Fourier optics filtering (Abbe's experiment)

```python
def gauss1D(x, x0, dx, n=1):
    """
    compute the gaussian function along a vector x, centered in x0 and with a
    FWHM i intensity of dx. n=1 is for the standart gaussian while n>1 defines
    a hypergaussian

    Parameters
    ----------
    x: (ndarray) first axis of the 2D gaussian
    x0: (float) the central position of the gaussian
    dx: (float) :the FWHM of the gaussian
    n=1 : an integer to define hypergaussian, n=1 by default for regular gaussian
    Returns
    -------
    out : vector
      the value taken by the gaussian along x axis

    """
    if dx <= 0:
        raise ValueError('dx should be strictly positive')
    if not isinstance(n, int):
        raise TypeError('n should be a positive integer')
    elif n < 0:
        raise ValueError('n should be a positive integer')
    out = np.exp(-2 * np.log(2) ** (1 / n) * (((x - x0) / dx)) ** (2 * n))
    return out

def gauss2D(x, x0, dx, y, y0, dy, n=1, angle=0):
    """
    compute the 2D gaussian function along a vector x, centered in x0 and with a
    FWHM in intensity of dx and smae along y axis. n=1 is for the standard gaussian while n>1 defines
    a hypergaussian. optionally rotate it by an angle in degree

    Parameters
    ----------
    x: (ndarray) first axis of the 2D gaussian
    x0: (float) the central position of the gaussian
    dx: (float) :the FWHM of the gaussian
    y: (ndarray) second axis of the 2D gaussian
    y0: (float) the central position of the gaussian
    dy: (float) :the FWHM of the gaussian
    n=1 : an integer to define hypergaussian, n=1 by default for regular gaussian
    angle: (float) a float to rotate main axes, in degree

    Returns
    -------
    out : ndarray 2 dimensions

    """
    if angle == 0:
        data = np.transpose(np.outer(gauss1D(x, x0, dx, n), gauss1D(y, y0, dy, n)))

    else:

        theta = np.radians(angle)
        c, s = np.cos(theta), np.sin(theta)
        R = np.array(((c, -s), (s, c)))
        (x0r, y0r) = tuple(R.dot(np.array([x0, y0])))

        data = np.zeros((len(y), len(x)))

        for indx, xtmp in enumerate(x):
            for indy, ytmp in enumerate(y):
                rotatedvect = R.dot(np.array([xtmp, ytmp]))
                data[indy, indx] = np.exp(
                    -2 * np.log(2) ** (1 / n) * ((rotatedvect[0] - x0r) / dx) ** (2 * n)) * np.exp(
                    -2 * np.log(2) ** (1 / n) * ((rotatedvect[1] - y0r) / dy) ** (2 * n))

    return data

```

```python
freqx0 = 0
dx = 1000
freqy0 = 0
dy= 50000
n=4

filter = gauss2D(freqx, freqx0, dx, freqy, freqy0, dy,n=2) + gauss2D(freqx, -freqx0, dx, freqy, -freqy0, dy, n=2)
#filter +=  gauss2D(freqx, freqy0, dy, freqy, freqx0, dx,n=2) + gauss2D(freqx, -freqy0, dy, freqy, -freqx0, dx,n=2)
#filter /= 2
filter[filter < 0.] = 0.
TF_grille_filtered = TF_grille * filter

magnitude=np.abs(fftshift(TF_grille_filtered))**2.0
plt.figure()
plt.imshow(fftshift(filter), cmap='gray',  extent=[min(freqx), max(freqx), min(freqy), max(freqy)])
plt.xlim([-50000, 50000])
plt.ylim([-50000, 50000])
plt.colorbar()
plt.figure()

plt.imshow(np.log10(magnitude), cmap='gray', vmin=1, extent=[min(freqx), max(freqx), min(freqy), max(freqy)])
plt.xlim([-50000, 50000])
plt.ylim([-50000, 50000])
plt.colorbar()

TF_TF_grille_filtered=ifft2(TF_grille_filtered)
spacex=fftfreq(shape(TF_TF_grille)[1],d=freqx[1]-freqx[0])
spacey=fftfreq(shape(TF_TF_grille)[0],d=freqy[1]-freqy[0])

new_grid=abs(TF_TF_grille_filtered)**2.0
plt.figure()
plt.imshow(new_grid,cmap='gray',vmin=0,extent=[min(spacex),max(spacex),min(spacey),max(spacey)],aspect=1)

plt.show()

```

```python

```

```python

```

```python

```


---
title: "Cross entropy — Part 2: Convexity of the objective function"
date: 2021-02-24
categories: [statistics, entropy]
tags: [statistics, cross entropy, softmax, convexity]
preview_image: /enrightward.github.io/assets/img/cross-entropy/convex.png
---

## 1. Introduction

In the [previous post](https://enrightward.github.io/enrightward.github.io/posts/cross-entropy-part-1/), I recalled the definition of the _cross entropy_ $H(p, q)$ of two discrete PDFs $p$ and $q$ over the same support $S = \{ x_{1}, \ldots, x_{n} \}$. It is a loose measure of similarity of $p$ and $q$, and so is used in machine learning to define objective functions for tasks where the goal is to learn a PDF $p$ implicit in training data by updating the internal parameters of a learnt PDF $q$. I also wrote down a proof that for fixed but arbitrary $p$, the function $q \mapsto H(p, q)$ obtains a global minimum at $q = p$.

In this post, I will show that $q \mapsto H(p, q)$ is a convex function, provided we restrict $p$ and $q$ to the space of all PDFs such that no event in $S$ is impossible, i.e. we assume the $p(x_{i})$ and $q(x_{i})$ are never zero. This assumption is not very restrictive for machine learning purposes, where learnt zero probabilities are rare. It also allows us to use the _softmax_ parametrisation for $q$, explained below.

## 2. Definition of cross entropy

Recall that the _cross entropy_ of a pair $(p, q)$ of discrete PDFs with the same support $S$ is defined to be:

\begin{equation}
H(p, q) := -\sum_{x \in S}^{n} p(x) \log(q(x)).
\end{equation}

## 3. The softmax parametrisation of a discrete PDF

For any collection $t_{1}, \ldots, t_{n}$ of real numbers, the $n$-tuple:

\begin{equation}
q(t_{1}, \ldots, t_{n}) := \left( \frac{e^{t_{1}}}{Z}, \ldots, \frac{e^{t_{n}}}{Z} \right),
\end{equation}

where $Z$ is the normalisation constant $\sum_{i=1}^{n} e^{t_{i}}$, is a PDF. Indeed, each entry of $q$ is positive-valued, since its numerator and denominator are sums of real-valued exponentials, and these entries sum to $1$, by construction of $Z$. Conversely, any PDF $q = (q_{1}, \ldots, q_{n})$ with all non-zero entries can be written in this form: Simply set $t_{i}$ equal to $\log(q_{i})$. We call this the _softmax parametrisation_ of $q$.

## 4. Proof that $q \mapsto H(p, q)$ is convex

We showed in the previous post that the function $q \mapsto H(p, q)$, with $p$ fixed and $q$ varying, has a global minimum at $q = p$. We now show that this is a convex function, assuming $p$ and $q$ have no zero entries. By the above assumption, we can write $q$ using the softmax parametrisation:

\begin{equation}
q(x_{j}) = \frac{e^{t_{j}}}{Z},
\end{equation}

where $Z$ is the normalisation constant $\sum_{i=1}^{n} e^{t_{i}}$.

### Step 1: Use the softmax parametrisation

Fix $p$ and re-write $H(p, q)$ by replacing $q(x_{i})$ with $e^{t_{i}}/Z$:

\begin{equation}
H(p, q) = -\sum_{i=1}^{n} p(x_{i}) \log(q(x_{i})) = -\sum_{i=1}^{n} p(x_{i}) \log \left( \frac{e^{t_{i}}}{Z} \right) =
\sum_{i=1}^{n} p(x_{i})(\log(Z) - t_{i}) = \log(Z) - \sum_{i=1}^{n} p(x_{i}) t_{i}.
\end{equation}

### Step 2: Compute first derivative

Now we find a local optimum for $H(p, q)$, regarded as a function of $q$ for fixed but arbitrary $p$. The first step is to solve $\partial H/\partial t_{j} = 0$. To do this, observe that because $Z = \sum_{i=1}^{n} e^{t_{i}}$, we have $\partial Z/\partial t_{j} = e^{t_{j}}$, so that:

\begin{equation}
\frac{\partial \log(Z)}{t_{j}} = \frac{e^{t_{j}}}{Z} = q(x_{j}),
\end{equation}

and hence:

\begin{equation}
\frac{\partial H(p, q)}{\partial t_{j}} = q(x_{j}) - p(x_{j}).
\end{equation}

This partial derivative is zero exactly when $q(x_{j}) = p(x_{j})$.

### Step 3: The Hessian is positive definite

The previous step implies that for fixed $p$, the quantity $H(p, q)$ is locally flat around $q = p$. To show this is a global minimum, we will show $H$ is "concave up" as a function in $q$. It suffices to show that the Hessian matrix:

\begin{equation}
\nabla^{2} H(p, q) = \left( \frac{\partial^{2} H(p, q)}{\partial t_{j} \partial t_{i}} \right)_{i, j=1}^{n}
\end{equation}

is positive definite, i.e. satisfies $v^{t} \nabla^{2} H(p, q) v \ge 0$ for all $v \in \mathbb{R}^{n}$, with equality only for $v = 0$. First let's compute the entries of the Hessian. From the partial derivative calculation in the previous step, we have:

\begin{equation}
\frac{\partial^{2} H(p, q)}{\partial t_{j} \partial t_{i}} = \frac{\partial}{\partial t_{j}}(q(x_{i}) - p(x_{i})) =
\frac{\partial q(x_{i})}{\partial t_{j}},
\end{equation}

because $p$ is constant. Now,

\begin{equation}
\frac{\partial q(x_{i})}{\partial t_{j}} = \frac{\partial}{\partial t_{j}} \left( \frac{e^{t_{i}}}{Z} \right) =
\frac{Z \cdot \frac{\partial e^{t_{i}}}{\partial t_{j}} - e^{t_{i}} \cdot \frac{\partial Z}{\partial t_{j}}}{Z^{2}} =
\frac{Z \cdot \frac{\partial e^{t_{i}}}{\partial t_{j}} - e^{t_{i} + t_{j}}}{Z^{2}},
\end{equation}
noting that $\frac{\partial Z}{\partial t_{j}} = e^{t_{j}}$ from Step 2. On the other hand, the quantity $\frac{\partial e^{t_{i}}}{\partial t_{j}}$ is equal either to zero, if $i$ and $j$ differ, or else $e^{t_{i}}$, if $i$ and $j$ are the same. It follows that:

\begin{equation}
\frac{\partial^{2} H(p, q)}{\partial t_{j} \partial t_{i}} = -\frac{e^{t_{i} + t_{j}}}{Z^{2}} = -q(x_{i})q(x_{j}),
\end{equation}

if $i \neq j$, and

\begin{equation}
\frac{\partial^{2} H(p, q)}{\partial t_{i}^2{}} = \frac{Z e^{t_{i}} - e^{2t_{i}}}{Z^{2}} = q(x_{i}) - q(x_{i})^{2}.
\end{equation}

The Hessian can thus be written:

\begin{equation}
\nabla^{2} H(p, q) = \textrm{diag}(Q) - QQ^{t},
\end{equation}

where $Q = (q(x_{1}), \ldots, q(x_{n}))$ is an $n$-dimensional column vector, and $\textrm{diag}(Q)$ is the $n \times n$ matrix whose diagonal is defined by $Q$, and whose off-diagonal entries are zero. Observe now that for $v \in \mathbb{R}^{n}$, we have:

\begin{equation}
v \nabla^{2} H(p, q) v^{t} = v \, \textrm{diag}(Q) v^{t} - v QQ^{t} v^{t} = \\
\sum_{i=1}^{n} v_{i}^{2} q(x_{i}) - \sum_{i=1}^{n} v_{i}^{2} q(x_{i})^{2} =
\sum_{i=1}^{n} v_{i}^{2} q(x_{i}) (1 - q(x_{i})).
\end{equation}
Here, each summand $v_{i}^{2} q(x_{i}) (1 - q(x_{i}))$ is non-negative, being the product of a square, a probability and its complementary probability. It follows that the whole sum is non-negative. Since $p$ and $q$ have no zero probabilities by assumption, this sum can be zero only if each $v_{i}$ is zero, so $\nabla^{2} H(p, q)$ is positive definite and the minimum $q = p$ is global.

## 5. Roundup

We introduced the softmax function, which can differentiably paramatrise any discrete PDF with no zero probabilities. Then, using the softmax parametrisation, we showed that the $p \mapsto H(p, q)$ is a convex function. This convexity is important because, as noted in the [previous post](https://enrightward.github.io/enrightward.github.io/posts/cross-entropy-part-1/), a common machine learning task is to update a parametrised PDF $q$ to more closely resemble an idealised PDF $p$ implicit in some data set — language modelling often uses this technique, for example. The above convexity result, combined with last post's proof that $q \mapsto H(p, q)$ achieves a minimum at $q = p$, implies this function can be optimised using a gradient descent algorithm.


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Riding the wave

## Convection problems

Welcome to *Riding the wave: Convection problems*, the third module of  ["Practical Numerical Methods with Python"](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about).

In the [first module](https://github.com/numerical-mooc/numerical-mooc/tree/master/lessons/01_phugoid), we learned about numerical integration methods for the solution of ordinary differential equations (ODEs). The [second module](https://github.com/numerical-mooc/numerical-mooc/tree/master/lessons/02_spacetime) introduced the finite difference method for numerical solution of partial differential equations (PDEs), where we need to discretize both *space* and *time*.

This module explores the convection equation in more depth, applied to a traffic-flow problem.  We already introduced convection in [Lesson 1 of Module 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_01_1DConvection.ipynb). This hyperbolic equation is very interesting because the solution can develop *shocks*, or regions with very high gradient, which are difficult to resolve well with numerical methods.

We will start by introducing the concept of a conservation law, closely related to the convection equation. Then we'll explore different numerical schemes and how they perform when shocks are present.

## Conservation laws

You know from (non relativistic) physics that mass is _conserved_. This is one example of a conserved quantity, but there are others (like momentum and energy) and they all obey a _conservation law_. Let's start with the more intuitive case of conservation of mass.

### Conservation of mass

In any closed system, we know that the mass $M$ in the system does not change, which we can write: $\frac{D\,M}{Dt} =0$. When we change the point of view from a closed system to what engineers call a _control volume_, mass can move in and out of the volume and conservation of mass is now expressed by:

Let's imagine the control volume as a tiny cylinder of cross-section dA and length dx, like in the sketch below.

#### Figure 1. Tiny control volume in the shape of a cylinder.

If we represent the mass density by $\rho$, then mass is equal to $\rho\times$ volume. For simplicity, let's assume that mass flows in or out of the control volume only in one direction, say, the $x$-direction. Express the 1D velocity component by $u$, and conservation of mass for the control volume is translated to a mathematical expression as follows:

$$
\begin{equation}
\frac{\partial}{\partial t}\int_{\text{cv}}\rho \, dV + \int_{\text{cs}}\rho \, u\, dA =0
\end{equation}
$$

where "cv" stands for control volume and "cs" stands for control surface. The first term represents the rate of change of mass in the control volume, and the second term is the rate of flow of mass, with velocity $u$, across the control surface.

Since the control volume is very small, we can take, to leading order, $\rho$ as a uniform quantity inside it, and the first term in equation (1) can be simplified to the time derivative of density multiplied by the volume of the tiny cylinder, $dAdx$:

$$
\frac{\partial}{\partial t}\int_{\text{cv}}\rho \, dV \rightarrow \frac{\partial \rho}{\partial t} dA dx
$$

Now, for the second term in equation (1), we have to do a little more work. The quantity inside the integral is now $\rho u$ and, to leading order, we have to take into consideration that this quantity can change in the distance $dx$. Take $\rho u$ to be the value in the center of the cylinder. Then the flow of mass on each side is illustrated in the figure below, where we use a Taylor expansion of the quantity $\rho u$ around the center of the control volume (to first order).

#### Figure 2. Flux terms on the control surfaces.

Subtracting the negative flux on the left to the positive flux on the right, we arrive at the total flux of mass across the control surfaces, the second term in equation (1):

$$
\int_{\text{cs}}\rho \, u\, dA \rightarrow \frac{\partial}{\partial x}(\rho u) dA dx
$$

We can now put together the equation of conservation of mass for the tiny cylindrical control volume, which after diving by $dA dx$ is:

$$
\begin{equation}
\frac{\partial \rho}{\partial t} + \frac{\partial}{\partial x}(\rho u)=0
\end{equation}
$$

This is the 1D mass conservation equation in differential form. If we take $u$ to be a constant and take it out of the spatial derivative this equation looks the same as the first PDE we studied: the linear convection equation in [Lesson 1 of Module 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_01_1DConvection.ipynb).
But in the form shown above, it is a typical _conservation law_. The term under the spatial derivative is called the _flux_, for reasons that should be clear from our discussion above: it represents amounts of the conserved quantity flowing across the boundary of the control volume.

##### Dig deeper

You can follow the derivation of the full three-dimensional equation of conservation of mass for a flow on this screencast by Prof. Barba (duration 12:47).

```python
from IPython.display import YouTubeVideo
YouTubeVideo('35unQgSaT88')
```

### General conservation laws

All conservation laws express the same idea: the variation of a conserved quantity inside a control volume is due to the total flux of that quantity crossing the boundary surface (plus possibly the effect of any sources inside the volume, but let's ignore those for now).

The _flux_ is a fundamental concept in conservation laws: it represents the amount of the quantity that crosses a surface per unit time. Our discussion above was limited to flow in one dimension, but in general the flux has any direction and is a vector quantity. Think about this: if the direction of flow is parallel to the surface, then no quantity comes in or out. We really only care about the component of flux perpendicular to the surface. Mathematically, for a vector flux $\vec{F}$, the amount of the conserved quantity crossing a small surface element is:

$$
\vec{F}\cdot d\vec{A}
$$

where $d\vec{A}$ points in the direction of the outward normal to the surface. A general conservation law for a quantity $e$ is thus (still ignoring possible sources):

$$
\begin{equation}
\frac{\partial}{\partial t}\int_{\text{cv}}e \, dV + \oint_{\text{cs}}\vec{F}\cdot d\vec{A} =0
\end{equation}
$$

To obtain a differential form of this conservation equation, we can apply the theorem of Gauss to the second integral, which brings the gradient of $\vec{F}$ into play. One way to recognize a conservation law in differential form is that the _fluxes appear only under the gradient operator_.

Recall the non-linear convection equation from [Lesson 1 of Module 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_01_1DConvection.ipynb). It was:

$$
\begin{equation}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} = 0
\end{equation}
$$

If we look closely at the spatial derivative, we can rewrite this equation as

$$
\begin{equation}
\frac{\partial u}{\partial t} + \frac{\partial}{\partial x} \left(\frac{u^2}{2} \right) = 0
\end{equation}
$$

which is the *conservation form* of the non-linear convection equation, with flux $F=\frac{u^2}{2}$.

## Traffic flow model

We've all experienced it: as rush hour approaches certain roads in or out of city centers start getting full of cars, and the speed of travel can reduce to a crawl. Sometimes, the cars stop altogether. If you're a driver, you know that the more cars on the road, the slower your trip will flow.

Traffic flow models seek to describe these everyday experiences with mathematics, to help engineers design better road systems.

Let's review the [Lighthill-Whitham-Richards](http://en.wikipedia.org/wiki/Macroscopic_traffic_flow_model) traffic model that was offered as an exercise at the end of Module 2. This model considers cars with a continuous *traffic density* (average number of cars per unit length of road) rather than keeping track of them individually. If $\rho(x)=0$, there are no cars at that point $x$ of the road.  If $\rho(x) = \rho_{\rm max}$, traffic is literally bumper to bumper.

If the number of cars on a bounded stretch of road changes, it means that cars are entering or leaving the road somehow. _Traffic density obeys a conservation law_ (!) where the flux is the number of cars leaving the road per unit time. It is given by $F=\rho u$—as with mass conservation, flux equals density times velocity. But don't forget your experience on the road: the speed of travel depends on the car density. Here, $u$ refers not to the speed of each individual car, but to the _traffic speed_ at a given point of the road.

You know from experience that with more cars on the road, the speed of travel decreases. It is also true that if you are traveling at fast speed, you are advised to leave a larger gap with cars ahead. These two considerations lead us to propose a monotonically decreasing $u=u(\rho)$ function. As a first approximation, we may consider the linear function:

$$
\begin{equation}
u(\rho) = u_{\rm max} \left(1-\frac{\rho}{\rho_{\rm max}}\right)
\end{equation}
$$

#### Figure 3. Traffic speed vs. traffic density.

The linear model of the behavior of drivers satisfies these experimental observations:
1.  All drivers will approach a maximum velocity $u_{max}$ when the road is empty.
2.  If the road is completely jampacked ($\rho \rightarrow \rho_{max}$), velocity goes to zero.

That seems like a reasonable approximation of reality!

Applying a conservation law to the vehicle traffic, the traffic density will obey the following transport equation:

$$
\begin{equation}
\frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0
\end{equation}
$$

where $F$ is the *traffic flux*, which in the linear traffic-speed model is given by:

$$
\begin{equation}
F = \rho u_{\rm max} \left(1-\frac{\rho}{\rho_{\rm max}}\right)
\end{equation}
$$

We can now use our numerical kung-fu to solve some interesting traffic situations, and check if our simple model gives realistic results!

### Green light!

Let's say that we are examining a road of length $4$ where the speed limit is $u_{\rm max}=1$, fitting $10$ cars per unit length $(\rho_{\rm max}=10)$. Now, imagine we have an intersection with a red light at $x=2$.  At the stoplight, traffic is bumper-to-bumper, and the traffic density decreases linearly to zero as we approach the beginning of our road. Ahead of the stoplight, the road is clear.

Mathematically, we can represent this situation with the following initial condition:

$$
\begin{equation}
\rho(x,0) = \left\{
\begin{array}{cc}
\rho_{\rm max}\frac{x}{2} & 0 \leq x < 2  \\
0 & 2 \leq x \leq 4 \\
\end{array}
\right.
\end{equation}
$$

Let's see what a plot of that looks like.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
def rho_green_light(x, rho_light):
    """
    Computes the "green light" initial condition.
    It consists of a shock with a linear distribution behind it.

    Parameters
    ----------
    x : numpy.ndaray
        Locations on the road as a 1D array of floats.
    rho_light : float
        Car density at the stoplight.

    Returns
    -------
    rho : numpy.ndarray
        The initial car density along the road as a 1D array of floats.
    """
    rho = numpy.zeros_like(x)
    mask = numpy.where(x < 2.0)
    rho[mask] = rho_light * x[mask] / 2.0
    return rho
```

```python
# Set parameters.
nx = 81  # number of locations on the road
L = 4.0  # length of the road
dx = L / (nx - 1)  # distance between two consecutive locations
nt = 30  # number of time step to compute
u_max = 1.0  # maximum speed allowed on the road
rho_max = 10.0  # maximum car density allowed on the road
rho_light = 10.0  # car density at the stoplight

# Discretize the road.
x = numpy.linspace(0.0, L, num=nx)

# Compute the initial traffic density.
rho0 = rho_green_light(x, rho_light)
```

```python
# Plot the initial car density on the road.
pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$\rho$')
pyplot.grid()
pyplot.plot(x, rho0, color='C0', linestyle='-', linewidth=2)
pyplot.xlim(0.0, L)
pyplot.ylim(-0.5, 11.0);
```

**How does the traffic behave once the light turns green?** Cars should slowly start moving forward: the density profile should move to the right. Let's see if the numerical solution agrees with that!

Before we start, let's define a function to calculate the traffic flux. We'll use it in each time step of our numerical solution.

```python
def flux(rho, u_max, rho_max):
    """
    Computes the traffic flux F = V * rho.

    Parameters
    ----------
    rho : numpy.ndarray
       Traffic density along the road as a 1D array of floats.
    u_max : float
        Maximum speed allowed on the road.
    rho_max : float
        Maximum car density allowed on the road.

    Returns
    -------
    F : numpy.ndarray
        The traffic flux along the road as a 1D array of floats.
    """
    F = rho * u_max * (1.0 - rho / rho_max)
    return F
```

### Forward-time/backward-space

Start by using a forward-time, backward-space scheme, like you used in Module 2. The discretized form of our traffic model is:

$$
\begin{equation}
\frac{\rho^{n+1}_i- \rho^n_{i}}{\Delta t}+ \frac{F^n_{i}-F^n_{i-1}}{\Delta x}=0
\end{equation}
$$

Like before, we'll step in time via a for-loop, and we'll operate on all spatial points simultaneously via array operations. In each time step, we also need to call the function that computes the flux. Here is a function that implements in code the forward-time/backward-space difference scheme:

```python
def ftbs(rho0, nt, dt, dx, bc_value, *args):
    """
    Computes the history of the traffic density on the road
    at a certain time given the initial traffic density.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial car density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_value : float
        The constant density at the first station.
    args : list or tuple
        Positional arguments to be passed to the flux function.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Advance in time.
        rho[1:] = rho[1:] - dt / dx * (F[1:] - F[:-1])
        # Set the left boundary condition.
        rho[0] = bc_value
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

We're all good to go!

**Note:** The code above saves the complete traffic density at each time step—we'll use that in a second to create animations with our results.

Running the numerical solution is easy now: we just need to call the function for evolving the initial condition with the forward-time/backward-space scheme.

Let's see how that looks.

```python
# Set the time-step size based on CFL limit.
sigma = 1.0
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = ftbs(rho0, nt, dt, dx, rho0[0], u_max, rho_max)
```

Let's now create an animation of the traffic flow density using the `animation` module of Matplotlib.

```python
from matplotlib import animation
from IPython.display import HTML
```

```python
# Plot the initial traffic density.
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$\rho$')
pyplot.grid()
line = pyplot.plot(x, rho0,
                    color='C0', linestyle='-', linewidth=2)[0]
pyplot.xlim(0.0, L)
pyplot.ylim(-0.5, 11.0)
fig.tight_layout()
```

```python
def update_plot(n, rho_hist):
    """
    Update the line y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    rho_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line.set_ydata(rho_hist[n])
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

**Yikes! The solution is blowing up.** This didn't happen in your traffic-flow exercise (coding assignment) for Module 2! (Thankfully.) What is going on? Is there a bug in the code?

No need to panic. Let's take a closer look at the equation we are solving:

$$
\begin{equation}
\frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0
\end{equation}
$$

Using the chain rule of calculus, rewrite is as follows:

$$
\begin{equation}
\frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial \rho} \frac{\partial \rho}{\partial x} = 0
\end{equation}
$$

This form of the equation looks like the nonlinear convection equation from [Lesson 1 of Module 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_01_1DConvection.ipynb), right? This is a wave equation where the wave speed is $u_{\rm wave} = \frac{\partial F}{\partial\rho}$. That term is:

$$
\begin{equation}
u_{\rm wave} = \frac{\partial F}{\partial \rho} = u_{\rm max} \left( 1-2\frac{\rho}{\rho_{\rm max}} \right)
\end{equation}
$$

See how the wave speed changes sign at $\rho = \rho_{\rm max}/2$? That means that for the initial conditions given for the green-light problem, the part of the wave under $\rho = \rho_{\rm max}/2$ will want to move right, whereas the part of the wave over this mark, will move left!

There is no real problem with that in terms of the model, but a scheme that is backward in space is *unstable* for negative values of the wave speed.

## Upwind schemes

Maybe you noticed that the backward-space discretization is spatially biased: we include the points $i$ and $i-1$ in the formula. Look again at the stencil and you'll see what we mean.

#### Figure 4. Stencil of forward-time/backward-space.

In fact, the spatial bias was meant to be in the direction of propagation of the wave—this was true when we solved the convection equation (with positive wave speed $c$), but now we have some problems. Discretization schemes that are biased in the direction that information propagates are called _upwind schemes_.

Remember when we discussed the characteristic lines for the linear convection equation in [lesson 1 of the previous module](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_01_1DConvection.ipynb)? Compare the sketch of the characteristic lines with the stencil above. The point is that there is an inherent directionality in the physics, and we want the numerical scheme to have the same directionality. This is one example of _choosing an appropriate scheme_ for the physical problem.

If we wanted to solve the convection equation with negative wave speed, $c<0$, we would need a spatial bias "slanting left," which we would obtain by using the points $i$ and $i+1$ in the formula.

But if we have waves traveling in both directions, we are in a bit of a bind. One way to avoid this problem with our traffic flow model is to simply use an initial condition that doesn't produce negative speed. This should work. But later we will learn about other numerical schemes that are able to handle waves in both directions.

Just for a sanity check, let's try the forward-time/backward-space scheme with the initial conditions

\begin{equation}\rho(x,0) = \left\{ \begin{array}{cc}
2.5 x & 0 \leq x < 2  \\
0 & 2 \leq x \leq 4 \\ \end{array} \right.\end{equation}

If all values of $\rho \leq \rho_{\rm max}/2$, then $\frac{\partial F}{\partial \rho}$ is positive everywhere. For these conditions, our forward-time/backward-space scheme shouldn't have any trouble, as all wave speeds are positive.

```python
# Modify some parameters.
nt = 40  # number of time step to compute
rho_light = 5.0  # car density at the stoplight

# Compute the initial traffic density.
rho0 = rho_green_light(x, rho_light)

# Plot the initial traffic density.
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$\rho$')
pyplot.grid()
line = pyplot.plot(x, rho0,
                   color='C0', linestyle='-', linewidth=2)[0]
pyplot.hlines(rho_max / 2.0, 0.0, L,
              label=r'$\rho_{max} / 2$',
              color='black', linestyle='--', linewidth=2)
pyplot.legend()
pyplot.xlim(0.0, L)
pyplot.ylim(-0.5, 11.0)
fig.tight_layout()
```

```python
# Compute the traffic density at all time steps.
rho_hist = ftbs(rho0, nt, dt, dx, rho0[0], u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the animation.
HTML(anim.to_html5_video())
```

Phew!  It works! Try this out yourself with different initial conditions.

Also, you can easily create a new function `ftfs` to do a forward-time/forward-space scheme, which is stable for negative wave speeds. Unfortunately, forward in space is unstable for positive wave speeds.  If you don't want it blowing up, make sure the wave speed is negative everywhere: $u_{\rm wave} = \frac{\partial F}{\partial \rho} < 0 \ \forall \ x$.

Look at that solution again, and you'll get some nice insights of the real physical problem. See how on the trailing edge, a shock is developing? In the context of the traffic flow problem, a shock is a sign of a traffic jam: a region where traffic is heavy and slow next to a region that is free of cars. In the initial condition, the cars in the rear end of the triangle see a mostly empty road (traffic density is low!).  They see an empty road and speed up, accordingly.  The cars in the peak of the triangle are moving pretty slowly because traffic density is higher there.  Eventually the cars that started in the rear will catch up with the rest and form a traffic jam.

## Beware the CFL!

[Lesson 2 of Module 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_02_CFLCondition.ipynb) discusses the CFL condition for the linear convection equation. To refresh your memory, for a constant wave speed $u_{\rm wave} = c$:

$$
\begin{equation}
\sigma = c\frac{\Delta t}{\Delta x} < 1
\end{equation}
$$

What happens for non-linear equations? The wave speed is space- and time-dependent, $u_{\rm wave} = u_{\rm wave}(x,t)$, and the CFL condition needs to apply for every point in space, at every instant of time. We just need $\sigma>1$ in one spot, for the whole solution to blow up!

Let's generalize the CFL condition to

$$
\begin{equation}
\sigma = \max\left[ \left| u_{\rm wave} \right| \frac{\Delta t}{\Delta x} \right] < 1
\end{equation}
$$

which in our case is

$$
\begin{equation}
\sigma = \max\left[ u_{\rm max} \left| 1-\frac{2 \rho}{\rho_{\rm max}} \right| \frac{\Delta t}{\Delta x} \right] < 1
\end{equation}
$$

Here, the closer $\rho$ is to zero, the more likely it is to be unstable.

### Green light and CFL

We know that the green-light problem with density at the stop light $\rho = \rho_{\rm light} = 4$ is stable using a forward-time/backward -space scheme. Earlier, we used $u_{\rm max} = 1$, and $\Delta t/\Delta x=1$, which gives a CFL $= 1$, when $\rho = 0$.

What if we change the conditions slightly, say $u_{\rm max} = 1.1$?

```python
# Set parameters.
rho_light = 4.0
u_max = 1.1

# Compute the initial traffic density.
rho0 = rho_green_light(x, rho_light)

# Compute the traffic density at all time steps.
rho_hist = ftbs(rho0, nt, dt, dx, rho0[0], u_max, rho_max)

# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

That failed miserably! Only by changing $u_{\rm max}$ to $1.1$, even an algorithm that we know is stable for this problem, fails. Since we kept $\Delta t/\Delta x=1$, the CFL number for $\rho=0$ is $1.1$. See where the instability begins? Beware the CFL!

## References

* Neville D. Fowkes and John J. Mahony, *"An Introduction to Mathematical Modelling,"* Wiley & Sons, 1994. Chapter 14: Traffic Flow.

* M. J. Lighthill and G. B. Whitham (1955), On kinematic waves. II. Theory of traffic flow and long crowded roads, _Proc. Roy. Soc. A_, Vol. 229, pp. 317–345. [PDF from amath.colorado.edu](https://amath.colorado.edu/sites/default/files/2013/09/1710796241/PRSA_Lighthill_1955.pdf), checked Oct. 14, 2014. [Original source](http://rspa.royalsocietypublishing.org/content/229/1178/317.short) on the Royal Society site.

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


```python
# %load ../../preconfig.py
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
plt.rcParams['axes.grid'] = False

#import numpy as np
#import pandas as pd

#import sklearn

#import itertools

import logging
logger = logging.getLogger()

def show_image(filename, figsize=None, res_dir=True):
    if figsize:
        plt.figure(figsize=figsize)

    if res_dir:
        filename = './res/{}'.format(filename)

    plt.imshow(plt.imread(filename))

```

11 Dimensionality Reduction
========================

### 11.1 Eigenvalues and Eigenvectors of Symmetric Matrices

#### 11.1.1 Definitions
$M e = \lambda e$, where $\lambda$ is an *eigenvalue* and $e$ is the corresponding *eigenvector*.

to make the eigenvector unique, we require that:

1. every eigenvector is a *unit vector*.

2. the first nonzero component of an eigenvector is positive.

#### 11.1.2 Computing Eigenvalues and Eigenvectors
To find *principal* eigenvector, we use **power iteration** method:
start with any unit vector $v$, and compute $M^i v$ iteratively until it converges. When $M$ is a stochastic matrix, the limiting vector is the *principal* eigenvector, and its corresponding eigenvalue is 1.

Another method $O(n^3)$: algebra solution.

#### 11.1.3 Finding Eigenpairs by Power Iteration
idea:
Start by computing the pricipal eigenvector, then modify the matrix to, in effect, remove the principal eigenvector.

1. To find the pricipal eigenpair

    power iteration:
    Start with any nonzero vector $x_0$ and then iterate:
    $$x_{k+1} := \frac{M x_k}{\| M x_k\|}$$

    $x$ is the pricipal eigenvector when it reaches convergence, and $\lambda_1 = x^T M x$.

    Attention: Since power iteration will introduce small errors, inaccuracies accumulate when we try to compute all eigenpairs.

2. To find the second eigenpair

    we create $M^* = M - \lambda_1 x x^T$, then use power iteration again.

    proof: the second eigenpair of $M^*$ is also that of $M$.
    略

#### 11.1.4 The Matrix of Eigenvectors
the eigenvectors of a symmetric matrix are orthonormal

#### 11.1.5 Exercises for Section 11.1

### 11.2 Pricipal-Component Analysis
idea:
treat the set of tuples as a matrix $M$ and find the eigenvectors for $M M^T$ or $M^T M$.

```python
plt.imshow(plt.imread('./res/fig11_2.png'))
```

#### 11.2.2 Using Eigenvectors for Dimensionality Reduction
Since $M^T M e = \lambda e = e \lambda$,
Let $E$ be the matrix whose columns are the eigenvectors, ordered as largest eigenvalue first. Define the matrix $L$ to have the eigenvalues of $M^T M$ along the diagonal, largest first, and 0's in all other entries.

$M^T M E = E L$

Let $E_k$ be the first $k$ columns of $E$. Then $M E_k$ is a $k$-dimensional representation of $M$.

#### 11.2.3 The Matrix of Distances
\begin{align}
    M^T M e &= \lambda e \\
    M M^T (M e) &= M \lambda e = \lambda (M e)
\end{align}

the eigenvalues of $M M^T$ are the eigenvalues of $M^T M$ plus additional 0's, and their eigenvectors are shared.

#### 11.2.4 Exercises for Section 11.2

### 11.3 Singular-Value Decomposition
#### 11.3.1 Definition of SVD
Let $M$ be an $m \times n$ matrix, and let the rank of $M$ be $r$.

$$M = U \Sigma V^T$$

+ $U$ is an $m \times r$ *column-orthnormal matrix* (each of its columns is a **unit vector** and the dot product of any two columns is 0).

+ $V$ is an $n \times r$ column-orthnormal maxtrix.

+ $\Sigma$ is a diagonal matrix. The elements of $\Sigma$ are called the *singular values* of $M$.

```python
show_image('fig11_5.png')
```

```python
show_image('fig11_7.png', figsize=(8, 10))
```

#### 11.3.2 Interpretation of SVD
viewing the $r$ columns of $U$, $\Sigma$, and $V$ as representing *concepts* that are hidden in the original matrix $M$.

In Fig 11.7:

+ *concepts*: "science fiction" and "romance".

+ $U$ connects peopel to concepts.

+ $V$ connects movies to concepts.

+ $\Sigma$ give the strength of each of the concepts.

#### 11.3.3 Dimensionality Reduction Using SVD

```python
show_image('fig11_9.png', figsize=(8, 10))
```

If we set the $s$ smallest singular values to 0, then we can also eliminate the corresponding $s$ rows of $U$ and $V$.

```python
show_image('fig11_10.png', figsize=(8, 10))
```

#### 11.3.4 Why Zeroing Low Singular Values Works
Let $M = P Q R$, $m_{i j} = \sum_k \sum_l p_{ik} q_{kl} r_{lj}$.

Then
\begin{align}
    \| M \|^2 &= \sum_i \sum_j (m_{ij})^2 \\
              &= \sum_i \sum_j (\sum_k \sum_l p_{ik} q_{kl} r_{lj})^2 \\
              &= \sum_i \sum_j \left ( \sum_k \sum_l \sum_n \sum_m p_{ik} q_{kl} r_{lj} p_{in} q_{nm} r_{mj} \right) \\
              &\text{as $Q$ is diagonal matrix, $q_{kl}$ and $q_{nm}$ will be 0 unless $k = l$ and $n = m$.} \\
              &= \sum_i \sum_j \sum_k \sum_n p_{ik} q_{kk} r_{kj} p_{in} q_{nn} r_{nj} \\
              &= \sum_j \sum_k \sum_n \color{blue}{\sum_i p_{ik} p_{in}} q_{kk} r_{kj} q_{nn} r_{nj} \\
              &\text{as } P = U, \sum_i p_{ik} p_{in} = 1 \text{ if } k = n \text{ and 0 otherwise} \\
              &= \sum_j \sum_k q_{kk} r_{kj} q_{kk} r_{kj} \\
              &= \sum_k (q_{kk})^2
\end{align}

How many singular values should we retain?

A useful rule of thumb is to retain enough singular values to make up 90\% of the *energy* in $\Sigma$.

#### 11.3.5 Querying  Using Concepts
Let $q$ is the vector of user Quincy

+ what movies he would like?
  "concept space": $q V$, select the one whose score is highest.

+ find users similar to Quincy?
  measure the similarity of users by their cosine distance in concept space.

#### 11.3.6 Computing the SVD of a Matrix
The SVD of a matrix $M$ is strongly connected to the eigenvalues of the symmetric matrices $M^T M$and $M M^T$.

$M^T = (U \Sigma V^T)^T = V \Sigma^T U^T = V \Sigma U^T$

\begin{align}
    M^T M &= V \Sigma U^T U \Sigma V^T \\
          &= V \Sigma^2 V^T \\
    M^T M V &= V \Sigma^2 V^T V \\
            & = V \Sigma^2
\end{align}

similar, $M M^T U = U \Sigma^2$.

#### 11.3.7 Exercises for Section 11.3

### 11.4 CUR Decomposition
SVD: even if $M$ is sparse, $U$ and $V$ will be dense.

CUR: if $M$ is sparse, $C$ and $R$ will be sparse.

#### 11.4.2 Choosing Rows and Columns Properly

Let $f = \sum_{i,j} m_{ij}^2$.

1. find $C$
   + pick $r$ rows, and each row is picked with $p_i = \frac{\sum_j m_{ij}^2}{f}$.
   + normailize: each row is divided by $\sqrt{r p_i}$.

2. find $R$
   selected in the analogous way.

3. Counstructing $U$.
   + find $M$, that is the intersection of the chosen columns of $C$ and $R$.
   + compute the SVD of $W$: $W = X \Sigma Y^T$.
   + compute $\Sigma^+$, the *Moore-Penrose pseudoinverse* of the diagonal matrix $\Sigma$: replace $\sigma$ by $1/\sigma$ if $\sigma \neq 0$.
   + $U = Y (\Sigma^+)^2 X^T$.

```python
show_image('fig11_13.png')
```

##### Eliminating Duplicate Rows and Columns
it is possible that a single row or column is selected more than once, how to deal with it?

+ let it go.

+ or, merge same rows and/or columns
  $W$ will not be a sqaure matrix, then we need transpose the result to get $\Sigma^+$.

```python
show_image('ex11_17.png')
```

```python
#Exercise
```

```python

```