# Part 1: Linear Regression

```python
# Execute this code block to install dependencies when running on colab
try:
    import torch
except:
    from os.path import exists
    from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
    platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())
    cuda_output = !ldconfig -p|grep cudart.so|sed -e 's/.*\.\([0-9]*\)\.\([0-9]*\)$/cu\1\2/'
    accelerator = cuda_output[0] if exists('/dev/nvidia0') else 'cpu'

    !pip install -q http://download.pytorch.org/whl/{accelerator}/torch-1.0.0-{platform}-linux_x86_64.whl torchvision
```

## Getting started

At its heart, PyTorch is just a library for manipulating tensors. We're going to start learning how to use
PyTorch by looking at how we can implement simple linear regression.

Code speaks better than words, so lets start by looking at a bit of pytorch code to generate some 2d data to regress:

```python
import torch

# Generate some data points on a straight line perturbed with Gaussian noise
N = 1000 # number of points
theta_true = torch.Tensor([[1.5], [2.0]]) # true parameters of the line

X = torch.rand(N, 2)
X[:, 1] = 1.0
y = X @ theta_true + 0.1 * torch.randn(N, 1) # Note that just like in numpy '@' represents matrix multiplication and A@B is equivalent to torch.mm(A, B)
```

The above code generates $(x,y)$ data according to $y = 1.5x + 2$, with the $x$'s chosen from a uniform distribution. The $y$'s are additionally purturbed by adding an amount $0.1z$, where $z\sim \mathcal{N}(0,1)$ is a sample from a standard normal distribution.

Note that we represent our $x$'s as a two-dimensional (row) vector with a 1 in the second element so that the offset can be rolled into the matrix multiplication for efficiency:

\begin{align}
    y &= \mathbf{X}\begin{bmatrix}
           1.5 \\
           2
         \end{bmatrix}
  \end{align}

Let's use `matplotlib` to draw a scatter so we can be sure of what our data looks like:

```python
%matplotlib inline
import matplotlib.pyplot as plt

plt.scatter(X[:,0].numpy(), y.numpy())
plt.show()
```

__Make sure you understand how the code above is generating data; feel free to change the parameters to see what effect they have.__

Now, lets consider the situation where we have been given the tensors $X$ and $y$ and wish to compute the regression parameters. Our model looks like $\mathbf{y} = \mathbf{X\theta}$, and we wish to recover the parameters $\theta$.

As the problem is both overcomplete (only two data pairs are required to find $\theta$), and the data is noisy, we can use the Moore-Penrose Pseudoinverse to find the least-squares solution to $\theta$: $\theta = \mathbf{X^+y}$. PyTorch has a built-in pseudoinverse method (`pinverse`) that can do all the work for us:

```python
# direct solution using moore-penrose pseudo inverse
X_inv = torch.pinverse(X)
theta_pinv = torch.mm(X_inv, y)
print(theta_pinv)
```

Running the above code should give you a solution vector for $\theta$ that is very similar to the true parameter vector (`theta_true`).

## Exercise: computing the pseudoinverse from the Singular Value Decomposition.

The standard way of computing the pseudoinverse is by using the Singular Value Decomposition (SVD). The SVD is defined as: $\mathbf{X} = \mathbf{U}\Sigma\mathbf{V}^\top$. The pseudoinverse is thus $\mathbf{X}^+ = \mathbf{V}\Sigma^{-1}\mathbf{U}^\top$ where $\Sigma^{-1}$ is a diagonal matrix in which the reciprocal of the corresponding non-zero elements in $\Sigma$ has been taken.

__Use the code block below to compute the parameter vector using the SVD directly rather than the through the `pinverse` method.__ You need to store your manually computed pseudoinverse in `X_inv_svd`. Useful methods will be `torch.svd()` to compute the SVD, `[Tensor].t()` to transpose a matrix and `torch.diag()` to form a diagonal matrix from a vector.

```python
# YOUR CODE HERE
raise NotImplementedError()

theta_pinv_svd = torch.mm(X_inv_svd, y)
print(theta_pinv_svd)
```

```python
assert(torch.all(torch.lt(torch.abs(torch.add(theta_pinv, -theta_pinv_svd)), 1e-6)))
```

## Gradient based Linear Regression

Fundamentally, with linear regression we are trying to find a solution vector, $theta$ that minimises $f(\theta) = 0.5\|\mathbf{X}\theta - \mathbf{y}\|_2^2$.

We've already seen how this can be minimised directly using the pseudoinverse, but it could also be minimised by using gradient descent: $\theta \gets \theta - \alpha f'(\theta)$. (_Interesting aside_: SVD (and thus the pseudoinverse) can also be solved using gradient methods - in fact this becomes the only practical way for really large matrices.).

__Use the following block to derive and write down the gradient, $f'(\theta)$, of $f(\theta)$__. Note that you can insert latex code by wrapping expressions in dollar symbols.

__Now complete the following code block to implement your gradient as pytorch code:__

```python
def linear_regression_loss_grad(theta, X, y):
    # theta, X and y have the same shape as used previously
    # YOUR CODE HERE
    raise NotImplementedError()
    return grad
```

```python
assert(linear_regression_loss_grad(torch.zeros(2,1), X, y).shape == (2,1))

```

Now we can plug that gradient function into a basic gradient descent solver and check that the solution is close to what we get with the pseudoinverse:

```python
alpha = 0.001
theta = torch.Tensor([[0], [0]])
for e in range(0, 200):
    gr = linear_regression_loss_grad(theta, X, y)
    theta -= alpha * gr

print(theta)
```

## Real data

Doing linear regression on synthetic data is a great way to understand how PyTorch works, but it isn't quite as satisfying as working with a real dataset. Let's now apply or understanding of computing linear regression parameters to a dataset of house prices in Boston.

We'll load the dataset using scikit-learn and perform some manipulations in the following code block:

```python
from sklearn.datasets import load_boston

X, y = tuple(torch.Tensor(z) for z in load_boston(True)) #convert to pytorch Tensors
X = X[:, [2,5]] # We're just going to use features 2 and 5, rather than using all of of them
X = torch.cat((X, torch.ones((X.shape[0], 1))), 1) # append a column of 1's to the X's
y = y.reshape(-1, 1) # reshape y into a column vector
print('X:', X.shape)
print('y:', y.shape)

# We're also going to break the data into a training set for computing the regression parameters
# and a test set to evaluate the predictive ability of those parameters
perm = torch.randperm(y.shape[0])
X_train = X[perm[0:253], :]
y_train = y[perm[0:253]]
X_test = X[perm[253:], :]
y_test = y[perm[253:]]
```

__Use the following code block to compute the regression parameters using the training data in the variable `theta` by solving using the pseudoinverse directly:__

```python
# compute the regression parameters in variable theta
# YOUR CODE HERE
raise NotImplementedError()
```

We can now print out the error achieved on the test set, as well as the parameter vector:

```python
assert(theta.shape == (3,1))

print("Theta: ", theta.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta, y_test))
```

Now let's try using gradient descent:

```python
alpha = 0.00001
theta_gd = torch.rand((X_train.shape[1], 1))
for e in range(0, 10000):
    gr = linear_regression_loss_grad(theta_gd, X_train, y_train)
    theta_gd -= alpha * gr

print("Gradient Descent Theta: ", theta_gd.t())
print("MSE of test data: ", torch.nn.functional.mse_loss(X_test @ theta_gd, y_test))
```

__Use the following block to note down any observations you can make about the choice of learning rate and number of iterations in the above code. What factors do you think influence the choice?__

```python
# YOUR CODE HERE
raise NotImplementedError()
```

Finally, just so we can visualise what our model has learned, we can plot the predicted house prices (from both the direct solution and from gradient descent) along with the true value for each of the houses in the test set (ordered by increasing true value):

```python
perm = torch.argsort(y_test, dim=0)
plt.plot(y_test[perm[:,0]].numpy(), '.', label='True Prices')
plt.plot((X_test[perm[:,0]] @ theta).numpy(), '.', label='Predicted (pinv)')
plt.plot((X_test[perm[:,0]] @ theta_gd).numpy(), '.', label='Predicted (G.D.)')
plt.xlabel('House Number')
plt.ylabel('House Price ($,000s)')
plt.legend()
plt.show()
```


# Scientific Computing: Introduction to Python

Dr. Ilian Iliev

Let's start with some examples of simple printing. Note that in Python 3 the brackets are required (not the case in Python 2). Quotes can be single or double.

```python
print('hello')
```

    hello

```python
print("hello")
```

    hello

## Using Python as a (quite powerful) calculator
You can print arithmetic expressions directly at the Python prompt and it will evaluate them and give you the result, just as a calculator will do. (Note: The pound sign (#) denotes the beginning of a comment – all characters between # and the end of the line are ignored
by the Python interpreter.). For example:

```python
5+9
```

    14

```python
3**50
```

    717897987691852588770249L

```python
2000/25  #division
```

    80

```python
2000//25 #integer division
```

    80

```python
2345677*25
```

    58641925

```python
36**0.5  #square root (Note: sqrt() is not in core Python!)
```

    6.0

Note that (unlike many other programming languages) Python can do arbitrary-length arithmetic (Note: When the number has more digits than normal arithmetic Python 2 appends 'L' at the end of the result, for 'Long'; Python 3 does not do that.).

```python
45%7   # modulus (remainder)
```

    3

# Note: Beware of integer division in Python 2!
Many people have been tripped by it. Try for example:

```python
print(3/2, 3/2.0, 3.0/2.0, float(3)/2, 3/float(2))
```

    (1, 1.5, 1.5, 1.5, 1.5)

The best solution is to make sure you always divide real numbers (floats). That is not an issue for Python 3, however.
# Exercise: Evaluate some more arithmetic expressions on your own.
Core Python supports only a few mathematical functions, shown with some examples below

```python
a=-7
```

```python
b={3,9,2,55}
```

```python
c=3.1415
```

```python
abs(a)  #absolute value
```

    7

```python
max(b)  # Largest element in a sequence
```

    55

```python
min(b)  # Smallest element in a sequence
```

    2

What about other mathematical functions like e.g. square root, logarithms, trigonometric functions, etc.? Those are not built into the core Python, but are available by loading the math module (we will also discuss other modules later).

There are three ways of accessing the functions available in a module:

```python
import math  # make the 'math' module available. This is the
             # RECOMMENDED method because it is economical on resources
             # and a good programming practice. This is the method used
             # by most programmers.
```

```python
print(math.log(math.sin(0.5))) # Then this is how you use it - each
                               # function gets 'math' prefix telling
                               # Python where to get this function from.
```

    -0.735166686385

You can also 'rename' modules when importing them, for brevity or other reasons:

```python
import math as m  # From now on module math has also this other name 'm'
```

```python
m.cosh(3)
```

    10.067661995777765

You can also import a module like this:

```python
from math import *   # import ALL math module functions
```
For small modules this could be OK, but for big ones with many functions like 'math' this is wasteful, and also a bad idea for other reasons.

Finally, you can also just import specific functions that you need, in which case you can use the functions without a module prefix:

```python
from math import log, sin
```

```python
print(log(sin(0.5)))
```

    -0.735166686385

The contents of a module could be printed as follows (for example to check if a certain function is available). This type of inquiry is usually called 'introspection':

```python
dir(math)
```

    ['__doc__',
     '__file__',
     '__name__',
     '__package__',
     'acos',
     'acosh',
     'asin',
     'asinh',
     'atan',
     'atan2',
     'atanh',
     'ceil',
     'copysign',
     'cos',
     'cosh',
     'degrees',
     'e',
     'erf',
     'erfc',
     'exp',
     'expm1',
     'fabs',
     'factorial',
     'floor',
     'fmod',
     'frexp',
     'fsum',
     'gamma',
     'hypot',
     'isinf',
     'isnan',
     'ldexp',
     'lgamma',
     'log',
     'log10',
     'log1p',
     'modf',
     'pi',
     'pow',
     'radians',
     'sin',
     'sinh',
     'sqrt',
     'tan',
     'tanh',
     'trunc']

Many of the above functions should be familiar to you. If you are not sure what a given function do or how to call it, you can invoke the Python help system, as follows:

```python
help(modf)
```

    Help on built-in function modf in module math:

    modf(...)
        modf(x)

        Return the fractional and integer parts of x.  Both results carry the sign
        of x and are floats.

Exercises:

1) The Universe is 13.6 billion years old, how many seconds old is it?

2) Compute 2 to the 266th power. This is (roughly) the number of atoms
in the Universe.

3) Import the math module and calculate:

\begin{equation}
\cos(\pi/4)
\\
\pi-4 atan(1)
\end{equation}

Do results conform with your expectations?

# Variables and important data types
In Python, all numbers (and everything else, including functions) are
OBJECTS (i.e. Python, like C++ is an object-oriented language). A variable can be used to store a certain value or object. There are several basic types of variables that we will need: integers, floats (real numbers), complex numbers, strings and combinations of these (lists, tuples and arrays).

Variables are typed dynamically, i.e. their type (which can be checked with 'type(var)') is set whenever the variable is assigned a value and can be changed by assigning another value:

```python
a=1  #integer
```

```python
type(a)
```

    int

```python
b=1.0 #float=real
```

```python
type(b)
```

    float

```python
c='1.0' #string
```

```python
type(c)
```

    str

```python
d=2+3.0j #complex number
```

```python
type(d)
```

    complex

A string is a sequence of characters enclosed in single or double quotes. Long strings continuing on several lines are defined using triple quotes.

Strings are concatenated with the plus (+) operator, whereas slicing (:) is used to extract a portion of the string:

```python
string1 = 'Press return to exit'
```

```python
string2 = 'the program'
```

```python
print(string1 + ' ' + string2) # Concatenation
```

    Press return to exit the program

```python
print(string1[0:12])  # Slicing
```

    Press return

Strings are immutable objects (i.e. its individual characters cannot be modified with an assignment statement; the whole string CAN be assigned a different value) with a fixed length.

An attempt to violate immutability will result in TypeError, as shown here:

```python
s='Press return to exit'
```

```python
s[0]  # Individual characters can be accessed
```

    'P'

```python
s[0]='T' # ... but they cannot be modified
```

```python
b=2; c=4 # b and c are of an integer type
```

```python
b=b*1.0; c=float(c) # two ways to convert them back into floats
```

```python
print(b,c) # Both b and c are floats now, the original values were destroyed
```

    (2.0, 4.0)

```python
d=int(b); print(d) # 'int' converts a number into an integer
```

    2

The function 'round' with a single argument rounds up a number to the nearest integer (up or down). You can also round a number with certain number of significant digits after the decimal point.

Strings also can be converted into numbers if they are made of numbers:

```python
a=5.78666; print(round(a)); print(round(a,3))
```

    6.0
    5.787

```python
s='357'; type(int(s))
```

    int

```python
print(int(s))
```

    357

# Exercises:

1) Create a string variable 'myname' that is initialised to
your full name - first, middle and last.

2) Using a slice operator, print your first name only.

3) Using a slice operator, print your last name only.

4) Using the slice and concatenation operators, print
your name in the form 'Last name, First name'.

5) Create a new string where your middle name is replaced by
your middle initial.

6) Use Python to check how long the 'myname' string is and
if it contains the letter 'a'.

7) What happens if you multiply a string and a number?

# Tuples and lists
A tuple is a sequence of arbitrary objects separated by commas and enclosed in parentheses. If the tuple contains a single object, a final comma is required; for example, 'x = (2,)'. Tuples support the same operations as strings and are also immutable. Here is an example where the tuple 'rec' contains another tuple '(6,23,68)':

```python
rec = ('Smith','John',(24,7,1988))   # This is a tuple
```

```python
lastName,firstName,birthdate = rec # Unpacking the tuple into its parts
```

```python
print(firstName)
```

    John

```python
birthYear=birthdate[2]; print(birthYear)
```

    1988

```python
name = rec[1] + ' ' + rec[0] # combine the two name strings
```

```python
print(name)
```

    John Smith

```python
print(rec[0:2]) # Slice the first 2 elements of the tuple 'rec'
```

    ('Smith', 'John')

### Important note: In Python sequences have zero offset, so that a[0] represents the first element of a, a[1] the second one, and so forth. Similarly, indices in Python always start from 0, so e.g. 2 is the third element, etc.
Lists are ordered sets of objects similar to tuples, but they are mutable, i.e. their elements and length can be changed. A list is identified by enclosing its elements in square brackets, [,]. The objects in a list could be of any type, e.g. real or integer numbers, strings, even other lists.

Below is a sampling of operations that can be performed on lists:

```python
a = [1.0,'two',3]  # Define a list
```

```python
a[0]  # First element of the list
```

    1.0

```python
a[1]  # Second element of the list
```

    'two'

```python
a[3]  # There is no such element, so this gives an error.
```

```python
a.append(33.0)  # Append another element (33.0) to the list
```

```python
print(a)
```

    [1.0, 'two', 3, 33.0]

```python
a.insert(0,22)  # Insert another element (22) at position 0
```

```python
print(a)
```

    [22, 1.0, 'two', 3, 33.0]

Question: What does a[-1] give? What about a[-3]?If a is a mutable object, such as a list, the assignment statement b=a
does not result in a new object b, but simply creates a new reference to a (i.e. now the same object has 2 names - a and b). Thus any changes made to b will be reflected in a (but not vice-versa). To create an independent copy of a list a, use the statement 'c = a[:]', as shown below:

```python
b=a  # 'b' becomes an alias of a
```

```python
b[0]=5  # change an element of b
```

```python
print(a) # a has changed
```

    [5, 1.0, 'two', 3, 33.0]

```python
c=a[:]  # 'c' is an independent copy of a
```

```python
c[0]=4  # change an element of c
```

```python
print(a) # a remains the same
```

    [5, 1.0, 'two', 3, 33.0]

There are various oprations that can be performed on sequences like lists. Some examples:

```python
print(a[1:3]) # slices a part of a (here the second and third elements)
```

    [1.0, 'two']

```python
len(a)        # how many elements does 'a' have?
```

    5

```python
'two' in a   # checks if a certain element can be found in the list
```

    True

```python
a[2]=7       # lets make the list wholly of numbers (strings cannot be compared to numbers!)
```

```python
max(a)       # what is the largest element of the sequence?
```

    33.0

```python
min(a)       # what is the smallest element in the sequence?
```

    1.0

```python
d=a+c        # concatenates the two sequences
```

```python
print(d)
```

    [5, 1.0, 7, 3, 33.0, 4, 1.0, 'two', 3, 33.0]

# Comparison operators
The comparison operators are:
< (less than),
> (greater than),
<= (less than or equal to),
>= (greater than or equal to),
== (equal to)
!= (not equal to).

They are quite useful for programming, as we shall see. The result is a logical expression ('True' or 'False'). These can be used to construct more complicated logical statements using also 'or' and 'and'.

Numbers of different type (integer, floating point, etc.) are converted to a common type before the comparison is made. Otherwise, objects of different type are considered to be unequal (e.g. 2 does not equal 2.0). Here are a few examples:

```python
a=5.7
```

```python
b=4.3
```

```python
print(a>b)
```

    True

```python
print(a==b)
```

    False

```python
c=3
```

```python
a>b and c<b
```

    True

# Exercises:

1) What is the result of the Boolean expression 'not 8>5'?

2) What is the result of the Boolean expression 'not(True and False)'? Is this what you expected?

3) Write a compound Boolean expression that returns True if the value    of the variable 'count' is between 1 and 10 inclusive.

# Conditionals
Conditionals are quite useful in construction of algorithms and are
an important program control structure. The 'if' construct

if condition:
    block

executes a block of statements (which MUST be indented since that
is how Python knows what is inside the 'if' and what is not!) if the
condition returns true. If the condition returns false, the block
is skipped. The if conditional can be followed by any number of
'elif' (short for “else if”) constructs

if condition:
    block1
elif condition:
    block2

which work in the same manner. The 'else' clause

else:
   block

can be used to define the block of statements that are to be executed if none of the if-elif clauses is true. The function below illustrates the use of the conditionals:

```python
def sign_of_a(a):
    if a < 0.0:
        sign = 'negative'
    elif a > 0.0:
        sign = 'positive'
    else:
        sign = 'zero'
    return sign

a = 1.5
print('a is ' + sign_of_a(a))
```

    a is positive

Here 'def' defines a function, to be discussed below (similar to Matlab
functions).
# Exercises:

1) If you have several nested 'if/else'  constructs, how does Python
know to which 'if' an 'else' belongs? Try to check your answer
using an example.

2) Write a construct that sets the value of a variable called
'grade' to the value 4 if a variable named 'score' is greater
than 70, 3 if 'score' is between 60 and 69, 2 if 'score' is
between 50 and 59, 1 if 'score' is between 40 and 49, and 0 otherwise.
This statement (roughly) converts British-style grades to American-style ones.

# Loops
One of the most typical tasks we use a computer for is to do a certain task multiple times (e.g. for different input values). Such repetitive tasks (at which computers are really good, since unlike people they do not get bored) are performed by loops. There are two basic types of loops - a 'while' loop and a 'for' loop.
## 'While' loops
The 'while' construct:

while condition:
    block

executes a block of (again, indented) statements if the condition is true. After execution of the block, the condition is evaluated again. If it is still true, the block is executed again. This process is continued until the condition becomes false. The 'else' clause

else:
    block

can be used to define the block of statements that are to be executed if the condition is false.

Here is an example that creates the list [1, 1/2, 1/3, . . .]:

```python
nMax=5
```

```python
n=1
```

```python
a=[]  # Create empty list
```

```python
while n < nMax:
    a.append(1.0/n) # Append element to list
    n = n + 1
    print(a)
```

    [1.0]
    [1.0, 0.5]
    [1.0, 0.5, 0.3333333333333333]
    [1.0, 0.5, 0.3333333333333333, 0.25]

Note the indentations above - they indicate what does or does not belong to the loop (since there is no explicit loop end) and are thus absolutely necessary. The IPython shell will do the indentations for you automatically (you indicate the end the loop by pressing 'Return' twice), but an external editor might not do it for you, in which case you have to do it yourself.
## The range() command

A special type of list is frequently required (often together with
for-loops, see below) and therefore a command exists to generate that
list: the 'range(stop)' command generates a list of integers starting
from 0 and going up to but NOT INCLUDING n. You can also specify
both the start and stop points, as well as (optionally) the step

range(start,stop[,step])

(see help for further details). Here are a few examples:

```python
for a in range(5):
    print(a)          # print a
```

    0
    1
    2
    3
    4

```python
for a in range(3,10,2):
    print(a)
```

    3
    5
    7
    9

# For loops
The 'for' loops are explicit in terms of how many times the loop statements will be executed, rather than conditional as the 'while' loops we discussed above:

for target in sequence:
    block of instructions

and repeat the operations in 'block' for all values in 'sequence'. Unlike many other programming languages (e.g. C, Fortran), the loops in Python need not be over an integer index (as were the 'range' examples above), but can be performed directly over the sequence elements, which need not even be numbers. Here is an example where we iterate over a list of strings and within it over the letters in each string:

```python
fruits = ['apple','banana','orange']
```

```python
for fruit in fruits:
    for letter in fruit:
         print(letter)
```

    a
    p
    p
    l
    e
    b
    a
    n
    a
    n
    a
    o
    r
    a
    n
    g
    e

Python provides further statements for controlling the flow of a program: break, continue and else.

The 'break' command issued inside a loop immediately ends the loop and moves execution to the statements immediately following that loop.

```python
x=0
```

```python
while True: # this will go on forever unless stopped
    x += 1  # add 1 to x and put the result back in x
    if not (x % 15 or x % 25):
        break
print(x,  'is divisible by both 15 and 25')
```

    (75, 'is divisible by both 15 and 25')

The 'continue' statement allows us to skip a portion of the statements in an iterative loop. If the interpreter encounters the continue statement, it immediately returns to the beginning of the loop to start the next iteration without executing the statements below 'continue'.

The following example compiles a list of all numbers between 1 and 99 that are divisible by 7:

```python
x = []  # Create an empty list
for i in range(1,100):
   if i%7!= 0: continue # If not divisible by 7, skip rest of loop
   x.append(i) # Append i to the list
print(x)
```

    [7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98]

Finally, a 'for' or 'while' loop may be followed by an 'else' block of statements, which is executed only if the loop finished 'normally', i.e. without using 'break'.
# Exercises:

1) Write a for loop which calculates the first 10 terms of the Madhava series:

\begin{equation}
\sqrt{12}\left(1-\frac1{3\times3}+\frac1{5\times3^2}-\frac1{7\times3^3}+...\right)
\end{equation}

(this is one way to calculate $\pi$ using series).

2) Use a 'while' loop to calculate

\begin{equation}
x_{k+1}=\frac12\left(x_k+\frac2{x_k}\right)
\end{equation}

with x(0)=1, until $abs(x(k+1)-\sqrt(2)) < 0.0001$. (As we will discuss later in the term, this is a way to calculate square roots using just addition and division).

3) The double factorial function, n!!, is the product of the positive odd integers up to and including n (which must itself be odd):

\begin{equation}
 n!!=\prod\limits_{i=1}^{(n+1)/2}(2i-1)=1\cdot3\cdot5\dots(n-2)\cdot n
\end{equation}

Write a routine to calculate n!! in Python.

4) A very elegant way to calculate the greatest common denominator of two numbers is the ver elegant Euclid agorithm:

In [7]: a,b=1071,462

In [8]: while b:

            a,b=b,a%b

In [9]: print a

21

Explain how it works.

# Reading input and writing output
The intrinsic Python 2.x function for accepting user input is

raw_input(prompt)

Note: In Python 3.x this function is renamed 'input()'. In the Python 3.x version of our textbook (Kyusalaas) this is incorrect and 'raw_input' is still being used.

The 'raw_input' function displays the prompt and then reads a line of input that is converted into a string. To convert the string back into a numerical value (if it is a number), you can use the function

eval(string)

```python
a = raw_input("Input a: ") # Ask for input
```

    Input a: 34

```python
print(a,type(a))  # Print a and its type
```

    ('34', <type 'str'>)

```python
b=eval(a)  # b is a number
```

```python
print(b,type(b))  # 'b' is a number (integer)
```

    (34, <type 'int'>)

As we saw earlier, output can be displayed with the print statement:

print(object1, object2, . . .)

which converts object1, object2, and so on to strings and prints them on the same line, separated by spaces. The newline character '\n'  can be used to force a new line. For example,

```python
a=1234.56789
```

```python
b=[2,4,6,8]
```

```python
print(a,b)
```

    (1234.56789, [2, 4, 6, 8])

```python
print 'a=',a,'\nb=',b
```

    a= 1234.56789
    b= [2, 4, 6, 8]

The modulo operator '%' can be used to format a tuple. The form of the conversion statement is

'%format1 %format2 ... ' % tuple

where format1, format2 ... are the format specifications for each object in the tuple. Typically used format specifications are:

wd      Integer
w.df    Floating point notation
w.de    Exponential notation

where 'w' is the width of the field and d is the number of digits after the decimal point. The output is right-justified in the specified field and padded with blank spaces (there are provisions for changing the justification and padding). A few examples:

```python
n=9876
```

```python
print('%7.2f'%a)
```

    1234.57

```python
print('n=%6d'%n)  # Pad with spaces
```

    n=  9876

```python
print('n=%06d'%n) # Pad with zeroes
```

    n=009876

```python
print('%12.4e %6d' %(a,n))
```

      1.2346e+03   9876

# Exercise:
Calculate and print on the screen a temperature conversion table from Fahrenheit to Celsius scales. The table should include temperatures from -300 to 210 degrees Fahrenheit (in steps of 5
degrees) and their Celsius equivalents, presented in 2 columns
with appropriate headings. Each column should be 10 characters wide, and each temperature should have 3 digits to the right of the decimal point. The conversion formula is:

C=(F-32)(5/9)
# Reading from and writing to a file
Up to this point, all data we have used has been either hard-coded into the program or has been obtained from or written to the screen. However, often we need to read from an external file, or write it to such a file. This is done through Python 'file' objects.

A file object is created by opening a file with a given filename and mode. The filename could be given as an absolute path (from the root directory/folder up) or relative one (i.e. from the current working directory). 'Mode' is a string with one of the values given below, which indicates what the file will be used for (reading, writing, or both) and what type of data is being read/written (text, binary).

mode        meaning
r           text, read-only (the default)
w           text, write (an existing file with the same name will be overwritten)
a           text, append to an existing file
r+          text, reading and writing
rb          binary, read-only
wb          binary, write (an existing file with the same name will be overwritten)
ab          binary, append to an existing file
rb+         binary, reading and writing

For example:

```python
f=open('my_file.txt','w')   # open a text file for writing
```
file objects are closed with the 'close' method:

```python
f.close()
```
Note: Python automatically closes any open files when a program terminates.The 'write' method of a file object writes a string to the file (in Python3 it also returns the number of characters being written):

```python
f=open('my_file.txt','w')   # open a text file for writing
```

```python
f.write('Hello')
```

```python
f.close()
```
To read n bytes from a file we use the 'read' method, f.read(n). If n is omitted , the entire file is read in (it is your problem if the file is too large, as stated in the official documentation).

'readline()' reads a single line from a file, up to nd including the newline ('\n') character. Each subsequent call to 'readline' reads another line from the file. Both 'read' and 'readline' return an empty string when they reach the end of the file. To read all of the lines into a list of strings in one go, use 'f.readlines()'.

A better method in practice is to use Numpy to read and write data to files (to be discussed later).
# Functions
We have already encountered many internal Python functions. One can also define new, custom functions which behave the same way as the internal ones. The structure of a Python function is

def  func_name(param1,param2,...):
    statements
    return return_values

(again, note the indentations!), where 'param1, param2, ...' are the parameters. A parameter can be any Python object, including another function. Parameters may be given default values, in which case the parameter in the function call is optional. If the return statement or return values are omitted, the function returns the null object (i.e. nothing).

The following example computes numerically the first two derivatives of arctan(x) using finite differences (we will discuss this method later in the course, for now focus on the Python constructs only):

```python
from math import atan
def finite_diff(f,x,h=0.0001): # h has a default value
   df =(f(x+h) - f(x-h))/(2.0*h)
   ddf =(f(x+h) - 2.0*f(x) + f(x-h))/h**2
   return df,ddf

x = 0.5
df,ddf = finite_diff(atan,x) # Uses default value of h
print 'First derivative =',df
print 'Second derivative =',ddf
```

    First derivative = 0.799999999573
    Second derivative = -0.639999991892

Note that the function atan (arctan) was passed to the function finite_diff as a parameter.

The number of input parameters in a function definition may be left arbitrary. For example, in the function definition

def func(x1,x2,*x3)

'x1' and 'x2' are the usual parameters, also called positional parameters, whereas 'x3' is a tuple of arbitrary length containing the excess parameters (indicated by the star in front of x3). Calling this function with e.g.

func(a,b,c,d,e)

results in the following correspondence between the parameters:

\begin{equation}
a \leftrightarrow x1, b \leftrightarrow x2, (c,d,e) \leftrightarrow x3
\end{equation}

The positional parameters must always be listed before the excess parameters!
If a mutable object, such as a list, is passed to a function where it is then modified, the changes will also appear in the calling program. Here is an example:

```python
def squares(a):
    for i in range(len(a)):
       a[i] = a[i]**2

a = [1, 2, 3, 4]
squares(a)
print(a)  # the list a is now modified
```

    [1, 4, 9, 16]

# * syntax
Sometimes we want to call a function with arguments taken from a list
or other sequence. The * syntax, used in a function call unpacks such a sequence into positional arguments to the function.

Here is an example, using the function 'math.hypot(a,b)'  (this is a function, in the 'math' package, which for given 'a' and 'b' returns sqrt{a^2+b^2}. If the two arguments are in a list or tuple, the following call will fail:

```python
import math

t=[3,4]

math.hypot(t)
```
This is because we tried to call the function with a single (list) argument 't', while it requires two arguments. One can just index the list explicitly:

```python
math.hypot(t[0],t[1])
```

    5.0

which works fine, but a more elegant method is to unpack the object:

```python
math.hypot(*t)
```

    5.0

# Lambda  Statement
If a function has the form of an expression, it can be defined with the
'lambda' statement:

func name = lambda param1, param2,...: expression

This is called an anonymous function (function literal) and useful for
defining small helper function that will be used only once. Here is an example:

```python
c = lambda x,y : x**2 + y**2
print c(3,4)
```

    25

# Exercises:
1) A year is a leap year if it is divisible by 4, unless it is a century that is not divisible by 400. Write a function that takes a year as a parameter and returns 'True' if the year is a leap one and 'False' otherwise.

2) Write a function that takes two parameters - a pay rate and the number of hours worked -- and returns the total pay. Any hours over 40 paid at 1.5 times higher rate.

ExamPrep/01 Intro to Python/__EXAMPLES 01 - Intro to Python.ipynb


# Resolviendo sistemas de ecuaciones lineales

En su forma más general un sistema de ecuaciones lineales se ve como

$$
\begin{align}
A_{11} x_{1} + A_{12} x_{2} + \ldots + A_{1M} x_M &= b_1 \nonumber \\
A_{21} x_{1} + A_{22} x_{2} + \ldots + A_{2M} x_M &= b_2  \nonumber \\
&\vdots  \nonumber \\
A_{N1} x_{1} + A_{N2} x_{2} + \ldots + A_{NM} x_M &= b_N  \nonumber \\
\end{align}
$$

donde

- $A_{ij}$ y $b_i$ son los coeficientes del sistema
- $N$ es la cantidad de ecuaciones del sistema
- $M$ es la cantidad de incógnitas del sistema

El sistema anterior puede escribirse de forma matricial como

$$
A x = b
$$

donde $A \in \mathbb{R}^{N \times M}$ y $b \in \mathbb{R}^N$

Revisemos a continuación como se resuelven problemas de este tipo utilizando `scipy.linalg`

```python
%matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import scipy.linalg
```

## Solución de un sistema cuadrado

Este es un caso particular donde la matriz $A$ tiene igual número de filas y columnas ($N=M$)

Asumiendo que la inversa de $A$ existe podemos resolver este sistema como

$$
\begin{align}
A x &=  b \nonumber \\
A^{-1} A x &= A^{-1} b \nonumber \\
x &= A^{-1} b \nonumber
\end{align}
$$

donde $A A^{-1} = I$

A continuación veremos como calcular la inversa de una matriz y resolver el sistema de ecuaciones cuadrado

**Ejemplo** Sea el sistema de tres ecuaciones y tres incognitas

$$
\begin{align}
x_1 - 2x_2 + 3x_3 &= 4 \\
2x_1 - 5x2 + 12x_3 &= 15 \\
2x_2 - 10x_3 &= -10
\end{align}
$$

que podemos reescribir como

$$
\begin{pmatrix}
1 & -2 & 3 \\
2 & -5 & 12 \\
0 & 2 & -10
\end{pmatrix} \cdot
\begin{pmatrix}
x_1 \\ x_2 \\ x_3
\end{pmatrix} =
\begin{pmatrix}
4 \\
15 \\
-10
\end{pmatrix}
$$

de donde es directo identificar $A$ y $b$

```python
A = np.array([[1, -2, 3], [2, -5, 12], [0, 2, -10]])
b = np.array([4, 15, -10])
```

Dado que $A$ es una matriz cuadrada podemos intentar invertla la función [`inv`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.inv.html#scipy.linalg.inv)

```python
Ainv = scipy.linalg.inv(A)
Ainv
```

    array([[-13. ,   7. ,   4.5],
           [-10. ,   5. ,   3. ],
           [ -2. ,   1. ,   0.5]])

Con la inversa podemos resolver el problema con

```python
np.dot(Ainv, b)
```

    array([8., 5., 2.])

:::{note}

Si $A$ hubiera sido singular, es decir no invertible, la función `inv` hubiera retornado un `LinAlgError`

:::

Podemos verificar la invertibilidad de la matriz comprobando que su determinante sea distinto de cero

```python
scipy.linalg.det(A)
```

    -1.9999999999999976

Otra forma de verificar si una matriz es invertible es comprobar que todas sus columnas sean linealmente independientes (LI)

Esto es equivalente a que su rango sea igual al número de columnas, lo cual se puede verificar con la función de `NumPy`

```python
np.linalg.matrix_rank(A) == A.shape[1]
```

    True

**Resolviendo sistemas cuadrados eficientemente**

En general si sólo nos interesa $x$, podemos no realizar el cálculo explícito de  $A^{-1}$. Si un sistema de ecuaciones es grande es preferible no calcular la inversa de $A$ debido al alto costo computacional necesario

Podemos encontrar $x$ directamente en un sistema cuadrado usando la función [`solve`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.solve.html#scipy.linalg.solve)

```python
scipy.linalg.solve(A, b)
```

    array([8., 5., 2.])

El resultado es idéntico al anterior

Veamos ahora la diferencia en eficiencia utilizando un sistema más grande

```python
N = 2000
A_big = np.random.rand(N, N) # Matriz cuadrada
b_big = np.random.rand(N, 1) # Vector
%timeit -r5 -n5 np.dot(scipy.linalg.inv(A_big), b_big)
%timeit -r5 -n5 scipy.linalg.solve(A_big, b_big)
```

    297 ms ± 15.7 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)
    193 ms ± 11.6 ms per loop (mean ± std. dev. of 5 runs, 5 loops each)

```python
np.allclose(scipy.linalg.solve(A_big, b_big), np.dot(scipy.linalg.inv(A_big), b_big))
```

    True

Usar `solve` toma un poco más de la mitad del tiempo de utilizar `inv`+`dot`

¿Cómo puede ser posible esto?

La respuesta es que `solve` realiza internamente una factorización del tipo

$$
\begin{align}
A x &= b \nonumber \\
LU x &= b \nonumber \\
L z &= b \nonumber
\end{align}
$$

Donde $L$ es una matriz triangular inferior (lower) y $U$ es una matriz triangular superior (upper)

$$
L = \begin{pmatrix}
l_{11} & 0 & 0 & \ldots & 0 & 0 \\
l_{21} & l_{22} & 0 &\ldots & 0 & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\
l_{N1} & l_{N2} & l_{N3} & \ldots & l_{N(N-1)} & l_{NN} \\
\end{pmatrix} \quad
U = \begin{pmatrix}
u_{11} & u_{11} & u_{13} & \ldots & u_{1(N-1)} & u_{1N} \\
u_{21} & u_{22} & u_{32} &\ldots & u_{2(N-1)} & 0 \\
\vdots & \vdots & \vdots &\ldots & \ddots & \vdots \\
u_{N1} & 0 & 0 & \ldots & 0 & 0\\
\end{pmatrix}
$$

Luego $z$ se puede obtener recursivamente

$$
z_1 = \frac{b_1}{l_{11}}
$$
$$
z_2 = \frac{b_2 - l_{21} z_1}{l_{22}}
$$
$$
z_i = \frac{b_i - \sum_{j=1}^{i-1} l_{ij} z_j}{l_{ii}}
$$

y $x$ se puede obtener recursivamente de $z$

En caso de necesitar los factores LU podemos realizar la factorización en `scipy` con [`linalg.lu`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lu.html#scipy.linalg.lu)

```python
P, L, U = scipy.linalg.lu(A)
L, U
```

    (array([[1.  , 0.  , 0.  ],
            [0.  , 1.  , 0.  ],
            [0.5 , 0.25, 1.  ]]),
     array([[  2. ,  -5. ,  12. ],
            [  0. ,   2. , -10. ],
            [  0. ,   0. ,  -0.5]]))

## Solución de un sistema rectangular

Consideremos que

- Las incógnitas de un sistema representan sus grados de libertad
- Las ecuaciones de un sistema representan sus restricciones

Si tenemos un sistema

- con más ecuaciones que incógnitas ($N>M$): el sistema está sobredeterminado
- con más incógnitas que ecuaciones ($M>N$): el sistema está infradeterminado

:::{warning}

En ambos casos la matriz $A$ ya no es cuadrada, es decir ya no podemos calcular la inversa

:::

Sin embargo podemos utilizar otros métodos, como mostraremos a continuación

**Caso N>M**

Sea el vector de error $e = Ax - b$ de un sistema con más ecuaciones que incognitas.

Podemos encontrar una solución aproximada minimizando la norma euclidiana del error

$$
\begin{align}
\hat x &= \min_x \|e\|_2^2 \nonumber \\
& = \min_x e^T e \nonumber \\
& = \min_x (Ax -b)^T (Ax -b) \nonumber \\
\end{align}
$$

Lo cual se conoce como el **Problema de mínimos cuadrados**

Para continuar tomamos la última expresión y derivamos con respecto a $x$

$$
\begin{align}
\frac{d}{dx}  (A x - b)^T (A x -b) &= 2 A^T (A x -b) \nonumber \\
&= 2A^T A x - 2A^T b = 0 \nonumber \\
\rightarrow \hat x &= (A^T A)^{-1} A^T b \nonumber \\
&= A^{\dagger} b \nonumber \\
\end{align}
$$

donde $A^{\dagger} = (A^T A)^{-1} A^T$ se conoce como la pseudo-inversa de [Moore-Penrose](https://en.wikipedia.org/wiki/Moore%E2%80%93Penrose_inverse)

**Caso M>N**

La consecuencia de que el sistema sea infradeterminado es que $A^T A$ no es invertible.

Para resolver el problema infradeterminado se debe agregar una restricción adicional. La más típica es que el vector solución tenga norma mínima, por ejemplo

$$
\min_\theta \| x \|_2^2 ~\text{s.a.}~ Ax =b
$$

que se resuelve usando $M$ [multiplicadores de Lagrange](https://es.wikipedia.org/wiki/Multiplicadores_de_Lagrange) $\lambda$ como sigue

$$
\begin{align}
\frac{d}{dx} \| x\|_2^2 + \lambda^T (b - Ax) &= 2x - \lambda^T A  \nonumber \\
&= 2Ax - A A^T \lambda \nonumber \\
&= 2b - A A^T \lambda = 0
\end{align}
$$

De donde obtenemos que

$$
\lambda = 2(AA^T)^{-1}b
$$

y por lo tanto

$$
\hat x = \frac{1}{2} A^T \lambda = A^T (A A^T)^{-1} b,
$$

donde $A^T (A A^T)^{-1}$ se conoce como la pseudo-inversa "por la derecha"

**Resolviendo el sistema rectangular con Python**

Sea un sistema de ecuaciones lineales con $N\neq M$. Podemos usar scipy para

- Calcular la matriz pseudo inversa: [`scipy.linalg.pinv`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.pinv.html)
- Obtener la solución del sistema directamente: [`scipy.linalg.lstsq`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.lstsq.html)

Si $N>M$ se utiliza la pseudoinversa por la izquierda y la solución de mínimos cuadrados. En cambio si $M>N$ se utiliza la pseudoinversa para la derecha y la solución de multiplicadores de Lagrange

**Ejemplo:** Sea el siguiente set de datos de un estudio realizado en los años 50

Referencia: [A handbook of small datasets](https://www.routledge.com/A-Handbook-of-Small-Data-Sets/Hand-Daly-McConway-Lunn-Ostrowski/p/book/9780367449667)

```python
df = pd.read_csv('data/helados.csv', index_col=0)
df.head()
```

      cons
      income
      price
      temp

      1
      0.386
      78
      0.270
      41

      2
      0.374
      79
      0.282
      56

      3
      0.393
      81
      0.277
      63

      4
      0.425
      80
      0.280
      68

      5
      0.406
      76
      0.272
      69

donde cada fila corresponde a un día las columnas corresponde a

- consumo de helados promedio ([pintas](https://en.wikipedia.org/wiki/Pint) per capita)
- ingreso familiar promedio (dolares)
- temperatura promedio (grados Fahrenheit)
- precio promedio de los helados (dolares)

Ahora consideremos la siguiente pregunta

> ¿Está el consumo de helados influenciado por la temperatura?

Intentemos responder esta pregunta en base al siguiente modelo

$$
\text{cons} = \theta_0 + \theta_1 \cdot \text{temp}
$$

que corresponde a un sistema de ecuaciones lineales de dos incognitas y

```python
f"{len(df)} ecuaciones"
```

    '30 ecuaciones'

Es decir que es un sistema sobredeterminado

$$
\begin{pmatrix}
cons[0] \\ cons[1] \\ \vdots \\ cons[{29}]
\end{pmatrix} =
\begin{pmatrix}
1&temp[0] \\ 1&temp[1] \\ \vdots & \vdots \\ 1 & temp[29]
\end{pmatrix}
\begin{pmatrix}
\theta_0 \\ \theta_1
\end{pmatrix}
$$

que podemos resolver utilizando `lstsq`

```python
A = np.ones(shape=(len(df), 2))
A[:, 1] = df["temp"].values
b = df["cons"].values

theta, residuals, rank, s = scipy.linalg.lstsq(A, b)
theta
```

    array([0.20686215, 0.00310736])

La tupla retornada contiene

- `theta`: El resultado buscado
- `residuals`: La norma del error al cuadrado
- `rank`: El rango de $A$
- `s`: Los valores singulares de $A$

Analicemos gráficamente la solución obtenida

```python
fig, ax = plt.subplots(tight_layout=True)
ax.scatter(df["temp"], df["cons"], s=5, c='k', label='observaciones')
ax.set_xlabel('Temperatura [F]')
ax.set_ylabel('Consumo promedio')
temp_modelo = np.linspace(df["temp"].min(), df["temp"].max(), num=200)
cons_modelo = temp_modelo*theta[1] + theta[0]
ax.plot(temp_modelo, cons_modelo, label='modelo')
ax.legend();
```

De donde podemos observar que el consumo promedio tiende al alza con la temperatura promedio

:::{note}

Lo que acabamos de resolver es un problema conocido como **regresión lineal**. Más adelante veremos una forma más general de este problema

:::

## Análisis de errores y *condition number*

Incluso aunque una matriz sea matemáticamente invertible (determinante distinto de cero), podríamos no ser capaces de resolver el problema numéricamente

Imaginemos una pequeña variación en $b$ denominada $\delta b$. Esta variación provoca a su vez una pequeña variación en $x$ denominada $\delta x$

Se puede encontrar una cota que compara el error relativo de $b$ y $x$ como

$$
\frac{\| \delta x \|}{\|x\|} \leq \frac{\| A^{-1} \|  \|\delta b\|}{\|x\|}  = \|A^{-1}\| \|A\| \frac{\| \delta b \|}{\|b\|}
$$

donde se usó que $A \delta x = \delta b$ (propiedad de linealidad)

:::{note}

Esto significa que un pequeño error relativo en $b$ puede causar un gran error en $x$

:::

El estimador de $\|A^{-1}\| \|A\|$ es lo que se conoce como *condition number*

Un sistema se dice "bien condicionado" si este valor es cercano a $1$ y "mal condicionado" si es mucho mayor que $1$.

Podemos calcular el *condition number* con la función de NumPy `cond` como se muestra a continuación

```python
np.linalg.cond(x, # Arreglo multidimensional
               p # El orden de la norma: 1, 2, 'fro',...
              )
```

```python

```


# Special Series Solutions

```python
%matplotlib inline
from sympy import *
init_printing()
```

```python
x, t = symbols('x, t')
```

SymPy can compute special series like formal power series and fourier series. This is a new feature released in SymPy 1.0

Let's try computing formal power series of some basic functions.

```python
exp_series = fps(exp(x), x)
exp_series
```

This looks very similar to what ``series`` has to offer, but unlike series a formal power series object returns an infinite expansion.

```python
exp_series.infinite # Infinite representation
```

$$\sum_{k=1}^{\infty} \begin{cases} \frac{x^{k}}{k!} & \text{for}\: \operatorname{Mod}{\left (k,1 \right )} = 0 \\0 & \text{otherwise} \end{cases} + 1$$

We can easily find out any term of the expansion (no need to recompute the expansion).

```python
exp_series.term(51) # equivalent to exp_series[51]
```

```python
exp_series.truncate(10) # return a truncated series expansion
```

# Exercise

Try computing the formal power series of $\log(1 + x)$. Try to look at the infinite representation. What is the 51st term in this case? Compute the expansion about 1.

```python
log_series = fps(log(1 + x), x)
log_series
```

```python
# infinite representation
log_series.infinite
```

```python
# 51st term
log_series.term(51)
```

```python
# expansion about 1
fps(log(1 + x), x, x0=1)
```

# Fourier Series

Fourier series for functions can be computed using ``fourier_series`` function.

A sawtooth wave is defined as:
   1. $$ s(x) = x/\pi \in (-\pi, \pi) $$
   2. $$ s(x + 2k\pi) = s(x) \in (-\infty, \infty) $$

Let's compute the fourier series of the above defined wave.

```python
sawtooth_series = fourier_series(x / pi, (x, -pi, pi))
sawtooth_series
```

```python
plot(sawtooth_series.truncate(50))
```

See https://en.wikipedia.org/wiki/Gibbs_phenomenon for why the fourier series has peculiar behavior near jump discontinuties.

Just like formal power series we can index fourier series as well.

```python
sawtooth_series[51]
```

It is easy to shift and scale the series using ``shift`` and ``scale`` methods.

```python
sawtooth_series.shift(10).truncate(5)
```

```python
sawtooth_series.scale(10).truncate(5)
```

# Exercise

Consider a square wave defined over the range of (0, 1) as:
   1. $$ f(t) = 1 \in (0, 1/2] $$
   2. $$ f(t) = -1 \in (1/2, 1) $$
   3. $$ f(t + 1) = f(t) \in (-\infty, \infty) $$

Try computing the fourier series of the above defined function. Also, plot the computed fourier series.

```python
square_wave = Piecewise((1, t <= Rational(1, 2)), (-1, t > Rational(1, 2)))
```

```python
square_series = fourier_series(square_wave, (t, 0, 1))
square_series
```

```python
plot(square_series.truncate(50))
```

# What next?

Try some basic operations like addition, subtraction, etc on formal power series, fourier series and see what happens.

tutorial_exercises/Advanced - Special Series Solutions.ipynb


## 1. Basic Design of the Model
1. Output of the model: 0 or 1 (Binary Classification)
2. Hypothesis to be tested: $Z = W \cdot X + b$
3. Activation Function: $\frac{1}{1 + e^{-x}} $ (Signmoid Function)

## 2. Import Packages

1. numpy
2. matplotlib
3. seaborn

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

# Next Libraries are unimportant, they just make everyhting look better

import matplotlib.style as style
import seaborn as sns

style.use('seaborn-poster') #sets the size of the charts
style.use('ggplot')
```

## 3. Loading the dataset

```python
dataset = np.load('dataset.npz', encoding='ASCII')
## Get the numpy arrays from the dictionary
X_train = dataset['X_train']
Y_train = dataset['Y_train']
X_test  = dataset['X_test']
Y_test  = dataset['Y_test']
```

```python
print(X_train.shape)
print(Y_train.shape)
print(X_test.shape)
print(Y_test.shape)
```

    (784, 8000)
    (1, 8000)
    (784, 2000)
    (1, 2000)

```python
idx = np.random.randint(X_train.shape[1])

plt.imshow(X_train[:, idx].reshape(28, 28))

label = "cat" if Y_train[:, idx][0] else "bat"
print(f"Label: {label}")
```

## 4. Normalizing the data

Normalizing the data with the following equation:

$$ X_{norm} = \frac {X - X_{min}}{X_{max} - X_{min}} $$

For this pixel data, $X_{max} = 255$ and $X_{min} = 0$

> After running the next cell, go back and view the raw array again

```python
## Normalizing the training and testing data
X_min = 0
X_max = 255
X_train = X_train / X_max
X_test = X_test / X_max
```

## 5. Helper functions for the Model:

### Sigmoid Function and Initialize Parameters function

```python
def sigmoid(z):
    """
    Computes the element sigmoid of scalar or numpy array(element wise)

    Arguments:
        z: Scalar or numpy array

    Returns:
        s: Sigmoid of z (element wise in case of Numpy Array)
    """
    s = 1/(1+np.exp(-z))

    return s
```

```python
def initialize_parameters(n_x):
    """
    Initializes w to a zero vector, and b to a 0 with datatype float

    Arguments:
        n_x: Number of features in each sample of X

    Returns:
        w: Initialized Numpy array of shape (1, n_x) (Weight)
        b: Initialized Scalar (bias)
    """

    w = np.full((1,X_train.shape[0]),0)
    b = 0

    return w, b

```

Here is a summary of the equations for Forward Propagation and Backward Propagation we have used so far:

For m training examples $ X_{train} $ and $ Y_{train} $:

### 5.1 Forward Propagation

$$ Z^{(i)} = w \cdot X_{train}^{(i)} + b $$

$$ \hat Y^{(i)} = A^{(i)} = \sigma(Z^{(i)}) = sigmoid(Z^{(i)}) $$

$$ \mathcal{L}(\hat Y^{(i)}, Y_{train}^{(i)}) = \mathcal{L}(A^{(i)}, Y_{train}^{(i)}) = -[Y_{train}^{(i)} \log(A^{(i)}) + (1 - Y_{train}^{(i)}) \log(1 - A^{(i)})] $$

$$ J = \frac{1}{m} \sum_1^m \mathcal{L} (A^{(i)}, Y_{train}^{(i)}) $$

### 5.2 Backward Propagation - Batch Gradient Descent

$$ \frac{\partial J}{\partial w} = \frac{1}{m} (A - Y) \cdot X^T $$

$$ \frac{\partial J}{\partial b} = \frac{1}{m} \sum_1^m (A - Y) $$

> Note: $ \frac{\partial J}{\partial w} $ is represented as dw, and $ \frac{\partial J}{\partial b}$ is represented as db

```python
def compute_cost(A, Y, m):
    """
    Calculates the Cost using the Cross Entropy Loss

    Arguments:
        A: Computer Probabilities, numpy array
        Y: Known Labels, numpy array

    Returns:
        cost: The computed Cost
    """
    cost = np.sum(((- np.log(A))*Y + (-np.log(1-A))*(1-Y)))/m

    return np.squeeze(cost)
```

```python
def propagate(w, b, X, Y):
    """
    Performs forward and backward propagation for the Logistic Regression model

    Arguments:
        w: The Weight Matrix of dimension (1, n_x)
        b: Bias
        X: Input Matrix, with shape (n_x, m)
        Y: Label Matrix of shape (1, m)

    Returns:
        dw: Gradient of the weight matrix
        db: Gradient of the bias
        cost: Cost computed on Calculated Probability, and output Label
    """
    m = X.shape[1]

    A = sigmoid((w @ X)+b)
    cost = compute_cost(A, Y, m)
    dw = (np.dot(X,(A-Y).T).T)/m
    db = (np.sum(A-Y))/m

    assert(dw.shape == w.shape)
    assert(db.dtype == float)
    return dw, db, cost

```

### 5.3 Optimization

For a parameter $ \theta $, the gradient descent update rule is given by:
$$ \theta := \theta - \alpha \frac{\partial J}{\partial \theta} $$

where $\alpha$ is the learning rate

```python
def fit(w, b, X, Y, num_iterations, learning_rate, print_freq=100):
    """
    Given the parameters of the model, fits the model corresponding to the given Input Matrix aand output labels, by performing batch gradient descent for given number of iterations.

    Arguments:
        w: The Weight Matrix of dimension (1, n_x)
        b: Bias
        X: Input Matrix, with shape (n_x, m)
        Y: Label Matrix of shape (1, m)
        num_iterations: The number of iteratios of bgd to be performed
        print_freq: Frequency of recording the cost
    Returns:
        w: Optimized weight matrix
        b: optimized bias
        costs: print the cost at frequency given by print_freq, no prints if freq is 0
    """

    costs = []
    for i in range(num_iterations):
        ## 1. Calculate Gradients and cost
        dw, db, cost = propagate(w, b, X, Y)

        costs.append(cost)

        if print_freq and i % print_freq == 0:
            print(f"Cost after iteration {i}: {cost}")

        ## 2. Update parameters
        w = w - (learning_rate*dw)
        b = b - (learning_rate*db)

    return w, b, costs

```

### 5.4 Prediction
Using the following equation to determine the class that a given sample belongs to:

$$
\begin{equation}
    Y_{prediction}^{(i)} =
    \begin{cases}
        1 \text{, if } \hat Y^{(i)} \ge 0.5\\
        0 \text{, if } \hat Y^{(i)} \lt 0.5\\
    \end{cases}
\end{equation}
$$

```python
def predict(w, b, X):
    """
    Predicts the class which the given feature vector belongs to given Weights and Bias of the model

    Arguments:
        w: The Weight Matrix of dimension (1, n_x)
        b: Bias
        X: Input Matrix, with X.shape[0] = n_X
    Returns:
        Y_prediction: Predicted labels
    """

    m = X.shape[1]
    Y_prediction = np.full((1,m),0)
    A = sigmoid((w @ X) + b)
    Y_prediction = (A >= 0.5) * 1.0

    return Y_prediction
```

## 6. Building the Model

Now we have assembled all the individual pieces required to create the Logistic Regression model.
Next function is creating the model and calculating its train and test accuracy.

```python
def model(X_train, Y_train, X_test, Y_test, num_iterations, learning_rate, print_freq):
    """
    Creates a model and fit it to the train and test data. Use this model to compute the train and test accuracy after 2500 iterations

    Arguments:
        X_train: Training Data X
        Y_train: Training Data Y
        X_test: Testing Data X
        Y_test: Testing data Y
        num_iterations: Number of iterations of bgd to perform
        learning_rate: Learning Rate of the model
        print_freq: Frequency of recording the cost
    Returns:
        -None-
    """

    w, b = initialize_parameters(X_train.shape[0])
    w, b, costs = fit(w, b, X_train, Y_train, num_iterations, learning_rate, print_freq)

    Y_prediction_train = predict(w, b, X_train)
    Y_prediction_test = predict(w, b, X_test)

    costs = np.squeeze(costs)

    print(f"train accuracy: {100 - np.mean(np.abs(Y_prediction_train - Y_train)) * 100} %")
    print(f"test accuracy: {100 - np.mean(np.abs(Y_prediction_test - Y_test)) * 100} %")

    plt.plot(costs)

    plt.ylabel('cost')
    plt.xlabel('iterations (per hundreds)')
    plt.title(f"Learning rate = {learning_rate}")
    plt.show()

```

```python
model(X_train, Y_train, X_test, Y_test, num_iterations=2000, learning_rate=0.1, print_freq=100)
```


# ODE

We will solve the following linear Cauchy model

\begin{align}
y^{\prime}(t) &= \lambda y(t)\\
y(0) & = 1
\end{align}

whose exact solution is

$$
y(t) = e^{\lambda t}
$$

```python
%matplotlib inline
from numpy import *
from matplotlib.pyplot import *
import scipy.linalg
import numpy.linalg
```

```python
l = -5.   # lamba (see formula above)
t0 = 0.   # initial time
tf = 10.  # final time
y0 = 1.   # boundary condition

s = linspace(t0,tf,5000)
exact = lambda x: exp(l*x)  # exact solution
```

### Forward Euler

$$
\frac{y_{n}-y_{n-1}}{h} = f(y_{n-1}, t_{n-1})
$$

```python
def fe(l,y0,t0,tf,h):
    timesteps = arange(t0,tf+1e-10, h)
    sol = zeros_like(timesteps)
    sol[0] = y0
    for i in range(1,len(sol)):
        sol[i] = sol[i-1]*(1+l*h)

    return sol, timesteps

y, t = fe(l,y0,t0,tf,0.1)
_ = plot(t,y, 'o-')
_ = plot(s,exact(s))

error = numpy.linalg.norm(exact(t) - y, 2)
print(error)
```

### Backward Euler

$$
\frac{y_{n}-y_{n-1}}{h} = f(y_{n}, t_{n})
$$

```python
def be(l,y0,t0,tf,h):
    timesteps = arange(t0, tf+1e-10, h)
    sol = zeros_like(timesteps)
    sol[0] = y0
    for i in range(1, len(sol)):
        sol[i] = sol[i-1]/(1-l*h)

    return sol, timesteps

y,t = be(l, y0, t0, tf, 0.1)
_ = plot(t, y, 'o-')
_ = plot(s, exact(s))

error = numpy.linalg.norm(exact(t)-y, infty)
print(error)
```

### $\theta$-method

$$
\frac{y_{n}-y_{n-1}}{h} = \theta\, f(y_{n}, t_{n}) + (1-\theta)\,f(y_{n-1}, t_{n-1})
$$

```python
def tm(theta,l,y0,t0,tf,h):
    timesteps = arange(t0, tf+1e-10, h)
    sol = zeros_like(timesteps)
    sol[0] = y0
    for i in range(1, len(sol)):
        sol[i] = theta*sol[i-1]/(1-l*h) + (1. - theta)*sol[i-1]*(1+l*h)

    return sol, timesteps

y,t = tm(0.5, l, y0, t0, tf, 0.1)
_ = plot(t, y, 'o-')
_ = plot(s, exact(s))

error = numpy.linalg.norm(exact(t) - y, infty)
print(error)
```

### Simple adaptive time stepper

For each time step:
- Compute solution with CN
- Compute solution with BE
- Check the difference
- If the difference satisfy a given tolerance:
    - keep the solution of higher order
    - double the step size
    - go to the next step
- Else:
    - half the step size and repeat the time step

```python
def adaptive(l,y0,t0,tf,h0, hmax=0.9,tol=1e-3):
    sol = []
    sol.append(y0)
    t = []
    t.append(t0)
    h = h0
    while t[-1] < tf:
        #print 'current t =', t[-1], '            h=', h
        current_sol = sol[-1]
        current_t = t[-1]
        sol_cn, _ = tm(0.5,l,current_sol,current_t, current_t + h, h)
        sol_be, _ = tm(1.,l,current_sol,current_t, current_t + h, h)

        if (abs(sol_cn[-1] - sol_be[-1]) < tol): #accept
            sol.append(sol_cn[-1])
            t.append(current_t+h)
            h *= 2.
            if h > hmax:
                h=hmax
        else:
            h /= 2.

    return sol, t

y,t = adaptive(l,y0,t0,tf,0.9)
_ = plot(t,y, 'o-')
_ = plot(s,exact(array(s)))

error = numpy.linalg.norm(exact(array(t)) - y, infty)
print(error, len(y))
```


# 逆行列を求める方法

- `np.linalg` に `inv` という関数がある

```python
import numpy as np
```

```python
a = np.array([[3, 1, 1], [1, 2, 1], [0, -1, 1]])
```

```python
np.linalg.inv(a)
```

    array([[ 0.42857143, -0.28571429, -0.14285714],
           [-0.14285714,  0.42857143, -0.28571429],
           [-0.14285714,  0.42857143,  0.71428571]])

# ひとつの連立方程式を解く方法

下記の連立方程式を解くには逆行列を求めるよりも `solve` 関数を使うほうが良い。(高速かつ数値安定的なアルゴリズムを背後で利用しているため)

\begin{equation}
\begin{pmatrix}
3& 1& 1\\
1& 2& 1\\
0& -1& 1
\end{pmatrix}
\begin{pmatrix}
x \\ y\\ z
\end{pmatrix}
=
\begin{pmatrix}
1 \\ 2 \\ 3
\end{pmatrix}
\end{equation}

```python
b = np.array([[3, 1, 1], [1, 2, 1], [0, -1, 1]])
```

```python
c = np.array([1, 2, 3])
```

```python
np.linalg.solve(b, c)
```

    array([-0.57142857, -0.14285714,  2.85714286])

# 同じ係数行列からなく複数の連立方程式を解く方法

\begin{equation}
Ax=b_1, Ax=b_2, \dots, Ax=b_m
\end{equation}
となる連立方程式があったときは、 $A^{-1}$ を計算することで、

\begin{equation}
A^{-1}b_1, A^{-1}b_2, \dots, A^{-1}b_m
\end{equation}

と解が計算できる。しかし、もっと良い方法がある。

## LU分解

$A=PLU$ の形に分解することで連立方程式を高速かつ数値安定的に解くことができる。

ここで $L$ は下三角行列で対角成分が $1$ となるもの、 $U$ は上三角行列、 $P$ は各行に $1$ となる成分がただひとつだけある行列でそのほかの成分は $0$(置換行列)

\begin{equation}
PLUx = b
\end{equation}

という連立方程式は次の3つの方程式を逐次的に解くことで解 $x$ を求めることができる。

\begin{align}
Uz &= b \\
Ly &= z \\
Px &= y
\end{align}

```python
# scipy を利用
from scipy import linalg
```

```python
a = np.array([[3, 1, 1], [1, 2, 1], [0, -1, 1]])
b = np.array([1, 2, 3])
```

```python
lu, p = linalg.lu_factor(a)
linalg.lu_solve((lu, p), b)
```

    array([-0.57142857, -0.14285714,  2.85714286])

```python

```


# Mixed Integer Linear Programming (MILP)

## Introduction

* Some variables are restricted to be integers
* NP-complete
* Applications
    * Production planning
    * Scheduling
    * Many more...

## The standard form

\begin{align}
\text{maximize}\  & \mathbf{c}^T\mathbf{x} + \mathbf{k}^T\mathbf{y} \\
\text{subject to } & \\
& A\mathbf{x} &&\leq \mathbf{b} \\
& D\mathbf{y} &&\leq \mathbf{e} \\
& \mathbf{x},\mathbf{y} &&\geq 0 \\
& \mathbf{x} \in \mathbb{Z}^n
\end{align}
where $A, D \in \mathbb{R}^{m\times n}$ are matrices, $\mathbf{b}, \mathbf{e}\in\mathbb{R}^{m}$ are constants, $\mathbf{c}, \mathbf{k} \in \mathbb{R}^{n}$ objective function coefficients, and $\mathbf{x}, \mathbf{y} \in\mathbb{R}^{n}$ are the decision variables.

## CPLEX basics: Mixed Integer Programming Model
## Mathematical Model
\begin{align}
\text{maximize}\  & 2x + y + 3z \\
\text{subject to } & \\
& x+2y+z &&\leq 4 \\
& 2z + y &&\leq 5 \\
& x + y &&\geq 1 \\
& x &&\in \{0,1\} \\
& y, z \geq 0 \\
& z \in \mathbb{Z}
\end{align}

# Code in Python using docplex
## Step 1: Importing Model from docplex package

```python
from docplex.mp.model import Model
```

## Step 2: Create an optimization model

```python
milp_model = Model(name = "MILP")
```

## Step 3: Add decision variables

```python
x = milp_model.binary_var(name = 'x')
y = milp_model.continuous_var(name = 'y', lb = 0)
z = milp_model.integer_var(name="z", lb=0)
```

## Step 4: Add the constraints

```python
# Add constraint: x + 2 y + z <= 4
c1 = milp_model.add_constraint(x + 2 * y + z <= 4, ctname = "c1")

# Add constraint: 2 z + y <= 5 \\
c2 = milp_model.add_constraint(2 * z + y <= 5, ctname = "c2")

# Add constraint x + y >= 1
c3 = milp_model.add_constraint(x + y >= 1, ctname = "c3")
```

## Step 5: Define the objective function

```python
obj_fn = 2 * x + y + 3 * z
milp_model.set_objective('max', obj_fn)

milp_model.print_information()
```

## Step 6: Solve the model

```python
milp_model.solve()
```

## Step 7: Output the result

```python
milp_model.print_solution()
```

```python

```


# Lab 3
## Introduction
In this lab we will analyse population dynamics under the logisitic model with managed harvesting.

First import the modules we need.

```python
from plotly.figure_factory import create_quiver
from plotly import graph_objs as go
from numpy import meshgrid, arange, sqrt, linspace
from scipy.integrate import odeint
```

## Harvesting of fish
A population of fish in a lake, left to its own devices, is modelled by the logistic differential equation
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y),
\end{align}
where the population $y$ is in units of thousands of fish and time $t$ is measured in years.

First define a function for $\mathrm{d}y/\mathrm{d}x$ in terms of $y$ and $x$.

```python
def diff_eq(y, x):
    return 4 * y * (1 - y)
```

Next define a function that creates a Plotly Figure object that contains a slope field and, optionally, a few solutions to initial value problems.

It automates a few things we did in the last lab.

- `diff_eq` is the differential equation to be plotted
- `x` and `y` should be outputs from `meshgrid`.
- `args` is any additional arguments to `diff_eq` (we will use that below).
- `initial_values` is a list (or array) of starting $y$ values from which approximate solutions will start. The corresponding $x$ value is the minimum element of `x`.

Note that the numerical solutions will plotted for the whole range of $x$ values in `x`, so if they blow up you will probably get a warning and less-than-useful plot.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.8*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=True, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
```

The slope field below should hopefully give you some idea for the fish population dynamics.

Note that we use `arange` rather than `linspace` this week so that we can carefully control the increments between our grid points. `arange(0, 1.1, 0.25)` returns an array that starts with 0 and increments by 0.25 until it exceeds 1.1.

The plot also contains the solution curves for
$y(0) = 1$ and $y(0) = 0.4$. Edit the cell to also include the solution curve for $y(0)=1.4$.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, initial_values=(0.4, 1))
fig.show('png')
```

### Equilibrium solutions
Looking back to our differential equation, $\mathrm{d}y/\mathrm{d}t = 0$ when $y(t) = 0$ or $y(t) = 1$. Looking at the slope field, we see that the equilibrium solution $y(t) = 1$ is stable (this is the carrying capacity here, corresponding to 1000 fish), whereas the equilibrium solution $y(t) = 0$ is unstable. Any non-zero initial population will eventually stabilise at 1000 fish.

### What will happen if harvesting is now commenced at a steady rate?
For the simplest harvesting model, assume that $H$ units (thousands) of fish are taken
continuously (smoothly) over the year, rather than at one instant each year.
Note that the units of $H$ are the same as those of $\mathrm{d}y/\mathrm{d}t$, thousands of fish per year, so we simply subtract $H$ from the RHS of our existing equation to give the DE with harvesting as
\begin{align}
\frac{\mathrm{d}y}{\mathrm{d}{t}} = 4y(1-y) - H.
\end{align}
Again, the (constant) equilibrium solutions are found by setting $\mathrm{d}y/\mathrm{d}t = 0$, giving from the quadratic formula (check this),
\begin{align}
y(t) = \frac{4\pm\sqrt{16-16H}}{8} = \frac{1\pm\sqrt{1-H}}{2}.
\end{align}
What happens after harvesting starts will depend on the equilibrium solutions, their
stability and the initial number of fish $y(0)$.

Start by redefining `diff_eq` to include the `H` parameter. Note that defining `diff_eq` again overides our original definition.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
```

Now set $H = 0.6$ and plot the slope field. This is done by setting `args=(0.6,)` when we call `create_slope_field`. This is exactly how you would pass additional arguments like this one to `odeint` if you were calling it directly.

```python
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.6,))
fig.show('png')
```

From the solutions to the quadratic equation above, the equilibrium solutions of the DE are found to be $y(t) \approx 0.184$ and $y(t) \approx 0.816$. The previous equilibrium solution with no harvesting at $y(t) = 0$ has moved up to $y(t) \approx 0.184$, while the previous equilibrium solution with no harvesting at $y(t) = 1$ has moved down to $y(t) \approx 0.816$.

From the slope field, we see that the equilibrium solution $y(t) \approx 0.184$ is unstable, whereas the equilibrium solution $y(t) \approx 0.816$ is stable. If the population ever falls below about 0.184, or 184 fish, it will then drop to 0. This is a new feature, introduced by harvesting.

In the cell below, use `create_slope_field` to experiment by plotting the solutions to the initial value problems $y(0)=0.183$ and $y(0)=0.25$. Extend the $x$ range of your slope field until the top line is close to equlibrium. Note that if you extend it too far you will break `odeint` (why?). You may also like to increase the increments in `arange` to make the plot clearer.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
x, y = meshgrid(arange(0, 2.0, 0.1), arange(-0.4, 1.41, 0.1))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(0.183, 0.25))
fig.show('png')
```

## Exercises

In this lab you will experiment with the population dynamics given by the logistic equation with harvesting that we started analysing in the lab.

This week the questions will be a combination of plots and written answers.

1. Assume that the harvest is 600 fish per year. **On the same figure,**
  a. plot the slope field,
  b. plot the equilibrium solutions that we found in above, and
  c. plot the solution curves for $y(0)=1$, $y(0)=0.3$, and $y(0)=0.15$.

```python
def diff_eq(y, x, H=0):
    return 4 * y * (1 - y) - H
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.9*min(x[0][1]-x[0][0], y[1][0]-y[0][0])
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=500,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
x, y = meshgrid(arange(0, 0.9, 0.05), arange(-0.05, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.6,), initial_values=(1, 0.3, 0.15))
fig.show()
```

1.  d. In the cell below, describe the behaviour of the fish population for each of these five initial numbers of fish.

When y(0)=0.15, the curve tends to diverge.

2. a. i. Assume that $H=0.8$. Plot the slope field and five solutions, one for each equilibrium solution and one for each region between, above, or below them. You can use the equation from the lab to calculate the equilibrium solutions.

```python
def create_slope_field(diff_eq, x, y, args=(), initial_values=()):
    S = diff_eq(y, x, *args)
    L = sqrt(1 + S**2)
    scale = 0.8*min(x[0][1]-x[0][0], y[1][0]-y[0][0])  # assume a regular grid
    fig = create_quiver(x, y, 1/L, S/L, scale=scale, arrow_scale=1e-16)
    fig.layout.update(yaxis=dict(scaleanchor='x',
                                 scaleratio=1,
                                 range=[y.min()-scale, y.max()+scale]),
                      xaxis=dict(range=[x.min()-scale, x.max()+scale]),
                      showlegend=False, width=400,
                      height=0.8*(y.max()-y.min())/(x.max()-x.min())*500)
    x = linspace(x.min(), x.max())
    for y0 in initial_values:
        y = odeint(diff_eq, y0, x, args).flatten()
        fig.add_trace(go.Scatter(x=x, y=y))
    return fig
x, y = meshgrid(arange(0, 1.1, 0.05), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(0.5,), initial_values=(0.6,))
fig.show('png')
```

2. a. ii. In the cell below, describe the limiting behaviour of each line.

2. b. i. Assume that $H=1$. Plot the slope field and three solutions for the equilibrium solution and the regions above and below it.

```python
x, y = meshgrid(arange(0, 0.8, 0.1), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(1.0,), initial_values=(1, 0.3, 0.15,0.183, 0.25))
fig.show('png')
```

2. b. ii. Describe the limiting behaviour of each line.

2. c. i. Assume that  𝐻=1.2. Plot the slope field and two or three solutions.

```python
x, y = meshgrid(arange(0.0, 0.7, 0.1), arange(-0.4, 1.41, 0.05))
fig = create_slope_field(diff_eq, x, y, args=(1.2,), initial_values=(1, 0.3, 0.15,0.183, 0.25))
fig.show()
```

2. c. iii. Describe the limiting behaviour of the lines.

3. Summarize what happens to the equilibrium solutions and their stability as $H$
is increased from 0 to beyond 1. Refer to your plots to support your answers.

As H value increases, the lower the initial value of y it requires to diverge as seen on the plot.

4. What is a reasonable strategy for sustainable fishing in this case?
Don’t forget to allow qualitatively for minor catastrophes, such as disease or temporary overfishing.

THe best way to sustain fishing is to set a certain threshold on harvesting fish yearly. If we harvest too many on a yearly rate, then there's a strong likelihood that the population will plummet.


## FowardDiff.jl

[The JuliaDiff website](http://www.juliadiff.org/) describes the advantages (and some of the implementation) of autodifferentiation. The basic idea is as follows. For normal derivative approximations, you do calculations like:

```julia
Df = (f(x+ϵ) - f(x)) / ϵ
```

with ϵ sufficiently small. However, when ϵ is small [catestrophic cancellation](https://en.wikipedia.org/wiki/Loss_of_significance) occurs which causes numerical error. This puts numerical differentiation methods in a bind: a small epsilon is required to not have derivative error (since it's defined as the limit), but small epsilon results in floating point errors due to cancelation. The result is that numerical derivative approximations are very error prone.

Symbolic differentiation isn't always possible since the length of the expressions grows exponentially. But a newer technique, called autodifferentiation, can do this. The central idea is that you can directly define the derivatives of basic expressions (like `sin`), and then use the chain rule to know how to propagate derivatives. This allows you to efficiently calculate derivatives through entire codes without using small epsilon values, meaning it doesn't result in the high numerical errors that plague numerical differentiation. The result is fast and accurate derivatives.

ForwardDiff.jl is an implementation of autodifferentiation which uses [dual numbers](https://en.wikipedia.org/wiki/Dual_number).  The idea is you have a two dimensional number, and you define operators like `+` so that way the first part is the value and the second part is the derivative. For example, `sin(Ax) = sin(Ax) + cos(A)ϵ`, and then you can use the chain rule to define how the epsilon gets passed down. The resulting derivative at the end of the calculation is then simply the coefficient on the epsilon part. Here, ϵ is part of a number type, and this is all done by multiple dispatch and defining new dispatches on basic mathematical functions for Dual numbers which also compute the derivative.

### Problem

The ForwardDiff.jl documentation can be found here: http://www.juliadiff.org/ForwardDiff.jl/stable/user/api.html

1. Use the `ForwardDiff.derivative` function to take the derivative of `x^5 + 3x^2`. Compare the result to the analytical solution by plotting both with Plots.jl, overlaying one on top of the other (hint: `plot!` can be helpful!).

2. The transformation from spherical to cartesian coordinates is given by:

$$
\begin{align}
x &= r\sin(\theta)\cos(\phi) \\
y &= r\sin(\theta)\sin(\phi) \\
z &= r\cos(\theta)
\end{align}
$$

Use ForwardDiff.jl to calculate the Jacobian of the transformation. Compare the determinant of the Jacobian against the analytical solution $r^2 \sin(\theta)$.


# ODE boundary value problems

Let's have a look at how the DifferentialEquations.jl package handles boundary value problems. Let's solve the simple harmonic equation
$$
u'' + 3 u = 0
$$
on the interval $t \in [0,2\pi]$ where $u(0)=7$ and $u(2 \pi)=0$. The exact solution is given by
$$
\begin{align}
u(t) = 7 \cos \left(\sqrt{3}t \right) - 7 \cot \left(2\sqrt{3}\pi \right) \sin \left(\sqrt{3}t \right).
\end{align}
$$

The second order equation can be rewritten as the first order system
$$
\begin{align}
\left( u \right)' =& u', \\
\left( u' \right)' =& - 3 u.
\end{align}
$$
So let's solve this two point boundary value problem numerically using Julia.

```julia
#using Pkg
#Pkg.add( "BoundaryValueDiffEq" )
```

```julia
using DifferentialEquations, BoundaryValueDiffEq, Plots
using BenchmarkTools
```

```julia
function harmonic!(du,u,p,t)
  du[1] = u[2]
  du[2] = - 3 * u[1]
end
```

    harmonic! (generic function with 1 method)

```julia
function boundary_conditions!(residual, u, p, t)
    residual[1] = u[1][1] - 7
    residual[2] = u[end][1]
end
```

    boundary_conditions! (generic function with 1 method)

```julia
tspan = (0.0,2*pi)
tstep = 0.05
initial_guess = [7.0,-1.0]
bvp = TwoPointBVProblem( harmonic!, boundary_conditions!, initial_guess, tspan )
sol = solve( bvp, MIRK4(), dt=tstep);
```

```julia
plot( sol, vars=(1), lw=2, label="Numerical solution" )

function exact( t )
     return 7*( cos(sqrt(3)*t) - cot(2*sqrt(3)*pi) * sin(sqrt(3)*t) )
end

plot!( sol.t, t->exact(t), lw=2,ls=:dash, label="Exact solution" )
```

```julia
error = map( exact, sol.t )
for i in 1:length(sol.t)
    error[i] = error[i] - sol.u[i][1]
end
plot( sol.t, error, label="Error", lw=2 )
```

Let's try a slightly more difficult example such as the Blasius equation
$$
f''' + f f'' = 0,
$$
with the conditions $f(0)=f'(0)=0$ and $f'(\infty)=1$. We can rewrite the Blasius equation as the first order system
$$
\begin{align}
\left( f \right)' =& f', \\
\left( f' \right)' =& f'', \\
\left( f'' \right)' =& -f f''.
\end{align}
$$

```julia
function Blasius!(du,u,p,t)
    du[1] = u[2]
    du[2] = u[3]
    du[3] = -u[1] * u[3]
end
```

    Blasius! (generic function with 1 method)

```julia
function Blasius_bcs!(residual, u, p, t)
    residual[1] = u[1][1]
    residual[2] = u[1][2]
    residual[3] = u[end][2] - 1
end
```

    Blasius_bcs! (generic function with 1 method)

```julia
tinf = 20
N = 200
tspan = (0.0,20.0)
tstep = (tspan[2] - tspan[1])/N
# Make an initial guess
t = tspan[1]:tstep:tspan[2]
function f_guess( t )
    return t * ( 1.0 - exp( -t ) )
end
function f_d_guess( t )
    return ( 1.0 - exp( -t ) ) + t * exp( -t )
end
function f_dd_guess( t )
    return 2 * exp( -t ) - t * exp( -t )
end
f = map( f_guess, t )
f_d = map( f_d_guess, t )
f_dd = map( f_dd_guess, t )
#initial_guess = [f,f_d,f_dd]
initial_guess = [1.0,1.0,0.0]
bvp_Blasius = TwoPointBVProblem( Blasius!, Blasius_bcs!, initial_guess, tspan )
@btime sol_Blasius = solve( bvp_Blasius, MIRK4(), dt=tstep );
```

      2.528 s (40327661 allocations: 3.56 GiB)

```julia
plot( sol_Blasius, xlims=(0,10), ylims=(0,2), label=["f" "f'" "f''"] )
```

Adding the initial guesses doesn't seem to help much with speed or the memory allocation. How can the performance be improved?

```julia

```


# Lucas_task 19 - Motor Control
### Introduction to modeling and simulation of human movement
https://github.com/BMClab/bmc/blob/master/courses/ModSim2018.md

### 1) Find the extrema in the function f(x)=x3−7.5x2+18x−10 analytically and determine if they are minimum or maximum.

Import Python libraries

```python
import numpy as np
%matplotlib inline
import matplotlib
import matplotlib.pyplot as plt
import sympy as sym
from sympy.plotting import plot
import pandas as pd
from IPython.display import display
from IPython.core.display import Math
```

Finding the roots

```python
x = sym.symbols('x')
F = x**3 - 7.5*x**2 + 18*x - 10
Fdiff = sym.expand(sym.diff(F, x))
roots = sym.solve(Fdiff, x)
display(Math(sym.latex('Roots:') + sym.latex(roots)))
```

$$Roots:\left [ 2.0, \quad 3.0\right ]$$

Finding the minimum and maximum

```python
x1 = 2
x2 = 3
F1 = x1**3 - 7.5*x1**2 + 18*x1 - 10
F2 = x2**3 - 7.5*x2**2 + 18*x2 - 10

print("F(x1): ", F1)
print("\nF(x2): ", F2)

```

    F(x1):  4.0

    F(x2):  3.5

```python
print("Minimim:", x2,"\nMaximum:", x1)

```

    Minimim: 3
    Maximum: 2

### 2) Find the minimum in the f(x)=x3−7.5x2+18x−10 using the gradient descent algorithm.

```python
cur_x = 6               # The algorithm starts at x=6
gamma = 0.01            # step size multiplier
precision = 0.00001
step_size = 1           # initial step size
max_iters = 10000       # maximum number of iterations
iters = 0               # iteration counter

f  = lambda x: x**3 - 7.5*x**2 + 18*x - 10  # lambda function for f(x)
df = lambda x: 3*x**2 - 15*x  + 18   # lambda function for the gradient of f(x)

while (step_size > precision) & (iters < max_iters):
    prev_x = cur_x
    cur_x -= gamma*df(prev_x)
    step_size = abs(cur_x - prev_x)
    iters+=1

print('True local minimum at {} with function value {}.'.format(9/4, f(9/4)))
print('Local minimum by gradient descent at {} with function value {}.'.format(cur_x, f(cur_x)))
```

    True local minimum at 2.25 with function value 3.921875.
    Local minimum by gradient descent at 3.000323195755751 with function value 3.5000001567170003.

```python

```


```python
import numpy as np
```

```python
# matrix multiplication
a = np.array([[0,6],[2,8],[4,10]])
b = np.array([[1,3,5],[7,9,11]])
print(a)
print(b)
print(np.dot(a,b))
```

    [[ 0  6]
     [ 2  8]
     [ 4 10]]
    [[ 1  3  5]
     [ 7  9 11]]
    [[ 42  54  66]
     [ 58  78  98]
     [ 74 102 130]]

```python
print(np.matmul(a,b))
```

    [[ 42  54  66]
     [ 58  78  98]
     [ 74 102 130]]

```python
# sigmoid activation function
def sigmoid(x):
    return 1 / (1 + np.exp(-x))
```

```python
x1=1
x2=1
score = 4*x1 + 5*x2 - 9
```

```python
sigmoid(score)
```

    0.5

```python
w1=3
w2=5
b=-2.2

sigmoid(w1*0.4 + w2*0.6 + b)
```

    0.8807970779778823

```python
# eigenvector and eigenvalues
A = np.array([[3, 0],[-9, 6]])
x = np.array([1, 3])

np.dot(A, x)
```

    array([3, 9])

```python
# picture size calculation
#     W  = input volume
#     K  = Kernel size
#     P =  padding
#     S = stride

np.int_(((130-3+0)/1)+1)
```

    128

$[(W−K+2*P)/S]+1$

```python
# calculating padding that has kernel_size of 7 for CNN that is the same x-y size as an input array
# see above equation; using P = 3
x = (-7 + 2*3/1) + 1 # results in zero; W - 0 = W
print(np.int_(x))
```

    0

```python
from sympy import Symbol

Symbol('w') + x == Symbol('w') # W = W; CNN has same x-y size as input array
```

    True

```python
# nlp: Subsampling equation
# text with 1 million words in it
# word "learn" appears 700 times in this text
# threshold is 0.0001
```

$p = 1 - \sqrt{\frac{t}{f(x_[i])}}$

```python
# -> calculate probability that we will discard the word "learn"
f = 700
w_i = 1e6
t = 1e-4
```

```python
p = 1 - np.sqrt(t/(f/w_i))
print(p)
```

    0.6220355269907727

```python
# For an input [23, 702, 89, 15, 99] and R=2, what will the returned context be if we pass in the token at idx=1 (702)?
def get_target(words, idx, R):
    start = idx - R if (idx - R) > 0 else 0
    stop = idx + R
    target_words = words[start:idx] + words[idx+1:stop+1]

    return list(target_words)
```

```python
print(get_target(words=[23, 702, 89, 15, 99], idx=1, R=2))
```

    [23, 89, 15]


# sympy

Since we want to work interactively with `sympy`, we will import the complete module. Note that this polutes the namespace, and is not recommended in general.

```python
from sympy import *
```

Enable pretty printing in this notebook.

```python
init_printing()
```

## Expression manipulation

Define a number of symbols to work with, as well as an example expression.

```python
x, y, a, b, c = symbols('x y a b c')
```

```python
expr = (a*x**2 - b*y**2 + 5)/(c*x + y)
```

Check the expression's type.

```python
expr.func
```

    sympy.core.mul.Mul

Although the expression was defined as a divisino, it is represented as a multiplicatino by `sympy`.  The `args` attribute of an expressions stores the operands of the top-level operator.

```python
expr.args
```

Although the first factor appears to be a division, it is in fact a power.  The denominator of this expression would be given by:

```python
expr.args[0].func
```

    sympy.core.power.Pow

```python
expr.args[0].args[0]
```

The expression $\frac{1}{a x + b}$ can alternatively be defined as follows, which highlights the internal representation of expressions.

```python
expr = Pow(Add(Mul(a, x), b), -1)
```

```python
pprint(expr)
```

       1
    ───────
    a⋅x + b

```python
expr.args
```

```python
expr.args[0].args[0]
```

This may be a bit surprising when you look at the mathematical representation of the expression, but the order of the terms is different from its rendering on the screen.

```python
expr.args[0].args
```

Since the addition operation is commutative, this makes no difference mathematically.

```python
expr.args[0].args[1].args
```

```python
expr = x**2 + 2*a*x + y**2
```

```python
expr2 = expr.subs(y, a)
expr2
```

Most expression manipulation algorithms can be called as functions, or as methods on expressions.

```python
factor(expr2)
```

```python
expr2.factor()
```

```python
x, y = symbols('x y', positive=True)
```

```python
(log(x) + log(y)).simplify()
```

## Calculus

### Series expansion

```python
x, a = symbols('x a')
```

```python
expr = sin(a*x)/x
```

```python
expr2 = series(expr, x, 0, n=7)
```

```python
expr2
```

A term of a specific order in a given variable can be selected easily.

```python
expr2.taylor_term(2, x)
```

When the order is unimportant, or when the expression should be used to define a function, the order term can be removed.

```python
expr2.removeO()
```

Adding two series deals with the order correctly.

```python
s1 = series(sin(x), x, 0, n=7)
```

```python
s2 = series(cos(x), x, 0, n=4)
```

```python
s1 + s2
```

### Derivatives and integrals

```python
expr = a*x**2 + b*x + c
```

```python
expr.diff(x)
```

```python
expr.integrate(x)
```


# SymPy
The SymPy package is useful for symbolic algebra, much like the commercial software Mathematica.

We won't make much use of SymPy during the boot camp, but it is definitely useful to know about
for mathematics courses.

```python
import sympy as sp
sp.init_printing()
```

# Symbols and Expressions
We'll define `x` and `y` to be `sympy` symbols, then do some symbolic algebra.

```python
x, y = sp.symbols("x y")
```

```python
expression = (x+y)**4
```

```python
expression
```

```python
sp.expand(expression)
```

```python
expression = 8*x**2 + 26*x*y + 15*y**2
```

```python
expression
```

```python
sp.factor(expression)
```

```python
expression - 20 *x*y - 14*y**2
```

```python
sp.factor(expression - 20*x*y - 14*y**2)
```

# Lambdify: Making python functions from sympy expressions

```python
expression
```

```python
f = sp.lambdify((x,y), expression, 'numpy')
```

```python
f(3,4)
```

```python
8 * 3**2 + 26 * 3 * 4 + 15 * 4**2
```

# Calculus
You can use `sympy` to perform symbolic integration or differentiation.

```python
expression = 5*x**2 * sp.sin(3*x**3)
```

```python
expression
```

```python
expression.diff(x)
```

```python
expression = sp.cos(x)
```

```python
expression.integrate(x)
```

```python
expression.integrate((x, 0, sp.pi / 2))
```

You can also create unevalated integrals or derivatives. These can later be evaluated with their `doit` methods.

```python
deriv = sp.Derivative(expression)
```

```python
deriv
```

```python
deriv.doit()
```

```python
inte = sp.Integral(expression, (x, 0, sp.pi / 2))
```

```python
inte
```

```python
inte.doit()
```

```python

```


# Activity: Logistic Functions

```python
from cyllene import *
from sympy import solve,log
from f_special import logistic_function, logistic_plot_L, second_derivative
```

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
%matplotlib inline

from ipywidgets import interact, interactive
from IPython.display import Image

```

## The definition of a logistic function

A *logistic function* is a function of the form

$f(x) = \dfrac{L}{1+C \, e^{-kx}},$

where $L, C, k$ are *parameters*.

Logistic functions *initially* exhibit growth that is similar to *exponential*, but after a while the growth begins to slow down as a saturation property takes hold of the growth process.

```python
f = function('1/(1+10*exp(-x))')
```

```python
graph(f(x))
```

### The parameter $L$

Let us take a look at the paramater $L$. Use the slider below to change the value of $L$.
What effect does this have on the graph of a logistic function?

```python
interactive_plot = interactive(logistic_plot_L, L=(1, 8, 0.5))
output = interactive_plot.children[-1]
output.layout.height = '300px'
interactive_plot
```

    interactive(children=(FloatSlider(value=4.0, description='L', max=8.0, min=1.0, step=0.5), Output(layout=Layou…

We see that $L$ determines the **height of the graph**. It is the **limiting value** of the logistic function. The values of $f$ will approach $L$ as $x$ goes to infinity. The parameter $L$ is also called the **carrying capacity**.

## The inflection point

Let us again consider a simple logistic function:

```python
f = function('8/(1+10*exp(-2*x))')
```

To find possible inflection points, we need to compute the second derivative:

```python
f2 = second_derivative(f(x))
f2
```

$\displaystyle - \frac{320 e^{- 2 x}}{\left(1 + 10 e^{- 2 x}\right)^{2}} + \frac{6400 e^{- 4 x}}{\left(1 + 10 e^{- 2 x}\right)^{3}}$

Try to find the zeros for this expression.

You can check your answer by running the code below:

```python
p = solve(f2,x)[0]
solve(f2,x)[0]
```

$\displaystyle \log{\left(\sqrt{10} \right)}$

Now plug the solution into the original function:

```python
f(p)
```

$\displaystyle 4$

Repeat this with different values for the parameters $L, C, k$. Do you observe a relation between $L$ and the location of the inflection point?

### **FACT**: The inflection point of a logistic function occurs where $f(x) = L/2$.

```python
Image(filename='logistic_function.png', width=600)
```

---

## Logistic modeling of COVID-19 data

In the following cell, we read the current Coronavirus data from CSSE at Johns Hopkins University. We extract a time series for the US, giving us for each day since January 22 the total number of infected persons.

```python
url = "https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv"
df = pd.read_csv(url)
# country = df[df["Country/Region"] == "Italy"]
# country = df[df["Country/Region"] == "Sweden"]
# country = df[df["Country/Region"] == "Germany"]
country = df[df["Country/Region"] == "US"]

interval_len = 200

x_data = list(range(interval_len))
y_data = list(country.iloc[int(0),int(4):int(interval_len+4)])
```

We run a logistic regression for our data and the logistic model.

```python
fit = curve_fit(logistic_function, x_data, y_data)
```

```python
k = round(fit[0][0],3)
C = round(fit[0][1],3)
L = int(fit[0][2])

print("We estimated the following parameters:")
print("k = ", k)
print("C = ", C)
print("L = ", L)
```

    We estimated the following parameters:
    k =  0.025
    C =  106.736
    L =  8580027

### Question: What does the parameter $L$ reflect in this case?

#### Plotting the data

```python
plt.figure(figsize=(10,8))

plt.scatter(list(range(interval_len)),y_data,label="Real data",color="red")

# Predicted logistic curve
t = np.arange(0, interval_len, 1)
s = L/(1+C*np.exp(-k*t))
plt.plot(t,s)

plt.xlabel("Days since January 22")
plt.ylabel("Total number of infected people")

plt.show()
```

```python

```