# Introduction to NumPy

## Topics
- Basic Synatx
    - creating vectors matrices
    - special: ones, zeros, identity eye
    - add, product, inverse
- Mechanics: indexing, slicing, concatenating, reshape, zip
- Numpy load, save data files
- Random numbers $\rightarrow$ distributions
- Similarity: Euclidian vs Cosine
- Example Nearest Neighbor search
- Example Linear Regression

## References
- Quick Start Tutorial https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
- NumPy Basic https://docs.scipy.org/doc/numpy-dev/user/basics.html
- NumPy Refernce https://docs.scipy.org/doc/numpy-dev/reference/index.html

# Basics
this section uses content created by Rob Hicks http://rlhick.people.wm.edu/stories/linear-algebra-python-basics.html

## Loading libraries

The python universe has a huge number of libraries that extend the capabilities of python. Nearly all of these are open source, unlike packages like stata or matlab where some key libraries are proprietary (and can cost lots of money).  In lots of my code, you will see this at the top:

```python
%matplotlib inline
import math
import numpy as np
import matplotlib.pyplot as plt
##import seaborn as sbn
##from scipy import *
```

This code sets up Ipython Notebook environments (lines beginning with `%`), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

1. sympy: provides for symbolic computation (solving algebra problems)
2. numpy: provides for linear algebra computations
3. matplotlib.pyplot: provides for the ability to graph functions and draw figures
4. scipy: scientific python provides a plethora of capabilities
5. seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.

```python
%ls
```

    APD-Crime-Data_orig.ipynb           LinearRegression.ipynb
    HW03/                               LinearRegression_orig.ipynb
    Introduction_class.ipynb            README.md
    Introduction_orig.ipynb             SeabornDemo.ipynb
    linear-algebra-python-basics.ipynb

## Creating arrays, scalars, and matrices in Python

Scalars can be created easily like this:

```python
y = 1/2
print y
```

    0

```python
x = .5
print x
```

    0.5

**Vectors and Lists**

The numpy library (we will reference it by np) is the workhorse library for linear algebra in python.  To creat a vector simply surround a python list ($[1,2,3]$) with the np.array function:

```python
x_list = [1,2,3]
print x_list
print type(x_list)
```

    [1, 2, 3]
    <type 'list'>

```python
x_vector = np.array([1,2,3])
print x_vector
print type(x_vector)
```

    [1 2 3]
    <type 'numpy.ndarray'>

```python
x_list*2
```

    [1, 2, 3, 1, 2, 3]

```python
x_vector*2
```

    array([2, 4, 6])

We could have done this by defining a python list and converting it to an array:

```python
c_list = [1,2]
print "The list:",c_list
print "Has length:", len(c_list)

c_vector = np.array(c_list)
print "\nThe vector:", c_vector
print "Has shape:",c_vector.shape
```

    The list: [1, 2]
    Has length: 2

    The vector: [1 2]
    Has shape: (2,)

```python
len(c_vector)
```

    2

```python
z = [5,6]
print "This is a list, not an array: z = ",z
print '\ntype = '+str(type(z))
```

    This is a list, not an array: z =  [5, 6]

    type = <type 'list'>

```python
A = np.array([[0, 1, 2], [5, 6, 7]])
print 'A='+str(A)+'\n'
print 'A.shape='+str(A.shape)+'\n'
print 'type = '+str(type(A))
```

    A=[[0 1 2]
     [5 6 7]]

    A.shape=(2, 3)

    type = <type 'numpy.ndarray'>

```python
v = np.array([1,2,3,4,5,6,7,8,9,10,11,12])
print v.shape
print len(v)
```

    (12,)
    12

```python
v = v.reshape([3,2,2])
print v
print v.shape
print len(v)
```

    [[[ 1  2]
      [ 3  4]]

     [[ 5  6]
      [ 7  8]]

     [[ 9 10]
      [11 12]]]
    (3, 2, 2)
    3

```python
for element in v:
    print element
```

    [[1 2]
     [3 4]]
    [[5 6]
     [7 8]]
    [[ 9 10]
     [11 12]]

```python
A = np.ones([4,4], dtype='float')
print A
```

    [[ 1.  1.  1.  1.]
     [ 1.  1.  1.  1.]
     [ 1.  1.  1.  1.]
     [ 1.  1.  1.  1.]]

```python
B = np.zeros([3,2])
print(B)
```

    [[ 0.  0.]
     [ 0.  0.]
     [ 0.  0.]]

```python
C = np.eye(3)
print(C)
```

    [[ 1.  0.  0.]
     [ 0.  1.  0.]
     [ 0.  0.  1.]]

##  Matrix Addition and Subtraction

### Adding or subtracting a scalar value to a matrix

To learn the basics, consider a small matrix of dimension $2 \times 2$, where $2 \times 2$ denotes the number of rows $\times$ the number of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Consider adding a scalar value (e.g. 3) to the A.
$$
\begin{equation}
	A+3=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}+3
	=\begin{bmatrix}
	  a_{11}+3 & a_{12}+3 \\
	  a_{21}+3 & a_{22}+3
	\end{bmatrix}
\end{equation}
$$
The same basic principle holds true for A-3:
$$
\begin{equation}
	A-3=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}-3
	=\begin{bmatrix}
	  a_{11}-3 & a_{12}-3 \\
	  a_{21}-3 & a_{22}-3
	\end{bmatrix}
\end{equation}
$$
Notice that we add (or subtract) the scalar value to each element in the matrix A.  A can be of any dimension.

This is trivial to implement, now that we have defined our matrix A:

```python
A = np.array(range(6)).reshape([2,3])
print(A)
```

    [[0 1 2]
     [3 4 5]]

```python
result = A + 3
#or
result = 3 + A
print result
```

    [[3 4 5]
     [6 7 8]]

### Adding or subtracting two matrices
Consider two small $2 \times 2$ matrices, where $2 \times 2$ denotes the \# of rows $\times$ the \# of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$ and $B$=$\bigl( \begin{smallmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{smallmatrix} \bigr)$.  To find the result of $A-B$, simply subtract each element of A with the corresponding element of B:

$$
\begin{equation}
	A -B =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix} -
	\begin{bmatrix} b_{11} & b_{12} \\
	  b_{21} & b_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  a_{11}-b_{11} & a_{12}-b_{12} \\
	  a_{21}-b_{21} & a_{22}-b_{22}
	\end{bmatrix}
\end{equation}
$$

Addition works exactly the same way:

$$
\begin{equation}
	A + B =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix} +
	\begin{bmatrix} b_{11} & b_{12} \\
	  b_{21} & b_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  a_{11}+b_{11} & a_{12}+b_{12} \\
	  a_{21}+b_{21} & a_{22}+b_{22}
	\end{bmatrix}
\end{equation}
$$

An important point to know about matrix addition and subtraction is that it is only defined when $A$ and $B$ are of the same size.  Here, both are $2 \times 2$.  Since operations are performed element by element, these two matrices must be conformable- and for addition and subtraction that means they must have the same numbers of rows and columns.  I like to be explicit about the dimensions of matrices for checking conformability as I write the equations, so write

$$
A_{2 \times 2} + B_{2 \times 2}= \begin{bmatrix}
  a_{11}+b_{11} & a_{12}+b_{12} \\
  a_{21}+b_{21} & a_{22}+b_{22}
\end{bmatrix}_{2 \times 2}
$$

Notice that the result of a matrix addition or subtraction operation is always of the same dimension as the two operands.

Let's define another matrix, B, that is also $2 \times 2$ and add it to A:

```python
B = np.random.randn(2,2)
print B
```

    [[ 0.63936233  1.40645192]
     [ 1.43495976  0.95392152]]

```python
# A = np.array([[1,0], [0,1]])
A = np.eye(2)
A
```

    array([[ 1.,  0.],
           [ 0.,  1.]])

```python
A+B
```

    array([[ 1.63936233,  1.40645192],
           [ 1.43495976,  1.95392152]])

```python

```

## Matrix Multiplication

### Multiplying a scalar value times a matrix

As before, let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Suppose we want to multiply A times a scalar value (e.g. $3 \times A$)

$$
\begin{equation}
	3 \times A = 3 \times \begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  3a_{11} & 3a_{12} \\
	  3a_{21} & 3a_{22}
	\end{bmatrix}
\end{equation}
$$

is of dimension (2,2).  Scalar multiplication is commutative, so that $3 \times A$=$A \times 3$.  Notice that the product is defined for a matrix A of any dimension.

Similar to scalar addition and subtration, the code is simple:

```python

```

```python
A
```

    array([[ 1.,  0.],
           [ 0.,  1.]])

```python
A * 3
```

    array([[ 3.,  0.],
           [ 0.,  3.]])

### Multiplying two matricies

Now, consider the $2 \times 1$ vector $C=\bigl( \begin{smallmatrix} c_{11} \\
  c_{21}
\end{smallmatrix} \bigr)$

Consider multiplying matrix $A_{2 \times 2}$ and the vector $C_{2 \times 1}$.  Unlike the addition and subtraction case, this product is defined.  Here, conformability depends not on the row **and** column dimensions, but rather on the column dimensions of the first operand and the row dimensions of the second operand.  We can write this operation as follows

$$
\begin{equation}
	A_{2 \times 2} \times C_{2 \times 1} =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}_{2 \times 2}
    \times
    \begin{bmatrix}
	c_{11} \\
	c_{21}
	\end{bmatrix}_{2 \times 1}
	=
	\begin{bmatrix}
	  a_{11}c_{11} + a_{12}c_{21} \\
	  a_{21}c_{11} + a_{22}c_{21}
	\end{bmatrix}_{2 \times 1}
\end{equation}
$$

Alternatively, consider a matrix C of dimension $2 \times 3$ and a matrix A of dimension $3 \times 2$

$$
\begin{equation}
	A_{3 \times 2}=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
	,
	C_{2 \times 3} =
	\begin{bmatrix}
		  c_{11} & c_{12} & c_{13} \\
		  c_{21} & c_{22} & c_{23} \\
	\end{bmatrix}_{2 \times 3}
	\end{equation}
$$

Here, A $\times$ C is

$$
\begin{align}
	A_{3 \times 2} \times C_{2 \times 3}=&
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
	\times
	\begin{bmatrix}
	  c_{11} & c_{12} & c_{13} \\
	  c_{21} & c_{22} & c_{23}
	\end{bmatrix}_{2 \times 3} \\
	=&
	\begin{bmatrix}
	  a_{11} c_{11}+a_{12} c_{21} & a_{11} c_{12}+a_{12} c_{22} & a_{11} c_{13}+a_{12} c_{23} \\
	  a_{21} c_{11}+a_{22} c_{21} & a_{21} c_{12}+a_{22} c_{22} & a_{21} c_{13}+a_{22} c_{23} \\
	  a_{31} c_{11}+a_{32} c_{21} & a_{31} c_{12}+a_{32} c_{22} & a_{31} c_{13}+a_{32} c_{23}
	\end{bmatrix}_{3 \times 3}
\end{align}
$$

So in general, $X_{r_x \times c_x} \times Y_{r_y \times c_y}$ we have two important things to remember:

* For conformability in matrix multiplication, $c_x=r_y$, or the columns in the first operand must be equal to the rows of the second operand.
* The result will be of dimension $r_x \times c_y$, or of dimensions equal to the rows of the first operand and columns equal to columns of the second operand.

Given these facts, you should convince yourself that matrix multiplication is not generally commutative, that the relationship $X \times Y = Y \times X$ does **not** hold in all cases.
For this reason, we will always be very explicit about whether we are pre multiplying ($X \times Y$) or post multiplying ($Y \times X$) the vectors/matrices $X$ and $Y$.

For more information on this topic, see this
http://en.wikipedia.org/wiki/Matrix_multiplication.

```python
# Let's redefine A and C to demonstrate matrix multiplication:
A = np.arange(6).reshape((3,2))
C = np.random.randn(2,2)

print A.shape
print C.shape
```

    (3, 2)
    (2, 2)

We will use the numpy dot operator to perform the these multiplications. You can use it two ways to yield the same result:

```python
print A.dot(C)
print np.dot(A,C)
```

    [[ 1.25114834  0.40428894]
     [ 3.73789373 -1.32820512]
     [ 6.22463911 -3.06069918]]
    [[ 1.25114834  0.40428894]
     [ 3.73789373 -1.32820512]
     [ 6.22463911 -3.06069918]]

```python
# What would happen to
C.dot(A)
```

## Matrix Division
The term matrix division is actually a misnomer.  To divide in a matrix algebra world we first need to invert the matrix.  It is useful to consider the analog case in a scalar work.  Suppose we want to divide the $f$ by $g$.  We could do this in two different ways:
$$
\begin{equation}
	\frac{f}{g}=f \times g^{-1}.
\end{equation}
$$
In a scalar seeting, these are equivalent ways of solving the division problem.  The second one requires two steps: first we invert g and then we multiply f times g.  In a matrix world, we need to think about this second approach.  First we have to invert the matrix g and then we will need to pre or post multiply depending on the exact situation we encounter (this is intended to be vague for now).

###Inverting a Matrix

As before, consider the square $2 \times 2$ matrix $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22}\end{smallmatrix} \bigr)$.  Let the inverse of matrix A (denoted as $A^{-1}$) be

$$
\begin{equation}
	A^{-1}=\begin{bmatrix}
             a_{11} & a_{12} \\
		     a_{21} & a_{22}
           \end{bmatrix}^{-1}=\frac{1}{a_{11}a_{22}-a_{12}a_{21}}	\begin{bmatrix}
		             a_{22} & -a_{12} \\
				     -a_{21} & a_{11}
		           \end{bmatrix}
\end{equation}
$$

The inverted matrix $A^{-1}$ has a useful property:
$$
\begin{equation}
	A \times A^{-1}=A^{-1} \times A=I
\end{equation}
$$
where I, the identity matrix (the matrix equivalent of the scalar value 1), is
$$
\begin{equation}
	I_{2 \times 2}=\begin{bmatrix}
             1 & 0 \\
		     0 & 1
           \end{bmatrix}
\end{equation}
$$
furthermore, $A \times I = A$ and $I \times A = A$.

An important feature about matrix inversion is that it is undefined if (in the $2 \times 2$ case), $a_{11}a_{22}-a_{12}a_{21}=0$.  If this relationship is equal to zero the inverse of A does not exist.  If this term is very close to zero, an inverse may exist but $A^{-1}$ may be poorly conditioned meaning it is prone to rounding error and is likely not well identified computationally.  The term $a_{11}a_{22}-a_{12}a_{21}$ is the determinant of matrix A, and for square matrices of size greater than $2 \times 2$, if equal to zero indicates that you have a problem with your data matrix (columns are linearly dependent on other columns).  The inverse of matrix A exists if A is square and is of full rank (ie. the columns of A are not linear combinations of other columns of A).

For more information on this topic, see this
http://en.wikipedia.org/wiki/Matrix_inversion, for example, on inverting matrices.

```python
# A x = y
# x = y/A <- NO
# x = inv(A) y
```

```python
A = np.zeros([3,3])
print A
np.linalg.inv(A)
```

```python
A = np.linspace(0., 2., 6).reshape([3,2])
print(A)
np.linalg.inv(A)
```

```python
# note, we need a square matrix (# rows = # cols), use C:
C = np.random.randn(3,3)
print C, '\n'
C_inverse = np.linalg.inv(C)
print C_inverse
```

    [[ 1.43157718  0.07796517  3.5566182 ]
     [-0.4352189  -0.2912122  -0.77983248]
     [-2.07158701  0.74363178 -0.85274384]]

    [[-0.41124495 -1.34624151 -0.48408255]
     [-0.61786254 -3.05220825  0.21426124]
     [ 0.46024075  0.60878465  0.1901516 ]]

Check that $C\times C^{-1} = I$:

```python
print C.dot(C_inverse)
print "\nIs identical to:\n"
print C_inverse.dot(C)
```

    [[  1.00000000e+00  -8.46539965e-18  -2.66324353e-17]
     [  2.46608390e-17   1.00000000e+00   5.51929816e-17]
     [  4.25023507e-17   1.38097700e-16   1.00000000e+00]]

    Is identical to:

    [[  1.00000000e+00  -3.75173322e-17  -1.29392279e-16]
     [ -1.00065300e-16   1.00000000e+00  -6.91489326e-16]
     [ -1.56315316e-17   1.79331409e-17   1.00000000e+00]]

```python
A = np.matrix([[1,0,3],[4,5,6],[7,-8,9]])
# or
A = np.matrix('[1  0 3; 4,5,6; 7,-8,9]')
print A
```

    [[ 1  0  3]
     [ 4  5  6]
     [ 7 -8  9]]

```python
print "trace A = ", A.trace()
print "det(A) = ", np.linalg.det(A)
```

    trace A =  [[15]]
    det(A) =  -108.0

```python
print A.I
```

    [[-0.86111111  0.22222222  0.13888889]
     [-0.05555556  0.11111111 -0.05555556]
     [ 0.62037037 -0.07407407 -0.0462963 ]]

```python
print B
```

    [[ 1  0  3]
     [ 4  5  6]
     [ 7 -8  9]]

```python
B = np.array(A)

print "different behaviors of arrays and matrices"
print "A*A: \n", A*A
print "B*B: \n", B*B
print "matrix mult B.dot(B): \n", B.dot(B)
```

    different behaviors of arrays and matrices
    A*A:
    [[  22  -24   30]
     [  66  -23   96]
     [  38 -112   54]]
    B*B:
    [[ 1  0  9]
     [16 25 36]
     [49 64 81]]
    matrix mult B.dot(B):
    [[  22  -24   30]
     [  66  -23   96]
     [  38 -112   54]]

```python
A.I*A
```

    matrix([[  1.00000000e+00,  -4.44089210e-16,   3.05311332e-16],
            [  6.93889390e-17,   1.00000000e+00,   4.16333634e-17],
            [  2.08166817e-17,   1.11022302e-16,   1.00000000e+00]])

```python
print A.A
print A.A1
```

    [[ 1  0  3]
     [ 4  5  6]
     [ 7 -8  9]]
    [ 1  0  3  4  5  6  7 -8  9]

## Transposing a Matrix

At times it is useful to pivot a matrix for conformability- that is in order to matrix divide or multiply, we need to switch the rows and column dimensions of matrices.  Consider the matrix
$$
\begin{equation}
	A_{3 \times 2}=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
\end{equation}
$$
The transpose of A (denoted as $A^{\prime}$) is
$$
\begin{equation}
   A^{\prime}=\begin{bmatrix}
	  a_{11} & a_{21} & a_{31} \\
	  a_{12} & a_{22} & a_{32} \\
	\end{bmatrix}_{2 \times 3}
\end{equation}
$$

```python
A = np.arange(6).reshape((6,1))
B = np.arange(6).reshape((1,6))
```

```python
A
```

    array([[0],
           [1],
           [2],
           [3],
           [4],
           [5]])

```python
B
```

    array([[0, 1, 2, 3, 4, 5]])

```python
A.dot(B)
```

    array([[ 0,  0,  0,  0,  0,  0],
           [ 0,  1,  2,  3,  4,  5],
           [ 0,  2,  4,  6,  8, 10],
           [ 0,  3,  6,  9, 12, 15],
           [ 0,  4,  8, 12, 16, 20],
           [ 0,  5, 10, 15, 20, 25]])

```python
B.dot(A)
```

    array([[55]])

```python
B.T
```

    array([[0],
           [1],
           [2],
           [3],
           [4],
           [5]])

```python
A.T
```

    array([[0, 1, 2, 3, 4, 5]])

```python
B.T.dot(A.T)
```

    array([[ 0,  0,  0,  0,  0,  0],
           [ 0,  1,  2,  3,  4,  5],
           [ 0,  2,  4,  6,  8, 10],
           [ 0,  3,  6,  9, 12, 15],
           [ 0,  4,  8, 12, 16, 20],
           [ 0,  5, 10, 15, 20, 25]])

```python
A.T.dot(B.T)
```

    array([[55]])

```python
A = np.arange(6).reshape((3,2))
B = np.arange(8).reshape((2,4))
print "A is"
print A

print "The Transpose of A is"
print A.T
```

    A is
    [[0 1]
     [2 3]
     [4 5]]
    The Transpose of A is
    [[0 2 4]
     [1 3 5]]

One important property of transposing a matrix is the transpose of a product of two matrices.  Let matrix A be of dimension $N \times M$ and let B of of dimension $M \times P$.  Then
$$
\begin{equation}
	(AB)^{\prime}=B^{\prime}A^{\prime}
\end{equation}
$$
For more information, see this http://en.wikipedia.org/wiki/Matrix_transposition on matrix transposition.  This is also easy to implement:

```python
print B.T.dot(A.T)
print "Is identical to:"
print (A.dot(B)).T
```

    [[ 4 12 20]
     [ 5 17 29]
     [ 6 22 38]
     [ 7 27 47]]
    Is identical to:
    [[ 4 12 20]
     [ 5 17 29]
     [ 6 22 38]
     [ 7 27 47]]

```python
B.shape
```

    (2, 4)

```python
print B, '\n'
B[0, 3]
```

    [[0 1 2 3]
     [4 5 6 7]]

    3

```python
A = np.arange(12).reshape((3,4))
A
```

    array([[ 0,  1,  2,  3],
           [ 4,  5,  6,  7],
           [ 8,  9, 10, 11]])

```python
A[2,:].shape
```

    (4,)

```python
A[:,1].reshape(1,3).shape
```

    (1, 3)

# Mechanics

## Indexing and Slicing
examples from https://www.tutorialspoint.com/numpy/numpy_indexing_and_slicing.htm

```python
a = np.arange(10)
s = slice(2,7,2)
print a
print a[s]
```

    [0 1 2 3 4 5 6 7 8 9]
    [2 4 6]

```python
a = np.arange(10)
b = a[2:7:2]
print a
print b
```

    [0 1 2 3 4 5 6 7 8 9]
    [2 4 6]

```python
a = np.arange(10)
b = a[5]
print a
print b
```

    [0 1 2 3 4 5 6 7 8 9]
    5

```python
a = np.arange(10)
print a
print a[2:5]
```

    [0 1 2 3 4 5 6 7 8 9]
    [2 3 4]

```python
print a[2:5]
print a[5:8]
print np.concatenate([a[2:5],a[5:8]])
print a[2:8]
```

    [2 3 4]
    [5 6 7]
    [2 3 4 5 6 7]
    [2 3 4 5 6 7]

```python
a = np.arange(10)
print a
print a[-1]
print a[-2:-5:-1]
```

    [0 1 2 3 4 5 6 7 8 9]
    9
    [8 7 6]

```python
a = np.array([[1,2,3],[3,4,5],[4,5,6]])
print a

# slice items starting from index
print 'Now we will slice the array from the index a[1:]'
print a[1:]
print "slicing along 2 dimensions"
print a[1:,2]
print "aside: if a is a matrix, we keep the original shape of the slice..."
print np.matrix(a)[1:,2]
```

    [[1 2 3]
     [3 4 5]
     [4 5 6]]
    Now we will slice the array from the index a[1:]
    [[3 4 5]
     [4 5 6]]
    slicing along 2 dimensions
    [5 6]
    aside: if a is a mtrix, we keep the original shape of the slice...
    [[5]
     [6]]

```python
# array to begin with
a = np.array([[1,2,3],[3,4,5],[4,5,6]])

print 'Our array is:'
print a
print '\n'

# this returns array of items in the second column
print 'The items in the second column are:'
print a[...,1]
print '\n'

# Now we will slice all items from the second row
print 'The items in the second row are:'
print a[1,...]
print '\n'

# Now we will slice all items from column 1 onwards
print 'The items column 1 onwards are:'
print a[...,1:]
```

    Our array is:
    [[1 2 3]
     [3 4 5]
     [4 5 6]]

    The items in the second column are:
    [2 4 5]

    The items in the second row are:
    [3 4 5]

    The items column 1 onwards are:
    [[2 3]
     [4 5]
     [5 6]]

```python
# in-class exercise:
a = np.matrix('[4 5 6 7; 4 1 0 1; 5 0 1 3; 9 8 3 2]')
print a
# how can you get the [[1,0],[0,1]] matrix in the middle of a?
```

    [[4 5 6 7]
     [4 1 0 1]
     [5 0 1 3]
     [9 8 3 2]]

```python

```

```python

```

```python

```

```python

```

```python
# answer:
a[1:3,1:3]
```

    matrix([[1, 0],
            [0, 1]])

```python
# in-class exercise:
a = np.arange(18)
# what is the transpose of the 3x3 matrix formed from taking the even elements of a?
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
# answer:
a[0:18:2].reshape([3,3]).T
```

    array([[ 0,  6, 12],
           [ 2,  8, 14],
           [ 4, 10, 16]])

## Logic, Comparison

```python
A = np.random.rand(5,5)*10
print A, '\n'
print (A < 5)
```

    [[ 2.3494965   2.12481172  9.41409716  3.61456994  3.95928254]
     [ 3.87873229  4.17960296  7.15367797  5.82959523  8.72072764]
     [ 1.83427056  0.70719589  0.77183064  6.51119661  3.30665102]
     [ 4.41900782  3.96434475  6.55068733  7.73096267  3.83840644]
     [ 8.40243264  3.59926057  8.25263237  0.7361585   7.96197924]]

    [[ True  True False  True  True]
     [ True  True False False False]
     [ True  True  True False  True]
     [ True  True False False  True]
     [False  True False  True False]]

```python
np.all(A<5)
```

    False

```python
A.flatten() <5
```

    array([ True,  True, False,  True,  True,  True,  True, False, False,
           False,  True,  True,  True, False,  True,  True,  True, False,
           False,  True, False,  True, False,  True, False], dtype=bool)

```python
sum(A.flatten()<5)
```

    15

```python
print A[A < 5]
```

    [ 2.3494965   2.12481172  3.61456994  3.95928254  3.87873229  4.17960296
      1.83427056  0.70719589  0.77183064  3.30665102  4.41900782  3.96434475
      3.83840644  3.59926057  0.7361585 ]

```python
A[A<5] = 0
A
```

    array([[ 0.        ,  0.        ,  9.41409716,  0.        ,  0.        ],
           [ 0.        ,  0.        ,  7.15367797,  5.82959523,  8.72072764],
           [ 0.        ,  0.        ,  0.        ,  6.51119661,  0.        ],
           [ 0.        ,  0.        ,  6.55068733,  7.73096267,  0.        ],
           [ 8.40243264,  0.        ,  8.25263237,  0.        ,  7.96197924]])

```python

```

```python
A[A>=5] = 1
```

```python
A
```

    array([[ 0.,  0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  1.,  1.],
           [ 0.,  0.,  0.,  1.,  0.],
           [ 0.,  0.,  1.,  1.,  0.],
           [ 1.,  0.,  1.,  0.,  1.]])

```python
A[:2,:2][np.array([[True,False],[1==2,True]])]
```

    array([ 0.,  0.])

```python

```

```python

```

```python

```

```python
# in-class exercise
A = np.random.randn(4,4)
B = np.arange(16).reshape([4,4])
print A, '\n\n', B
# find the elements of A that are less than 0, and replace them with corresponding elements of B
```

    [[-0.65359979  0.51322216 -1.10197091 -1.32989247]
     [-1.02473805 -2.89062776  0.07370503  1.1674123 ]
     [ 0.40994164  0.03098448  0.12507655  0.43065601]
     [ 0.25247169 -0.09467224  0.81080887 -0.16973108]]

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]]

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
A[A<0]=B[A<0]
A
```

    array([[  0.        ,   0.51322216,   2.        ,   3.        ],
           [  4.        ,   5.        ,   0.07370503,   1.1674123 ],
           [  0.40994164,   0.03098448,   0.12507655,   0.43065601],
           [  0.25247169,  13.        ,   0.81080887,  15.        ]])

```python

```

```python

```

```python

```

```python

```

```python

```

```python
A = np.random.randn(4,4)
A
```

    array([[ 0.37304455,  0.1140263 ,  0.45516422, -0.47696553],
           [ 0.28975316,  2.39686238,  1.05551098, -0.47443002],
           [ 1.90313068, -0.46075897, -0.36204442,  0.07785992],
           [ 1.48329968,  0.22641051,  1.0591176 , -0.91079241]])

```python
A>-1 and A<1
```

```python
np.logical_and(A>-1,A<1)
```

    array([[ True,  True,  True,  True],
           [ True, False, False,  True],
           [False,  True,  True,  True],
           [False,  True, False,  True]], dtype=bool)

```python
# in-class exercise: combining logical conditions
# USE np.logical_and()
A = np.random.randn(4,4)
B = np.arange(16).reshape([4,4])
print A, '\n\n', B
# find the elements of A that are between -1 and 1, and replace them with elements of B
```

    [[ 0.81347907  0.2938777  -1.92768272 -0.88420703]
     [ 0.9215361   0.11586535  0.34359852 -1.37552356]
     [ 0.87847905  1.34448599  0.87814617 -1.67135974]
     [ 0.31357061  2.75238453  0.7967456  -0.1075649 ]]

    [[ 0  1  2  3]
     [ 4  5  6  7]
     [ 8  9 10 11]
     [12 13 14 15]]

```python

```

```python

```

```python

```

```python

```

```python

```

```python
A[np.logical_and(A > -1, A < 1)] = B[np.logical_and(A > -1,A < 1)]
A
```

    array([[  0.        ,   1.        ,  -1.92768272,   3.        ],
           [  4.        ,   5.        ,   6.        ,  -1.37552356],
           [  8.        ,   1.34448599,  10.        ,  -1.67135974],
           [ 12.        ,   2.75238453,  14.        ,  15.        ]])

## Concatenate, Reshape

```python
np.ones((10,5), int)
```

```python
np.zeros((10,5), int)
```

```python
np.eye(5, dtype="int")
```

```python

```

```python

```

## Random Numbers

```python
np.random.seed(100)
```

```python
v1 = np.random.rand(500)
v2 = np.random.randn(500)
plt.plot(range(v1.shape[0]), v1, '.')
plt.scatter(range(len(v2)), v2)
plt.xlabel('Index')
plt.ylabel('Random Value')
plt.title('Some random numbers')
plt.show()
```

```python
plt.hist(v1, bins=20);
```

```python
v2 = np.random.randn(10000)
plt.hist(v2, bins=100)
;
```

```python
v3 = np.random.beta(3,2, 10000)
plt.hist(v3, bins=100)
;
```

```python

```

```python

```

```python

```

```python

```

```python
# in-class exercise
# generate a 1000 points in 2-d uniformly distributed on the rectangle [-1,1]x[0,0.5], then plot the points.
```

```python

```

```python

```

```python

```

```python

```

```python

```

```python
points = zip(np.random.rand(1000)*2-1,np.random.rand(1000)*0.5)
plt.scatter(np.array(points).T[0],np.array(points).T[1]);
# plt.scatter(*zip(*points));
```

# Numpy load, save data files

```python
%ls -l HW03/
```

    total 244
    -rw-r--r-- 1 klimka klimka 72580 Sep 13 19:58 Camera_cleaned.csv
    -rw-rw-r-- 1 klimka klimka 87053 Jun 23 07:58 Camera.csv
    -rw-rw-r-- 1 klimka klimka 72580 Sep  6 16:20 Camera_mod.csv
    -rw-rw-r-- 1 klimka klimka  3661 Sep  8 17:56 HW3_submission.ipynb
    -rwxrwxr-x 1 klimka klimka   244 Sep  6 16:09 preprocess_data.sh*
    -rw-rw-r-- 1 klimka klimka  3184 Sep  8 17:51 README.md

```sh
%%sh
./HW03/preprocess_data.sh HW03/Camera.csv HW03/Camera_cleaned.csv
head HW03/Camera.csv
head HW03/Camera_cleaned.csv
```

    Model;Release date;Max resolution;Low resolution;Effective pixels;Zoom wide (W);Zoom tele (T);Normal focus range;Macro focus range;Storage included;Weight (inc. batteries);Dimensions;Price
    STRING;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE;DOUBLE
    Agfa ePhoto 1280;1997;1024.0;640.0;0.0;38.0;114.0;70.0;40.0;4.0;420.0;95.0;179.0
    Agfa ePhoto 1680;1998;1280.0;640.0;1.0;38.0;114.0;50.0;0.0;4.0;420.0;158.0;179.0
    Agfa ePhoto CL18;2000;640.0;0.0;0.0;45.0;45.0;0.0;0.0;2.0;0.0;0.0;179.0
    Agfa ePhoto CL30;1999;1152.0;640.0;0.0;35.0;35.0;0.0;0.0;4.0;0.0;0.0;269.0
    Agfa ePhoto CL30 Clik!;1999;1152.0;640.0;0.0;43.0;43.0;50.0;0.0;40.0;300.0;128.0;1299.0
    Agfa ePhoto CL45;2001;1600.0;640.0;1.0;51.0;51.0;50.0;20.0;8.0;270.0;119.0;179.0
    Agfa ePhoto CL50;1999;1280.0;640.0;1.0;34.0;102.0;0.0;0.0;8.0;0.0;0.0;179.0
    Canon PowerShot 350;1997;640.0;0.0;0.0;42.0;42.0;70.0;3.0;2.0;320.0;93.0;149.0
    Model;Release_date;Max_resolution;Low_resolution;Effective_pixels;Zoom_wide_W;Zoom_tele_T;Normal_focus_range;Macro_focus_range;Storage_included;Weight_inc._batteries;Dimensions;Price
    1;1997;1024.0;640.0;0.0;38.0;114.0;70.0;40.0;4.0;420.0;95.0;179.0
    2;1998;1280.0;640.0;1.0;38.0;114.0;50.0;0.0;4.0;420.0;158.0;179.0
    3;2000;640.0;0.0;0.0;45.0;45.0;0.0;0.0;2.0;0.0;0.0;179.0
    4;1999;1152.0;640.0;0.0;35.0;35.0;0.0;0.0;4.0;0.0;0.0;269.0
    5;1999;1152.0;640.0;0.0;43.0;43.0;50.0;0.0;40.0;300.0;128.0;1299.0
    6;2001;1600.0;640.0;1.0;51.0;51.0;50.0;20.0;8.0;270.0;119.0;179.0
    7;1999;1280.0;640.0;1.0;34.0;102.0;0.0;0.0;8.0;0.0;0.0;179.0
    8;1997;640.0;0.0;0.0;42.0;42.0;70.0;3.0;2.0;320.0;93.0;149.0
    9;1996;832.0;640.0;0.0;50.0;50.0;40.0;10.0;1.0;460.0;160.0;139.0

```python
DATA = np.genfromtxt('HW03/Camera_cleaned.csv', delimiter=';', names=True, dtype='float')
```

```python
DATA
```

    array([ (1.0, 1997.0, 1024.0, 640.0, 0.0, 38.0, 114.0, 70.0, 40.0, 4.0, 420.0, 95.0, 179.0),
           (2.0, 1998.0, 1280.0, 640.0, 1.0, 38.0, 114.0, 50.0, 0.0, 4.0, 420.0, 158.0, 179.0),
           (3.0, 2000.0, 640.0, 0.0, 0.0, 45.0, 45.0, 0.0, 0.0, 2.0, 0.0, 0.0, 179.0),
           ...,
           (1036.0, 2001.0, 2048.0, 1024.0, 3.0, 35.0, 98.0, 80.0, 10.0, 8.0, 340.0, 107.0, 62.0),
           (1037.0, 2001.0, 2400.0, 1200.0, 3.0, 35.0, 98.0, 80.0, 10.0, 16.0, 340.0, 107.0, 62.0),
           (1038.0, 2002.0, 1600.0, 800.0, 1.0, 38.0, 38.0, 40.0, 20.0, 8.0, 180.0, 86.0, 129.0)],
          dtype=[('Model', '<f8'), ('Release_date', '<f8'), ('Max_resolution', '<f8'), ('Low_resolution', '<f8'), ('Effective_pixels', '<f8'), ('Zoom_wide_W', '<f8'), ('Zoom_tele_T', '<f8'), ('Normal_focus_range', '<f8'), ('Macro_focus_range', '<f8'), ('Storage_included', '<f8'), ('Weight_inc_batteries', '<f8'), ('Dimensions', '<f8'), ('Price', '<f8')])

```python
DATA['Max_resolution'].max()
```

    5616.0

```python
np.nanargmin(DATA['Max_resolution'])
```

    370

```python
np.savetxt('Cameras_TMP.csv', DATA, delimiter=',')
```

```sh
%%sh
ls -l
rm Cameras_TMP.csv
```

    total 3024
    -rw-rw-r-- 1 klimka klimka  650999 Aug 27 16:42 APD-Crime-Data_orig.ipynb
    -rw-r--r-- 1 klimka klimka  337203 Sep 13 20:17 Cameras_TMP.csv
    drwxrwxr-x 3 klimka klimka    4096 Sep 11 18:41 HW03
    -rw-rw-r-- 1 klimka klimka  232486 Sep 13 20:16 Introduction_class.ipynb
    -rw-rw-r-- 1 klimka klimka  135673 Sep 11 16:25 Introduction_orig.ipynb
    -rw-rw-r-- 1 klimka klimka   58770 Aug 27 16:42 linear-algebra-python-basics.ipynb
    -rw-rw-r-- 1 klimka klimka 1047866 Aug 27 16:42 LinearRegression.ipynb
    -rw-rw-r-- 1 klimka klimka  520177 Aug 27 16:42 LinearRegression_orig.ipynb
    -rw-rw-r-- 1 klimka klimka     484 Sep  6 16:08 README.md
    -rw-rw-r-- 1 klimka klimka   92411 Aug 27 16:42 SeabornDemo.ipynb

```python
import pickle

with open('Cameras.pkl','wb') as f:
    pickle.dump(DATA, f) # pickle.dump(DATA, open('Cameras.pkl','wb'))

# Shortcut:
np.save('Cameras2.pkl', DATA, allow_pickle=True)

%ls -l
```

    total 3376
    -rw-rw-r-- 1 klimka klimka  650999 Aug 27 16:42 APD-Crime-Data_orig.ipynb
    -rw-r--r-- 1 klimka klimka  108368 Sep 13 20:20 Cameras2.pkl.npy
    -rw-r--r-- 1 klimka klimka  363158 Sep 13 20:20 Cameras.pkl
    drwxrwxr-x 3 klimka klimka    4096 Sep 11 18:41 HW03/
    -rw-rw-r-- 1 klimka klimka  233610 Sep 13 20:20 Introduction_class.ipynb
    -rw-rw-r-- 1 klimka klimka  135673 Sep 11 16:25 Introduction_orig.ipynb
    -rw-rw-r-- 1 klimka klimka   58770 Aug 27 16:42 linear-algebra-python-basics.ipynb
    -rw-rw-r-- 1 klimka klimka 1047866 Aug 27 16:42 LinearRegression.ipynb
    -rw-rw-r-- 1 klimka klimka  520177 Aug 27 16:42 LinearRegression_orig.ipynb
    -rw-rw-r-- 1 klimka klimka     484 Sep  6 16:08 README.md
    -rw-rw-r-- 1 klimka klimka   92411 Aug 27 16:42 SeabornDemo.ipynb

```python
with open('Cameras.pkl','rb') as f:
    datapkl = pickle.load(f)

# Shortcut:
datapkl2 = np.load('Cameras2.pkl.npy')
```

```python
datapkl
```

    array([ (1.0, 1997.0, 1024.0, 640.0, 0.0, 38.0, 114.0, 70.0, 40.0, 4.0, 420.0, 95.0, 179.0),
           (2.0, 1998.0, 1280.0, 640.0, 1.0, 38.0, 114.0, 50.0, 0.0, 4.0, 420.0, 158.0, 179.0),
           (3.0, 2000.0, 640.0, 0.0, 0.0, 45.0, 45.0, 0.0, 0.0, 2.0, 0.0, 0.0, 179.0),
           ...,
           (1036.0, 2001.0, 2048.0, 1024.0, 3.0, 35.0, 98.0, 80.0, 10.0, 8.0, 340.0, 107.0, 62.0),
           (1037.0, 2001.0, 2400.0, 1200.0, 3.0, 35.0, 98.0, 80.0, 10.0, 16.0, 340.0, 107.0, 62.0),
           (1038.0, 2002.0, 1600.0, 800.0, 1.0, 38.0, 38.0, 40.0, 20.0, 8.0, 180.0, 86.0, 129.0)],
          dtype=[('Model', '<f8'), ('Release_date', '<f8'), ('Max_resolution', '<f8'), ('Low_resolution', '<f8'), ('Effective_pixels', '<f8'), ('Zoom_wide_W', '<f8'), ('Zoom_tele_T', '<f8'), ('Normal_focus_range', '<f8'), ('Macro_focus_range', '<f8'), ('Storage_included', '<f8'), ('Weight_inc_batteries', '<f8'), ('Dimensions', '<f8'), ('Price', '<f8')])

```python
datapkl2
```

    array([ (1.0, 1997.0, 1024.0, 640.0, 0.0, 38.0, 114.0, 70.0, 40.0, 4.0, 420.0, 95.0, 179.0),
           (2.0, 1998.0, 1280.0, 640.0, 1.0, 38.0, 114.0, 50.0, 0.0, 4.0, 420.0, 158.0, 179.0),
           (3.0, 2000.0, 640.0, 0.0, 0.0, 45.0, 45.0, 0.0, 0.0, 2.0, 0.0, 0.0, 179.0),
           ...,
           (1036.0, 2001.0, 2048.0, 1024.0, 3.0, 35.0, 98.0, 80.0, 10.0, 8.0, 340.0, 107.0, 62.0),
           (1037.0, 2001.0, 2400.0, 1200.0, 3.0, 35.0, 98.0, 80.0, 10.0, 16.0, 340.0, 107.0, 62.0),
           (1038.0, 2002.0, 1600.0, 800.0, 1.0, 38.0, 38.0, 40.0, 20.0, 8.0, 180.0, 86.0, 129.0)],
          dtype=[('Model', '<f8'), ('Release_date', '<f8'), ('Max_resolution', '<f8'), ('Low_resolution', '<f8'), ('Effective_pixels', '<f8'), ('Zoom_wide_W', '<f8'), ('Zoom_tele_T', '<f8'), ('Normal_focus_range', '<f8'), ('Macro_focus_range', '<f8'), ('Storage_included', '<f8'), ('Weight_inc_batteries', '<f8'), ('Dimensions', '<f8'), ('Price', '<f8')])

```python
%rm Cameras.pkl
%rm Cameras2.pkl.npy
```

```python
# For saving multiple variables in workspace: import shelve
```

# Example Nearest Neighbor search
Nearest Neighbor search is a common technique in Machine Learning

```python
points = [[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]]
qPoint = [4,5,3]
# which point in points is closest to qPoint?
```

```python
# Euclidean distance between 2 points [x1,y1,z1] and [x2,y2,z2] is:
# sqrt((x2-x1)**2+(y2-y1)**2+(z2-z1)**2)
```

```python
### Pure iterative Python ###
points = [[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]]
qPoint = [4,5,3]

minIdx = -1
minDist = -1
for idx, point in enumerate(points):  # iterate over all points
        print "index is %d, point is %s" % (idx, point)
        dist = sum([(dp-dq)**2 for dp,dq in zip(point,qPoint)])**0.5  # compute the euclidean distance for each point to q
        if dist < minDist or minDist < 0:  # if necessary, update minimum distance and index of the corresponding point
            minDist = dist
            minIdx = idx

print 'Nearest point to q: ', points[minIdx]
```

    index is 0, point is [9, 2, 8]
    index is 1, point is [4, 7, 2]
    index is 2, point is [3, 4, 4]
    index is 3, point is [5, 6, 9]
    index is 4, point is [5, 0, 7]
    index is 5, point is [8, 2, 7]
    index is 6, point is [0, 3, 2]
    index is 7, point is [7, 3, 0]
    index is 8, point is [6, 1, 1]
    index is 9, point is [2, 9, 6]
    Nearest point to q:  [3, 4, 4]

```python
# # # Equivalent NumPy vectorization # # #
import numpy as np
points = np.array([[9,2,8],[4,7,2],[3,4,4],[5,6,9],[5,0,7],[8,2,7],[0,3,2],[7,3,0],[6,1,1],[2,9,6]])
qPoint = np.array([4,5,3])
minIdx = np.argmin(np.linalg.norm(points-qPoint,axis=1))  # compute all euclidean distances at once and return the index of the smallest one
print 'Nearest point to q: ', points[minIdx]
```

    Nearest point to q:  [3 4 4]

```python

```

# Example: Linear Regression

**Linear regression** is an approach for modeling the relationship between a scalar dependent variable $y$ and one or more explanatory variables (or independent variables) denoted $X$. The case of one explanatory variable is called *simple linear regression*. For more than one explanatory variable, the process is called *multiple linear regression*.$^1$ (This term is distinct from multivariate linear regression, where multiple correlated dependent variables are predicted, rather than a single scalar variable.$^2$

We assume that the equation

$y_i = \beta_0 + \beta_1 X_i + \epsilon_i$ where $\epsilon_i \approx N(0, \sigma^2)$

***
$^1$ David A. Freedman (2009). Statistical Models: Theory and Practice. Cambridge University Press. p. 26. A simple regression equation has on the right hand side an intercept and an explanatory variable with a slope coefficient. A multiple regression equation has two or more explanatory variables on the right hand side, each with its own slope coefficient

$^2$ Rencher, Alvin C.; Christensen, William F. (2012), "Chapter 10, Multivariate regression – Section 10.1, Introduction", Methods of Multivariate Analysis, Wiley Series in Probability and Statistics, 709 (3rd ed.), John Wiley & Sons, p. 19, ISBN 9781118391679.

```python
n = 100  # numeber of samples
Xr = np.random.rand(n)*99.0
y = -7.3 + 2.5*Xr + np.random.randn(n)*27.0
plt.plot(Xr, y, "o", alpha=0.5);
```

Let's add the *bias*, i.e. a column of $1$s to the explanatory variables

```python
X = np.vstack((np.ones(n), Xr)).T
print X.shape
X[0:10,:]
```

    (100, 2)

    array([[  1.        ,  61.97352568],
           [  1.        ,  62.30662457],
           [  1.        ,   7.02231372],
           [  1.        ,  38.69648292],
           [  1.        ,  66.41528033],
           [  1.        ,   5.76376214],
           [  1.        ,  67.46176025],
           [  1.        ,  36.92342092],
           [  1.        ,  98.75991963],
           [  1.        ,  72.29709866]])

## Closed-form Linear Regression

And compute the parametes $\beta_0$  and $\beta_1$ according to
$$ \beta = (X^\prime X)^{-1} X^\prime y $$
***
**Note:**
This not only looks elegant but can also be written in Julia code. However, matrix inversion $M^{-1}$ requires $O(d^3)$ iterations for a $d\times d$ matrix.<br />
https://www.coursera.org/learn/ml-regression/lecture/jOVX8/discussing-the-closed-form-solution

```python
beta = np.linalg.inv(X.T.dot(X)).dot(X.T).dot(y)
```

```python
yhat = X.dot(beta)
yhat.shape
```

    (100,)

```python
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=1, color="red")
```

```python
beta
```

    array([-1.9323329 ,  2.44874906])

## Multiple Linear Regression

```python
n = 100  # numeber of samples
X1 = np.random.rand(n)*99.0
X2 = np.random.rand(n)*51.0 - 26.8
X3 = np.random.rand(n)*5.0 + 6.1
X4 = np.random.rand(n)*1.0 - 0.5
X5 = np.random.rand(n)*300.0

y_m = -7.3 + 2.5*X1 + -7.9*X2 + 1.5*X3 + 10.0*X4 + 0.13*X5 + np.random.randn(n)*7.0
plt.hist(y_m, bins=20)
;
```

```python
X_m = np.vstack((np.ones(n), X1, X2, X3, X4, X5)).T
X_m.shape
X_m[:5]
```

    array([[  1.00000000e+00,   7.87254578e+01,   5.79392004e+00,
              9.23904879e+00,  -4.30247149e-01,   2.46031647e+02],
           [  1.00000000e+00,   2.91761682e+01,  -4.31353400e+00,
              9.81830252e+00,   3.81949696e-01,   2.28261565e+02],
           [  1.00000000e+00,   6.91467395e+01,  -2.12907404e+01,
              7.22969351e+00,  -4.19864300e-01,   1.71674799e+02],
           [  1.00000000e+00,   4.66818561e+01,   1.90678555e+01,
              7.79765125e+00,   8.72368223e-02,   1.19724373e+02],
           [  1.00000000e+00,   2.13247283e+01,   1.99083214e+01,
              1.03699124e+01,   3.22850499e-01,   7.61743075e+01]])

```python
beta_m = np.linalg.inv(X_m.T.dot(X_m)).dot(X_m.T).dot(y_m)
beta_m
```

    array([-8.31523529,  2.50466068, -7.90093854,  1.46755969,  9.06000428,
            0.13216386])

```python
yhat_m = X_m.dot(beta_m)
yhat_m.shape
plt.hist(yhat_m, bins=20);
```

## Evaluation: Root-mean-square Deviation
The root-mean-square deviation (RMSD) or root-mean-square error (RMSE) is a frequently used measure of the differences between values (sample and population values) predicted by a model or an estimator and the values actually observed. The RMSD represents the sample standard deviation of the differences between predicted values and observed values. These individual differences are called residuals when the calculations are performed over the data sample that was used for estimation, and are called prediction errors when computed out-of-sample. The RMSD serves to aggregate the magnitudes of the errors in predictions for various times into a single measure of predictive power. RMSD is a good measure of accuracy, but only to compare forecasting errors of different models for a particular variable and not between variables, as it is scale-dependent.$^1$

***
$^1$  Hyndman, Rob J. Koehler, Anne B.; Koehler (2006). "Another look at measures of forecast accuracy". International Journal of Forecasting. 22 (4): 679–688. doi:10.1016/j.ijforecast.2006.03.001.

```python
import math
RSMD = math.sqrt(np.square(yhat_m-y_m).sum()/n)
print RSMD
```

    7.32215370465

## Regularization, Ridge-Regression

Regularization, in mathematics and statistics and particularly in the fields of machine learning and inverse problems, is a process of introducing additional information in order to solve an ill-posed problem or to prevent overfitting.

In general, a regularization term $R(f)$ is introduced to a general loss function:

for a loss function  $V$ that describes the cost of predicting $f(x)$ when the label is
$y$, such as the square loss or hinge loss, and for the term
$\lambda$  which controls the importance of the regularization term.
$R(f)$ is typically a penalty on the complexity of
$f$, such as restrictions for smoothness or bounds on the vector space norm.$^1$

A theoretical justification for regularization is that it attempts to impose Occam's razor on the solution, as depicted in the figure. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters.

Regularization can be used to learn simpler models, induce models to be sparse, introduce group structure into the learning problem, and more.

We're going to add the L2 term $\lambda||\beta||_2^2$ to the regression equation, which yields to$^2$

$$ \beta = (X^\prime X + \lambda I)^{-1} X^\prime y $$

***
$^1$ Bishop, Christopher M. (2007). Pattern recognition and machine learning (Corr. printing. ed.). New York: Springer. ISBN 978-0387310732.

$^2$ http://stats.stackexchange.com/questions/69205/how-to-derive-the-ridge-regression-solution

```python
p = X.shape[1]  ## get number of parameters
lam = 10.0
p, lam
```

```python

beta2 = np.linalg.inv(X.T.dot(X) + lam*np.eye(p)).dot(X.T).dot(y)
```

```python
yhat2 = X.dot(beta2)
```

```python
RSMD2 = math.sqrt(np.square(yhat2-y).sum()/n)
print RSMD2
```

```python
##n = float(X.shape[0])
print "      RMSE = ", math.sqrt(np.square(yhat-y).sum()/n)
print "Ridge RMSE = ", math.sqrt(np.square(yhat2-y).sum()/n)
plt.plot(X[:,1], y, "o", alpha=0.5)
plt.plot(X[:,1], yhat, "-", alpha=0.7, color="red")
plt.plot(X[:,1], yhat2, "-", alpha=0.7, color="green")
```

```python

```

```python

```

```python

```

```python

```

```python

```


### Example 2: Nonlinear convection in 2D

Following the initial convection tutorial with a single state variable $u$, we will now look at non-linear convection (step 6 in the original). This brings one new crucial challenge: computing a pair of coupled equations and thus updating two time-dependent variables $u$ and $v$.

The full set of coupled equations is now

\begin{aligned}
\frac{\partial u}{\partial t} + u \frac{\partial u}{\partial x} + v \frac{\partial u}{\partial y} = 0 \\
\\
\frac{\partial v}{\partial t} + u \frac{\partial v}{\partial x} + v \frac{\partial v}{\partial y} = 0\\
\end{aligned}

and rearranging the discretized version gives us an expression for the update of both variables

\begin{aligned}
u_{i,j}^{n+1} &= u_{i,j}^n - u_{i,j}^n \frac{\Delta t}{\Delta x} (u_{i,j}^n-u_{i-1,j}^n) - v_{i,j}^n \frac{\Delta t}{\Delta y} (u_{i,j}^n-u_{i,j-1}^n) \\
\\
v_{i,j}^{n+1} &= v_{i,j}^n - u_{i,j}^n \frac{\Delta t}{\Delta x} (v_{i,j}^n-v_{i-1,j}^n) - v_{i,j}^n \frac{\Delta t}{\Delta y} (v_{i,j}^n-v_{i,j-1}^n)
\end{aligned}

So, for starters we will re-create the original example run in pure NumPy array notation, before demonstrating
the Devito version. Let's start again with some utilities and parameters:

```python
from examples.cfd import plot_field, init_hat
import numpy as np
import sympy
%matplotlib inline

# Some variable declarations
nx = 101
ny = 101
nt = 80
c = 1.
dx = 2. / (nx - 1)
dy = 2. / (ny - 1)
sigma = .2
dt = sigma * dx
```

Let's re-create the initial setup with a 2D "hat function", but this time for two state variables.

```python
#NBVAL_IGNORE_OUTPUT

# Allocate fields and assign initial conditions
u = np.empty((nx, ny))
v = np.empty((nx, ny))

init_hat(field=u, dx=dx, dy=dy, value=2.)
init_hat(field=v, dx=dx, dy=dy, value=2.)

plot_field(u)
```

Now we can create the two stencil expression for our two coupled equations according to the discretized equation above. We again use some simple Dirichlet boundary conditions to keep the values on all sides constant.

```python
#NBVAL_IGNORE_OUTPUT
for n in range(nt + 1): ##loop across number of time steps
    un = u.copy()
    vn = v.copy()
    u[1:, 1:] = (un[1:, 1:] -
                 (un[1:, 1:] * c * dt / dy * (un[1:, 1:] - un[1:, :-1])) -
                  vn[1:, 1:] * c * dt / dx * (un[1:, 1:] - un[:-1, 1:]))
    v[1:, 1:] = (vn[1:, 1:] -
                 (un[1:, 1:] * c * dt / dy * (vn[1:, 1:] - vn[1:, :-1])) -
                 vn[1:, 1:] * c * dt / dx * (vn[1:, 1:] - vn[:-1, 1:]))

    u[0, :] = 1
    u[-1, :] = 1
    u[:, 0] = 1
    u[:, -1] = 1

    v[0, :] = 1
    v[-1, :] = 1
    v[:, 0] = 1
    v[:, -1] = 1

plot_field(u)
```

Excellent, we again get a wave that resembles the one from the oiginal examples.

Now we can set up our coupled problem in Devito. Let's start by creating two initial state variables $u$ and $v$, as before, and initialising them with our "hat function.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Grid, TimeFunction

# First we need two time-dependent data fields, both initialized with the hat function
grid = Grid(shape=(nx, ny), extent=(2., 2.))
u = TimeFunction(name='u', grid=grid)
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)

v = TimeFunction(name='v', grid=grid)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

plot_field(u.data[0])
```

Using the two `TimeFunction` objects we can again derive our discretized equation, rearrange for the forward stencil point in time and define our variable update expression - only we have to do everything twice now! We again use forward differences for time via `u.dt` and backward differences in space via `u.dxl` and `u.dyl` to match the original tutorial.

```python
from devito import Eq, solve

eq_u = Eq(u.dt + u*u.dxl + v*u.dyl)
eq_v = Eq(v.dt + u*v.dxl + v*v.dyl)

# We can use the same SymPy trick to generate two
# stencil expressions, one for each field update.
stencil_u = solve(eq_u, u.forward)
stencil_v = solve(eq_v, v.forward)
update_u = Eq(u.forward, stencil_u, subdomain=grid.interior)
update_v = Eq(v.forward, stencil_v, subdomain=grid.interior)

print("U update:\n%s\n" % update_u)
print("V update:\n%s\n" % update_v)
```

    U update:
    Eq(u(t + dt, x, y), dt*(-u(t, x, y)*Derivative(u(t, x, y), x) - v(t, x, y)*Derivative(u(t, x, y), y) + u(t, x, y)/dt))

    V update:
    Eq(v(t + dt, x, y), dt*(-u(t, x, y)*Derivative(v(t, x, y), x) - v(t, x, y)*Derivative(v(t, x, y), y) + v(t, x, y)/dt))

We then set Dirichlet boundary conditions at all sides of the domain to $1$.

```python
x, y = grid.dimensions
t = grid.stepping_dim
bc_u = [Eq(u[t+1, 0, y], 1.)]  # left
bc_u += [Eq(u[t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(u[t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(u[t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(v[t+1, 0, y], 1.)]  # left
bc_v += [Eq(v[t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(v[t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(v[t+1, x, 0], 1.)]  # bottom
```

And finally we can put it all together to build an operator and solve our coupled problem.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

# Reset our data field and ICs
init_hat(field=u.data[0], dx=dx, dy=dy, value=2.)
init_hat(field=v.data[0], dx=dx, dy=dy, value=2.)

op = Operator([update_u, update_v] + bc_u + bc_v)
op(time=nt, dt=dt)

plot_field(u.data[0])
```

Excellent, we have now a scalar implementation of a convection problem, but this can be written as a single vectorial equation:

$\frac{d U}{dt} + \nabla(U)U = 0$

Let's now use devito vectorial utilities and implement the vectorial equation

```python
from devito import VectorTimeFunction, grad

U = VectorTimeFunction(name='U', grid=grid)
init_hat(field=U[0].data[0], dx=dx, dy=dy, value=2.)
init_hat(field=U[1].data[0], dx=dx, dy=dy, value=2.)

plot_field(U[1].data[0])

eq_u = Eq(U.dt + grad(U)*U)
```

We now have a vectorial equation. Unlike in the previous case, we do not need to play with left/right derivatives
as the automated staggering of the vectorial function takes care of this.

```python
eq_u
```

$\displaystyle \left[\begin{matrix}\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \frac{\partial}{\partial t} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}\\\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\partial}{\partial t} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}\end{matrix}\right] = 0$

Then we set the nboundary conditions

```python
x, y = grid.dimensions
t = grid.stepping_dim
bc_u = [Eq(U[0][t+1, 0, y], 1.)]  # left
bc_u += [Eq(U[0][t+1, nx-1, y], 1.)]  # right
bc_u += [Eq(U[0][t+1, x, ny-1], 1.)]  # top
bc_u += [Eq(U[0][t+1, x, 0], 1.)]  # bottom
bc_v = [Eq(U[1][t+1, 0, y], 1.)]  # left
bc_v += [Eq(U[1][t+1, nx-1, y], 1.)]  # right
bc_v += [Eq(U[1][t+1, x, ny-1], 1.)]  # top
bc_v += [Eq(U[1][t+1, x, 0], 1.)]  # bottom
```

```python
# We can use the same SymPy trick to generate two
# stencil expressions, one for each field update.
stencil_U = solve(eq_u, U.forward)
update_U = Eq(U.forward, stencil_U, subdomain=grid.interior)
```

And we have the updated (stencil) as a vectorial equation once again

```python
update_U
```

$\displaystyle \left[\begin{matrix}\operatorname{U_{x}}{\left(t + dt,x + \frac{h_{x}}{2},y \right)}\\\operatorname{U_{y}}{\left(t + dt,x,y + \frac{h_{y}}{2} \right)}\end{matrix}\right] = \left[\begin{matrix}dt \left(- \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} - \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} + \frac{\operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)}}{dt}\right)\\dt \left(- \operatorname{U_{x}}{\left(t,x + \frac{h_{x}}{2},y \right)} \frac{\partial}{\partial x} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} - \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} \frac{\partial}{\partial y} \operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)} + \frac{\operatorname{U_{y}}{\left(t,x,y + \frac{h_{y}}{2} \right)}}{dt}\right)\end{matrix}\right]$

We finally run the operator

```python
#NBVAL_IGNORE_OUTPUT
op = Operator([update_U] + bc_u + bc_v)
op(time=nt, dt=dt)

# The result is indeed the expected one.
plot_field(U[0].data[0])
```

```python
from devito import norm
assert np.isclose(norm(u), norm(U[0]), rtol=1e-5, atol=0)
```


```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
N = 11
h = 1/(N-1)
x = linspace(0,1,N)
```

```python
f = ones((N,))
```

```python
A = zeros((N,N))
for i in range(1,N-1):
    A[i, i-1] = A[i, i+1] = -1
    A[i,i] = 2
A[0,0] = A[-1,-1] = 1
f[0] = f[-1] = 0

A = A/h**2
```

```python
A, f
```

    (array([[ 100.,    0.,    0.,    0.,    0.,    0.,    0.,    0.,    0.,
                0.,    0.],
            [-100.,  200., -100.,    0.,    0.,    0.,    0.,    0.,    0.,
                0.,    0.],
            [   0., -100.,  200., -100.,    0.,    0.,    0.,    0.,    0.,
                0.,    0.],
            [   0.,    0., -100.,  200., -100.,    0.,    0.,    0.,    0.,
                0.,    0.],
            [   0.,    0.,    0., -100.,  200., -100.,    0.,    0.,    0.,
                0.,    0.],
            [   0.,    0.,    0.,    0., -100.,  200., -100.,    0.,    0.,
                0.,    0.],
            [   0.,    0.,    0.,    0.,    0., -100.,  200., -100.,    0.,
                0.,    0.],
            [   0.,    0.,    0.,    0.,    0.,    0., -100.,  200., -100.,
                0.,    0.],
            [   0.,    0.,    0.,    0.,    0.,    0.,    0., -100.,  200.,
             -100.,    0.],
            [   0.,    0.,    0.,    0.,    0.,    0.,    0.,    0., -100.,
              200., -100.],
            [   0.,    0.,    0.,    0.,    0.,    0.,    0.,    0.,    0.,
                0.,  100.]]),
     array([0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 0.]))

```python
u = linalg.solve(A, f)
```

```python
plot(x, u, 'o-b')
plot(x, 0.5*(x*(1-x)), 'r')
```

```python
exact = 0.5*(x*(1-x))
error = max(abs(exact-u))
```

```python
error
```

    4.163336342344337e-17

```python
exact = x*(1-x)*(x-.85)
```

```python
import sympy as sym
t = sym.var('x')
exact_t = t*(1-t)*(t-.85)
fsymbol = sym.lambdify(t, -exact_t.diff(t, 2) )

x = linspace(0,1,N)
f = fsymbol(x)
```

```python
x = linspace(0,1,N)
f = fsymbol(x)
f[0] = f[-1] = 0
```

```python
plot(x,f)
```

```python
u = linalg.solve(A, f)
```

```python
plot(x, u, 'ob-')
plot(x, exact)
```

```python
max(abs(u - exact))
```

    3.469446951953614e-17

```python
x = sym.var('x')
h = sym.var('h')
g = sym.Function('g')
```

```python
def cfd_II(x,h,g):
    return (g(x+h)- 2*g(x) + g(x-h))/h**2

def back_fd(x,h,g):
    return (g(x+h)- g(x))/h

def forward_fd(x,h,g):
    return (g(x)- g(x-h))/h

def central_fd(x,h,g):
    return (g(x+h)- g(x-h))/(2*h)
```

```python
sym.series(back_fd(x, h, g), x=h, x0=0, n=2)
```

$\displaystyle \left. \frac{d}{d \xi_{1}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }} + \frac{h \left. \frac{d^{2}}{d \xi_{1}^{2}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }}}{2} + O\left(h^{2}\right)$

```python
sym.series(forward_fd(x, h, g), x=h, x0=0, n=2)
```

$\displaystyle \left. \frac{d}{d \xi_{1}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }} - \frac{h \left. \frac{d^{2}}{d \xi_{1}^{2}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }}}{2} + O\left(h^{2}\right)$

```python
sym.series(central_fd(x, h, g), x=h, x0=0, n=3)
```

$\displaystyle \left. \frac{d}{d \xi_{1}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }} + \frac{h^{2} \left. \frac{d^{3}}{d \xi_{1}^{3}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }}}{6} + O\left(h^{3}\right)$

```python
sym.series(cfd_II(x, h, g), x=h, x0=0, n=5)
```

$\displaystyle \left. \frac{d^{2}}{d \xi_{1}^{2}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }} + \frac{h^{2} \left. \frac{d^{4}}{d \xi_{1}^{4}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }}}{12} + \frac{h^{4} \left. \frac{d^{6}}{d \xi_{1}^{6}} g{\left(\xi_{1} \right)} \right|_{\substack{ \xi_{1}=x }}}{360} + O\left(h^{5}\right)$

slides/Lecture 10 - LH - LAB - Introduction to PDEs - Finite Differences in 1D.ipynb


# 1. City level spread dynamics

## 1.1 Heterogeneity in getting infected

I start modeling city level spread dynamic from simple SIR model
https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology#The_SIR_model_without_vital_dynamics
$$
\begin{align}
& \frac{dS}{dt} = - r_0 I S, \\[3pt]
& \frac{dI}{dt} =- \frac{dS}{dt}- I, \\[3pt]
&R = 1-S-I,\\[3pt]
&I(0)=I_0,\,S(0)=1-I_0
\end{align}
$$
Here the city population being normalized to 1.

$S$ - The proportion of the of susceptible population

$I$ - The proportion of the of infectious population

$R$ - The proportion of the of removed (and immune) or deceased population

The time unit is equal to mean disease recovery (or death) duration and for COVID-19 is something like 15-17 days.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import scipy.stats as scst
import scipy.special as scsp
import scipy.optimize as scop
from scipy.integrate import odeint
from scipy.integrate import quad

import matplotlib.ticker as ticker
plt.rc('legend',fontsize=12, handlelength=2) # using a size in points

plt.style.use('seaborn-darkgrid')
```

I decided to perform modeling for 8 time units.

```python
modeling_period=8.0
t_step=0.005
m_t=np.arange(0.0, modeling_period+t_step, t_step,dtype=np.float64)
n_steps=len(m_t)-1
```

Now define  parameters,equation and solve it

```python
def SIR(y,t,r0):
    S,I=y[0],y[1]
    dS = -r0*I*S
    dI= -dS - I
    return [dS, dI]

r0,I0=7.0,0.0001
SIR_y0= [1.0 -I0, I0]

SIR_sol = odeint(SIR,SIR_y0, m_t,(r0,)).T
S,I,R=SIR_sol[0],SIR_sol[1],1.0-SIR_sol[0]-SIR_sol[1]
```

Let's look at the result

```python
fig, ax = plt.subplots(dpi=120)
plt.plot(m_t, S , color='C0', label=f'$S(t)$')
plt.plot(m_t, I, color ='C1',  label=f'$I(t)$')
plt.plot(m_t, R, color='C2',  label=f'$R(t)$')
plt.xlabel('t')
plt.legend()
plt.show()
```

SIR model above suggests homogeneity of the population. And the model doesn't explain the fact that COVID-19 infection rate reaches maximum after around 10%-20% of the population being ever infected and then substantially slows down.  This was observed in many cities around the world. In many cases it can be explained by public measures to limit the disease spread. But the roughly same picture was in countries where there was no public measures or the measures was inefficient.

Inhomogeneity of the population could give the explanation. Indeed, individuals are different in the intensity of contacts with other people (social inhomogeneity), and in ability to catch the disease (biological inhomogeneity). Social and biological are together forming total population inhomogeneity in ability to catch the disease. As a result some persons are much more exposed to the disease catching (and thus transmitting) and these persons catch the disease among the first and become immune (at least for some time). It is a reason why average disease spreading ability drops in time.

To deal with inhomogeneity we start from even more simple model SI, where susceptible at some moment becomes infectious and leave infectious for ever. For this model at first I suggest some reasonable inhomogeneity dealing technique (IDT) and next propose more speculative one. The second IDT is much more simple to implement and I will show, that although having more speculative nature it is produces almost the same results as the first one. So it is kind of digital approximation of the reasonable IDT and will be used in the later modeling.

I use lognormally distributed random variable $X \sim LogN(\mu_0,v_0)$ to model the population ability to get the disease, so $\log(X) \sim N(\mu_0,v_0)$, where $v_0$ - variance of $\log(X)$.
$$f_{X}(x)=f_{X}(x,\mu_0,v_0)= \frac {1} {\sqrt{2\pi\,v_0}} \frac{1}{x} \exp\left( -\frac{(\ln x-\mu_0)^2}{2 v_0} \right)$$
So, for now log normally distributed variable $X$ describe the **whole population ability to catch  the disease and it does not change in time**.

Log normal distribution is quite common in describing social and biological processes. One of its main feature is large range of values. It makes separate consideration of superspread events unnecessary because heavy distribution tail covers superspread events.

Let $s(x,t)$ is the portion (at moment t) of susceptible population with ability to catch the disease $x$. At the beginning it is equal to one ($s(x,0)=1$) and moves down to zero in time as more and more individuals get infected. The total portion of susceptible population is getting by integrating over x distribution:
$$S(t)=\int_0^{\infty} s(x,t)f_X(x)dx$$

The part of $s(x,t)$ becoming infectious at the interval $t,t+dt$ is proportional to $s(x,t)$ itself, ability to catch and spread the disease $x$, and total portion of infectious population:
$$ds(x,t)=-\alpha x\,s(x,t)I(t)dt$$
where
$$I(t)=1-S(t)=1-\int_0^{\infty} s(x,t)f_X(x)\,dx$$

Finally
$$\frac{ds}{dt}=-\alpha\,x\,s\,\left(1-\int_0^{\infty} s(x,t)f_X(x)\,dx\right)$$

Let $s(x,t)=e^{-x\,u(t)}$ then we have
$$-x\,s\,\frac{du(t)}{dt}=-\alpha\,x\,s\,\left(1-\int_0^{\infty} e^{-x\,u(t)}f_X(x)\,dx\right)$$
or
$$\frac{du}{dt}=\alpha\,\left(1-\int_0^{\infty} f_X(x)e^{-x\,u}\,dx\right)$$

The integral on the right is the Laplace transform of the $X$ distribution https://en.wikipedia.org/wiki/Laplace_transform#Formal_definition
$$\mathcal{L}(u)= \int_0^{\infty} f_X(x)e^{-x\,u}\,dx $$

so we get the equation looking rather simple:
$$\frac{du}{dt}=\alpha\,\left(1-\mathcal{L}(u)\right)$$
$$u(0)=\mathcal{L}^{-1}(S(0)),\:S(t)=\mathcal{L}(u(t)),\: I(t)=1-S(t)$$

The problem is the Laplace transform of the lognormal distribution
$$\mathcal{L}(u,\mu_0,v_0)= \int_0^{\infty}e^{-x\,u} f_X(x,\mu_0,v_0)dx$$

$$=\frac {1} {\sqrt{2\pi\,v_0}} \int_0^{\infty}  \frac{1}{x}\exp(-x\,u) \exp\left(-\frac{(\ln x-\mu_0)^2}{2 v_0}\right) dx$$
is not easily computable function.

Let $M(\mu_0,v_0)=\exp(\mu_0+v_0/2) $ is expectation of lognormal variable $X \sim LogN(\mu_0,v_0)$

It can be shown that for any for any $a$
$$\mathcal{L}(u,\mu_0,v_0)=\mathcal{L}(u\,e^a,\mu_0-a,v_0)$$
also
$$ \frac{\mathcal{L}(u,\mu_0,v_0)}{du}=-M(\mu_0,v_0)\mathcal{L}(u,\mu_0+v_0,v_0)=-M(\mu_0,v_0)\mathcal{L}(u\,e^{v_0},\mu_0,v_0)$$

So we get forward differential equation for $\mathcal{L}(u)$
$$\frac{d\mathcal{L}(u)}{du}=-M\,\mathcal{L}(u\,e^{v_0})$$

But all these things does not make us any closer to calculation of Laplace transform of the lognormal distribution. Yet what these equations does allow us — they are allowed us to check the quality of different approaches to the calculation.

First I use this approximation https://en.wikipedia.org/wiki/Log-normal_distribution#Characteristic_function_and_moment_generating_function

$$\mathcal{L}^*(u,\mu_0,v_0)=\frac{\exp\left(-\frac{W(\theta)(W(\theta)+ 2)}{2v_0} \right)}{\sqrt{1+W(\theta)}}$$
where
$$\theta=u\,v_0\,e^{\mu_0}$$

and $W$ is the Lambert W function.

Let's check how good the approximation is by defining the function for checking the ratio
$$-\frac{d\mathcal{L}^*(u,\mu_0,v_0)/du}{M\,\mathcal{L}^*(u,\mu_0+v_0,v_0)}$$
it should be close to 1.

```python
def check_ratio(L,u,mu0,v0):
    u_step=(u[-1]-u[0])/(len(u)-1)
    d=L(u,mu0,v0)
    deriv=(d[:-1]-d[1:])/u_step
    M=np.exp(mu0+0.5*v0)
    V=M*L(u,mu0+v0,v0)
    return 0.5*(u[:-1]+u[1:]),2.*deriv/(V[:-1]+V[1:])
```

Also define Laplace transform approximation function

```python
def LaplTrLNApprox(u,mu0,v0):
    theta=u*v0*np.exp(mu0)
    W=np.real(scsp.lambertw(theta))
    return np.exp(-W*(W+2)/(2*v0))/np.sqrt(1+W)

```

And now we can make the check:

```python
mu0=1.5
v_step=0.5
v_min=1.5
v_max=4.0
v_range=np.arange(v_min,v_max+v_step, v_step,dtype=np.float64)
u_step=0.005
u_min=0.0
u_max=4.0

u_range=np.arange(u_min,u_max+u_step, u_step,dtype=np.float64)

fig, ax = plt.subplots(dpi=120)
for v0 in v_range:
    u,r=check_ratio(LaplTrLNApprox,u_range,mu0,v0)
    plt.plot(u, r, label=f'$v=${v0}')
plt.xlabel('u')
ax.set_ylim([0.4,1.2])
plt.legend(loc='lower right')
plt.show()

```

Let's take closer look what happens nearby 0.

```python
u_mid=0.25
u_subrange=np.arange(u_min,u_mid+u_step, u_step,dtype=np.float64)
fig, ax = plt.subplots(dpi=120)
for v0 in v_range:
    u,r=check_ratio(LaplTrLNApprox,u_subrange,mu0,v0)
    plt.plot(u, r, label=f'$v=${v0}')
plt.xlabel('u')
ax.set_ylim([0.4,1.2])
plt.legend(loc='lower right')
plt.show()
```

So the approximation is quite good for $u>1$, but nearby 0 it is unsatisfactory. And the range nearby 0 is very important for modelling because it is the beginning of the disease and small difference in the infection level will be amplified.

Now we define the Laplace transform by direct integrating

```python
def LaplTrLNDirect(u,mu0,v0):
    max_exp=744.0 # np.exp(-max_exp) - minimal double value to calculate without overflow
    a=max_exp*np.exp(-(np.sqrt(2.0*max_exp*v0)+mu0)/2.0)
    def LLN(v): #calculates the Laplace transform for scalar v
        if v==0:
            return np.sqrt(2.0*np.pi*v0)

        def F(y): #integrand
            s=np.exp(y+mu0)*v+0.5*y**2/v0
            return np.exp(-s)

        lb=-np.sqrt(2.0*max_exp*v0) #low integration bound
        # for smaller values integrand will be calculated as zero
        # anyway due to overflow
        ub=3.*np.log(max_exp/(v+a))-mu0 #upper integration bound

        # for bigger values integrand will be calculated as zero
        # anyway due to overflow
        if lb>=ub:
            return 0.0
        return quad(F,lb,ub,epsabs=1.0e-9, epsrel=1.0e-9)[0] #integration
    m=1.0/np.sqrt(2.0*np.pi*v0)
    return m*np.vectorize(LLN,otypes=[float])(u)

```

It is really much more accurate but also much slower

```python
fig, ax = plt.subplots(dpi=120)
for v0 in v_range:
    u,r=check_ratio(LaplTrLNDirect,u_subrange,mu0,v0)
    plt.plot(u, r, label=f'$v=${v0}')
plt.xlabel('u')
ax.set_ylim([0.4,1.2])
plt.legend(loc='lower right')
plt.show()
```

Below alternative much faster and more simple to implement inhomogeneity dealing technique.

Previously we use lognormally distributed variable $X$ to model the **whole** population ability to catch the disease. The distribution was fixed and all dynamics was described by $s(x,t)$ — the portion  of susceptible population with ability to catch and spread the disease $x$. Now we use lognormally distributed variable $X_s(t)$ to model **susceptible** population ability to catch the disease and also value of $S(t)$ — the proportion of the of susceptible population (of all the population)

Suppose at a time $t$ we have log normal $X_s$ distribution with parameters $\mu(t)=E[\log(X_s(t))]$ and $v(t)=Var[\log(X_s(t))]$. Let's calculate new values $\mu(t+dt)=E[\log(X_s(t+dt))]$ and $v(t+dt)=Var[\log(X_s(t+dt))]$ at moment $t+dt$. Of cause the distribution at moment $t+dt$ will loose lognormality **but we suppose the distribution somehow adjust itself to become lognormal with new parameters** $\mu(t+dt)$ and $v(t+dt)$.

Let now $s(x,t)$ is the "density" of portion of susceptible population namely: the susceptible portion of total population having (at a time t) ability to catch and spread the disease in the interval $[x,x+dx]$ is $s(x,t)dx$ and $s(x,t)=S(t)\cdot f_{X_s}(x)$:
$$f_{X_s}(x,t)=f_{X_s}(x,\mu(t),v(t))= \frac {1} {\sqrt{2\pi\,v}} \frac{1}{x} \exp\left( -\frac{(\ln x-\mu)^2}{2 v} \right)$$

Vice versa
$$f_{X_s}(x,t)=s(x,t)/S(t)$$
$$S(t)= \int_0^{\infty}s(x,t)dx$$

We start from calculation $s(t+dt)$ assuming that individuals get infected with probability equal to the product of $dt$,$x$ and some coefficient $c$ for the infection "intensity":
$$s(x,t+dt)=s(x,t)-c x\, s(x,t)dt=S(t)f_{X_s}(x,t)(1-c x\, dt)$$
integrating it over x we get
$$S(t+dt)= \int_0^{\infty}S(t)f_{X_s}(x,t)(1-c x\, dt)\,dx=S(t)(1-c E_{X_s(t)}[x]dt)=S(t)(1-c M\, dt)$$
where $M=M(t)=M(\mu,v)=\exp(\mu+v/2)$ and
$$f_{X_s}(t+dt)=s(x,t+dt)/S(t+dt)=\frac{f_{X_s}(t)(1-c x\, dt)}{1-c M\, dt}$$
so
$$\mu(t+dt)=E_{X_s(t+dt)}\,[\ln(x)]=\frac{E_{X_s(t)}[(1-c x\,dt)\ln(x)]}{1-c M dt}=\frac{E_{X_s(t)}[\ln(x)]-c E_{X_s(t)}[x\ln(x)]\,dt}{1-c M dt}$$

$$=\mu(t)-c M(\mu,v) v(t)\,dt +o(dt)$$
also omitting some math

$$v(t+dt)=v(t)-c M(\mu,v) v^2(t)\,dt +o(dt) $$

As we have assumed above $f_{X_s}(t+dt)$ somehow adjust itself to become lognormal $X_s(t+dt)\sim LogN(\mu(t+dt),v(t+dt))$ and we can apply the above procedure to step further on another $dt$. Thus, the distribution $ X_s (t) $ evolves while remaining lognormal with the parameters $ \mu (t) $ and $ v (t) $, which change in accordance with the equations:
$$\frac{d S}{dt}=-c  M S$$
$$\frac{d\mu}{dt}=-c M v$$
$$\frac{d v}{dt}=-c M v^2$$
Coefficient  $c$  for infection "intensity" is just $\alpha I=\alpha (1-S)$ (remember, we're still discussing SI model) so finally we get the system:
$$\frac{d S}{dt}=- \alpha M (1-S)S$$
$$\frac{d\mu}{dt}=- \alpha M (1-S)v $$
$$\frac{d v}{dt}=-\alpha M (1-S)v^2 $$

If compared to SIR model this model suggest non-constant $r_0$. Now $r_0$ changes in time $r_0(t)=\alpha M(t)$ where $\alpha$ is a constant calculated as $\alpha =r_0(0)/M(0)$ so
$$r_0= \alpha\, M(\mu,v)$$
$$\frac{d S}{dt}=- r_0 (1-S)S$$
$$\frac{d\mu}{dt}=- r_0 (1-S)v $$
$$\frac{d v}{dt}=-r_0 (1-S)v^2 $$
$$S(0)=1-I_0,\:\mu(0)=\mu_0,\:v(0)=v_0,\: \alpha=\frac{r_0(0)}{M(\mu_0,v_0)}$$

Let's compare approach with direct calculation of the Laplace transform (or its approximation using Lambert W function) and evolving lognormal distribution (the alternative).

For the first one we need a function to calculate initial value of $u$

```python
def M(mu,v):return np.exp(mu+0.5*v)

def get_u0(LLN,I0,mu0,v0,guess=3.0e-03):
    '''
    calculates u_0 given S_0 and the distribution parameters
    LLN - function to calculate the Laplace transfom of lognormal variable
    mu0,v0 - parameters of the disribution namely
    expectation and variance of the lognormal variable logarithm
    '''
    def cost_f(v):
        if v<=0:
            vM=v*M(mu0,v0)
            return vM**2*np.exp(v0)-vM #the minimize function can pass a negative v
        return LLN(v,mu0,v0)-1.0+I0
    m=scop.root(cost_f,guess)
    return np.atleast_1d(m.x)[0]

```

Next define ODE and function to integrate it returning $I$

```python
def get_I(LTLN,mu0,v0):
    '''
    calculates I for SI model with lognormal population inhomogeneity
    LTLN - function to calculate the Laplace transfom of lognormal variable
    mu0,v0 - parameters of the disribution namely
    expectation and variance of the lognormal variable logarithm
    '''
    α=r0/M(mu0,v0)
    def ODE(y,t):
        u=y[0]
        du = α*(1-LTLN(u,mu0,v0))
        return [du]

    y0= [get_u0(LTLN,I0,mu0,v0)]
    sol = odeint(ODE,y0, m_t).T
    S=LTLN(sol[0],mu0,v0)
    return 1.0-S
```

The same for the alternative method

```python
def get_I_alt(mu0,v0):
    α=r0/M(mu0,v0)
    def ODE(y,t):
        S,mu,v=y[0],y[1],y[2]

        minus_r0_mult_I=-α*M(mu,v)*(1.0-S)

        dS = minus_r0_mult_I*S
        dmu = minus_r0_mult_I*v
        dv = dmu*v
        return [dS, dmu, dv]
    y0= [1.0 - I0,mu0,v0]
    sol = odeint(ODE,y0, m_t).T
    return 1.0-sol[0]

```

Now compare the results

```python
for v0 in v_range:
    fig, ax = plt.subplots(dpi=120)
    I_exact=get_I(LaplTrLNDirect,mu0,v0)
    I_approx=get_I(LaplTrLNApprox,mu0,v0)
    I_alt=get_I_alt(mu0,v0)
    plt.plot(m_t, I_exact, color='C0', label=f'exact')
    plt.plot(m_t, I_approx,color='C1', label=f'approximation')
    plt.plot(m_t, I_alt,color='C3', label=f'alternative')
    ax.yaxis.set_major_locator(ticker.MultipleLocator(0.2))
    ax.set_ylim([-0.05,1.05])
    ax.set_ylabel(f'$I$ value')
    plt.legend(loc='upper left')
    ax2 = ax.twinx()
    plt.plot(m_t, I_approx/I_exact, color='C2', label=f'approximation/exact ')
    plt.plot(m_t, I_alt/I_exact, color='C4', label=f'alternative/exact ')
    ax2.set_ylim([-0.05,1.30])
    ax2.yaxis.set_major_locator(ticker.MultipleLocator(0.25))
    ax2.grid(False)
    ax2.set_ylabel('Ratio')
    plt.legend(loc='lower right')
    plt.title(f'$v=${v0}')
    plt.xlabel(f'$t$')
    plt.show()
```

Thus, the alternative method although uses suspicious hypothesis of somehow adjusted log normal distribution gives quite close result to exact solution.

Exact solution uses slow calculated Laplace transform moreover I can not to imaging how to implement it many environments for example in pymc3 Bayesian inference.

Now let's apply the approach to SIR model. Modified SIR equations are
$$\begin{align}
& r_0= \alpha\, M(\mu,v), \\[3pt]
& \frac{d S}{dt}=- r_0 I\,S, \\[3pt]
& \frac{dI}{dt} =- \frac{dS}{dt}- I,\\[3pt]
& \frac{d\mu}{dt}=- r_0 I\,v, \\[3pt]
& \frac{d v}{dt}=-r_0 I\,v^2, \\[3pt]
&R = 1-S-I
\end{align}$$
$$I(0)=I_0,\:S(0)=1-I_0,\:\mu(0)=\mu_0,\:v(0)=v_0,\: \alpha=\frac{r_0(0)}{M(\mu_0,v_0)}$$

```python
def getSIRdS(r0_0,I0,v0):
    α=r0_0/M(0.0,v0)

    def ODE(y,t):
        S,I,mu,v=y

        minus_r0_mult_I=-α*M(mu,v)*I

        dS = minus_r0_mult_I*S
        dI = -dS - I
        dmu = minus_r0_mult_I*v
        dv = dmu*v
        return [dS, dI, dmu, dv]

    y0=[1.0-I0,I0,0.0,v0]

    sol = odeint(ODE,y0, m_t).T
    dS=-ODE(sol,m_t)[0]
    S,I,mu,v=sol
    R=1.0-S-I
    return S,I,R,dS
```

The function above calculates $S$,$I$,$R$ values, also infection rate $|dS|$ and $r_0(t)/r_0(0)$.

We use 0.001 as initial infectious portion $I_0$. To get $r_0(0)$ for COVID_19 we uses the fact that it was reported around 30% daily growth in initial stage. Combining with, say, 16 days mean disease recovery (or death) duration it gives us a value for $r_0$.

$$r_0(0)=\ln(1.3)\cdot 16 +1 \approx 5.2$$

We need only to set up initial $v$ - $v_0$. And we guess it to 2.5.

To check $v_0$ values we use a function calculating the ratio of top 10% *bound* to low 10% *bound* given lognormal distribution with parameter $v_0$.
$$r=\exp\left((\Phi^{-1}(0.9)-\Phi^{-1}(0.1))\cdot\sqrt{v_0}\right)$$
where $\Phi()$ - standard normal PDF.

For guessed $v_0=2.5$ the range is

```python
range0901=scst.norm.ppf(0.9)-scst.norm.ppf(0.1)
r=np.exp(range0901*np.sqrt(2.5))
print(f'ratio of top 10% to low 10% for v0=2.5 is {r}')
```

    ratio of top 10% to low 10% for v0=2.5 is 57.548144094658774

I also divide total ratio on social part $r_s$ and biological part $r_b$ assuming $r_s$ and $r_b$ having joint log normal distribution and independent of each other. We will consider equally divided $r_s$ and $r_b$ version and version with predefined $r_s$ equal say 20.0

For equally divided
$$r_s=r_b=\exp\left(\frac{(\Phi^{-1}(0.9)-\Phi^{-1}(0.1))\cdot\sqrt{v_0}}{\sqrt{2}}\right)$$
For given $r_s$
$$r_b=\exp\left(\sqrt{(\Phi^{-1}(0.9)-\Phi^{-1}(0.1))^2\cdot v_0-r_s^2}\right)$$

```python
def get_range(v0,rs=None):
    temp=range0901**2*v0
    r=np.exp(np.sqrt(temp))
    if rs is None:
        rs=np.exp(np.sqrt(0.5*temp))
        rb=rs
    else:
        rb=np.exp(np.sqrt(temp-np.log(rs)**2))
    return r,rs,rb

r,rs,rb=get_range(2.5)
print(f'ratio of top 10% to low 10% for v0=2.5 is {r},\nrs={rs} and rb={rb} (equally divded)')
```

    ratio of top 10% to low 10% for v0=2.5 is 57.548144094658774,
    rs=17.56022527907195 and rb=17.56022527907195 (equally divded)

Let's see the results

```python
v0=2.5
r0_0=5.2
I0=0.001
S,I,R,dS= getSIRdS(r0_0,I0,v0)

fig, ax = plt.subplots(dpi=120)
plt.plot(m_t, S, color='C0', label=f'$S$')
plt.plot(m_t, I,color='C1', label=f'$I$')
plt.plot(m_t, R,color='C3', label=f'$R$')
plt.plot(m_t, dS/(I*S)/r0_0,color='C4', label=f'$r_0(t)/r_0(0)$')
ax.set_ylim([-0.05,1.05])
ax.yaxis.set_major_locator(ticker.MultipleLocator(0.1))
ax.set_ylabel(f'$SIR$ values, $r_0(t)/r_0(0)$')
plt.legend(loc='center left')
plt.xlabel(f'$t$')
ax2 = ax.twinx()
plt.plot(m_t, dS, color='C2', label=f'$|dS/dt|$')
ax2.set_ylim([-0.01,0.26])
ax2.yaxis.set_major_locator(ticker.MultipleLocator(0.025))
ax2.grid(False)
ax2.set_ylabel(f'The number of new cases per unit of time $|dS/dt|$')
plt.legend(loc='upper right')
plt.title(f'$v=${v0}, $r_0(0)=${r0_0}')

plt.show()

```

How many has ever been infected at the maximum of the infection rate $|dS/dt|$?

```python
print(f'R+I at max |dS/dt| is {(R+I)[np.argmax(dS)]}.')
```

    R+I at max |dS/dt| is 0.15232846641336895.

For $v_0=2.5$ infection rate peaked at 15% of the population has been ever infected. Let's call the percentage of the population has been ever infected (i.e. $R+I$) at infection rate peak as peaked infected percent or PIP.

Next we try to find $v_0$ so that PIP is equal 10%.

```python
goal=0.1
guess=3.0
def cost_f(v):
    S,_,_,dS= getSIRdS(r0_0,I0,v[0])
    return (1.0-goal)-S[np.argmax(dS)]
m=scop.root(cost_f,guess,tol=0.001)
print(f'PIP is equal 10% when v0 is {m.x}')
```

    PIP is equal 10% when v0 is [3.54164561]

So total inhomogeneity of the population described by  $v_0$ must be equal 3.54. How soundness this figure we can see calculating appropriate ratios.

```python
v0=3.54
r,rs,rb=get_range(v0)
print(f'ratio of top 10% to low 10% for v0={v0} is {r},\nrs={rs} and rb={rb} (equally divded)')
```

    ratio of top 10% to low 10% for v0=3.54 is 124.26914817869788,
    rs=30.264848679665814 and rb=30.264848679665814 (equally divded)

30 as ratio of top 10% to low 10% for social contacts rate seems to me possible although 20 seems more reasonable. Let's try 20 and see what happens with biological inhomogeneity.

```python
r,rs,rb=get_range(v0,20.0)
print(f'ratio of top 10% to low 10% for v0={v0} is {r},\nif rs={rs} then rb={rb} ')
```

    ratio of top 10% to low 10% for v0=3.54 is 124.26914817869788,
    if rs=20.0 then rb=43.77670064450303

How sound biological inhomogeneity to COVID-19 presuming top 10% to bottom 10% ratio of 30 or even 43? From one hand it is a big range. On the other hand we know that some ill patients have no symptoms at all, others soon painfully dies. So COVID-19 acts really different on individuals and it is possible that biological ability to catch the decease also greatly varies.

Also there are the articles showing that the individuals ability to spread COVID-19 substantioanally differs:
https://www.sciencemag.org/news/2020/05/why-do-some-covid-19-patients-infect-many-others-whereas-most-don-t-spread-virus-all

https://www.theatlantic.com/health/archive/2020/09/k-overlooked-variable-driving-pandemic/616548/

Anyway using our model we draw some usefull dependencies for $v_0$, $r_s$,$r_b$.

```python
def PIP(r0,I0,v):
    def F(d):
        _,I,R,dS=getSIRdS(r0,I0,d)
        return (R+I)[np.argmax(dS)]
    return 100*np.vectorize(F,otypes=[float])(v)
```

```python
v_r=np.arange(2.5,6.1,0.1)
r,_,_=get_range(v_r)
pip=PIP(r0,I0,v_r)
fig, ax = plt.subplots(dpi=120)
plt.plot(v_r,pip, color='C0', label=f'PIP')
ax.yaxis.set_major_locator(ticker.MultipleLocator(2.5))
ax.set_ylim([0,17.5])
ax.set_ylabel(f'PIP %')
plt.xlabel(f'$v_0$')
plt.legend(loc='center left')
ax2 = ax.twinx()
plt.plot(v_r,r, color='C1', label=f'$r$')
ax2.set_ylim([0,700])
ax2.yaxis.set_major_locator(ticker.MultipleLocator(100))
ax2.grid(False)
ax2.set_ylabel('ratio top 10% to bottom 10%')
plt.legend(loc='center right')
plt.title(f'PIP values and top 10% to bottom 10% ratio for differend $v_0$')
plt.show()

```

Now look at the ratios of biological and social inhomogeneity for different PIP values

```python
_,rs,_=get_range(v_r)
_,_,rb=get_range(v_r,20.0)
fig, ax = plt.subplots(dpi=120)
plt.plot(pip,rs, color='C0', label=f'biological (and  social) inhomogeneity equally divided')
plt.plot(pip,rb, color='C1', label=f'biological inhomogeneity assuming $r_s=20$')
ax.set_ylabel(f'ratio top 10% to bottom 10%')
plt.legend()
plt.xlabel(f'PIP')
plt.show()

```

PS To get the Laplace transform of lognormal variable by direct integration there is the much faster version:

```python
def LaplTrLNFast(u,mu0,v0):
    '''
    fast version of direct integration
    '''
    max_exp=700.0 # np.exp(max_exp) - maximal double value to calculate without overflow
    sm=np.sqrt(2.0*v0)
    def F(v,x):
        y=x**3+x
        z=sm*y+mu0
        s=np.exp(np.where(z<max_exp,z,max_exp))*v+y**2
        return np.exp(-s)*(3.0*x**2+1.0)
    v=np.atleast_1d(u)[None,:]
    step=5.5/512.0
    lb=-250.5*step
    x=np.arange(lb,0.0,step)[:,None]
    res=(sm*step/np.sqrt(2.0*np.pi*v0))*np.sum(F(v,-x)+F(v,x),axis=0)
    if len(res)==1:return res[0]
    return res
```

I have found the idea behind here
    https://www.sciencedirect.com/science/article/pii/S0167668719303993
and it is rooted in this article

*Takahasi, H. and Mori, M. (1974). Double exponential formulas for numerical integration. Publ. RIMS Kyoto Univ, 9:721-741.*

Still it is much more computational expensive compared to proposed alternative approach.

```python

```