# Numpy (Numeric python)
> - 패키지 이름과 같이 **수리적 파이썬 활용**을 위한 파이썬 패키지
> - **선형대수학 구현**과 **과학적 컴퓨팅 연산**을 위한 함수를 제공
> - (key) `nparray` 다차원 배열을 사용하여 **벡터의 산술 연산**이 가능
> - **브로드캐스팅**을 활용하여 shape(형태 혹은 모양)이 다른 데이터의 연산이 가능
>> - 기존 언어에서는 제공 X
>> - 굉장히 파워풀한 기능으로서 빅데이터 연산에 굉장히 효율이 좋음

## Numpy 설치 와 import
> - 선행 학습을 통해 클래스와 함수에서 클래스를 불러들여 사용할 수 있다고 배웠습니다.
- 다만 직접 작성한 클래스가 아닐경우, 그리고 현재 컴퓨터에 사용해야 할 패키지가 없을경우 간단한 명령어로 설치가능.
>> - `pip`, `conda` 명령어 : python 라이브러리 관리 프로그램으로 오픈소스라이브러리를 간편하게 설치 할 수 있도록 하는 명령어

> - 1. 콘솔창에서 실행 시
**`pip` `install` `[패키지명]`** 혹은
**`conda` `install` `[패키지명]`**
> - 2. 주피터 노트북으로 실행 시
**`!pip` `install` `[패키지명]`**
> - 아나콘다 환경으로 python 환경설정 시 기본적으로 Numpy 설치가 되어있음

```python
# 주피터 노트북에서 Numpy 설치
!pip install numpy
```

    DEPRECATION: Configuring installation scheme with distutils config files is deprecated and will no longer work in the near future. If you are using a Homebrew or Linuxbrew Python, please see discussion at https://github.com/Homebrew/homebrew-core/issues/76621
    Requirement already satisfied: numpy in /opt/homebrew/lib/python3.9/site-packages (1.21.2)

```python
# Numpy 사용을 위해 패키지 불러들이기
import numpy as np
# 관례적으로 np라는 약자를 많이 사용하게 됩니다.
# 파이썬을 사용하는 대부분의 유저들이 사용하고 있는 닉네임이니 이건 꼭 지켜서 사용해주시는 것을 추천드립니다.
```

## 데이터분석을 위한 잠깐의 선형대수학
numpy는 기본적으로 수리적 컴퓨팅을 위한 패키지 입니다. 선형대수학을 약간만 이해한다면 데이터를 훨씬 더 깊이있게 다룰 수 있습니다.

출처 : https://art28.github.io/blog/linear-algebra-1/

### 데이터의 구분에 따른 표현 방법과 예시

#### 스칼라
    1, 3.14, 실수 혹은 정수

#### 벡터
    [1, 2, 3, 4], "문자열"

#### 3 X 4 매트릭스
    [[1, 2, 3, 4],
     [5, 6, 7, 8],
     [9, 0, 11, 12]]

#### 2 X 3 X 4 텐서
    [[[1, 2, 3, 4],
     [5, 6, 7, 8],
     [9, 0, 11, 12]],
     [[1, 2, 3, 4],
     [5, 6, 7, 8],
     [9, 0, 11, 12]]]

### 데이터로 표현한 선형대수학

출처 : https://m.blog.naver.com/nabilera1/221978354680

### 데이터 형태에 따른 사칙연산

> 스칼라 +, -, *, / -> 결과도 스칼라
벡터 +, -, 내적 -> +, - 결과는 벡터, 내적 결과는 스칼라
매트릭스 +, -, *, /
텐서 +, -, *, /

### 데이터분석에 자주 사용하는 특수한 연산
벡터와 벡터의 내적

$$\begin{bmatrix}1 & 2 & 3 & 4 \end{bmatrix} \times \begin{bmatrix}1 \\ 2 \\ 3 \\ 4 \end{bmatrix} = 1 * 1 + \
2 * 2 + 3 * 3 + 4 * 4 = 30$$
# $$ A^TA $$

#### 벡터와 벡터의 내적이 이루어지려면

    1. 벡터가 마주하는 shape의 갯수(길이)가 같아야 합니다.
    2. 연산 앞에 위치한 벡터는 전치(transpose) 되어야 합니다.

출처 : https://ko.wikipedia.org/wiki/%EC%A0%84%EC%B9%98%ED%96%89%EB%A0%AC

#### 벡터 내적으로 방정식 구현

$$y = \begin{bmatrix}1 & 2 & 1 \end{bmatrix} \times \begin{bmatrix}x_1 \\ x_2 \\ x_3 \\ \end{bmatrix} = 1 * x_1 + \
2 * x_2 + 1 * x_3 = x_1 + 2x_2 + x_3$$

## 브로드캐스팅
> 파이썬 넘파이 연산은 브로드캐스팅을 지원합니다.
벡터연산 시 shape이 큰 벡터의 길이만큼 shape이 작은 벡터가 연장되어 연산됩니다.

출처 : http://www.astroml.org/book_figures/appendix/fig_broadcast_visual.html

```python
np.arange(3).reshape((3,1))+np.arange(3)
```

    array([[0, 1, 2],
           [1, 2, 3],
           [2, 3, 4]])

## Numpy function(유니버셜함수)
> `numpy`는 컴퓨팅연산을 위한 다양한 연산함수를 제공합니다.
>> 연산함수 기본구조
ex) **`np.sum`**(연산대상, axis=연산방향)
**`dtype()`**

### 수리연산
- **`prod()`**
- **`dot()`**
- **`sum()`**
- **`cumprod()`**
- **`cumsum()`**
- **`abs()`**
- **`sqaure()`**
- **`sqrt()`**
- **`exp()`**
- **`log()`**

### 통계연산
- **`mean()`**
- **`std()`**
- **`var()`**
- **`max()`**
- **`min()`**
- **`argmax()`**
- **`argmin()`**

### 로직연산
- **`arange()`**
- **`isnan()`**
- **`isinf()`**
- **`unique()`**

### 기하
- **`shape()`**
- **`reshape()`**
- **`ndim()`**
- **`transpose()`**

각종 연산 함수 참고: https://numpy.org/doc/stable/reference/routines.math.html

### numpy 함수 실습

```python
# 함수 예제를 위한 데이터셋
test_list = [1, 2, 3, 4]
test_list2 = [[1, 3], [5, 7]]
test_flist = [1, 3.14, -4.5]
test_list_2nd = [[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]]
test_list_3rd = [[[1, 2, 3, 4],
              [5, 6, 7, 8]],

              [[1, 2, 3, 4],
               [5, 6, 7, 8]],

              [[1, 2, 3, 4],
               [5, 6, 7, 8]]]
test_exp = [0, 1, 10]
test_nan = [0, np.nan, np.inf]
```

```python
# 곱연산
np.prod(test_list)
```

    24

```python
# 합연산
np.sum(test_list)
```

    10

```python
# 누적곱연산 벡터형태
np.cumprod(test_list)
```

    array([ 1,  2,  6, 24])

```python
# 누적합연산
# 매일/매월 매출에 대한 계산을 할 때 자주 사용
np.cumsum(test_list)
```

    array([ 1,  3,  6, 10])

```python
# 절대값
np.abs(test_flist)
```

    array([1.  , 3.14, 4.5 ])

```python
# 제곱
np.sqrt(test_list)
```

    array([1.        , 1.41421356, 1.73205081, 2.        ])

```python
# 루트

```

```python
# exp

```

```python
# 로그

```

### 통계값

```python
# 평균
np.mean(test_list)
```

    2.5

```python
# 표준편차
np.std(test_list)
```

    1.118033988749895

```python
# 분산
np.var(test_list)
```

    1.25

```python
# 최대값

```

```python
# 최소값

```

```python
test_list_2nd
```

    [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

```python
# 최대값이 존재하고 있는 인덱스 넘버를 리턴
# 출력값이 인덱스
np.argmax(test_list_2nd)
```

    8

```python
# 최소값 인덱스
np.argmin(test_list_2nd)
```

    0

```python
# 범위설정
# range() 함수와 동일하게 작동함
# for i in range(0, 100, 10):
#     print(i)
#(시작포인트, 마지막포인트+1, 스텝수)
np.arange(10, 100, 10)
```

    array([10, 20, 30, 40, 50, 60, 70, 80, 90])

```python
# 범위설정2
#(시작포인트, 마지막포인트+1, 데이터수)
np.linspace(0, 10, 50)  # 0~10까지 50개 동일간격으로
```

    array([ 0.        ,  0.20408163,  0.40816327,  0.6122449 ,  0.81632653,
            1.02040816,  1.2244898 ,  1.42857143,  1.63265306,  1.83673469,
            2.04081633,  2.24489796,  2.44897959,  2.65306122,  2.85714286,
            3.06122449,  3.26530612,  3.46938776,  3.67346939,  3.87755102,
            4.08163265,  4.28571429,  4.48979592,  4.69387755,  4.89795918,
            5.10204082,  5.30612245,  5.51020408,  5.71428571,  5.91836735,
            6.12244898,  6.32653061,  6.53061224,  6.73469388,  6.93877551,
            7.14285714,  7.34693878,  7.55102041,  7.75510204,  7.95918367,
            8.16326531,  8.36734694,  8.57142857,  8.7755102 ,  8.97959184,
            9.18367347,  9.3877551 ,  9.59183673,  9.79591837, 10.        ])

```python
test_nan
```

    [0, nan, inf]

```python
# 결측 확인(nan 확인)
np.isnan(test_nan)
```

    array([False,  True, False])

```python
# 발산 확인(무한대 확인)
np.isinf(test_nan)
```

    array([False, False,  True])

```python
test_list_3rd
```

    [[[1, 2, 3, 4], [5, 6, 7, 8]],
     [[1, 2, 3, 4], [5, 6, 7, 8]],
     [[1, 2, 3, 4], [5, 6, 7, 8]]]

```python
# 고유값 확인
np.unique(test_list_3rd)
len(np.unique(test_list_3rd))
```

    8

```python
# 데이터 구조(모양)확인(차원 확인)
# 안쪽(열)shape부터 확인하자
np.shape(test_list_3rd)
```

    (3, 2, 4)

```python
# 데이터 shape 변경
# 어떤 조건에서 reshape가능한가? 데이터 내부에 존재하는 속성 갯수가 같아야 함.
np.reshape(test_list_3rd, (4,6))
np.reshape(test_list_3rd, (2,2,6))
```

    array([[[1, 2, 3, 4, 5, 6],
            [7, 8, 1, 2, 3, 4]],

           [[5, 6, 7, 8, 1, 2],
            [3, 4, 5, 6, 7, 8]]])

```python
test_list_3rd
```

    [[[1, 2, 3, 4], [5, 6, 7, 8]],
     [[1, 2, 3, 4], [5, 6, 7, 8]],
     [[1, 2, 3, 4], [5, 6, 7, 8]]]

```python
# 데이터 차원확인
np.ndim(test_list_3rd)
# 기하학적 데이터의 차원수를 이야기하고 데이터분식시 열기준으로 차원을 이야기한다
```

    3

```python
test_list_2nd
```

    [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

```python
# 전치행렬
np.transpose(test_list_2nd)
```

    array([[1, 4, 7],
           [2, 5, 8],
           [3, 6, 9]])

```python
test_list
```

    [1, 2, 3, 4]

```python

```

## Numpy array (배열, 행렬)
> - numpy 연산의 기본이 되는 데이터 구조입니다.
- 리스트보다 간편하게 만들 수 있으며 **연산이 빠른** 장점이 있습니다.
- **브로드캐스팅 연산을 지원**합니다.
- 단, **같은 type**의 데이터만 저장 가능합니다.
- array 또한 numpy의 기본 함수로서 생성 가능합니다.
>> array 함수 호출 기본구조
ex) **`np.array(배열변환 대상 데이터)`**
ex) **`np.arange(start, end, step_forward)`**

### numpy array 실습

```python
# 기존 데이터 구조를 array로 변환
test_array = np.array(test_list)
test_array2 = np.array(test_list2)
test_farray = np.array(test_flist)
test_array_2nd = np.array(test_list_2nd)
test_array_3rd = np.array(test_list_3rd)
```

```python
# array 생성 확인
test_array
```

    array([1, 2, 3, 4])

```python
array_list = [1,2,4.5]
```

```python
# 같은 타입의 데이터만 들어가는지 확인
array_test = np.array(array_list)
```

```python
array_test   # 정수, 실수, 문자열 순으로 전체 타입이 설정됨
```

    array([1. , 2. , 4.5])

```python
# 2차원 배열 확인
test_list_2nd
# 2차원 배열을 좀 더 편하게 보여줌
test_array_2nd
```

    array([[1, 2, 3],
           [4, 5, 6],
           [7, 8, 9]])

```python
# 3차원 배열 확인
test_list_3rd
test_array_3rd
```

    array([[[1, 2, 3, 4],
            [5, 6, 7, 8]],

           [[1, 2, 3, 4],
            [5, 6, 7, 8]],

           [[1, 2, 3, 4],
            [5, 6, 7, 8]]])

```python
# np.arange 함수로 생성
np.arange(25).reshape(5,5)
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19],
           [20, 21, 22, 23, 24]])

```python
np.arange(1,25).reshape(2,12)
```

    array([[ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12],
           [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]])

### 특수한 형태의 array를 함수로 생성
함수 호출의 기본구조
> ex) **`np.ones([자료구조 shape])`**
>> 자료구조 shape은 정수, **[ ]**리스트, **( )** 튜플 로만 입력가능합니다.
- ones()
- zeros()
- empty()
- eye()

```python
# 1로 초기화한 array 생성
np.ones([3,3])
```

    array([[1., 1., 1.],
           [1., 1., 1.],
           [1., 1., 1.]])

```python
# 0으로 초기화
np.zeros((5,5))
```

    array([[0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.],
           [0., 0., 0., 0., 0.]])

```python
# 빈 값으로 초기화
np.empty((2,2))
```

    array([[0., 0.],
           [0., 0.]])

```python
# 항등행렬 초기화
# A X 항등행렬 = A
# shape가 안맞는 경우 연산이 가능하도록 만들때 사용
np.eye(4,4)

# 예시
# 날짜 매출
# 날짜 매출
# 날짜 매출
# 날짜 매출
# 날짜 매출

# 날짜 날짜 날짜 날짜 날짜
# 매출 없고 없고 없고 없고
# 없고 매출 없고 없고 없고
# 없고 없고 매출 없고 없고
```

    array([[1., 0., 0., 0.],
           [0., 1., 0., 0.],
           [0., 0., 1., 0.],
           [0., 0., 0., 1.]])

```python
np.arange(16).reshape(4,4) @ np.eye(4,4)
```

    array([[ 0.,  1.,  2.,  3.],
           [ 4.,  5.,  6.,  7.],
           [ 8.,  9., 10., 11.],
           [12., 13., 14., 15.]])

```python
test_array @ np.transpose(test_array)
```

    array([1, 2, 3, 4])

### array 속성 및 내장함수
`np.array` 에는 유용한 수리, 통계 연산을 위한 함수가 갖추어져 있습니다. 다차원 기하학적 연산을 위한 함수도 함께 살펴보겠습니다.

> array 내장 속성 호출 기본구조
ex) **`test_array.ndim`**
자주 사용하는 속성 `shape`, `dtype`, `ndim`

> array 내장함수 호출 기본구조
ex) **`test_array.prod()`**

위에 학습한 np.sum() 과는 달리 array 변수의 인자를 받아 그대로 사용합니다.

#### array 속성

```python
test_array_3rd
```

    array([[[1, 2, 3, 4],
            [5, 6, 7, 8]],

           [[1, 2, 3, 4],
            [5, 6, 7, 8]],

           [[1, 2, 3, 4],
            [5, 6, 7, 8]]])

```python
# 데이터 타입확인
test_array_3rd.dtype
```

    dtype('int64')

```python
# 데이터 타입 확인
test_array_3rd.shape
```

    (3, 2, 4)

```python
# 데이터 차원 확인
test_array_3rd.ndim
```

    3

```python
# 전치행렬
np.transpose(test_array_3rd)
test_array_3rd.T
```

    array([[[1, 1, 1],
            [5, 5, 5]],

           [[2, 2, 2],
            [6, 6, 6]],

           [[3, 3, 3],
            [7, 7, 7]],

           [[4, 4, 4],
            [8, 8, 8]]])

#### array 내장함수

```python
# 내장함수 호출
test_array.mean()
np.sqrt(test_array)
```

    array([1.        , 1.41421356, 1.73205081, 2.        ])

```python
# numpy 함수와 키워드가 같습니다.
test_array.prod()
test_array.sum(axis=)
...
...
```

### array 연산
컴퓨팅 연산을 위한 패키지인 만큼 편리한 배열 연산 기능을 지원합니다. 여러 array 연산을 통해 다른 자료구조와의 연산 차이를 알아봅시다.

```python
test_list = [1,2,3,4,5]
test_list2 = [x*2 for x in test_list]
test_list2
```

    [2, 4, 6, 8, 10]

```python
# array 덧셈, 뺄셈, 곱셈, 나눗셈
test_array = np.array(test_list)
test_array > 1
```

    array([False,  True,  True,  True,  True])

```python
# 실제 연산속도 차이를 확인하기 위한 큰 데이터 생성
big_list = [x for x in range(400000)]
big_array = np.array(big_list)
len(big_list), len(big_array)
```

    (400000, 400000)

```python
# for로 각 리스트에 1씩 값을 더함
big_list2 = [x+1 for x in big_list]  # 일반적인 for문보다 빠름(파이썬은 줄단위로 값을 확인)

# for index, item in enumerate(big_list):
#     big_list[index] = item + 1
```

    UsageError: Line magic function `%%time` not found.

```python
# array성질을 이용
big_array + 1
```

    array([     1,      2,      3, ..., 399998, 399999, 400000])

```python
# 행렬내적
first_array = np.arange(15).reshape(5, 3)
second_array = np.arange(15).reshape(3, 5)
```

```python
first_array.shape, second_array.shape
```

    ((5, 3), (3, 5))

```python
# 행렬내적 연산
first_array @ second_array
```

    array([[ 25,  28,  31,  34,  37],
           [ 70,  82,  94, 106, 118],
           [115, 136, 157, 178, 199],
           [160, 190, 220, 250, 280],
           [205, 244, 283, 322, 361]])

### 벡터 가중합
벡터의 내적은 가중합을 계산할 때 쓰일 수 있다. **가중합(weighted sum)**이란 복수의 데이터를 단순히 합하는 것이 아니라 각각의 수에 어떤 가중치 값을 곱한 후 이 곱셈 결과들을 다시 합한 것을 말한다.
만약 데이터 벡터가 $x=[x_1, \cdots, x_N]^T$이고 가중치 벡터가 $w=[w_1, \cdots, w_N]^T$이면 데이터 벡터의 가중합은 다음과 같다.

$$
\begin{align}
w_1 x_1 + \cdots + w_N x_N = \sum_{i=1}^N w_i x_i
\end{align}
$$

이 값을 벡터 $x$와 $w$의 곱으로 나타내면 $w^Tx$ 또는 $x^Tw$ 라는 간단한 수식으로 표시할 수 있다.

쇼핑을 할 때 각 물건의 가격은 데이터 벡터, 각 물건의 수량은 가중치로 생각하여 내적을 구하면 총금액을 계산할 수 있다.

```python
# 벡터의 가중합 연습문제
# 삼성전자, 셀트리온, 카카오로 포트폴리오를 구성하려한다.
# 각 종목의 가격은 80,000원, 270,000원, 160,000원이다.
# 삼성전자 100주, 셀트리온 30주, 카카오 50주로 구성하기 위한 매수금액을 구하시오
price = np.array([80000,270000,160000])
item = np.array([100,30,50])
price @ item.T
# shape가 맞지 않는데도 결과가 나옴 => @ 연산을하면 item.T로 작동함
price @ item
```

    24100000

### array 인덱싱, 슬라이싱(매우중요)
> 기본적으로 자료구조란 데이터의 묶음, 그 묶음을 관리 할 수 있는 바구니를 이야기 합니다.
데이터 분석을 위해 자료구조를 사용하지만 자료구조안 내용에 접근을 해야 할 경우도 있습니다.
>> **인덱싱**이란?
데이터 바구니 안에 든 내용 하나에 접근하는 명령, 인덱스는 내용의 순번
>> **슬라이싱**이란?
데이터 바구니 안에 든 내용 여러가지에 접근 하는 명령

기본적으로 인덱싱과 슬라이싱의 색인은 리스트와 동일합니다.

#### 인덱싱, 슬라이싱 실습

```python
# 10부터 19까지 범위를 가지는 array생성
test = np.arange(10,20)
```

```python
# 0부터 3번 인덱스까지
test[:3]
```

    array([10, 11, 12])

```python
# 4번 인덱스부터 마지막 인덱스까지
test[4:]
```

    array([14, 15, 16, 17, 18, 19])

```python
# 마지막 인덱스부터 뒤에서 3번째 인덱스까지
test[-3:]
```

    array([17, 18, 19])

```python
# 0부터 3씩 증가하는 인덱스
test[::3]
```

    array([10, 13, 16, 19])

#### 여러가지 인덱싱 및 슬라이싱 방법을 시도해봅시다

```python
index_test2 = np.array(range(25)).reshape([5, 5])
index_test2
```

    array([[ 0,  1,  2,  3,  4],
           [ 5,  6,  7,  8,  9],
           [10, 11, 12, 13, 14],
           [15, 16, 17, 18, 19],
           [20, 21, 22, 23, 24]])

```python
index_test2[2:,1:4]
```

    array([[11, 12, 13],
           [16, 17, 18],
           [21, 22, 23]])

```python
index_test2[:2,2:]
```

    array([[2, 3, 4],
           [7, 8, 9]])

```python
index_test3 = np.arange(40).reshape(2, 5, 4)
index_test3
```

    array([[[ 0,  1,  2,  3],
            [ 4,  5,  6,  7],
            [ 8,  9, 10, 11],
            [12, 13, 14, 15],
            [16, 17, 18, 19]],

           [[20, 21, 22, 23],
            [24, 25, 26, 27],
            [28, 29, 30, 31],
            [32, 33, 34, 35],
            [36, 37, 38, 39]]])

```python
index_test3[0,3:5,1:3]
```

    array([[13, 14],
           [17, 18]])

```python
index_test3[0:,0:,1]
```

    array([[ 1,  5,  9, 13, 17],
           [21, 25, 29, 33, 37]])

## 팬시인덱싱
numpy에서 벡터연산을 통해 bool 형태의 벡터를 기준으로 데이터를 선별하는 방법

```python
pet = np.array(['개','고양이','고양이','햄스터','개','햄스터'])
num = np.array([1,2,3,4,5,6])
indexing_test = np.random.randn(6,5)  # randn : 난수로 행렬생성
```

```python
indexing_test
```

    array([[ 1.21892182, -0.57278769, -2.01837856,  0.37751666, -2.14632145],
           [ 1.72627741,  0.19978955, -0.45559384,  2.3741275 ,  0.61280794],
           [ 0.38249301, -0.10195438,  1.1064851 ,  0.14316647, -0.01103692],
           [ 0.89879404, -0.08147537, -0.02147208,  0.77555509, -0.45326144],
           [ 0.12338784, -0.00431352, -1.31633818,  0.85516829,  0.29007829],
           [-0.17169369,  0.42409219,  1.69908292,  1.43223254, -1.29304307]])

```python
pet == '개'
```

    array([ True, False, False, False,  True, False])

```python
indexing_test[pet=='개']
```

    array([[ 1.21892182, -0.57278769, -2.01837856,  0.37751666, -2.14632145],
           [ 0.12338784, -0.00431352, -1.31633818,  0.85516829,  0.29007829]])

```python
indexing_test[~(pet=='개')]
```

    array([[ 1.72627741,  0.19978955, -0.45559384,  2.3741275 ,  0.61280794],
           [ 0.38249301, -0.10195438,  1.1064851 ,  0.14316647, -0.01103692],
           [ 0.89879404, -0.08147537, -0.02147208,  0.77555509, -0.45326144],
           [-0.17169369,  0.42409219,  1.69908292,  1.43223254, -1.29304307]])

```python
(pet=='개') | (pet=='햄스터')
```

    array([ True, False, False,  True,  True,  True])

```python
indexing_test[(pet=='개') | (pet=='햄스터')]
```

    array([[ 1.21892182, -0.57278769, -2.01837856,  0.37751666, -2.14632145],
           [ 0.89879404, -0.08147537, -0.02147208,  0.77555509, -0.45326144],
           [ 0.12338784, -0.00431352, -1.31633818,  0.85516829,  0.29007829],
           [-0.17169369,  0.42409219,  1.69908292,  1.43223254, -1.29304307]])

```python
num > 3
```

    array([False, False, False,  True,  True,  True])

```python
indexing_test[num>3]
```

    array([[ 0.89879404, -0.08147537, -0.02147208,  0.77555509, -0.45326144],
           [ 0.12338784, -0.00431352, -1.31633818,  0.85516829,  0.29007829],
           [-0.17169369,  0.42409219,  1.69908292,  1.43223254, -1.29304307]])

```python
# bool 형태는 1,0으로 계산이 된다
(pet =='개').sum()
```

    2

```python
# true가 1개라도 있으면 true
(pet =='개').any()
```

    True

```python
(pet =='개').all()
```

    False


```python
from scipy import stats as ss

import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

plt.ion()
```

# 1 - Moedas Justa com a Binomial

Vamos explorar a probabilidade de uma moeda ser justa usando estatística e amostragem (conceitos não exclusivos).
Lembrando, temos um espaço amostral:

\begin{align}
\mathcal{S} &= \{h, t\} \\
P(h) &= 0.5 \\
P(t) &= 0.5
\end{align}

```python
p = 0.5 # probabilidade de heads/tails
n = 30  # temos 33 jogadas
```

```python
x = np.arange(0, 31)
x
```

    array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
           17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30])

```python
p = 0.5 # probabilidade de heads/tails
n = 30  # temos 33 jogadas
x = np.arange(0, 31)
prob_binom = ss.distributions.binom.pmf(x, n, p)
plt.step(x, prob_binom, 'r-')
plt.xlabel('Num Caras - x')
plt.ylabel('P(sair x caras)')
```

```python
ss.distributions.binom.pmf(22, n, p) + \
ss.distributions.binom.pmf(23, n, p) + \
ss.distributions.binom.pmf(24, n, p) + \
ss.distributions.binom.pmf(25, n, p) + \
ss.distributions.binom.pmf(26, n, p) + \
ss.distributions.binom.pmf(27, n, p) + \
ss.distributions.binom.pmf(28, n, p) + \
ss.distributions.binom.pmf(29, n, p) + \
ss.distributions.binom.pmf(30, n, p)
```

    0.0080624008551239308

```python
x_extreme = np.arange(22, 31)
x_extreme
```

    array([22, 23, 24, 25, 26, 27, 28, 29, 30])

```python
ss.distributions.binom.pmf(x_extreme, n, p).sum()
```

    0.0080624008551239291

# 2 - Moedas Justas Simulando

Vamos simular sem se preocupar com uma binomial. Só jogar uma moeda para cima várias vezes.

```python
# Jogando uma única moeda
np.random.randint(2)
```

    0

```python
# Jogando 30 moedas
np.random.randint(2, size=30)
```

    array([0, 0, 1, 0, 1, 0, 0, 1, 1, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 0, 0, 1,
           1, 0, 1, 1, 0, 1, 1])

```python
NUM_SIMULACOES = 100000
resultados = 0
n = 30
for i in range(NUM_SIMULACOES):
    jogadas = np.random.randint(2, size=n) # joga 30 moedas para cima
    n_caras = (jogadas == 1).sum()         # conta quantas foram == 1, caras
    if n_caras >= 22:
        resultados += 1                    # quantas vezes vi >= 22 caras
print(resultados / NUM_SIMULACOES)
```

    0.00824

## 3 Caso onde Batman está certo

```python
p = 0.9 # probabilidade de heads/tails
n = 30  # temos 33 jogadas
x = np.arange(0, 31)
prob_binom = ss.distributions.binom.pmf(x, n, p)
plt.step(x, prob_binom, 'r-')
plt.xlabel('Num Caras - x')
plt.ylabel('P(sair x caras)')
```

```python
NUM_SIMULACOES = 100000
resultados = 0
n = 30
for i in range(NUM_SIMULACOES):
    jogadas = np.random.rand(30) < 0.9
    n_caras = (jogadas == 1).sum()
    if n_caras >= 22:
        resultados += 1
print(resultados / NUM_SIMULACOES)
```

    0.99784

## 4 Tem vezes que precisamos testar a coroa também

```python
p = 0.2 # probabilidade de heads/tails
n = 30  # temos 33 jogadas
x = np.arange(0, 31)
prob_binom = ss.distributions.binom.pmf(x, n, p)
plt.step(x, prob_binom, 'r-')
plt.xlabel('Num Caras - x')
plt.ylabel('P(sair x caras)')
```

```python

```


# Orbit Perturbations

$$
\vec{\ddot{r}} = -\frac{\mu}{r^3} \vec{r} + \vec{a}_p \;,
$$

where $\vec{a}_p$ is the acceleration due to perturbations from thrust,
atmospheric drag, gravitational perturbations, or any other force.

## Atmospheric drag

## Gravitational perturbations

The rotationally symmetric perturbation $\Phi (r, \phi)$ is

$$
\Phi (r, \phi) = \frac{\mu}{r} \sum_{k=2}^{\infty} J_k \left( \frac{R}{r} \right)^k P_k (\cos \phi) \;,
$$

where $J_k$ are the zonal harmonics of the planet, $R$ is the equatoriall radius ($R/r < 1$), and $P_k$ are Legendre polynomials of order $k$.

For the Earth, the six zonal harmonics are:

$$
\begin{align}
J_2 &= 0.00108263 \\
J_3 &= -2.33936 \times 10^{-3} J_2 \\
J_4 &= -1.49601 \times 10^{-3} J_2 \\
J_5 &= -0.20995 \times 10^{-3} J_2 \\
J_6 &= 0.49941 \times 10^{-3} J_2 \\
J_7 &= 0.32547 \times 10^{-3} J_2
\end{align}
$$

The perturbing acceleration is $\vec{a}_P = - \nabla \Phi$.

The gravitational perturbation due to $J_2$ is

$$
\vec{a}_P = \frac{3}{2} \frac{J_2 \mu R^2}{r^4} \left[ \frac{x}{r} \left( 5 \frac{z^2}{r^2} - 1 \right) \hat{I} + \frac{y}{r} \left( 5 \frac{z^2}{r^2} - 1 \right) \hat{J} + \frac{z}{r} \left( 5 \frac{z^2}{r^2} - 3 \right) \hat{K} \right]
$$

The gravitational perturbation due to $J_3$ is

$$
\vec{a}_P = \frac{1}{2} \frac{J_3 \mu R^3}{r^5} \left[ 5 \frac{x}{r} \left( 7 \frac{z^3}{r^3} - 3 \frac{z}{r} \right) \hat{I} + 5 \frac{y}{r} \left( 7 \frac{z^3}{r^3} - 3 \frac{z}{r} \right) \hat{J} + \left( 35 \frac{z^4}{r^4} - 30 \frac{z^2}{r^2} + 3 \right) \hat{K} \right]
$$

Then, solving the equation of motion yields the perturbed orbit:

$$
\vec{\ddot{r}} = -\frac{\mu}{r^3} \vec{r} + \vec{a}_P
$$


Consider a curve given by the function $f(x)$ on the interval $[0,1]$ with the property $f(0)=f(1)=0$ and $f(x) > 0$
everywhere else. The length of the curve is $l \le 0$. What curve will have the maximum area under the function?

We want to maximize the area functional:
$$
S[f] = \int_0^1 f(x) d x
$$
subject to the constraint
$$
A[f] = \int_0^1 \sqrt{1+f'(x)^2} dx = l
$$
So we must extremize the modified functional:
$$
\tilde S[f, \lambda] = \int_0^1 \left(f(x)+\lambda \sqrt{1+f'(x)^2}\right) d x
$$
The Euler-Lagrange equations
$$
{\delta \tilde S\over\delta f(x)} - {d\over d x} {\delta \tilde S\over\delta f'(x)} = 0
$$
become
$$
1-{d\over d x}\left(\lambda {f'(x)\over \sqrt{1+f'(x)^2}}\right) = 0
$$
Integrating:
$$
x-\lambda {f'(x)\over \sqrt{1+f'(x)^2}} = c_1
$$

```
import sympy.interactive
sympy.interactive.init_printing()
```

Let's assume $c_1 = 0$ for now...

```
from sympy import var, Symbol, Function, sqrt, solve, integrate, simplify, refine, Q, python, Derivative, Abs, Id, powsimp
var("x c1")
lam = Symbol("lambda")
f = Function("f")
L = f(x) + lam*sqrt(1 + f(x).diff(x)**2)
eq0 = L.diff(f(x)) - Derivative(L.diff(f(x).diff(x)), x)
eq0
```

```
# integrate can't handle it directly (it will work in the next release)
# eq = integrate(eq0, x)
eq = x - lam * f(x).diff(x) / sqrt(1+f(x).diff(x)**2) #- c1
eq
```

```
sqrt(f(x)).diff(f(x))
```

```
dfx = solve(eq, f(x).diff(x))[1]
dfx
```

```
fx = integrate(dfx, x)
fx
```

```
fx = simplify(fx)
fx
```

```
fx = refine(fx, Q.positive(x))
fx
```

```
refine(fx, Q.positive(lam**2-x**2))   # Unfortunately does not work...
```

$$\frac{- \lambda^{2} + x^{2}}{\sqrt{\lvert{\lambda^{2} - x^{2}}\rvert}}$$

```
-refine(simplify(-fx), Q.positive(lam**2-x**2))
```

$$- \frac{\lambda^{2} - x^{2}}{\sqrt{\lvert{\lambda^{2} - x^{2}}\rvert}}$$

So the solution is $f(x) = -\sqrt{\lambda^2 - x^2}$. The minus sign should not be there, it probably got in at some point...

```

```

tutorial_exercises/Variation with Lagrange multipliers.ipynb

sympy/Variation with Lagrange multipliers.ipynb


```python
from sympy import *
init_printing()
import numpy as np
from collections import OrderedDict
import matplotlib.pyplot as plt
%matplotlib inline
```

## Generic fit for approximating with a polynomial

```python
f = Symbol('f')  # Function to approximate
f_approx = Symbol('fbar')   # Approximating function
w = Symbol('w') # weighting function
chi2 = Symbol('chi^2')

f, f_approx, w, chi2
```

```python
M = Symbol('M', integer=True)
k = Symbol('k', integer=True,positive=True)
a = IndexedBase('a',(M,))  # coefficient
h = IndexedBase('h',(M,))  # basis function
ak = Symbol('a_k')  # Temporary symbols to make some derivatives easier
hk = Symbol('h_k')  #    Basis function (function of r)
hj = Symbol('h_j')
r = Symbol('r',positive=True)
j = Symbol('j',integer=True)
poly_approx = Sum(a[k]*h[k],(k,0,M))
poly_approx_j = Sum(a[j]*h[j],(j,0,M)) # replace summation variable
poly_approx
```

Define a quality-of-fit measure, $\chi^2$

```python
eq1 = Eq(chi2, Integral(w(r)*(f(r)-f_approx(r,ak))**2,r))
eq1
```

We want to minimized $\chi^2$. Take derivative wrt the coefficients ($a_k$) and set to zero.  Will get $k$ equations.

```python
eq2 = Eq(0,diff(eq1.rhs, ak))
eq2
```

```python
eq3 = Eq(diff(poly_approx,ak,evaluate=False), hk)
eq3
```

```python
eq4 = Eq(0, Integral(eq2.rhs.args[0].subs(diff(f_approx(r,ak),ak),hk(r)),r))
eq4
```

```python
eq5 = Eq(0, Integral(eq4.rhs.args[0].subs(f_approx(r,ak), poly_approx_j),r))
eq5
```

```python
eq6 = Eq(0, Integral(-eq5.rhs.args[0]/2,r))
eq6
```

```python
base7 = expand(eq6.rhs.args[0])
eq7 = Eq(Integral(-base7.args[1],r),Integral(base7.args[0],r))
eq7
```

```python
int7 = eq7.lhs.args[0]
eq8 = Eq(Sum(a[j]*Integral(Mul(*int7.args[1:]),r),(j,0,M)), eq7.rhs)
eq8
```

Finally, we get a linear system - the lhs is a matrix times a vector of coefficients and the rhs is a vector of values.

```python

```


(c) Juan Gomez 2019. Thanks to Universidad EAFIT for support. This material is part of the course Introduction to Finite Element Analysis

# Two-dimensional truss elements

The simple code implemented for the springs elements in a previous notebook can be easily adapted to consider an assemblage of two-dimensional truss elements. In fact the code can be converted into a general program to consider a variety of (finite) elements just by adding subroutines with the force-displacement (stiffness) equations for each element.

In this particular problem a typical element of axial stiffness $\frac{AE}{\mathcal l}$ is shown in its local reference system in the figure below:

Note that since the element has only axial stiffness the force-displacement relationship in its local reference system is of the form:

$$
\begin{Bmatrix}f_1\\f_2\end{Bmatrix}=\frac{AE}{\mathcal l}\begin{bmatrix}1&-1\\-1&1\end{bmatrix}\begin{Bmatrix}u_1\\u_2\end{Bmatrix}
$$

which is also analogous to the one in the spring element.

### Assemblage of truss elements.

Cosnsider the two-elements assemblage shown below.

To obtain the total stiffness from the structure it is now necessary to consider the stiffness contribution from all the elements in a common (**Global**) reference system.

Let:

* $U, F$ : Displacements degrees of freedom and forces in the global reference system.
* $u, f$ : Displacements degrees of freedom and forces in the local reference system.

and related by the rotational transformation matrix $\lambda$ like;

$$u=\lambda U.$$

Using the fact that the virtual energy of the forces upon imposition of virtual displacements is a reference-independent scalar quantity gives:

$$\delta U^TF=\delta u^Tf.$$

Using the first equation in the second yields;

$$\delta U^TF=\delta U^T\lambda^Tf$$

from which:

$$F=\lambda^Tf.$$

Now, conisdering the equilibrium relation for the element in the local system:

$$f=ku$$

where $k$ is the local stiffness matrix we can write:

$$\begin{array}{l}\lambda^Tf=\lambda^Tku\\\lambda^Tf=\lambda^Tk\lambda U\\F=KU\end{array}$$

from which:

$$K=\lambda^Tk\lambda$$

where $K$ is the stiffness matrix for the two-dimensional truss element in the global reference system. It must be observed that in the global reference system the element has two degrees of freedom per node.

In the actual implementation all the information required to compute $K$ is passed as input paramters to the elemental subroutine **uel()** as described below.

**Question: Find the rotational transformation matrix $\lambda$ required for the formulation of the stiffness matrix in the global reference system.**

### Example

The modifications that must be applied to the spring-elements based code are only related to the fact that now there are 2 degrees of freedom are each nodal point.

(The required input files containing the input data for the nodes, elements, materials parameters and loads are in the folder **files** of this REPO.)

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym
```

Read the input files from the **files** folder.

```python
def readin():
    nodes    = np.loadtxt('files/' + 'Cnodes.txt', ndmin=2)
    mats     = np.loadtxt('files/' + 'Cmater.txt', ndmin=2)
    elements = np.loadtxt('files/' + 'Celes.txt' , ndmin=2)
    loads    = np.loadtxt('files/' + 'Cloads.txt', ndmin=2)

    return nodes, mats, elements, loads
```

**eqcounter** counts equations and generates the boundary conditions array in its second instance.

```python
def eqcounter(nodes):
    nnodes = nodes.shape[0]
    IBC = np.zeros([nnodes, 2], dtype=np.integer)
    for i in range(nnodes):
        for k in range(2):
            IBC[i , k] = int(nodes[i , k+3])
    neq = 0
    for i in range(nnodes):
        for j in range(2):
            if IBC[i, j] == 0:
                IBC[i, j] = neq
                neq = neq + 1

    return neq, IBC
```

**DME** computes the assembly operator.

```python
def DME(nodes, elements):

    nels = elements.shape[0]
    IELCON = np.zeros([nels, 2], dtype=np.integer)
    DME    = np.zeros([nels, 4], dtype=np.integer)
    neq , IBC = eqcounter(nodes)
    ndof   = 4
    nnodes = 2
    for i in range(nels):
        for j in range(nnodes):
            IELCON[i, j] = elements[i, j+3]
            kk = IELCON[i, j]
            for l in range(2):
                DME[i, 2*j+l] = IBC[kk, l]
    return DME , IBC , neq
```

**assembly** uses the model and the **DME** operator to compute the global stiffness matrix.

```python
def assembly(elements, mats, nodes, neq, DME, uel=None):

    IELCON = np.zeros([2], dtype=np.integer)
    KG = np.zeros((neq, neq))
    nels = elements.shape[0]
    nnodes = 2
    ndof = 4
    for el in range(nels):
        elcoor = np.zeros([nnodes, 2])
        im     = np.int(elements[el , 2])
        par0= mats[im ,0]
        par1= mats[im , 1]
        for j in range(nnodes):
            IELCON[j] = elements[el , j+3]
            elcoor[j , 0] = nodes[IELCON[j], 1]
            elcoor[j , 1] = nodes[IELCON[j], 2]
        kloc = ueltruss2D(elcoor, par0, par1)
        dme = DME[el , :ndof]
        for row in range(ndof):
            glob_row = dme[row]
            if glob_row != -1:
                for col in range(ndof):
                    glob_col = dme[col]
                    if glob_col != -1:
                        KG[glob_row, glob_col] = KG[glob_row, glob_col] +\
                                                 kloc[row, col]

    return KG
```

**ueltruss2D** uses the nodal point coordinates and the material parameters to compute the local stiffness matrix transformed to the global reference system.

**Question: Add comments to explains the different steps in the following subroutine. In particular identify the computation of the rotational transformation matrix $\lambda$.**

```python
def ueltruss2D(coord, A, Emod):
    vec = coord[1, :] - coord[0, :]
    length = np.linalg.norm(vec)
    nx = vec[0]/length
    ny = vec[1]/length
    Q = np.array([
        [nx, ny , 0 , 0],
        [0,  0, nx , ny]])
    kl = (A*Emod/length) * np.array([
        [1, -1],
        [-1, 1]])
    kG = np.dot(np.dot(Q.T, kl), Q)
    return kG
```

**loadassem** forms the vector of nodal loads.

```python
def loadasem(loads, IBC, neq, nl):
    """Assembles the global Right Hand Side Vector RHSG

    Parameters
    ----------
    loads : ndarray
      Array with the loads imposed in the system.
    IBC : ndarray (int)
      Array that maps the nodes with number of equations.
    neq : int
      Number of equations in the system after removing the nodes
      with imposed displacements.
    nl : int
      Number of loads.

    Returns
    -------
    RHSG : ndarray
      Array with the right hand side vector.

    """
    RHSG = np.zeros([neq])
    for i in range(nl):
        il = int(loads[i, 0])
        ilx = IBC[il , 0]
        ily = IBC[il , 1]
        if ilx != -1:
            RHSG[ilx] = loads[i, 1]
        if ily != -1:
            RHSG[ily] = loads[i, 2]

    return RHSG
```

The main program still retains the same structure as follows:

* Reads the model
* Builds the DME() operator
* Assembles the global system of equations
* Solves for the global displacements $UG$

```python
nodes, mats, elements, loads = readin()
DME , IBC , neq = DME(nodes, elements)
KG   = assembly(elements, mats, nodes, neq, DME)
RHSG = loadasem(loads, IBC, neq, 1)
UG = np.linalg.solve(KG, RHSG)
print(UG)
```

    [ 0.53009278 -0.17789364]

### Proposed problems
#### Problem 1
Implement a subroutine to compute the nodal forces in each element and verify the equilibrium of the system.

#### Problem 2
Find the lateral stiffness of the structure using the relation:

$$k=\frac P\delta$$

#### Problem 3
By introducing the necessary modifications to the input files, find the resulting displacement solution after adding a third bar connecting nodes $0$ and $2$. Comment on your results.

#### Problem 4
Fix the truss structure shown in the figure by adding elements and/or imposing appropiate displacement boundary conditions. (Create a new set of input files.)

### References

* Bathe, Klaus-Jürgen. (2006) Finite element procedures. Klaus-Jurgen Bathe. Prentice Hall International.

* Juan Gómez, Nicolás Guarín-Zapata (2018). SolidsPy: 2D-Finite Element Analysis with Python, <https://github.com/AppliedMechanics-EAFIT/SolidsPy>.

```python
from IPython.core.display import HTML
def css_styling():
    styles = open('./nb_style.css', 'r').read()
    return HTML(styles)
css_styling()
```

<link href='http://fonts.googleapis.com/css?family=Fenix' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>

```python

```


# Immersed Boundary Method
---

### Author: Marin Lauber

```python
import numpy as np
import matplotlib.pyplot as plt
import NSsolver as ns
try:
    plt.style.use("jupyter")
except OSError:
    print("Delaut syle in use")
```

Charles S Peskin (1972) developed the immersed boundary method (IBM) to tackle the problem of heart valves. Dirac delta function source terms impose the kinematic boundary condition in the fluid by regularizing a force density on the background mesh

\begin{equation}
\begin{split}
    &\frac{\partial \vec{u}}{\partial t} + (\vec{u}\cdot\nabla)\vec{u} = -\nabla p + \frac{1}{Re}\nabla^2\vec{u} + \int\vec{f}(\vec{\xi}(s, t))\delta_h(\vec{\xi} - \vec{x}) \text{ d}\vec{x},\\
    &\nabla\cdot\vec{u} = 0,\\
    &\vec{u}(\vec{\xi}(s, t)) = \int_{\vec{x}}\vec{u}(\vec{x})\delta(\vec{x} - \vec{\xi} ) \text{ d}\vec{x} = \vec{u}_B(\vec{\xi}(s, t)).
\end{split}
\end{equation}

A constitutional relationship, Hooke’s law, is used to derive this force density based on the motion of the filament, convected by the fluid. Different discrete Dirac delta function can be used within the IBM (three point delta function)

\begin{equation}
\delta_h(r) = \begin{cases}
    \frac{1}{6\Delta r}\left[ 5-3\frac{|r|}{\Delta r} -\sqrt{-3\left(1-\frac{|r|}{\Delta r}\right)^2 +1}\right] & \text{for  } 0.5\Delta r \le |r| \le 1.5\Delta r\\
    \frac{1}{3\Delta r}\left[ 1+\sqrt{-3\left(\frac{r}{\Delta r}\right)^2 +1}\right] & \text{for  } |r| \le 0.5\Delta r\\
    0 & \text{else}
       \end{cases}
\end{equation}

or the 3-point kernel (used in the following)

\begin{equation}
\delta_h(r) = \begin{cases}
    \frac{4}{3} - r^2 & \text{for  } |r| \le 1.5\\
    5(2.25-3|d|+r^2)&\text{for  } |r| \le 0.5\\
    0 & \qquad \text{else}
       \end{cases}
\end{equation}

```python
def kernel(d):
    return np.where(abs(d)<=.5, .75-d**2, np.where(abs(d)<=1.5,.5*(2.25-3*abs(d)+d**2), 0))

def Ic(x, X, dx, f):
    # interpolation operator
    return np.sum(f*kernel((X - x)/dx))

def Sc(x, X, F, ds=1):
    # spreading operator
    return kernel((X - x)/dx)*ds*F

def f(x, X, u, V, dx, dt, rho):
    # interpolate to Lagrangian point
    Ur = Ic(x, X, dx, u)
    # compute Lagrangian force density
    F = rho/dt*(V - Ur)
    return Sc(x, X, F) # extrapolate to Eulerian points
```

```python
d = np.linspace(-3,3,256)
plt.plot(d, kernel(d));
```

We use a fractional-step algorithm to solve the following system of equations.

\begin{split}
    u^* &= r_{\Delta t}(u^n) +  \Delta t \mathcal{S}_c[\xi - x](\mathcal{I}_c[\xi-x](u^n)),\\
    \nabla\cdot(\nabla p) &= \nabla\cdot u^*,\\
    u^{n+1} &= u^* - \frac{\Delta t}{\rho}\nabla p.
\end{split}

where $\mathcal{S}_c$ and $\mathcal{I}_c$ are the spreading and interpolation kernels.

```python
def update(x, X, u0, V, dx, dt=1, rho=1, Nsteps=1):

    u_n = u0

    for i in range(Nsteps):

        # first step
        u_1 = u_n + dt*ns.r(u_n)
        u_star = u_1 + dt*f(x, X, u_1, V, dx, dt, rho)

        sigma = ns.div(u_star, dx)

        p = ns.solve_pressure(np.ones_like(sigma), sigma, dx, verbose=True)

        u_n = u_star - dt/rho*(ns.grad(p, dx))

    return u_star, sigma, p, u_n
```

```python
N = 32
x, dx = np.linspace(-1, 1, N, retstep=True)
xs = x + 0.5*dx # scalar cell-centred values
X = 0.0
V = 1
u0 = np.zeros_like(x)
dt = 1.

us, sigma, p, u_n = update(x, X, u0, V, dx, dt)
print("Interface at X: %.2f" % X)
print(r"L inf: %.3e" % np.max(np.abs(u_n -  V)))
```

    Jacobi solver:
    	res0: 3.425e-01
     	res: 3.390e-11
     	iter: 548
    Interface at X: 0.00
    L inf: 9.688e-01

```python
ns.draw_results(x, xs, X, u0, u_n, p, sigma)
```

Clearly, the correct pressure is not obtained with a single step of the fractional-step method. In fact any number of sub-iteration of this sytem fails to capture the correct pressure (and thus solution).


```python
# Models from Introduction to Algorithmic Marketing
# https://algorithmicweb.wordpress.com/
#
# Markov chain-based LTV model predicts customer lifetime value
# using the probabilities of transition between different customer states
```

```python
%matplotlib inline
import sympy as sy
import numpy as np
import matplotlib.pyplot as plt
```

```python
p1 = 0.7                # Probability of a purchase after 1 month of silence
p2 = 0.5                # Probability of a purchase after 2 months of silence
p3 = 0.3                # Probability of a purchase after 3 months of silence
r = 100                 # Revenue associated with one purchase, $
c = 5                   # Cost of one promotion
d = 0.001               # Inflation rate for one month
projection_horizon = 10 # LTV projection horizon, months
```

```python
# Transition probabilities
P = np.matrix([
[  p1,  1-p1,   0,    0   ],
[  p2,   0,    1-p2,  0   ],
[  p3,   0,     0,   1-p3 ],
[  0,    0,     0,    0]
])

# Expected revenue
G = np.matrix([
[  r-c  ], # If customer goes to the initial state (purchases), we get the revenue minus promotion cost
[  -c   ], # If customer is silent for one month, we send promotion and lose c dollars
[  -c   ], # If customer is silent for two months, we send promotion and lose c dollars
[  0    ]  # If customer is silent for three month, we stop to send promotions
])
```

```python
# Calculate and visualize the LTV. Each line on the plot corresponds to a customer state
#
# For example, the topmost line corresponds to the first state (just purchased)
# It indicates that the cumulative revenue from this cusomer is likely to grow over
# next projection_horizon months.
#
# The bottommost line corresponds to customers who are silent for 3 months
# This customers are considred lost, so this line is constantly zero
ltv_dynamics = [
   sum(
      (1/((1+d)**(t-1))) * (P**t)*G
      for t in range(1, T+1)
   ) for T in range(1, projection_horizon)
]
plt.plot(np.concatenate(ltv_dynamics, axis=1).T);
```


# Variational Quantum Optimization (VQO) of CHSH Violation

This notebook demonstrates how quantum machine learning can be applied to train a quantum circuit to violate the Bell inequality.

* $x,y\in\{0,1\}$ and $a,b\in\{-1,1\}$.
* Local qubit measurements $\Pi_a^x$ and $\Pi_b^y$ are projective.
* State preparation $\Phi^{AB}$ is a static, bipartite quantum state.

$$
I_{CHSH} = \sum_{x,y\in\{0,1\}} (-1)^{x\wedge y}\langle A_xB_y\rangle\leq 2
$$

**Goal:** Find state preparation and measurements that maximally violate the CHSH inequality.

## Requirements

This code requires the `pennylane` and `matplotlib` python libraries.

```python
# quantum machine learning framework
import pennylane as qml

# numpy extension for pennylane
from pennylane import numpy as np

# plotting utility
import matplotlib.pyplot as plt
```

## CHSH Ansatz

We consider a simple ansatz circuit $U_{CHSH}^{AB}(\vec{\theta})$ for the CHSH scenario.

* **State Preparation:** local qubit rotations about the $y$-axis followed by a CNOT gate. Can prepare any pure state in the Bell basis or omputational basis.
* **Measurement:** The computational basis is rotated about the $y$-axis allowing any projective measurement in the $x$-$z$-plane to be realized.

$$
\begin{align}
    U^{AB}_{CHSH}(\vec{\theta})|00\rangle^{AB} &= \left(R^A_y(\theta_2)\otimes R^B_y(\theta_3)\right)U^{AB}_{CNOT}\left(R^A_y(\theta_0)\otimes R^B_y(\theta_1)\right)|00\rangle^{AB} \\
    &= \left(R^A_y(\theta_2)\otimes R^B_y(\theta_3)\right)|\psi(\theta_0,\theta_1)\rangle^{AB} \\
    &= |\psi_{CHSH}(\vec{\theta})\rangle^{AB}\\
\end{align}
$$

This ansatz parameterizes real-valued states and measurements, but is sufficient to find nonlocal behaviors.

```python
def chsh_ansatz(settings, wires):
    # state prepartion
    qml.RY(settings[0], wires=wires[0])
    qml.RY(settings[1], wires=wires[1])
    qml.CNOT(wires=wires[0:2])

    # measurement basis rotations
    qml.RY(settings[2], wires=wires[0])
    qml.RY(settings[3], wires=wires[1])
```

## Ansatz Parameterization

The CHSH scenario has a static state preparation and two inputs, $x,y \in \{0,1\}$ that select local measurement bases.

We organize our settings into preparation and measurement settings:

$$
\vec{\theta} = (\{\theta\}^P, \{\theta\}^M) = \left([\theta_0,\theta_1], \begin{bmatrix}
    \theta_2^{x=0} & \theta_2^{x=1} \\ \theta_3^{x=0} & \theta_3^{x=1}
\end{bmatrix}\right)
$$

while $\vec{\theta}_{x,y} = [\theta_0, \theta_1, \theta^x_2, \theta^y_3]$.
We now write a function create randomized initial parameters.

```python
def random_scenario_settings():
    preparation_settings = np.random.random(2)
    measurement_settings = np.random.random((2,2))

    return [preparation_settings, measurement_settings]

random_scenario_settings()
```

    [tensor([0.37895832, 0.00818022], requires_grad=True),
     tensor([[0.14476653, 0.61528113],
             [0.4289039 , 0.1094481 ]], requires_grad=True)]

## PennyLane Devices

PennyLane uses a `Device` class to model quantum hardware.
* Devices can be either a classical simulator or quantum hardware.
* Devices can run locally on your laptop or executed remotely by a third party.

For this demo, we use the default state-vector simulator.
We only need two qubits (`wires`).
This device will numerically evaluate the quantum circuit expectations rather than constructing a distribution through sampling.

```python
chsh_dev = qml.device("default.qubit", wires=[0,1])
```

## PennyLane QNodes

A quantum node (`QNode`) class describes a quantum circuit run by a particular device.
The `QNode` is a differentiable function with customizeable optimizer interfaces and gradient methods.
A qnode

The `@qml.qnode` decorator specifies that the `chsh_dev` device is used to execute the `chsh_correlator` function.

The CHSH correlator extends our ansatz with a joint observable expectation

$$
\begin{align}
    \langle A_xB_y\rangle(\vec{\theta}_{x,y}) &= \langle\psi(\theta_0,\theta_1)|A_x(\theta_2)B_y(\theta_3)|\psi(\theta_0,\theta_1)\rangle\\
        &= \sum_{i,j\in\{0,1\}}(-1)^{i\oplus j}\left|\langle ij|R^A_y(\theta_2)\otimes R^B_y(\theta_3)|\psi(\theta_0,\theta_1)\rangle^{AB}\right|^2 \\
        &= \sum_{i,j\in\{0,1\}}(-1)^{i\oplus j}\left|\langle ij|U^{AB}_{CHSH}(\vec{\theta}_{x,y})|00\rangle^{AB}\right|^2 \\
\end{align}
$$

Here $|ij\rangle$ is the computational basis. The labels $x$ and $y$ on the observables correspond to different parameter vectors $\vec{\theta}_{x,y}$

```python
@qml.qnode(chsh_dev)
def chsh_correlator(settings):
    wires = chsh_dev.wires
    chsh_ansatz(settings, wires)

    # computational basis measurement
    return qml.expval(
        qml.PauliZ(wires[0]) @ qml.PauliZ(wires[1])
    )
```

### Drawing the Ansatz Circuit

```python
chsh_correlator([0,0,0,0])
print(chsh_correlator.draw())
```

     0: ──RY(0)──╭C──RY(0)──╭┤ ⟨Z ⊗ Z⟩
     1: ──RY(0)──╰X──RY(0)──╰┤ ⟨Z ⊗ Z⟩

### Evaluating a QNode

```python
print("00 prep and ZZ meas : ", chsh_correlator([0,0,0,0]))
```

    00 prep and ZZ meas :  1.0

```python
print("01 prep and ZZ meas : ", chsh_correlator([0,np.pi,0,0]))
```

    01 prep and ZZ meas :  -1.0

```python
print("00 prep and XX meas : ", chsh_correlator([0,0,-np.pi/2,-np.pi/2]))
```

    00 prep and XX meas :  0.0

```python
print("bell prep and ZZ meas : ", chsh_correlator([np.pi/2,0,0,0]))
```

    bell prep and ZZ meas :  1.0

```python
print("bell prep and XX meas : ", chsh_correlator([np.pi/2,0,-np.pi/2,-np.pi/2]))
```

    bell prep and XX meas :  0.9999999999999998

```python
print("bell prep and ZX meas : ", chsh_correlator([np.pi/2,0,0,-np.pi/2]))
```

    bell prep and ZX meas :  2.220446049250313e-16

## The CHSH Cost Function

The CHSH inequality is expressed as:

$$
I_{CHSH}(\vec{\theta}) = \sum_{x,y\in\{0,1\}} (-1)^{x\wedge y}\langle A_xB_y\rangle(\vec{\theta}_{x,y})
$$

The corresponding cost function is expressed as:

$$
\text{Cost}(\vec{\theta}) = - I_{CHSH}(\vec{\theta})
$$

```python
def chsh_cost(scenario_settings):

    score = 0
    for x, y in [[0,0],[0,1],[1,0],[1,1]]:
        # construct the settings for inputs `x` and `y`
        qnode_settings = [
            *scenario_settings[0],
            scenario_settings[1][0,x],
            scenario_settings[1][1,y]
        ]

        # evaluate the `chsh_correlator` qnode
        score += (-1)**(x * y) * chsh_correlator(qnode_settings)

    # invert score because cost is minimized
    return -(score)
```

## Gradient Descent

Our goal is to find $\arg\min_{\vec{\theta}} \text{Cost}(\vec{\theta})$ using automatic differentiation and gradient descent.
This minimization problem is equivalent to maximizing the CHSH violation.

$$
\begin{align}
    \vec{\theta}' &= \vec{\theta} - \eta\nabla_{\vec{\theta}}\text{Cost}(\vec{\theta}) \\
    &= \vec{\theta} + \eta\nabla_{\vec{\theta}}I_{CHSH}(\vec{\theta})
\end{align}
$$

PennyLane handles the automatic differentiation of the `chsh_correlator` QNode.
We add a loop that iteratively computes the gradient and updates the settings.
Each step in this training procedure is called an epoch.

```python
def gradient_descent(cost, settings, step_size=0.1, num_steps=20):
    # optimizer work horse
    opt = qml.GradientDescentOptimizer(stepsize=step_size)

    scores = []
    settings_list = []
    for i in range(num_steps):
        # log data
        score = -(cost(settings))
        scores.append(score)
        settings_list.append(settings)

        # print progress
        if i % 5 == 0:
            print("iteration : ", i, ", score : ", score)

        # update settings using gradient evaluated at `settings`
        settings = opt.step(cost, settings)

    # log data for final score and settings
    final_score = -(cost(settings))
    scores.append(final_score)
    settings_list.append(settings)

    # find the maximum value and optimal settings
    max_score = max(scores)
    max_id = scores.index(max_score)
    opt_settings = settings_list[max_id]

    return {
        "max_score" : max_score,
        "opt_settings" : opt_settings,
        "max_id" : max_id,
        "samples" : range(num_steps + 1),
        "scores" : scores,
        "settings" : settings_list
    }
```

## VQO of a Maximally Nonlocal CHSH Protocol

```python
%%time

# initialize random parameters
init_settings = random_scenario_settings()
print("initial settings :\n", init_settings, "\n")

opt_dict = gradient_descent(chsh_cost, init_settings, step_size=0.5);

print("\nmax score : ", opt_dict["max_score"])
print("optimal settings : ", opt_dict["opt_settings"], "\n")
print("theoretical max : ", 2 * np.sqrt(2), "\n")
```

    initial settings :
     [tensor([0.96123244, 0.55235034], requires_grad=True), tensor([[0.37514297, 0.50044924],
            [0.29133242, 0.46641267]], requires_grad=True)]

    iteration :  0 , score :  1.5088222141327297
    iteration :  5 , score :  2.010436716885594
    iteration :  10 , score :  2.5808417015638088
    iteration :  15 , score :  2.8284193404598703

    max score :  2.828427124707118
    optimal settings :  [array([ 1.57079231e+00, -1.06968197e-06]), array([[ 0.71599797, -0.85479238],
           [-0.06939297,  1.50139766]])]

    theoretical max :  2.8284271247461903

    CPU times: user 436 ms, sys: 21.8 ms, total: 458 ms
    Wall time: 448 ms

```python
plt.plot(opt_dict["samples"], opt_dict["scores"], "o--", label="Optimization")
plt.plot(opt_dict["samples"], [2]*len(opt_dict["samples"]), label="Classical Bound")
plt.plot(opt_dict["samples"], [2*np.sqrt(2)]*len(opt_dict["samples"]), label="Quantum Bound")

plt.title("Optimization of CHSH Violation\n")
plt.ylabel("CHSH Score")
plt.xlabel("Epoch")
plt.legend()

plt.show()
```

```python

```


```python
from sympy import *
from sympy.abc import x,y
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
%matplotlib inline

```

## Bissection Method

$$
Let\ f:\mathbb{R}\rightarrow\mathbb{R}\ and\ a<b \in\mathbb{R} : f'(a) \times f'(b)<0\\
  \
$$

```python
def bissection(fct,a,b,err):
  print("Function :",fct)
  fct_prime = fct.diff(x)

  f_prime = lambdify(x,fct_prime)
  f = lambdify(x,fct)

  f_a = f_prime(a)
  f_b = f_prime(b)

  if f_a*f_b<0:
    print("Success")
    while (abs(b-a)>err) and (f_prime(a)*f_prime(b))<0:
      c = (a+b)/2
      # print(c)
      f_c = f_prime(c)
      if f_c <= 0 : a = c
      else: b = c
    print("2D: [x,f(x)] == ",[a,f(a)])
    return [a,f(a)]
  else: print("Failed")

def bissection_3d(fct,x1,y1,x2,y2,err):

  print("Function :",fct)
  fct_primeX = fct.diff(x)
  print("Derivée par rapport à x :\n",fct_primeX)
  fct_primeY = fct.diff(y)
  print("Derivée par rapport à Y :\n",fct_primeY)

  f = lambdify([x,y],fct)
  f_primeX = lambdify([x,y],fct_primeX)
  f_primeY = lambdify([x,y],fct_primeY)

  f_Xa = f_primeX(x1,y1)
  f_Xb = f_primeX(x2,y2)

  f_Ya = f_primeY(x1,y1)
  f_Yb = f_primeY(x2,y2)

  # print(f_prime(a))
  # print(f_prime(b))

  if f_Xa*f_Xb<0:
    print("Par rapport à x : Success")
    while (abs(x2-x1)>err) and (f_primeX(x1,y1)*f_primeX(x2,y2))<0:
      cx = (x1+x2)/2
      cy = (y1+y2)/2
      f_Xc = f_primeX(cx,cy)
      # print(f_c)
      if f_Xc <= 0 : x1 = cx
      else: x2 = cx

  else: print("Failed")

  if f_Ya*f_Yb<0:
      print("Par rapport à y : Success")
      while (abs(y2-y1)>err) and (f_primeY(x1,y1)*f_primeY(x2,y2))<0:
        cx = (x1+x2)/2
        cy = (y1+y2)/2
        f_Yc = f_primeY(cx,cy)
        if f_Yc <= 0 : y1 = cy
        else: y2 = cy
      print("3D: [(x,y),f(x,y)] == ",[(x1,y1),f(x2,y2)])
      return [(x1,y1),f(x2,y2)]

  else: print("Failed")

```

```python
expr = ( (x+2*y-7)**2 )+( (2*x+y-5)**2 )
f = lambdify([x,y],expr)
expr_prime = expr.diff(x,y)
expr_prime
```

$\displaystyle 8$

### Test functions :
$$
Booth\ Function :\ f_1 = (\ (x+2y-7)^2)+(2x+y-5)^2) \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \, (x,y)\in[-100,100]^2\\
f_2 = sin(2x) \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \  \, x\in[-2,0]
$$

```python
f1 = ( (x+2*y-7)**2 )+( (2*x+y-5)**2 )
f2 = sin(2*x)

lst1 = bissection(f2,-2,0,0.02)
lst2 = bissection_3d(f1,-100,-100,100,100,0.02)

f_lambda2 = lambdify(x,f2)
f_lambda1 = lambdify([x,y],f1)
# print(fx)
# fx(1,2)

fig1 = plt.figure(figsize=(10, 10))
x2 = np.linspace(-5, 5, 50)
ax1 = fig.gca()

# X2 = np.meshgrid(x2)
plt.plot(x2,f_lambda2(x2))
plt.grid(b=True, which='major', color='#666666', linestyle='-')
plt.minorticks_on()
plt.grid(b=True, which='minor', color='#999999', linestyle='-', alpha=0.2)
# plt.show()

fig = plt.figure(figsize=(10, 10))
x1 = np.linspace(-100, 100, 5150)
y1 = np.linspace(-100, 100, 150)
X, Y = np.meshgrid(x1, y1)
Z = f_lambda1(X, Y)
ax = fig.gca(projection='3d')
ax.view_init(30, -30)
# sc = ax.plot_surface(x1,y1,x2,y2)
surf = ax.plot_surface(X, Y, Z,rstride=3, cstride=3)

plt.show()
```

## Secant Method

$$
Secant's\ Method\ :\ x_{k+1}=x_{k}-\frac{f(x_k)}{ \left[ \frac{f(x_k)-f(x_{k-1})}{x_k - x_{k-1}}\right]}\\
To\ find\ the\ minimal\ point\ of\ the\ function\ f:\mathbb{R}\rightarrow \mathbb{R}\\
x_{k+1}=x_{k}-\frac{f'(x_k)}{ \left[ \frac{f'(x_k)-f'(x_{k-1})}{f(x_k) - f(x_{k-1})}\right]}
$$

```python
def secant(fct,x0,x_1,err=0.05):
  print("Function :",fct)
  fct_prime = fct.diff(x)
  print(fct_prime)

  x_k = x_1
  x_k_1 = x0

  f_prime = lambdify(x,fct_prime)
  f = lambdify(x,fct)

  f_prime_xk = f_prime(x_k)
  f_prime_xk_1 = f_prime(x_k_1)

  while (abs(x_k - x_k_1) > err) and (f_prime_xk > err):
    x_k1 = x_k-(f_prime_xk/ ((f_prime_xk-f_prime_xk_1)/(f(x_k) - f(x_k_1))) )
    x_k_1 = x_k
    x_k = x_k1
    print("x_k: ",x_k)
    print("x_k-1: ",x_k_1)
    f_prime_xk = f_prime(x_k)
    f_prime_xk_1 = f_prime(x_k_1)
    print("f_prime_xk : ",f_prime_xk)
    print("f_prime_xk-1 : ",f_prime_xk_1)

  if f_prime_xk < err:
    print([x_k,f(x_k)])
    return [x_k,f(x_k)]
  print("ERREUR DE CALCUL")

```

```python
f4 = x**2+5*x
lst1 = newton(f3,3.5)

f_lambda4 = lambdify(x,f4)

fig3 = plt.figure(figsize=(10, 10))
x4 = np.linspace(-10, 10, 5500)
ax3 = fig.gca()

plt.plot(x4,f_lambda4(x4))
plt.grid(b=True, which='major', color='#666666', linestyle='-')
plt.minorticks_on()
plt.grid(b=True, which='minor', color='#999999', linestyle='-', alpha=0.2)
plt.show()
```

## Newton-Raphson Method

$$
Newton's\ Method\ :\ x_{k+1}=x_{k}-\frac{f(x_k)}{f'(x_k)}\\
To\ find\ the\ minimal\ point\ of\ the\ function\ f:\mathbb{R}\rightarrow \mathbb{R}\\
x_{k+1}=x_{k}-\frac{f'(x_k)}{f''(x_k)}
$$

```python
def newton(fct,x0,err=0.05):
  print("Function :",fct)
  fct_prime = fct.diff(x)
  fct_second = fct.diff(x,2)
  print(fct_prime)
  print(fct_second)

  x_k = x0

  f_prime = lambdify(x,fct_prime)
  f_second = lambdify(x,fct_second)
  f = lambdify(x,fct)

  f_prime_xk = f_prime(x0)
  f_second_xk = f_second(x0)
  if (f_second_xk != 0) and (f_second_xk > 0):
    while (f_prime_xk > err):
      x_k1 = x_k-(f_prime_xk/f_second_xk)
      x_k = x_k1
      print("x_k: ",x_k)
      f_prime_xk = f_prime(x_k)
      print("f_prime_xk : ",f_prime_xk)
      f_second_xk = f_second(x_k)
      print("f_second_xk : ",f_second_xk)

      if f_prime_xk < err:
        print([x_k,f(x_k)])
        return [x_k,f(x_k)]
  print("ERREUR DE CALCUL")

```

```python
f3 = x**2+5*x
lst1 = newton(f3,3.5)

f_lambda3 = lambdify(x,f3)

fig2 = plt.figure(figsize=(10, 10))
x3 = np.linspace(-10, 10, 5500)
ax2 = fig.gca()

plt.plot(x3,f_lambda3(x3))
plt.grid(b=True, which='major', color='#666666', linestyle='-')
plt.minorticks_on()
plt.grid(b=True, which='minor', color='#999999', linestyle='-', alpha=0.2)
plt.show()
```

## False Position Method - Regula Falsi

```python

```


# Gaussian Mixture Models Implementation

```python
import matplotlib.pyplot as plt
import numpy as np
import random
from sklearn.cluster import KMeans
import scipy.stats as stats
import warnings
warnings.filterwarnings("ignore", category=RuntimeWarning)
```

Since we do not have any real data, we generate it randomly from normal distributions using the below fuction.

```python
def generate_random_sample(num_gaussians, num_observations):

    samples = []
    samples_tup = []
    parameters = []
    size = int(num_observations/num_gaussians)
    for x in range(num_gaussians):
        mean, sd = random.randint(-30, 30), random.randint(1, 3)
        samples.append(np.random.normal(loc=mean, scale=sd, size=size))
        parameters.append((mean, sd))
        samples_tup.append((mean, samples[x]))

    print('Parameters of the random distributions: ' +str(parameters))
    samples_array = np.array([x for sample_x in samples for x in sample_x])

    return {'parameters': parameters, 'X': samples_array.reshape((len(samples_array), 1)), 'samples': samples, 'samp_tup': samples_tup}
```

```python
def gaussian(X, mu, cov):
    n = X.shape[1]
    diff = (X - mu).T
    return np.diagonal(1 / ((2 * np.pi) ** (n / 2) * np.linalg.det(cov) ** 0.5) * np.exp(-0.5 * np.dot(np.dot(diff.T, np.linalg.inv(cov)), diff))).reshape(-1, 1)
```

Let us now implement the Gaussian density function. Ahem... I know you can use numpy functions for that, but I believe it is actually interesting to see how things work internally. Our aim is to create a function that implements this:

\begin{equation}
\large
p(\mathbf x | \mathbf\mu, \mathbf\Sigma) = \frac 1 {({2\pi})^{n/2}|\Sigma|^{1/2}}\exp\left(-\frac 1 2 (\mathbf x -\mathbf\mu)^T\mathbf\Sigma^{-1}(\mathbf x -\mathbf\mu)\right)
\end{equation}

__Step 1__

This is the initialization step of the GMM. At this point, we must initialise our parameters $\pi_k$, $\mu_k$, and $\Sigma_k$. In this case, we are going to use the results of KMeans as an initial value for $\mu_k$, set $\pi_k$ to one over the number of clusters and $\Sigma_k$ to the identity matrix. We could also use random numbers for everything, but using a sensible initialisation procedure will help the algorithm achieve better results.

```python
def initialize_clusters(X, n_clusters):
    clusters = []
    idx = np.arange(X.shape[0])

    # We use the KMeans centroids to initialise the GMM

    kmeans = KMeans(n_clusters, n_init = 100).fit(X)

    cluster_id = kmeans.labels_

    mu_k = kmeans.cluster_centers_

    for i in range(n_clusters):
        clusters.append({
            'pi_k': 1.0 / n_clusters,
            'mu_k': mu_k[i],
            'cov_k': np.identity(X.shape[1], dtype=np.float64)
        })

    return clusters, cluster_id
```

**Step 2 (Expectation step)**

We should now calculate $\gamma(z_{nk})$. We can achieve this by means of the following expression:

\begin{equation}
\large
\gamma{(z_{nk})}=\frac {\pi_k\mathcal N(\mathbf x_n| \mathbf\mu_k, \mathbf\Sigma_k)}{\sum_{j=1}^K\pi_j\mathcal N(\mathbf x_n| \mathbf\mu_j, \mathbf\Sigma_j)}
\end{equation}

For convenience, we just calculate the denominator as a sum over all terms in the numerator, and then assign it to a variable named __totals__

```python
def expectation_step(X, clusters):
    totals = np.zeros((X.shape[0], 1), dtype=np.float64)

    for cluster in clusters:
        pi_k = cluster['pi_k']
        mu_k = cluster['mu_k']
        cov_k = cluster['cov_k']

        gamma_nk = (pi_k * gaussian(X, mu_k, cov_k)).astype(np.float64)

        for i in range(X.shape[0]):
            totals[i] += gamma_nk[i]

        cluster['gamma_nk'] = gamma_nk
        cluster['totals'] = totals

    for cluster in clusters:
        cluster['gamma_nk'] /= cluster['totals']
```

**Step 3 (Maximization step):**

Let us now implement the maximization step. Since $\gamma(z_{nk})$ is common to the expressions for $\pi_k$, $\mu_k$ and $\Sigma_k$, we can simply define:

\begin{equation}
\large
N_k=\sum_{n=1}^N\gamma({z_{nk}})
\end{equation}

And then we can calculate the revised parameters by using:

\begin{equation}
\large
\pi_k^*=\frac {N_k} N
\end{equation}

\begin{equation}
\large
\mu_k^*=\frac 1 {N_k} \sum_{n=1}^N\gamma({z_{nk}})\mathbf x_n
\end{equation}

\begin{equation}
\large
\Sigma_k^*=\frac 1 {N_k} \sum_{n=1}^N\gamma({z_{nk}})(\mathbf x_n-\mathbf\mu_k)(\mathbf x_n-\mathbf\mu_k)^T
\end{equation}

Note: To calculate the covariance, we define an auxiliary variable __diff__ that contains $(x_n-\mu_k)^T$.

```python
def maximization_step(X, clusters):
    N = float(X.shape[0])
    Q = []
    for cluster in clusters:
        gamma_nk = cluster['gamma_nk']
        cov_k = np.zeros((X.shape[1], X.shape[1]))

        N_k = np.sum(gamma_nk, axis=0)
        pi_k = N_k / N
        mu_k = np.sum(gamma_nk * X, axis=0) / N_k

        for j in range(X.shape[0]):
            diff = (X[j] - mu_k).reshape(-1, 1)
            cov_k += gamma_nk[j] * np.dot(diff, diff.T)

        cov_k /= N_k

        cluster['pi_k'] = pi_k
        cluster['mu_k'] = mu_k
        cluster['cov_k'] = cov_k

        Q_K = np.sum(np.multiply(gamma_nk, np.log(pi_k)))

```

Let us now determine the log-likelihood of the model. It is given by:

\begin{equation}
\large
\ln p(\mathbf X)=\sum_{n=1}^N\ln\sum_{k=1}^K\pi_k\mathcal N(\mathbf x_n|\mu_k,\Sigma_k)
\end{equation}

However, the second summation has already been calculated in the __expectation_step__ function and is available in the __totals__ variable. So we just make use of it.

```python
def get_likelihood(X, clusters):
    sample_likelihoods = np.log(np.array([cluster['totals'] for cluster in clusters]))
    return np.sum(sample_likelihoods), sample_likelihoods
```

Finally, let's put everything together! First, we are going to initialise the parameters by using the __initialise_clusters__ function, and then perform several expectation-maximization steps. In this case, we set the number of iterations of the training procedure to a fixed __n_epochs__ number. I have done this on purpose to generate graphs of the log-likelihood later.

```python
def train_gmm(X, n_clusters, n_epochs):

    clusters, cluster_id = initialize_clusters(X, n_clusters)
    likelihoods = np.zeros((n_epochs, ))
    scores = np.zeros((X.shape[0], n_clusters))
    history = []

    for i in range(n_epochs):
        clusters_snapshot = []

        # This is just for our later use in the graphs
        for cluster in clusters:
            clusters_snapshot.append({
                'mu_k': cluster['mu_k'].copy(),
                'cov_k': cluster['cov_k'].copy()
            })

        history.append(clusters_snapshot)

        expectation_step(X, clusters)
        maximization_step(X, clusters)

        likelihood, sample_likelihoods = get_likelihood(X, clusters)
        likelihoods[i] = likelihood

    for i, cluster in enumerate(clusters):
        scores[:, i] = np.log(cluster['gamma_nk']).reshape(-1)
    return clusters, likelihoods, scores, sample_likelihoods, history, cluster_id
```

# Find the best number of Gaussians

The below function estimates the fit error of each gaussian using the Kolmogorov–Smirnov test (stats.ks_2samp). It returns the Kolmogorov–Smirnov test values and the parameters of each Gaussian (mean and standard deviation)

```python
def compare(sol, clusters, n_clusters, num_observations, cluster_Xs):

    bins = 50
    fitness = []
    parameters = []

    for i in range(n_clusters):

        X = cluster_Xs[cluster_Xs[:,1] == i][:,0]
    size = int(num_observations/n_clusters)
    for n in range(n_clusters):
        mean = float(clusters[n]['mu_k'])
        standard_deviation = np.sqrt(float(clusters[n]['cov_k']))

        X = cluster_Xs[cluster_Xs[:,1] == n][:,0]
        rvs1 = stats.norm.rvs(size=size, loc=mean, scale=standard_deviation)
        fitness.append(stats.ks_2samp(rvs1, X)[0])
        x_values = np.arange(-50, 50, 0.01)
        y_values = stats.norm(mean, standard_deviation)

        parameters.append((mean, standard_deviation))

    return np.mean(fitness), parameters

```

```python
def show_best(X,n_clusters):
    left, right = min(X), max(X)
    n_epochs = 2
    clusters, likelihoods, scores, sample_likelihoods, history, cluster_Xs = train_gmm(X, n_clusters, n_epochs)
    cluster_Xs = cluster_Xs.reshape((len(cluster_Xs), 1))
    cluster_Xs =  np.column_stack((X, cluster_Xs))

    bins = 20

    plt.figure(figsize=(15, 6), dpi=80)
    for i in range(n_clusters):

        X = cluster_Xs[cluster_Xs[:,1] == i][:,0]
        plt.hist(X, density=True, bins = bins, alpha = 0.4)[0]

    size = int(num_observations/n_clusters)
    for n in range(n_clusters):
        mean = float(clusters[n]['mu_k'])
        standard_deviation = np.sqrt(float(clusters[n]['cov_k']))

        X = cluster_Xs[cluster_Xs[:,1] == n][:,0]
        rvs1 = stats.norm.rvs(size=size, loc=mean, scale=standard_deviation)
        x_values = np.arange(left*1.2, right*1.2, 0.01)
        y_values = stats.norm(mean, standard_deviation)

        plt.plot(x_values, y_values.pdf(x_values), linewidth=4)

    plt.show()

```

```python
def gaussian_mixture(X, max_gaussians = 10):
    num_observations = len(X)
    fitness = []
    parameters = []
    for i in range(max_gaussians):
        n_clusters = i+1
        n_epochs = 2
        clusters, likelihoods, scores, sample_likelihoods, history, cluster_Xs = train_gmm(X, n_clusters, n_epochs)
        cluster_Xs = cluster_Xs.reshape((len(cluster_Xs), 1))
        cluster_Xs =  np.column_stack((X, cluster_Xs))
        fit, param = compare(sol, clusters, n_clusters, num_observations, cluster_Xs)
        fitness.append(fit)
        parameters.append(param)
    best_n_clusters = fitness.index(min(fitness))+1
    best_parameters = parameters[fitness.index(min(fitness))]
    print('The best Nº of Guassian fits is ' + str(fitness.index(min(fitness))+1))
    print('The paramters are: ' + str(parameters[fitness.index(min(fitness))]))
    print('The fit looks like: ')
    show_best(X,best_n_clusters)

    return parameters[fitness.index(min(fitness))]
```

```python
num_gaussians = 5
num_observations = 1000
sol = generate_random_sample(num_gaussians,num_observations)
X = sol['X']

parameters = gaussian_mixture(X)
```


# barrier_reach

Synthesize a probabilistic barrier certificate to get a lower bound on the probability that a target set is reached.
## Mathematics

Consider the polynomial dynamics

$$
x(t+1) = f(x(t), t, w),  \quad  w \sim \mathcal N(0, \Sigma)
$$

and the sets

\begin{align}
X = \{x : g(x) \geq 0 \} \\
X_0 = \{x : g_0(x) \geq 0\} \\
X_T = \{x : g_T(x) \geq 0\}
\end{align}

We want to find a lower bound on the property that we reach $X_T$ given that we start in $X_0$ and remain in $X$ during the whole transition.

$$
p_{01} = \mathbb{P} \left[ x(T) \in X_T \; \mid \; x(0) \in X_0, x(t) \in X \right]
$$

Consider a certificate $B(x, t)$ satisfying

\begin{align}
    B(x, t) \geq 0        \quad & x \in X  \\
    B(x, 0) \leq \gamma   \quad & x \in X_0     \\
    B(x, T) \geq 1        \quad & x \not \in X_T, x \in X\\
    \mathbb{E} \left[ B(x(t+1), t+1)) \; \mid \; x(t) \right]  \leq B(x(t), t) + c  \quad & x \in X
\end{align}

It obviously follows that with $B(t) = B(x(t), t)$ the last constraint implies $\mathbb{E} \left[ B(T) \right] < B(0) + cT$. Since $B$ is positive we can use Markov's inequality

$$
\mathbb{P} \left[ B(T) \leq 1 \right] = 1 - \mathbb{P} \left[ B(T) \geq 1 \right] \geq 1 - \mathbb{E} [B(T)] \geq 1 - (B(0) + cT)
$$

If $A \impliedby B$ and $C \implies D$, then $\mathbb{P}(A \mid C) \geq \mathbb{P}(B \mid D)$. From the constraints we have that

\begin{align}
    x(T) \in X_T \impliedby B(T) \leq 1, \\
    x(0) \in X_0 \implies B(0) \leq \gamma.
\end{align}

Therefore,
$$
p_{01} = \mathbb{P}\left[x(T) \in X_T \; \mid \; x(0) \in X_0 \right] \geq \mathbb{P}\left[ B(T) \leq 1 \; \mid \; B(0) \leq \gamma \right] \geq 1 - (\gamma + cT).
$$

Thus we search for a barrier certificate that satisfies the conditions above while minimizing $\gamma + cT$.

```python
from sympy.abc import x, y, w
import posipoly as pp

# parameters
n = 2          # dimension
T = 4          # time horizon
sigma = 0.1    # noise standard deviation
tot_deg = 6    # overall degree of ppp

# polynomials defining sets
g  = pp.Polynomial.from_sympy(1, [x,y])
g0 = pp.Polynomial.from_sympy(0.25**2 - x**2 - y**2, [x,y])
gT = pp.Polynomial.from_sympy(0.5**2 - (x-1)**2 - y**2, [x,y])

# dynamics:
#  x(t+1) = x(t) + 0.25 + w
#  y(t+1) = y(t)
ft_x = pp.Polynomial.from_sympy(x + 0.25 + w, [x,y,w])
ft_y = pp.Polynomial.from_sympy(y, [x,y,w])
```

We synhesize a separate barrier for each discrete time step, i.e. $B(x,t) = B_t(x)$. Using the S procedure we can rewrite the problem above as

\begin{align}
    c + B_t(x) - \mathbb{E} [B_{t+1}(f(x, t, w)) ] - \sigma_1^t(x) g(x)   \geq 0, & \qquad  t=0, ..., T-1 \\
    \gamma - B_0(x) - \sigma_2(x) g_0(x)  \geq 0, & \\
    B_T(x) - 1 + \sigma_{3a}(x) g_1(x) - \sigma_{3b}g(x)      \geq 0, & \\
    B_t(x) - \sigma_4^t(x) g(x)              \geq 0, & \qquad  t=0, ..., T \\
    \sigma_1^t(x) \geq 0,    & \qquad  t=0, ..., T-1 \\
    \sigma_4^t(x) \geq 0,    & \qquad  t=0, ..., T \\
    \sigma_2(x), \sigma_{3a}(x), \sigma_{3b}(x) \geq 0, \\
\end{align}

```python
deg_B = tot_deg   # need to subtract if deg f larger than 1
deg_g0 = tot_deg-g0.d
deg_gT = tot_deg-gT.d
deg_g = tot_deg-g.d

prog = pp.PPP()

prog.add_var('c', n, 0, 'coef')      # scalar variable = polynomial of degree 0
prog.add_var('gamma', n, 0, 'coef')  # scalar variable

for t in range(T+1):
    prog.add_var('B{}'.format(t), n, deg_B, 'coef')

for t in range(T):
    prog.add_var('s1{}'.format(t), n, deg_g, 'pp')

prog.add_var('s2', n, deg_g0, 'pp')
prog.add_var('s3a', n, deg_gT, 'pp')
prog.add_var('s3b', n, deg_g, 'pp')

for t in range(T+1):
    prog.add_var('s4{}'.format(t), n, deg_g, 'pp')
```

Prepare the transformations

```python
# Identity for scalars to tot_deg
T1 = pp.PTrans.eye(n0=n, d0=0, n1=n, d1=tot_deg)

# Identity for scalars to scalar
T10 = pp.PTrans.eye(n0=n, d0=0, n1=n, d1=0)

# B(x,y) -> E_w[ B(fx(x,y,w), fy(x,y,w)) ]
TBp = pp.PTrans.gaussian_expectation(n0=3, d0=deg_B, i=2, sigma=sigma) \
      * pp.PTrans.composition(n0=n, d0=deg_B, g_list=[ft_x, ft_y])

# Identity for B
TB  = pp.PTrans.eye(n, deg_B)

# Multiplication with g, g0, g1
Tg  = pp.PTrans.mul_pol(n, deg_g, g)
Tg0 = pp.PTrans.mul_pol(n, deg_g0, g0)
TgT = pp.PTrans.mul_pol(n, deg_gT, gT)

# Identity for free variables
Tfr = pp.PTrans.eye(n, tot_deg)
```

These are the constraints in algebra form

    T1.c + TB.Bt - Tbp.B(t+1) - Tg.s1t pp   t=0, ..., T-1    (1)
    T1.gamma - TB.B0 - Tg0.s2          pp                    (2)
    TB.BT + TgT.s3a - Tg.s3b - 1       pp                    (3)
    TB.Bt - Tg.s4t                     pp   t=0, ..., T      (4)

```python
# add (1)
for t in range(T):
    Aop = {'c': T1,
           'B{}'.format(t): TB,
           'B{}'.format(t+1): -TBp,
           's1{}'.format(t): -Tg}
    prog.add_constraint(Aop, pp.Polynomial.zero(n), 'pp')

# add (2)
prog.add_constraint({'gamma': T1, 'B0': -TB, 's2': -Tg0}, pp.Polynomial.zero(n), 'pp')

# add (3)
prog.add_constraint({'B{}'.format(T): TB, 's3a': TgT, 's3b': -Tg}, pp.Polynomial.one(n), 'pp')

# add (4)
for t in range(T+1):
    Aop = {'B{}'.format(t): TB,
           's4{}'.format(t): -Tg}
    prog.add_constraint(Aop, pp.Polynomial.zero(n), 'pp')

# add inequality: gamma <= 1  (to exclude large solutions)
prog.add_constraint({'gamma': T10}, pp.Polynomial.one(n), 'iq')

# set c=0
# prog.add_row({'c': T10}, pp.Polynomial.zero(n), 'eq')

# add objective
prog.set_objective({'c': pp.PTrans.eval0(n, 0)* T, 'gamma': pp.PTrans.eval0(n, 0)})
```

Time to solve it!

```python
sol, status = prog.solve('psd')

c = prog.get_poly('c')(0,0)
gamma = prog.get_poly('gamma')(0,0)

lb = 1-(gamma+c*T)

print('got c={:.2f} and gamma={:.2f}'.format(c, gamma))
print('lower bound is {:.2f}'.format(lb))

import scipy.stats
import numpy as np
print("real probablity ~ {:.2f}".format(scipy.stats.norm.cdf(0.25/np.sqrt(4*sigma**2))))
```

    optimizing...
    solved in 0.31s
    solsta.optimal
    got c=-0.14 and gamma=0.95
    lower bound is 0.62
    real probablity ~ 0.89

From the certificates we can now plot a lower bound on the probability that the target set is reached. From the proof above it follows that

$$
\mathbb{P}(x(T) \in X_1 \mid x(t) = x) \geq 1 - (c(T-t) + B_t(x)).
$$

```python
%matplotlib notebook
import matplotlib.pyplot as plt
from matplotlib.pyplot import cm

xx = np.linspace(-0.5, 1.5, 50)
yy = np.linspace(-0.8, 0.8, 50)

XX, YY = np.meshgrid(xx, yy)

g0_val = np.vectorize(g0)(XX, YY)
gT_val = np.vectorize(gT)(XX, YY)

levels = sorted([0, 0.4, 0.8, 0.95, 0.99, lb])

for t in [0, 1, 2, 3, 4]:
    Bt = prog.get_poly('B{}'.format(t))
    proba_min = 1 - (np.vectorize(Bt)(XX, YY) + c * (T-t))
    plt.figure()
    CS = plt.contour(XX, YY, proba_min, levels=levels, cmap=cm.get_cmap('plasma'), vmin=0, vmax=1)
    plt.clabel(CS, inline=1, fontsize=10)

    plt.contour(XX, YY, g0_val, levels=[0], colors='black', linestyles='dotted')
    plt.contour(XX, YY, gT_val, levels=[0], colors='black', linestyles='dotted')
    plt.title('t={}'.format(t))

plt.show()
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>