# Linear-Feedback Shift Register (LFSR)

In an LFSR, the output from a standard shift register is fed back into its
input causing an endless cycle. The feedback bit is the result of a linear
combination of the shift register content and the feedback coefficients.

The foolowing are the equation ruling the LFSR block scheme:
$$
\begin{align}
    b[t]       = &\; s_0[t] \\
    s_j[t]     = &\; s_{j+1}[t-1] \\
    s_{m-1}[t] = &\; \oplus_{j=0}^{m-1} p_{m-j} \otimes s_{j}[t-1] \\
\end{align}
$$

**Example**:
- LFSR length: 3
- feeedback polynomial: $x^3 + x + 1$
- initial state: 0b111
- LFSR cycle:
|     state | output | feedback |
|:---------:|:------:|:--------:|
| 0b111 (7) |    1   |     0    |
| 0b011 (3) |    1   |     1    |
| 0b101 (5) |    1   |     0    |
| 0b010 (2) |    0   |     0    |
| 0b001 (1) |    1   |     1    |
| 0b100 (4) |    0   |     1    |
| 0b110 (6) |    0   |     1    |
| 0b111 (7) |    1   |     0    |

```python
import math
from operator import xor
from functools import reduce
from itertools import compress
from itertools import islice
```

## LFSR generator

As first step, we implement an LFSR as a generator.

**Inputs**:
 - **Feedback Polynomial**: `list` of `int` representing the degrees of the non-zeros coefficients. Example: [12, 6, 4, 1, 0] represents the polynomial $x^{12}+x^6+x^4+x+1$.
 - **shift-register initial state** (optional, default all bits to 1): `int` or `list` of bits representing the LFSR initial state. Example: 0xA65 for 0b101001100101.

**Yield**:
- **Output bit**: `bool` representing the LFSR output bit

**Template**:
```python
def lfsr_generator(poly, state=None):
    ''' generator docstring '''

    # check inputs validity
    # define variables storing the internal state

    while True:
        # LFSR iteration:
        # - update state
        # - compute output
        yield output
```

### implentation with lists

In this first implementation both the shift register `state` and the feedback polynomial `poly` are `list` of `bool`. This is the most straightforward choice as it directly maps the LFSR block scheme.

| variable/
operation | definition | implementation |
|:-:|:-:|:--|
| feeedback polynomial | $$x^3 + x + 1$$ | `poly = [True, True, False, True]` |
| initial state | 0b111 | `state = [True, True, True]` |
| state update | $$s_j[t] = s_{j+1}[t-1]$$ | `state = state[1:] + [feedback]` |
| output bit | $$s_0[t]$$ | `output = state[0]` |
| feedback bit | $$\oplus_{j=0}^{m-1} p_{m-j} \otimes s_{j}[t]$$ | `feedback = reduce(xor, compress(state[::-1], poly[1:]))` |

For debugging purposes, a `verbose` flag is added as argument to enable the print of the internal state at each iteration.

```python
def lfsr_generator(poly, state=None, verbose=False):
    '''
    Generator implementing a Linear Feedback Shift Register (LFSR)

    Parameters
    ----------
    poly: list of int,
        feedback polynomial expressed as list of integers corresponding to
        the degrees of the non-zero coefficients.
    state: int, optional (default=None),
        shift register initial state. If None, all bits of state are set to 1.
    verbose: bool, optional (default=False),
        If True, the internal state is printed at each iteration

    Return
    ------
    bool, LFSR output bit
    '''

    # ==== check inputs ====
    # ==> verbose
    if verbose:
        # define a function to print the lfsr internal state
        def print_lfsr(state, output, feedback):
            state_bits = ''.join(str(int(s)) for s in state)[::-1]
            state_int = int(state_bits, 2)
            print(f'{state_bits} ({state_int})  {int(output)}  {int(feedback)}')

    # ==> poly
    length = max(poly) # LFSR length as the max degree of the polynomial
    poly = [i in poly for i in range(length+1)] # turn poly into a list of bool

    # ==> state
    if state is None: # default value for state is all ones (True)
        state = [True for _ in range(length)]
    else: # convert integer into list of bool
        state = [bool(int(s)) for s in (f'{state:0{length}b}')[::-1]]

    # ==== initial state ====
    # compute output bit and feedback bit
    output = state[0]
    feedback = reduce(xor, compress(state[::-1], poly[1:]))
    if verbose: # print initial state
        print(' state   b fb')
        print_lfsr(state, output, feedback)

    # ==== infinite loop ====
    while True:
        state = state[1:] + [feedback]                          # update state
        output = state[0]                                       # output bit
        feedback = reduce(xor, compress(state[::-1], poly[1:])) # feedback bit
        if verbose: # print current state
            print_lfsr(state, output, feedback)

        yield output # return current output
```

Here we check the generator functioning, by replication the example showed above where the feedback polynomial is $x^3+x+1$ and the initial state is `0x7`=`0b111`

```python
poly = [3, 1, 0] # feedback polynomial x^3 + x + 1
state = 0x7      # initial state
niter = 7        # number of iterations

# lfsr definition
lfsr = lfsr_generator(poly, state, verbose=True)

# just iter over the LFSR generator
for b in islice(lfsr, niter):
    # do nothing, just let the generator print its internal state
    pass
```

     state   b fb
    111 (7)  1  0
    011 (3)  1  1
    101 (5)  1  0
    010 (2)  0  0
    001 (1)  1  1
    100 (4)  0  1
    110 (6)  0  1
    111 (7)  1  0

### implentation with integers

In this implementation both the shift register `state` and the feedback polynomial `poly` are `int`. With this choice, bit-wise logical operation, as well as bit-shift, are easy to perform, while XOR of multiple bits or reversing the bit order are less straightforward.

For efficiency reasons, it is choosen to store the bits of the state in reverse order. At each iteration, the computation of the feedback bit requires a bit-wise AND between the current state and the polynomial feedback. Since these quantities are defined with inverted index, it is convinient to keep stored the bits of either state or poly in reverse order. To be compliant with common implementation in other programming languages (like C/C++) we choose to store the bits of the state in reverse order.

| variable/
operation |         definition         | implementation |
|:-:|:-:|:--|
| feeedback
 polynomial |   $x^3 + x + 1 $ | `poly = 0b1011`, `length = 3` |
| initial state | 0b111 | `state = 0b111` |
| state update |   $s_j[t] = s_{j+1}[t-1] $ | `statemask = (1 << length) - 1`
 `state = ((state << 1) \| feedback) & statemask` |
| output bit |   $s_0[t] $ | `outmask = 1 << (length-1)`
 `output = bool(state & outmask)` |
| feedback bit |   $\oplus_{j=0}^{m-1} p_{m-j} \otimes s_{j}[t] $   | `feedback = parity(state & poly)` |

Let us define a function `reverse` to reverse the bit order of an integer

```python
def reverse(x, nbit=None):
    ''' reverse the bit order of the input `x` (int) represented with `nbit`
    (int) bits (e.g., nbit: 5, x: 13=0b01101 -> output: 22=0b10110) '''
    if nbit is None:
        nbit = math.ceil(math.log2(x))
    return int(f'{x:0{nbit}b}'[::-1][:nbit], 2)
```

```python
x, nbit = 0b1101011000101010, 16
y = reverse(x, nbit)
print(f'{x:0{nbit}b} -> {y:0{nbit}b}')
```

    1101011000101010 -> 0101010001101011

Let us also define a function `parity` that computes the parity bit for a given integer.

```python
def parity(x):
    ''' compute the parity bit of an integer `x` (int) '''
    return bool(sum(int(b) for b in f'{x:b}') % 2)
```

```python
x = 0b1101011010101010
y = parity(x)

print(f'{x:0{nbit}b} -> parity bit: {y:b}')

```

    1101011010101010 -> parity bit: 1

Now we can define the generator that implements the LFSR.

```python
def lfsr_generator(poly, state=None, verbose=False):
    '''
    Generator implementing a Linear Feedback Shift Register (LFSR)

    Parameters
    ----------
    poly: list of int,
        feedback polynomial expressed as list of integers corresponding to
        the degrees of the non-zero coefficients.
    state: int, optional (default=None),
        shift register initial state. If None, all bits of state are set to 1.
    verbose: bool, optional (default=False),
        If True, the internal state is printed at each iteration

    Return
    ------
    bool, LFSR output bit
    '''

    # ==== check inputs ====
    # ==> verbose
    if verbose:
        def print_lfsr(state, output, feedback, length):
            print(f'{state:0{length}b} ({state})',
                  f'{int(output)}  {int(feedback)}')

    # ==> poly
    length = max(poly) # LFSR length as the max degree of the polynomial
    poly = sum([2**p for p in poly]) >> 1  # ignoring p0 (always 1)

    # ==> state
    outmask = 1 << (length-1)      # mask to select the output bit from state
    statemask = (1 << length) - 1  # mask for state
    if state is None: # default value for state is all ones
        state = statemask
    state = reverse(state, length) # reverse the bit order of state

    # ==== initial state ====
    # compute output bit and feedback bit
    output = bool(state & outmask) # get output from current state
    feedback = parity(state & poly)  # compute feedback bit as the parity bit
    if verbose: # print initial state
        print(' state   b fb')
        print_lfsr(reverse(state, length), output, feedback, length)

    # ==== infinite loop ====
    while True:
        state = ((state << 1) | feedback) & statemask # update state
        output = bool(state & outmask)                # get output
        feedback = parity(state & poly)               # update feedback bit
        if verbose: # print current state
            _state = reverse(state, length)
            print_lfsr(_state, output, feedback, length)

        yield output
```

As we did above, we check the implementation by replicating the reference example.

```python
poly = [3, 1, 0]
state = 0x7
niter = 7

lfsr = lfsr_generator(poly, state, verbose=True)

for b in islice(lfsr, niter):
    pass
```

     state   b fb
    111 (7) 1  0
    011 (3) 1  1
    101 (5) 1  0
    010 (2) 0  0
    001 (1) 1  1
    100 (4) 0  1
    110 (6) 0  1
    111 (7) 1  0

Now, we can use the LFSR to generate a "long" (not really) sequence of bits.

```python
nbits = 1000
poly = [5, 2, 0]
lfsr = lfsr_generator(poly)

# iter over LFSR and store its output in a list
bits = [b for b in islice(lfsr, nbits)]

# print the list of bool as string of 0s and 1s
print(''.join([str(int(b)) for b in bits]))
```

    1111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011010010000101011101100011111001101001000010101110110001111100110100100001010111011000111110011

We can notice that the 31-bits sequence `1111001101001000010101110110001` is periodically repeated. This is compliant with an LFSR with length 5 and a primitive polynomial with degree 5 in the feedback.

## LFSR Iterator

Implementing the LFSR as generator does not allow the user to access to the internal state and variables. Indeed, we needed to code a internal print that was activate by the `verbose` flag for debug.

To solve this issue, we can implement the LFSR as an **iterator**, that is a class with the `__iter__` and `__next__` methods that allow to iter over it. Such solution allow the user to employ the LFSR with the same scheme used for the generator bu with the capability to access to the internal state and variables.

**Inputs**:
 - **Feedback Polynomial**: `list` of `int` representing the degrees of the non-zeros coefficients. Example: [12, 6, 4, 1, 0] represents the polynomial $x^{12}+x^6+x^4+x+1$.
 - **shift-register initial state** (optional, default all bits to 1): `int` or `list` of bits representing the LFSR initial state. Example: 0xA65 for 0b101001100101.

**Attributes**:
- `poly`: (`list` of `int`) list of the polynomial coefficients;
- `length`: (`int`) polynomial degree and length of the shift register;
- `state`: (`int`) LFSR state;
- `output`: (`bool`) output bit;
- `feedback`: (`bool`) feedback bit.

**Methods**:
- `__init__`: class constructor;
- `__iter__`: necessary to be an iterable;
- `__next__`: update LFSR state and returns output bit;
- `cycle`: returns a list of bool representing the full LFSR cycle ;
- `run_steps`: execute N LFSR steps and returns the corresponding output as list of bool (N is a input parameter, default N=1);
- `__str__`: return a string describing the LFSR class instance.

**Template**:
```python
class LFSR(object):
    ''' class docstring '''

    def __init__(self, poly, state=None):
        ''' constructor docstring '''
        ...
        self.poly = ...
        self.length = ...
        self.state = ...
        self.output = ...
        self.feedback = ...

    def __iter__(self):
        return self

    def __next__(self):
        ''' next docstring '''
        ...
        return self.output

    def run_steps(self, N=1):
        ''' run_steps docstring '''
        ...
        return list_of_bool

    def cycle(self, state=None):
        ''' cycle docstring '''
        ...
        return list_of_bool
```

We first define two functions to convert integers into, what we will call it, their *sparse* representation, which consist in the list of indexes corresponding to 1s in the integer binary representation. For example the integer 138 (`0b10001010`) is turned into the list `[1, 3, 7]`.

This function is employed to easily pass from the integer representation of the LFSR feedback polynomial (used during computations) to the representation as list of the degrees of the non-zeor coefficients (used as inputs and outputs).

We call it sparse as it reminds how sparse matrices are stored, i.e., as list of triplets comprising the row and column indexes and the corresponding value.

```python
def int_to_sparse(integer):
    ''' transform an integer into the list of indexes corresponding to 1
    in the binary representation (sparse representation)'''
    sparse = [i for i, b in enumerate(f'{integer:b}'[::-1]) if bool(int(b))]
    return sparse

def sparse_to_int(sparse):
    ''' transform a list of indexes (sparse representation) in an integer whose
    binary representation has 1s at the positions corresponding the indexes and
    0 elsewhere '''
    integer = sum([2**index for index in sparse])
    return integer
```

```python
integer = 0b10001010

sparse = int_to_sparse(integer)
print(f'int to sparse: {integer} ({bin(integer)}) -> {sparse}')

integer = sparse_to_int(sparse)
print(f'sparse to int: {sparse} -> {integer} ({bin(integer)})')
```

    int to sparse: 138 (0b10001010) -> [1, 3, 7]
    sparse to int: [1, 3, 7] -> 138 (0b10001010)

Here is the LFSR implemented as Iterator. We choose to implement both state and polynomial feedback as integers.

Note that:
- in Python, for attributes and methods there is not the concept of **private** or **public** . It is all public and it is up to the user make a good use of the class. However, there is the convention (supported also by IDE) that attributes starting with an underscore `_` are to be considered private:
```python
self.state # this is public
self._state # this is considered private (actually, it is public)
```

- Python meets the idea of [Uniform Access Principle](https://en.wikipedia.org/wiki/Uniform_access_principle) that states *all services offered by a module should be available through a uniform notation, which does not betray whether they are implemented through storage or through computation*.
    - This means that `lfsr.state` and `lfsr.state = 0` are much preferred than `lfsr.get_state()` and `lfsr.set_state(0)`.
    - If you need to wrap the accesses to the attributes inside methods (to deny setting, to check the value before setting, to return the value in a different format with respect to how it is stored internally) you can use the [decorator](https://wiki.python.org/moin/PythonDecorators) `@property` that allows you to hide a method `lfsr.set_state(0)` behind the expression `lfsr.state = 0`.

These features help in writing very readable code.

**Example**: let us assume that, internally we want the LFSR state stored as an integer with bits in reverse order to allow an efficient computation for the LFSR iterations, but, externally, i.e., when the state is read, we want to return the value with bits in correct order. We can define a private attribute `_state` for computations and a public attribute `state` that hides (i.e., invokes) a **getter method** that return the private attribute inverting the bit order.
```python
    @property
    def state(self):
        return reverse(self._state, len(self))
```
The same principle can be employed to let the attribute `state` invoke a **setter method** that perform the inverse operation:
```python
    @state.setter
    def state(self, state):
        self._state = reverse(state & self._statemask, len(self))
```
Note that the decorator now is `<attribute>.setter`.

Moreover, the setter method can be used to make attributes only readble and not writable. For example, we may want the LFSR length to be an only-read attribute.
```python
    @property
    def length(self):
        return self._length
    @length.setter
    def length(self, val):
        raise AttributeError('Denied')
```

Thanks to these feature we choose:
- `state` to be both redable and writable but setter and getter are defined in such a way to allow state to be internally stored with bits in reverse order.
- `poly` to be only redable and internally stored as integer.
- `length`, `output`
- `feedback` to be both redable and writable. The user can insert the state one bit at a time and we will see that some stream ciphers exploit this possibility.

```python
class LFSR(object):
    '''
    Class implementing a Linear Feedback Shift Register (LFSR)

    Attributes
    ----------
    poly: list of int,
        feedback polynomial expressed as list of integers corresponding to
        the degrees of the non-zero coefficients.
    state: int,
        state of the shift register.
    output: bool,
        last shift register output,
    length: int,
        length of the shift register as well as  maximum degree of the feedback
        polynomial.

    Methods
    -------
    run_steps(self, N=1)
        Execute multiple LFSR steps
    cycle(self, state=None)
        Execute a full cycle.
    '''

    def __init__(self, poly, state=None):
        '''
        Parameters
        ----------
        poly: list of int,
            feedback polynomial expressed as list of integers corresponding to
            the degrees of the non-zero coefficients.
        state: int, optional (default=None)
            shift register initial state.
            If None, state is set to all ones.
        '''
        length = max(poly)
        self._length = length
        self._poly = sparse_to_int(poly) >> 1 # p0 is omitted (always 1)

        self._statemask = (1 << length) - 1
        if state is None:
            state = self._statemask
        self.state = state

        self._outmask = 1 << (length-1)
        self._output = bool(self._state & self._outmask)

        self._feedback = parity(self._state & self._poly)

    # ==== state ====
    @property
    def state(self):
        # state is re-reversed before being read
        return reverse(self._state, len(self))
    @state.setter
    def state(self, state):
        if not isinstance(state, int):
            raise TypeError('input type is not supported')
        # ensure seed is in the range [1, 2**m-1]
        if (state < 1) or (state > len(self)):
            state = 1 + state % (2**len(self)-2)
        self._state = reverse(state & self._statemask, len(self))

    # ==== length ====
    @property
    def length(self):
        return self._length
    @length.setter
    def length(self, val):
        raise AttributeError('Denied')

    # ==== poly ====
    @property
    def poly(self):
        return int_to_sparse((self._poly << 1) | 1)[::-1]
    @poly.setter
    def poly(self, poly):
        raise AttributeError('Denied')

    # ==== output ====
    @property
    def output(self):
        return self._output
    @output.setter
    def output(self, val):
        raise AttributeError('Denied')

    # ==== feedback ====
    @property
    def feedback(self):
        return self._feedback
    @feedback.setter
    def feedback(self, feedback):
        self._feedback = bool(feedback)

    def __str__(self):
        poly = ' + '.join([
            (f'x^{d}' if d > 1 else ('x' if d==1 else '1'))
            for d in self.poly
        ])
        string = ', '.join([
            f'poly: "{poly}"',
            f'state: 0x{self.state:0{(self.length+1)//4}x}',
            f'output: {None if self.output is None else int(self.output)}'
        ])
        return string

    def __repr__(self):
        return f'LSFR({str(self)})'

    def __len__(self):
        return self.length

    def __iter__(self):
        return self

    def __next__(self):
        '''Execute a LFSR step and returns the output bit (bool)'''
        self._state = ((self._state << 1) | self._feedback) & self._statemask
        self._output = bool(self._state & self._outmask)
        self._feedback = parity(self._state & self._poly)
        return self._output

    def run_steps(self, n=1):
        '''
        Execute multiple LFSR steps.

        Parameters
        ----------
        n: int, optional (default=1)
            number of steps to execute.

        Output
        ------
        list of bool (len=n),
            LFSR output bit stream.
        '''
        return [next(self) for _ in range(n)]

    def cycle(self, state=None):
        '''
        Execute a full LFSR cycle (LFSR.len steps).

        Parameters
        ----------
        state: int or list of int or bools, optional (default=None)
            shift register state. If None, state is kept as is.

        Output
        ------
        list of bool (len=2**myLFSR.len - 1),
            LFSR output bit stream.
        '''
        if state is not None:
            self.state = state
        return self.run_steps(n=int(2**len(self)) - 1)

```

Again, we check the implementation with the reference example. This time, we did not need to use a verbose flag to print the internal state, since all internal variable are accessible by the user.

```python
poly = [3, 1, 0]
state = 0x7
niter = 7

def print_lfsr(lfsr):
    print(f'{lfsr.state:0{len(lfsr)}b} ({lfsr.state:d}) ',
          f'{int(lfsr.output):d}  {int(lfsr.feedback):d}')

# create and instance of the LFSR
lfsr = LFSR(poly, state)

print('\n state   b fb')
print_lfsr(lfsr) # print initial state
for b in islice(lfsr, niter):
    print_lfsr(lfsr)
```

     state   b fb
    010 (2)  0  0
    001 (1)  1  1
    100 (4)  0  1
    110 (6)  0  1
    111 (7)  1  0
    011 (3)  1  1
    101 (5)  1  0
    010 (2)  0  0

Here is a check for the implementation of `__str__` and `__repr__` methods.
Remind that the goal of :
- `__str__` is to provide a readable string
- `__repr__` is to be unambiguous

```python
print(lfsr)   # print calls `__str__`
display(lfsr) # display calls `__repr__`
```

    poly: "x^3 + x + 1", state: 0x2, output: 0

    LSFR(poly: "x^3 + x + 1", state: 0x2, output: 0)

Here is a check for the implementation of the methods `cycle` and `run_steps`. Since the former calls the latter, we just check the former.

```python
cycle = lfsr.cycle()
print(cycle)
print(''.join(f'{int(b)}' for b in cycle))
```

    [True, False, False, True, True, True, False]
    1001110

## Berlekamp-Massey Algorithm

The Berlekamp–Massey algorithm is an algorithm that finds the shortest LFSR for a given binary sequence.
The input consists in the binary sequence and the output is the feedback polynomial characterizing the shortest LFSR able to generate that sequence.

**Pseudocode**:
> **Input** $b = [b_0, b_1, \dots, b_N]$

> $P(x) \leftarrow 1, \quad m \leftarrow 0 $

> $Q(x) \leftarrow 1, \quad r \leftarrow 1 $

> **for** $\tau = 0, 1, \dots, N-1 $

> $\qquad d \leftarrow \oplus_{j=0}^{m} p_j \otimes b[\tau-j] $

> $\qquad $ **if** $d = 1 $ **then**

> $\qquad \qquad $ **if** $2m \le \tau $ **then**

> $\qquad \qquad \qquad R(x) \leftarrow P(x) $

> $\qquad \qquad \qquad P(x) \leftarrow P(x) + Q(x)x^r $

> $\qquad \qquad \qquad Q(x) \leftarrow R(x) $

> $\qquad \qquad \qquad m \leftarrow \tau + 1 - m $

> $\qquad \qquad \qquad r \leftarrow 0 $

> $\qquad \qquad $ **else**

> $\qquad \qquad \qquad P(x) \leftarrow P(x) + Q(x)x^r $

> $\qquad \qquad $ **endif**

> $\qquad $ **endif**

> $\qquad r \leftarrow r + 1 $

> **endfor**

> **Output** $P(x) $

**Template**:
```python
def berlekamp_massey(b):
    ''' function docstring '''
    # algorithm implementation
    return poly
```

**Example**:

Input: $ b = [1, 0, 1, 0, 0, 1, 1, 1] $

| $\tau$ | $b_\tau$ | $d$ |      $P(x)$     | $m$ |    $Q(x)$   | $r$ |
|:------:|:--------:|:---:|:---------------:|:---:|:-----------:|:---:|
|   $-$  |    $-$   | $-$ | $ 1           $ | $0$ | $ 1       $ | $1$ |
|   $0$  |    $1$   | $1$ | $ 1 + x       $ | $1$ | $ 1       $ | $1$ |
|   $1$  |    $0$   | $1$ | $ 1           $ | $1$ | $ 1       $ | $2$ |
|   $2$  |    $1$   | $1$ | $ 1 + x^2     $ | $2$ | $ 1       $ | $1$ |
|   $3$  |    $0$   | $0$ | $ 1 + x^2     $ | $2$ | $ 1       $ | $2$ |
|   $4$  |    $0$   | $1$ | $ 1           $ | $3$ | $ 1 + x^2 $ | $1$ |
|   $5$  |    $1$   | $1$ | $ 1 + x + x^3 $ | $3$ | $ 1 + x^2 $ | $2$ |
|   $6$  |    $1$   | $0$ | $ 1 + x + x^3 $ | $3$ | $ 1 + x^2 $ | $3$ |
|   $7$  |    $1$   | $0$ | $ 1 + x + x^3 $ | $3$ | $ 1 + x^2 $ | $4$ |

Output: $ P(x) = 1 + x + x^3 $

We first define a functions to transform a bit sequence (list of bool, list of 1/0, or a string composed by only '1'/'0') into an integer.

```python
def bits_to_int(bits):
    ''' transform a bit sequence (str of 1/0 or list of bool) into an int '''
    integer = sum(1 << i for i, bit in enumerate(bits) if bool(int(bit)))
    return integer
```

```python
bits = '101011010111101'
integer = bits_to_int(bits)
bin(integer)
```

    '0b101111010110101'

We choose to implement the Berlekamp-Massy algorithm by storing polynoimals $P(x)$ and $Q(x)$ as integers. Instead, the bit sequence $b$ is kept as list/string and at each iteration is transformed to an integer.

Note that, from an implementation point of view, the main steps are:
- the computation of the discrepancy bit
$$d \leftarrow \oplus_{j=0}^{m} p_j \otimes b[\tau-j]$$
This operation is very similar to the combination of state and polynominal coefficients in the computation of the LFSR feedback bit.
```python
d = parity(P & bits_to_int(bits[t-m:t+1][::-1]))
```

- the update of the polynomial $P(x)$
$$P(x) \leftarrow P(x) + Q(x)x^r$$
```python
P = P ^ (Q << r)
```
    - Assuming $Q(x) = \sum_{i=0}^l q_i x^i$, then $Q(x)x^r = \sum_{i=0}^l q_i x^{r+i}$. Therefore, if $Q(x)$ is represented as `[q0, q1, ..., ql]`, $Q(x)x^r$ is `[0, ..., 0, q0, q1, ..., ql]` where the number of 0s is $r$. With the integer representation, this is translated in a left shift of `Q` by `r` positions.
    - The $+$ operation in $GF(2^m)$ corresponds to the element-wise $+$ operation in $GF(2)$, that is the `xor`. With the integer representation the bit-wise xor is implemented with `^`.

As for the LFSR implemented as a generator, we make use of a `verbose` flag to print the internal variables for debugging purposes.

```python
def berlekamp_massey(bits, verbose=False):
    '''
    Find the shortest LFSR for a given binary sequence.

    Parameters
    ----------
    bits: list/string of 0/1,
        stream of bits.

    Return
    ------
    list of int,
        linear feedback polynomial expressed as list of the exponents related
        to the non-zero coefficients
    '''

    # variables initialization
    P, m = 0x1, 0
    Q, r = 0x1, 1

    if verbose:
        from math import log2
        lenP = 1 + int(log2(len(b)+1))
        print(f'  t b d {"poly":>{lenP}} m {"Q":>{lenP}} r')
        print(f'  - - - {P:{lenP}b} 0 {Q:{lenP}b} 1')

    for t in range(len(bits)):

        # compute discrepancy
        d = parity(P & bits_to_int(bits[t-m:t+1][::-1]))

        if d:
            if 2*m <= t: # A
                P, Q = P^(Q<<r), P
                m, r = t+1-m, 0
            else: # B
                P = P^(Q<<r)
        r += 1

        if verbose:
            print(f'{t:3d} {bits[t]} {d:d} {P:{lenP}b} {m} {Q:{lenP}b} {r}')

    # convert poly from integer to list of degree of non-zero coefficients
    poly = int_to_sparse(P)[::-1]
    return poly
```

We can check the implementation by replicating the reference example.

```python
b = '1010011101'
# b = '11101001110100111010'
poly = berlekamp_massey(b, verbose=True)
print(poly)
```

      t b d poly m    Q r
      - - -    1 0    1 1
      0 1 1   11 1    1 1
      1 0 1    1 1    1 2
      2 1 1  101 2    1 1
      3 0 0  101 2    1 2
      4 0 1    1 3  101 1
      5 1 1 1011 3  101 2
      6 1 0 1011 3  101 3
      7 1 0 1011 3  101 4
      8 0 0 1011 3  101 5
      9 1 0 1011 3  101 6
    [3, 1, 0]

A more deep functioning assessment can consist in generating a bit sequence with an LFSR characterized by a primitive polynomial in the feedback, and check that the Berlekamp-Massey algorithm is able to infer the LFSR polyonimal.

Use LFSR with different length!

```python
# poly = [3, 1, 0]
# poly = [4, 1, 0]
# poly = [5, 2, 0]
# poly = [6, 1, 0]
# poly = [7, 1, 0]
# poly = [8, 4, 3, 2, 0]
# poly = [9, 4, 0]
# poly = [10, 3, 0]
poly = [11, 2, 0]
# poly = [16, 12, 3, 1, 0]

# create an instance of an LFSR
lfsr = LFSR(poly)
print(lfsr)

# generate a bit sequence with length of at least one cycle
bitstream = lfsr.cycle()
print(''.join([str(int(b)) for b in bitstream]))

# use the BM algorithm to infer the polynomial
bm_poly = berlekamp_massey(bitstream)
print(bm_poly)
```

    poly: "x^11 + x^2 + 1", state: 0x002, output: 0
    1000000000010101010101110111011100100111001011110010111001111011000011000111111000001100111111100001100110101001011011101000111001000001101000000011101010101100010001001010000010111010101110010001001111010111100111011001011000111101101011001001000111101000001100010101001010001000010000010101111111011100110010011110000101100101011100001000110101000001110111111100100110011110100101100100001001011111101000110011101011010011101101111001001001101000010110111111011011001101101101001001001000010111101010001100010000011111010101100111011100001101100000011100000000110101010100100010001011111010111001101110010110110000100100101011110100010011011111010010011000101111000001001101010111100010001100000101001010100010111011111011000110010010100101111011110110010011011010000111000101011000001000111111101011001100010010110101111011011100111000110100111110001011001111101101001100011101001010011101000101100010100011111011111001101100111100011110011111001111001101001101001000011101000000110111111110001100110000011110000000110000000001111111111100110011001011010010111000101110010100011010001010010000010000101010000001000101010111110111011001110010010110000101110000001001111111101001100110111100001110011111100101100110100011110001010011010111010010001101111010110110010001110000101001111110111100110001100001111100000011001010101101000100011101011111001000110010111110000100110000001011010101000111011101011000110111000001110010101001111011101001101100010110110101110001110110000011011010101101101110110110110001110001111100101001100001000011111101010011001000101101000001001000000010111111111011001100111000011110010101100101110110100011011011111000111001101011000011101101010011100010000110101111110001001100101000011110111111001110011000011010010101101111011100011000110101101011011100010010011111010000110010000001111010101001101110111100011011001010010010111010000100111010100001101110101001001110111101001110011101001111001000011000010101101010001001000101000010100010101110101110110001001110000010110000000100101010100001000100000010100000000
    [11, 2, 0]

## Berlekamp-Massey Streaming Algorithm

An algorithm is streaming when the input is a stream of symbols. That means the input is a sequence of items and the algorithm is applied to only one item (or a group of successive items) at a time.

Since Berlekamp-Massey Algorithm may be applied to very long sequence of bits,  it may be convenient to implement it as a streaming algorithm. The streaming implementation allows for lower memory requirements as you need to store only the bits of the sequence that are significant for the identification of the LFSR polynomial.

The streaming algorithm must:
- take one bit at a time as input
- for each input bit, it must return the polynomial corresponding to the shortest LFSR that can generate the bit sequence observed up to the last input bit.

Since streaming algorithms needs an internal state to be updated at each new incoming symbol of the stream, they cannot be implemented a normal functions. We can implement it as a class.

**Template**:
```python
class BerlekampMassey():

    def __init__(self):
        # do stuff
        self.poly = ...

    def __call__(self, bit):
        # do stuff
        return self.poly
```
The special method `__call__` makes the class callable, that means you can call
it as a function.
```python
berlekamp_massey = BerlekampMassey()

for bit in bit_stream:
    poly = berlekamp_massey(bit)
```

Below is the implementation of the Berlekamp-Massey algorithm as a streaming algorithm.

Unlike the batch implementation, here the bits come one at a time and therefore the algorithm needs to internally store the bit sequence by appending each new incoming bit. We choose to store the bit sequence `bits` as an integer, and the new incoming bit `bit` is appended as least significant bit (LSB):
```python
self.bits = (self.bits << 1) | int(bit)
```

Moreover, one can notice that not all bits of the sequence are employed to determine the polynomial so we can drop them. It can be proven that the drop operation can be performed when $Q(x)$ is updated and $r$ is reset, by selection only the last $m$ bits of the sequence.
```python
self.bits &= (1 << self.m) - 1
```

```python
class BerlekampMassey():
    '''
    Berlekamp-Massey Algorithm.
    The algorithm finds the shortest LFSR for a given binary sequence.

    This class returns a function whose call method takes one bit at a time as
    input and returns the feedback polynomial of the shortest LFSR capable of
    generating the sequence of all bits received up to the last one.

    Attributes
    ----------
    poly: list of int,
        feedback polynomial expressed as list of integers corresponding to
        the degrees of the non-zero coefficients.

    Methods
    -------
    __call__(bit):
        update and return the polynomial of the shortest LFSR for the observed
        bit sequence.
    discrepancy():
        compute the discrepancy bit that is 0 (False) if the polynomial `poly`
        explain the observed bit sequence, or 1 (True) otherwise.
    '''

    def __init__(self):
        ''' class constructor '''
        # init algorithm's internal variables
        self.P, self.m = 0x1, 0
        self.Q, self.r = 0x1, 1
        # init an empty sequence of bits
        self.bits = 0 # self.bits = []
        self.tau = 0

    # ==== poly ====
    @property
    def poly(self):
        return int_to_sparse(self.P)[::-1]
    @poly.setter
    def poly(self, val):
        raise AttributeError('Denied')

    def discrepancy(self):
        b = self.bits & ((1 << (self.m + 1)) - 1)
        d = parity(self.P & b)
        return d

    def __call__(self, bit):
        '''
        Update the feedback polynomial characterizing the shortest LFSR capable
        of generating the sequence of all bits received.

        Parameters
        ----------
        bit: bool, int, or 0/1 str
            input bit

        Return
        ------
        poly: list of int,
            feedback polynomial expressed as list of integers corresponding to
            the degrees of the non-zero coefficients.
        '''
        # append
        self.bits = (self.bits << 1) | int(bit)  # self.bits.append(bit)
        if self.discrepancy():
            if 2*self.m <= self.tau: # A
                self.P, self.Q = self.P ^ (self.Q << self.r), self.P
                self.m, self.r = self.tau + 1 - self.m, 0
                self.bits &= (1<<self.m) - 1  # self.bits = self.bits[-self.m:]
            else: # B
                self.P = self.P ^ (self.Q << self.r)
        self.r += 1
        self.tau += 1

        return self.poly
```

Again, we can assess the implementation of the algorithm by generating a sequence with an LFSR featuring a primitive polynomial and check whether the algorithm is able to determine the polynomial.

```python
# poly = [3, 1, 0]
poly = [11, 2, 0]

lfsr = LFSR(poly)

berlekamp_massey = BerlekampMassey()
for bit in islice(lfsr, 2**len(lfsr)-1):
    poly = berlekamp_massey(bit)
poly
```

    [11, 2, 0]

A deeper check may be performed by trying to determine the polynomial pf the LFSR generating the sequence but trying with any possible initial state.

```python
import tqdm # very useful module to implement progress bars
```

```python
# poly = [3, 1, 0]
# poly = [9, 4, 0]
poly = [11, 2, 0]

# iter over every possible initial state for the LFSR, i.e., from 1 to 2^m-1
for state in tqdm.tqdm(range(1, 2**max(poly)-1), ncols=79):

    # create and instance of the LFSR with the initial state
    lfsr = LFSR(poly, state)

    # create an instance of the BM algorithm
    berlekamp_massey = BerlekampMassey()

    # iter over the bit sequence and apply the algorithm
    for bit in islice(lfsr, 2**max(poly)):
        poly_ = berlekamp_massey(bit)

    # check
    if poly_ != poly:
        print(f'algorithm failed for initial state 0x{state:X}.')
        break

else:
    print('algorithm succeded for all possible initial states.')
```

    100%|██████████████████████████████████████| 2046/2046 [01:30<00:00, 22.59it/s]

    algorithm succeded for all possible initial states.


# Введение

Эти интерактивные тетрадки подразумевают последовательное выполнение.
Нажмите `Shift+Enter` чтобы выполнить ячейку с кодом.

```python
print("Hello","World")
```

Для работы с символьными вычисленями будем пользоваться пакетом `sympy`:

```python
from sympy import *
```

Символы для вычислений создаются через `symbols`:

```python
x, y, t = symbols("x, y, t")
alpha, beta, gamma = symbols("alpha, beta, gamma", cls=Function)
alpha(t) + x
```

Матрицы создаются из списка списков:

```python
m = Matrix([
    [cos(alpha(t)), -sin(alpha(t)), 0],
    [sin(alpha(t)), cos(alpha(t)), 0],
    [0, 0, 1]
])
v = Matrix([
    [x],
    [y],
    [1]
])
```

```python
rotated = m * v
rotated
```

`sympy` позволяет легко дифферинцировать выражения:

```python
velocity = diff(rotated, t)
velocity[0]
```

```python
velocity[1]
```

Помимо этого, можно заменять части выражений.
Например, если
$$\alpha(t) = t$$
получим следующие скорости:

```python
velocity[0].replace(alpha(t), t).simplify()
```

```python
velocity[1].replace(alpha(t), t).simplify()
```

А если вращение происходит с постоянным угловым ускорением
$$ \alpha(t) = t^2 $$
получим следующие линейные скорости:

```python
velocity[0].replace(alpha(t), t**2).simplify()
```

```python
velocity[1].replace(alpha(t), t**2).simplify()
```


```python
import numpy as np
from math import tau
import matplotlib.pyplot as plt

from utils import perimeter_hm, perimeter_gm, perimeter_am, perimeter_rms, perimeter_parker, perimeter_bessel
from utils import find_fractional_approx
```

# First thing first

Matt, you've missed an opportunity to write your approximation as

\begin{equation}
3\tau \left( \frac{a}{5} + \frac{b}{8} \right)
\end{equation}

Here is my submission

\begin{equation}
\tau \left( \frac{7}{45} a + \frac{27}{41} \sqrt{\frac{a^2+b^2}{2}} + \frac{7}{37} b \right)
\end{equation}

# Calibration

I would like to make sure if the code for the perimeter of ellipse is correct.

```python
space = np.linspace(1, 5, 100)

hm = perimeter_hm(space)
gm = perimeter_gm(space)
am = perimeter_am(space)
rms = perimeter_rms(space)
parker = perimeter_parker(space)
bessel = perimeter_bessel(space)

plt.figure(figsize=(10, 5))
plt.plot(space, np.abs(1 - hm / bessel) * 100, "k--")
plt.plot(space, np.abs(1 - gm / bessel) * 100, "r-", label="gmean")
plt.plot(space, np.abs(1 - am / bessel) * 100, "g-", label="amean")
plt.plot(space, np.abs(1 - rms / bessel) * 100, "b-", label="rms")
plt.plot(space, np.abs(1 - parker / bessel) * 100, "k-", label="parker")
plt.legend()
```

Looks good to me as it follows the trend as in [Matt's video](https://www.youtube.com/watch?v=5nW3nJhBHL0&t=826s).

# Generalized Matt's approximation -> $\tau (x_0 a + x_1 b)$

I would like to find the best out of Matt's simple approximation. I used **numerical methods** to optimize the generalized model. The error would be computed at `np.linspace(1, 5, 1000)`.

```python
from scipy.optimize import minimize
```

```python
def perimeter_jcop(x, a, b=1):
    return tau * (x[0] * a + x[1] * b)

def error(x):
    space = np.linspace(1, 5, 1000)
    jcop = perimeter_jcop(x, space)
    bessel = perimeter_bessel(space)
    return (np.abs(1 - jcop / bessel) * 100).sum()
```

```python
x_guess = (0, 0)
bound = (-1.5, 1.5)
sol = minimize(error, x_guess, bounds=(bound, bound))
[find_fractional_approx(x) for x in sol.x]
```

    [(10, 17), (22, 59)]

Instead of using the optimal solution $\tau (0.58786245 a + 0.373148 b)$, I would also like to have a neat solution, so I computed the fraction approximation for the solution algorithmically. Hence, resulted in

\begin{equation}
\tau \left( \frac{10}{17}a + \frac{22}{59}b \right)
\end{equation}

# Let's see the error

```python
space = np.linspace(1, 5, 100)

gm = perimeter_gm(space)
am = perimeter_am(space)
rms = perimeter_rms(space)
parker = perimeter_parker(space)
bessel = perimeter_bessel(space)
jcop = perimeter_jcop((10/17, 22/59), space)

plt.figure(figsize=(10, 5))
plt.plot(space, np.abs(1 - gm / bessel) * 100, "r-", label="gmean")
plt.plot(space, np.abs(1 - am / bessel) * 100, "g-", label="amean")
plt.plot(space, np.abs(1 - rms / bessel) * 100, "b-", label="rms")
plt.plot(space, np.abs(1 - parker / bessel) * 100, "k-", label="parker")
plt.plot(space, np.abs(1 - jcop / bessel) * 100, "m-", label="jcop")
plt.legend()
```

# It's not as good at small and larger $a/b$ values

```python
space = np.linspace(1, 75, 200)

gm = perimeter_gm(space)
am = perimeter_am(space)
rms = perimeter_rms(space)
parker = perimeter_parker(space)
bessel = perimeter_bessel(space)
jcop = perimeter_jcop((10/17, 22/59), space)

plt.figure(figsize=(10, 5))
plt.plot(space, np.abs(1 - gm / bessel) * 100, "r-", label="gmean")
plt.plot(space, np.abs(1 - am / bessel) * 100, "g-", label="amean")
plt.plot(space, np.abs(1 - rms / bessel) * 100, "b-", label="rms")
plt.plot(space, np.abs(1 - parker / bessel) * 100, "k-", label="parker")
plt.plot(space, np.abs(1 - jcop / bessel) * 100, "m-", label="jcop")
plt.legend()
```

# Improvement with RMS

As we can see that out of all the mean computation (harmonic, arithmetic, geometric, and rms), rms performs the best. So let's include that in our model.

\begin{equation}
\tau \left( x_0 a + x_1 \sqrt{ \left( \frac{a^2+b^2}{2} \right) } + x_2 b \right)
\end{equation}

```python
def perimeter_jcop(x, a, b=1):
    return tau * (x[0] * a + x[1] * np.sqrt((a**2 + b**2)/2) + x[2] * b)

def error(x):
    space = np.linspace(1, 5, 1000)
    jcop = perimeter_jcop(x, space)
    bessel = perimeter_bessel(space)
    return (np.abs(1 - jcop / bessel) * 100).sum()
```

```python
x_guess = (0, 0, 0)
bound = (-1.5, 1.5)
sol = minimize(error, x_guess, bounds=(bound, bound, bound))
[find_fractional_approx(x) for x in sol.x]
```

    [(7, 45), (27, 41), (7, 37)]

I would again used the fractinal approximation just for aesthetic

```python
space = np.linspace(1, 5, 200)

gm = perimeter_gm(space)
am = perimeter_am(space)
rms = perimeter_rms(space)
parker = perimeter_parker(space)
bessel = perimeter_bessel(space)
jcop = perimeter_jcop((7/45, 27/41, 7/37), space)

plt.figure(figsize=(10, 5))
plt.plot(space, np.abs(1 - gm / bessel) * 100, "r-", label="gmean")
plt.plot(space, np.abs(1 - am / bessel) * 100, "g-", label="amean")
plt.plot(space, np.abs(1 - rms / bessel) * 100, "b-", label="rms")
plt.plot(space, np.abs(1 - parker / bessel) * 100, "k-", label="parker")
plt.plot(space, np.abs(1 - jcop / bessel) * 100, "m-", label="jcop")
plt.legend()
```

# It also worked quite well on large $a/b$

```python
space = np.linspace(1, 75, 200)

gm = perimeter_gm(space)
am = perimeter_am(space)
rms = perimeter_rms(space)
parker = perimeter_parker(space)
bessel = perimeter_bessel(space)
jcop = perimeter_jcop((7/45, 27/41, 7/37), space)

plt.figure(figsize=(10, 5))
plt.plot(space, np.abs(1 - gm / bessel) * 100, "r-", label="gmean")
plt.plot(space, np.abs(1 - am / bessel) * 100, "g-", label="amean")
plt.plot(space, np.abs(1 - rms / bessel) * 100, "b-", label="rms")
plt.plot(space, np.abs(1 - parker / bessel) * 100, "k-", label="parker")
plt.plot(space, np.abs(1 - jcop / bessel) * 100, "m-", label="jcop")
plt.legend()
```

# So this is my submission for Ellipse's perimeter approximation

\begin{equation}
C \approx \tau \left( \frac{7}{45} a + \frac{27}{41} \sqrt{\frac{a^2+b^2}{2}} + \frac{7}{37} b \right)
\end{equation}


Computational Seismology

Numerical derivatives based on the Fourier Transform

Seismo-Live: http://seismo-live.org

##### Authors:
* Fabian Linder ([@fablindner](https://github.com/fablindner))
* Heiner Igel ([@heinerigel](https://github.com/heinerigel))
* David Vargas ([@dvargas](https://github.com/davofis))
---

## Basic Equations
The derivative of function $f(x)$ with respect to the spatial coordinate $x$ is calculated using the differentiation theorem of the Fourier transform:

\begin{equation}
\frac{d}{dx} f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} ik F(k) e^{ikx} dk
\end{equation}

In general, this formulation can be extended to compute the n−th derivative of $f(x)$ by considering that $F^{(n)}(k) = D(k)^{n}F(k) = (ik)^{n}F(k)$. Next, the inverse Fourier transform is taken to return to physical space.

\begin{equation}
f^{(n)}(x) = \mathscr{F}^{-1}[(ik)^{n}F(k)] = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{\infty} (ik)^{n} F(k) e^{ikx} dk
\end{equation}

```python
# Import all necessary libraries, this is a configuration step for the exercise.
# Please run it before the simulation code!
import numpy as np
import matplotlib.pyplot as plt

# Show the plots in the Notebook.
plt.switch_backend("nbagg")
```

#### Exercise 1

Define a python function call "fourier_derivative(f, dx)" that compute the first derivative of a function $f$ using the Fourier transform properties.

```python
#################################################################
# IMPLEMENT THE FOURIER DERIVATIVE METHOD HERE!
#################################################################
```

#### Exercise 2

Calculate the numerical derivative based on the Fourier transform to show that the derivative is exact. Define an arbitrary function (e.g. a Gaussian) and initialize its analytical derivative on the same spatial grid. Calculate the numerical derivative and the difference to the analytical solution. Vary the wavenumber content of the analytical function. Does it make a difference? Why is the numerical result not entirely exact?

```python
# Basic parameters
# ---------------------------------------------------------------
nx = 128
x, dx = np.linspace(2*np.pi/nx, 2*np.pi, nx, retstep=True)
sigma = 0.5
xo = np.pi

#################################################################
# IMPLEMENT YOUR SOLUTION HERE!
#################################################################

```

#### Exercise 3

Now that the numerical derivative is available, we can visually inspect our results. Make a plot of both, the analytical and numerical derivatives together with the difference error.

```python
#################################################################
# PLOT YOUR SOLUTION HERE!
#################################################################
```


# Ecuaciones diferenciales

## Preparación

Proseguimos a activar el ambiente de trabajo y cargar los paquetes a utilizar.

```julia
import Pkg; Pkg.activate("."); Pkg.instantiate()

```

```julia
using DifferentialEquations

```

```julia
using Plots

```

La siguiente configuración por defecto se coloca para la versión específica de Julia, Plots y backend GR utilizada en construir este documento. Puede ser óptimo cambiarlo a otra para uso personal:

```julia
begin
	size = 50
	Plots.default(size = (2600,2000),titlefontsize = size, tickfontsize = size, legendfontsize = size, guidefontsize = size, legendtitlefontsize = size, lw = 5)
end

```

## Modelos de ecuaciones diferenciales ordinarias (ODEs)
### Preliminares
Gracias a la gran eficiencia de métodos iterativos y de álgebra lineal, la forma preferencial de pensar en ecuaciones diferenciales es en su forma de *sistema de ecuaciones de primer órden*. Esto siempre es posible de obtener mediante [reducción de órden](https://en.wikipedia.org/wiki/Ordinary_differential_equation##Reduction_of_order).

Una vez realizado, la forma que obtenemos, para ecuaciones autónomas (no dependientes de manera explícita del tiempo), es:

$$
x'(t) = F(x(t))
$$

donde $x: \mathbb{R} \rightarrow \mathbb{R}^n$ es una trayectoria en el espacio para la cual deseamos resolver y obedece la dinámica del campo vectorial:

$$F: \mathbb{R}^n \rightarrow \mathbb{R}^n$$

Esto define un sistema de $n$ ecuaciones a resolver. A continuación veremos algunos ejemplos.

### Ejemplo introductorio: La estructura general

Se ejemplifica el procedimiento general de resolución de ecuaciones diferenciales utilizando el paquete `DifferentialEquations` utilizando el ejemplo:

$$\frac{du}{dx} = f(u, p, t) = \alpha u$$

donde $u$ es el observable dinámico a encontrar, y la forma general $f(u, p, t)$ describe la evolución temporal de $u$ y puede depender de dicho observable: $u$, el tiempo paramétrico que describe su dinámica: $t$, y parámetros potencialmente exteriores que caracterizan el contexto específico del fenómeno (ejemplo: constantes físicas (masa, aceleración debido a gravedad, permitividad, etc.), indicadores socio-económicos (tasas constantes, natalidad, migración, etc.)).

Los pasos se presentan a continuación

#### 1. Definición de la función

```julia
f(u,p,t) = 1.01*u

```

Como descrito anteriormente, $f$ depende de $u, p, t$, pero solamente es obligatoria la dependencia de $u$ (salvo el caso trivial donde $f$ es constante y en dado caso no necesitamos resolución numérica), mientras que, en este caso, no tenemos parámetros movibles $p$ ni tampoco el tiempo $t$ explícito, aunque por supuesto $u$ se asume ser función del tiempo ($u = u(t)$).

#### 2. Definición de las condiciones iniciales y de frontera

```julia
u₀ = 1/2; tspan_u = (0.0,1.0)

```

Aquí $u_0$ es el valor inicial tomado por la variable dinámica $u$. El tiempo inicial se define en el intervalo `tspan_u`, que es una tupla de un valor inicial y un valor final ($t_0$, $t_f$).

Es este intervalo el cual será particionado en una cadena discreta de puntos $\{t_0, t_1, t_2, \ldots, t_{n-1}, t_f\}$ y obtener los valores de $u$ en dichos tiempos: $\{u(t_0), u(t_1), u(t_2), \ldots, u(t_{n-1}), u(t_f)\}$.

#### 3. Planteamiento del problema

```julia
prob_u = ODEProblem(f,u₀,tspan_u);

```

Este es un ejemplo de un problema planteado como una ecuación diferencial ordinaria de primer orden. En su lugar, pudimos haber definido una de orden superior, una estocástica, integrodiferencial, etc.

Ejemplos de estos serán mostrados luego.

#### 4. Resolución y exploración

```julia
sol_u = solve(prob_u, Tsit5(), reltol=1e-8, abstol=1e-8);

```

La función `solve` es una de las funciones más complejas (en cuanto a completitud y flexibilidad) del paquete, siendo la misma redefinida mediante multiple-dispatch para poder reaccionar sin aparente dificultad a cualquier tipo de problema planteado y permitiendo argumentos que personalicen el proceso de solución para cada una de ellas.

Como ejemplo, aquí tenemos que el problema `prob_u` se resuelve mediante un **solver** llamado `Tsit5`. La lista de solvers incluidos en el paquete puede encontrarse en su documentación [aquí](https://diffeq.sciml.ai/stable/tutorials/ode_example/##Choosing-a-Solver-Algorithm), y de éstos se habla en mayor profundidad luego.

Además, se ilustra que podemos elegir un error relativo y error absoluto que queremos aceptar para que el solver lo garantice. Leer más sobre los tipos de errores de aproximación [aquí](https://en.wikipedia.org/wiki/Approximation_error).

Posterior a resolver la ecuación diferencial, podemos explorar su comportamiento de manera gráfica:

```julia
begin
	plot(sol_u,linewidth=5, title="Solución aproximada al problema",
     xaxis="Time (t)", yaxis="u(t) (in μm)",label="Aproximación")
	plot!(sol_u.t, t->0.5*exp(1.01t),lw=3,ls=:dash,label="Solución exacta")
end

```

### Péndulo caótico
El péndulo doble es un sistema muy famosamente estudiado por exhibir un comportamiento **caótico** (cuya definición matemática es rigurosa). Sus ecuaciones del movimiento son las siguientes:

$$\frac{d}{dt}
\begin{pmatrix}
\alpha \\ l_\alpha \\ \beta \\ l_\beta
\end{pmatrix}=
\begin{pmatrix}
2\frac{l_\alpha - (1+\cos\beta)l_\beta}{3-\cos 2\beta} \\
-2\sin\alpha - \sin(\alpha + \beta) \\
2\frac{-(1+\cos\beta)l_\alpha + (3+2\cos\beta)l_\beta}{3-\cos2\beta}\\
-\sin(\alpha+\beta) - 2\sin(\beta)\frac{(l_\alpha-l_\beta)l_\beta}{3-\cos2\beta} + 2\sin(2\beta)\frac{l_\alpha^2-2(1+\cos\beta)l_\alpha l_\beta + (3+2\cos\beta)l_\beta^2}{(3-\cos2\beta)^2}
\end{pmatrix}$$

Resolveremos este sistema utilizando los métodos del paquetes `DifferentialEquations.jl`.

El movimiento generado por esta ecuaciones se ve similar a:

```julia
html"
<p align='center'>

</p>
"

```

```julia
## El prefijo const es opcional y no significa VALOR constante, si no tipo constante.
begin
	const m₁, m₂, L₁, L₂, g = 1, 2, 1, 2, 9.81
	initial = [0, π/3, 0, 3pi/5]
	tspan = (0., 50.)
end;

```

Se define una función auxiliar para transformar de coordenadas polares a cartesianas:

```julia
function polar2cart(sol; dt=0.02, l1=L₁, l2=L₂, vars=(2,4))
    u = sol.t[1]:dt:sol.t[end]
    p1 = l1*map(x->x[vars[1]], sol.(u))
    p2 = l2*map(y->y[vars[2]], sol.(u))
    x1 = l1*sin.(p1)
    y1 = l1*-cos.(p1)
    (u, (x1 + l2*sin.(p2),
     y1 - l2*cos.(p2)))
end

```

```julia
function double_pendulum(xdot,x,p,t)
    xdot[1] = x[2]
    xdot[2] = - ((g*(2*m₁+m₂)*sin(x[1]) + m₂*(g*sin(x[1]-2*x[3]) +
				2*(L₂*x[4]^2+L₁*x[2]^2*cos(x[1]-x[3]))*sin(x[1]-x[3])))/
		        (2*L₁*(m₁+m₂-m₂*cos(x[1]-x[3])^2)))
    xdot[3] = x[4]
    xdot[4] = (((m₁+m₂)*(L₁*x[2]^2+g*cos(x[1])) +
			   L₂*m₂*x[4]^2*cos(x[1]-x[3]))*sin(x[1]-x[3]))/
			   (L₂*(m₁+m₂-m₂*cos(x[1]-x[3])^2))
end

```

```julia
double_pendulum_problem = ODEProblem(double_pendulum, initial, tspan);

```

```julia
sol = solve(double_pendulum_problem, Vern7(), abs_tol=1e-10, dt=0.05)

```

```julia
begin
	ts, ps = polar2cart(sol, l1=L₁, l2=L₂, dt=0.01)
	plot(ps...)
end

```

### 3. Sistema de Hénon-Heiles
Es un sistema dinámico que modela el movimiento de una estrella orbitando alrededor de su centro galáctico mientras yace restringido en un plano. Éste es un ejemplo de un sistema Hamiltoniano, y tiene la forma:

$$
\begin{align}
\frac{d^2x}{dt^2}&=-\frac{\partial V}{\partial x}\\
\frac{d^2y}{dt^2}&=-\frac{\partial V}{\partial y}
\end{align}
$$

donde

$$V(x,y)={\frac {1}{2}}(x^{2}+y^{2})+\lambda \left(x^{2}y-{\frac {y^{3}}{3}}\right).$$

es conocido como el **potencial de Hénon–Heiles**. De éste puede derivarse su Hamiltoniano:

$$H={\frac {1}{2}}(p_{x}^{2}+p_{y}^{2})+{\frac {1}{2}}(x^{2}+y^{2})+\lambda \left(x^{2}y-{\frac {y^{3}}{3}}\right).$$

Esta cantidad representa un invariante del sistema dinámico: Una cantidad conservada. En este caso, es la energía total del sistema.

```julia
begin
	## Parámetros
	initial₂ = [0.,0.1,0.5,0]
	tspan₂ = (0,100.)
	## V: Potencial, T: Energía cinética total, E: Energía total
	V(x,y) = 1//2 * (x^2 + y^2 + 2x^2*y - 2//3 * y^3)
	E(x,y,dx,dy) = V(x,y) + 1//2 * (dx^2 + dy^2);
end;

```

Definimos el modelo en una función:

```julia
function Hénon_Heiles(du,u,p,t)
    x  = u[1]
    y  = u[2]
    dx = u[3]
    dy = u[4]
    du[1] = dx
    du[2] = dy
    du[3] = -x - 2x*y
    du[4] = y^2 - y -x^2
end

```

Resolvemos el problema:

```julia
begin
	prob₂ = ODEProblem(Hénon_Heiles, initial₂, tspan₂)
	sol₂ = solve(prob₂, Vern9(), abs_tol=1e-16, rel_tol=1e-16);
end

```

```julia
plot(sol₂, vars=(1,2), title = "Órbita del sistema de Hénon-Heiles", xaxis = "x", yaxis = "y", leg=false)

```

Parece estar correctamente resuelto pero... examinando la evolución de la energía total/Hamiltoniano podemos encontrar lo siguiente:

```julia
begin
	energy = map(x->E(x...), sol₂.u)
	plot(sol₂.t, energy .- energy[1], title = "Cambio de la energía en el tiempo", xaxis = "Número de iteraciones", yaxis = "Cambio en energía")
end

```

¡La energía total cambia! Eso quiere decir que la ley de conservación que esperamos que se cumpla físicamente no parece cumplirse en nuestra simulación. Esto delata un error en la resolución de la ecuación, específicamente por el método utilizado.

Para evitar eso, podemos utilizar algo conocido como un **integrador simplético** que considere la estructura de sistema Hamiltoniano que tiene nuestras ecuaciones.

Éste lo implementamos a continuación:

```julia
function HH_acceleration!(dv,v,u,p,t)
    x,y  = u
    dx,dy = dv
    dv[1] = -x - 2x*y
    dv[2] = y^2 - y -x^2
end;

```

Debemos ahora definir la condición inicial por separado, pues nuestro sistema, al ser Hamiltoniano, tiene una segmentación natural en esos pares de variables.

```julia
begin
	initial_positions = [0.0,0.1]
	initial_velocities = [0.5,0.0]
end;

```

Resolvemos el problema ahora como una ecuación de segundo orden pero con estructura simpléctica detectada:

```julia
begin
	prob₃ = SecondOrderODEProblem(HH_acceleration!,
								 initial_velocities,
								 initial_positions,
								 tspan₂)
	sol₃ = solve(prob₃, KahanLi8(), dt=1/10)
end;

```

Ahora podemos observar cómo la energía se mantiene muy cercana a cero, oscilando solamente por errores de precisión numérica pero sin existir una tendencia de crecimiento anómala.

```julia
begin
	energy₂ = map(x->E(x[3], x[4], x[1], x[2]), sol₃.u)
	plot(sol₃.t, energy₂ .- energy₂[1], title = "Cambio de la energía en el tiempo", xaxis = "Número de iteraciones", yaxis = "Cambio en energía")
end

```

## Ecuaciones diferenciales Estocásticas

## Bibliografía
* [Documentación de SciML](https://sciml.ai/)
* [Tutoriales de SciML](https://github.com/SciML/SciMLTutorials.jl)
* Hénon, Michel (1983), \"Numerical exploration of Hamiltonian Systems\", in Iooss, G. (ed.), Chaotic Behaviour of Deterministic Systems, Elsevier Science Ltd, pp. 53–170, ISBN 044486542X
* Aguirre, Jacobo; Vallejo, Juan C.; Sanjuán, Miguel A. F. (2001-11-27). \"Wada basins and chaotic invariant sets in the Hénon-Heiles system\". Physical Review E. American Physical Society (APS). 64 (6): 066208. doi:10.1103/physreve.64.066208. ISSN 1063-651X.
* Steven Johnson. 18.335J Introduction to Numerical Methods . Spring 2019. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.


# Fitting a Mixture Model with Gibbs Sampling

```python
%matplotlib inline
import pandas as pd
import numpy as np
import random
import matplotlib.pyplot as plt

from scipy import stats
from collections import namedtuple, Counter
```

Suppose we receive some data that looks like the following:

```python
data = pd.Series.from_csv("clusters.csv")
_=data.hist(bins=20)
```

```python
data.size
```

    1000

It appears that these data exist in three separate clusters. We want to develop a method for finding these _latent_ clusters. One way to start developing a method is to attempt to describe the process that may have generated these data.

For simplicity and sanity, let's assume that each data point is generated independently of the other. Moreover, we will assume that within each cluster, the data points are identically distributed. In this case, we will assume each cluster is normally distributed and that each cluster has the same variance, $\sigma^2$.

Given these assumptions, our data could have been generated by the following process. For each data point, randomly select 1 of 3 clusters from the distribution $\text{Discrete}(\pi_1, \pi_2, \pi_3)$. Each cluster $k$ corresponds to a parameter $\theta_k$ for that cluster, sample a data point from $\mathcal{N}(\theta_k, \sigma^2)$.

Equivalently, we could consider these data to be generated from a probability distribution with this probability density function:

$$
p(x_i \,|\, \pi, \theta_1, \theta_2, \theta_3, \sigma)=
    \sum_{k=1}^3 \pi_k\cdot
        \frac{1}{\sigma\sqrt{2\pi}}
        \text{exp}\left\{
            \frac{-(x_i-\theta_k)^2}{2\sigma^2}
         \right\}
$$

where $\pi$ is a 3-dimensional vector giving the _mixing proportions_. In other words, $\pi_k$ describes the proportion of points that occur in cluster $k$.

That is, _the probability distribution describing $x$ is a linear combination of normal distributions_.

We want to use this _generative_ model to formulate an algorithm for determining the particular parameters that generated the dataset above. The $\pi$ vector is unknown to us, as is each cluster mean $\theta_k$.

We would also like to know $z_i\in\{1, 2, 3\}$, the latent cluster for each point. It turns out that introducing $z_i$ into our model will help us solve for the other values.

The joint distribution of our observed data (`data`) along with the assignment variables is given by:

\begin{align}
p(\mathbf{x}, \mathbf{z} \,|\, \pi, \theta_1, \theta_2, \theta_3, \sigma)&=
       p(\mathbf{z} \,|\, \pi)
       p(\mathbf{x} \,|\, \mathbf{z}, \theta_1, \theta_2, \theta_3, \sigma)\\
       &= \prod_{i=1}^N p(z_i \,|\, \pi)
           \prod_{i=1}^N p(x_i \,|\, z_i, \theta_1, \theta_2, \theta_3, \sigma) \\
       &= \prod_{i=1}^N \pi_{z_i}
           \prod_{i=1}^N
           \frac{1}{\sigma\sqrt{2\pi}}
        \text{exp}\left\{
            \frac{-(x_i-\theta_{z_i})^2}{2\sigma^2}
         \right\}\\
       &= \prod_{i=1}^N
           \left(
           \pi_{z_i}
           \frac{1}{\sigma\sqrt{2\pi}}
        \text{exp}\left\{
            \frac{-(x_i-\theta_{z_i})^2}{2\sigma^2}
         \right\}
         \right)\\
       &=
           \prod_i^n
           \prod_k^K
           \left(
               \pi_k
               \frac{1}{\sigma\sqrt{2\pi}}
        \text{exp}\left\{
            \frac{-(x_i-\theta_k)^2}{2\sigma^2}
         \right\}
           \right)^{\delta(z_i, k)}
\end{align}

### Keeping Everything Straight

Before moving on, we need to devise a way to keep all our data and parameters straight. Following ideas suggested by [Keith Bonawitz](http://people.csail.mit.edu/bonawitz/Composable%20Probabilistic%20Inference%20with%20Blaise%20-%20Keith%20Bonawitz%20PhD%20Thesis.pdf), let's define a "state" object to store all of this data.

It won't yet be clear why we are defining some components of `state`, however we will use each part eventually! As an attempt at clarity, I am using a trailing underscore in the names of members that are fixed. We will update the other parameters as we try to fit the model.

```python
SuffStat = namedtuple('SuffStat', 'theta N')

def update_suffstats(state):
    for cluster_id, N in Counter(state['assignment']).iteritems():
        points_in_cluster = [x
            for x, cid in zip(state['data_'], state['assignment'])
            if cid == cluster_id
        ]
        mean = np.array(points_in_cluster).mean()

        state['suffstats'][cluster_id] = SuffStat(mean, N)

def initial_state():
    num_clusters = 3
    alpha = 1.0
    cluster_ids = range(num_clusters)

    state = {
        'cluster_ids_': cluster_ids,
        'data_': data,
        'num_clusters_': num_clusters,
        'cluster_variance_': .01,
        'alpha_': alpha,
        'hyperparameters_': {
            "mean": 0,
            "variance": 1,
        },
        'suffstats': [None, None, None],
        'assignment': [random.choice(cluster_ids) for _ in data],
        'pi': [alpha / num_clusters for _ in cluster_ids],
        'cluster_means': [-1, 0, 1]
    }
    update_suffstats(state)
    return state

state = initial_state()
```

```python
for k, v in state.items():
    print(k)
```

    num_clusters_
    suffstats
    data_
    cluster_means
    cluster_variance_
    cluster_ids_
    assignment
    pi
    alpha_
    hyperparameters_

### Gibbs Sampling

The [theory of Gibbs sampling](https://en.wikipedia.org/wiki/Gibbs_sampling) tells us that given some data $\bf y$ and a probability distribution $p$ parameterized by $\gamma_1, \ldots, \gamma_d$, we can successively draw samples from the distribution by sampling from

$$\gamma_j^{(t)}\sim p(\gamma_j \,|\, \gamma_{\neg j}^{(t-1)})$$

where $\gamma_{\neg j}^{(t-1)}$ is all current values of $\gamma_i$ except for $\gamma_j$. If we sample long enough, these $\gamma_j$ values will be random samples from $p$.

In deriving a Gibbs sampler, it is often helpful to observe that

$$
    p(\gamma_j \,|\, \gamma_{\neg j})
        = \frac{
            p(\gamma_1,\ldots,\gamma_d)
        }{
            p(\gamma_{\neg j})
        } \propto p(\gamma_1,\ldots,\gamma_d).
$$

The conditional distribution is proportional to the joint distribution. We will get a lot of mileage from this simple observation by dropping constant terms from the joint distribution (relative to the parameters we are conditioned on).

The $\gamma$ values in our model are each of the $\theta_k$ values, the $z_i$ values, and the $\pi_k$ values. Thus, we need to derive the conditional distributions for each of these.

Many derivation of Gibbs samplers that I have seen rely on a lot of handwaving and casual appeals to conjugacy. I have tried to add more mathematical details here. I would gladly accept feedback on how to more clearly present the derivations! I have also tried to make the derivations more concrete by immediately providing code to do the computations in this specific case.

#### Conditional Distribution of Assignment

For berevity, we will use

$$
p(z_i=k \,|\, \cdot)=
p(z_i=k \,|\,
        z_{\neg i}, \pi,
        \theta_1, \theta_2, \theta_3, \sigma, \bf x
        ).
        $$

Because cluster assignements are conditionally independent given the cluster weights and paramters,

\begin{align}
    p(z_i=k \,|\, \cdot)
        &\propto
           \prod_i^n
           \prod_k^K
           \left(
               \pi_k
               \frac{1}{\sigma\sqrt{2\pi}}
        \text{exp}\left\{
            \frac{-(x_i-\theta_k)^2}{2\sigma^2}
         \right\}
           \right)^{\delta(z_i, k)} \\
        &\propto
            \pi_k \cdot
            \frac{1}{\sigma\sqrt{2\pi}}
            \text{exp}\left\{
                \frac{-(x_i-\theta_k)^2}{2\sigma^2}
             \right\}
\end{align}

This equation intuitively makes sense: point $i$ is more likely to be in cluster $k$ if $k$ is itself probable ($\pi_k\gg 0$) and $x_i$ is close to the mean of the cluster $\theta_k$.

For each data point $i$, we can compute $p(z_i=k \,|\, \cdot)$ for each of cluster $k$. These values are the unnormalized parameters to a discrete distribution from which we can sample assignments.

Below, we define functions for doing this sampling. `sample_assignment` will generate a sample from the posterior assignment distribution for the specified data point. `update_assignment` will sample from the posterior assignment for each data point and update the `state` object.

```python
def log_assignment_score(data_id, cluster_id, state):
    """log p(z_i=k \,|\, \cdot)

    We compute these scores in log space for numerical stability.
    """
    x = state['data_'][data_id]
    theta = state['cluster_means'][cluster_id]
    var = state['cluster_variance_']
    log_pi = np.log(state['pi'][cluster_id])
    return log_pi + stats.norm.logpdf(x, theta, var)

def assigment_probs(data_id, state):
    """p(z_i=cid \,|\, \cdot) for cid in cluster_ids
    """
    scores = [log_assignment_score(data_id, cid, state) for cid in state['cluster_ids_']]
    scores = np.exp(np.array(scores))
    return scores / scores.sum()

def sample_assignment(data_id, state):
    """Sample cluster assignment for data_id given current state

    cf Step 1 of Algorithm 2.1 in Sudderth 2006
    """
    p = assigment_probs(data_id, state)
    return np.random.choice(state['cluster_ids_'], p=p)

def update_assignment(state):
    """Update cluster assignment for each data point given current state

    cf Step 1 of Algorithm 2.1 in Sudderth 2006
    """
    for data_id, x in enumerate(state['data_']):
        state['assignment'][data_id] = sample_assignment(data_id, state)
    update_suffstats(state)
```

#### Conditional Distribution of Mixture Weights

We can similarly derive the conditional distributions of mixture weights by an application of Bayes theorem. Instead of updating each component of $\pi$ separately, we update them together (this is called blocked Gibbs).

\begin{align}
p(\pi \,|\, \cdot)&=
p(\pi \,|\,
        \bf{z},
        \theta_1, \theta_2, \theta_3,
        \sigma, \mathbf{x}, \alpha
        )\\
&\propto
p(\pi \,|\,
        \mathbf{x},
        \theta_1, \theta_2, \theta_3,
        \sigma, \alpha
        )
p(\bf{z}\ \,|\,
        \mathbf{x},
        \theta_1, \theta_2, \theta_3,
        \sigma, \pi, \alpha
        )\\
&=
p(\pi \,|\,
        \alpha
        )
p(\bf{z}\ \,|\,
        \mathbf{x},
        \theta_1, \theta_2, \theta_3,
        \sigma, \pi, \alpha
        )\\
&=
\prod_{i=1}^K \pi_k^{\alpha/K - 1}
\prod_{i=1}^K \pi_k^{\sum_{i=1}^N \delta(z_i, k)} \\
&=\prod_{k=1}^3 \pi_k^{\alpha/K+\sum_{i=1}^N \delta(z_i, k)-1}\\
&\propto \text{Dir}\left(
    \sum_{i=1}^N \delta(z_i, 1)+\alpha/K,
    \sum_{i=1}^N \delta(z_i, 2)+\alpha/K,
    \sum_{i=1}^N \delta(z_i, 3)+\alpha/K
    \right)
\end{align}

Here are Python functions to sample from the mixture weights given the current `state` and to update the mixture weights in the `state` object.

```python
def sample_mixture_weights(state):
    """Sample new mixture weights from current state according to
    a Dirichlet distribution

    cf Step 2 of Algorithm 2.1 in Sudderth 2006
    """
    ss = state['suffstats']
    alpha = [ss[cid].N + state['alpha_'] / state['num_clusters_']
             for cid in state['cluster_ids_']]
    return stats.dirichlet(alpha).rvs(size=1).flatten()

def update_mixture_weights(state):
    """Update state with new mixture weights from current state
    sampled according to a Dirichlet distribution

    cf Step 2 of Algorithm 2.1 in Sudderth 2006
    """
    state['pi'] = sample_mixture_weights(state)
```

#### Conditional Distribution of Cluster Means

Finally, we need to compute the conditional distribution for the cluster means.

We assume the unknown cluster means are distributed according to a normal distribution with hyperparameter mean $\lambda_1$ and variance $\lambda_2^2$. The final step in this derivation comes from the normal-normal conjugacy. For more information see [section 2.3 of this](http://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf) and [section 6.2 this](https://web.archive.org/web/20160304125731/http://fisher.osu.edu/~schroeder.9/AMIS900/ech6.pdf).)

\begin{align}
p(\theta_k \,|\, \cdot)&=
p(\theta_k \,|\,
        \bf{z}, \pi,
        \theta_{\neg k},
        \sigma, \bf x, \lambda_1, \lambda_2
        ) \\
&\propto p(\left\{x_i \,|\, z_i=k\right\} \,|\, \bf{z}, \pi,
        \theta_1, \theta_2, \theta_3,
        \sigma, \lambda_1, \lambda_2) \cdot\\
    &\phantom{==}p(\theta_k \,|\, \bf{z}, \pi,
        \theta_{\neg k},
        \sigma, \lambda_1, \lambda_2)\\
&\propto p(\left\{x_i \,|\, z_i=k\right\} \,|\, \mathbf{z},
        \theta_k, \sigma)
        p(\theta_k \,|\,  \lambda_1, \lambda_2)\\
&= \mathcal{N}(\theta_k \,|\, \mu_n, \sigma_n)\\
\end{align}

$$ \sigma_n^2 = \frac{1}{
    \frac{1}{\lambda_2^2} + \frac{N_k}{\sigma^2}
    } $$

and

$$\mu_n = \sigma_n^2
    \left(
        \frac{\lambda_1}{\lambda_2^2} +
        \frac{n\bar{x_k}}{\sigma^2}
    \right)
$$

Here is the code for sampling those means and for updating our state accordingly.

```python
def sample_cluster_mean(cluster_id, state):
    cluster_var = state['cluster_variance_']
    hp_mean = state['hyperparameters_']['mean']
    hp_var = state['hyperparameters_']['variance']
    ss = state['suffstats'][cluster_id]

    numerator = hp_mean / hp_var + ss.theta * ss.N / cluster_var
    denominator = (1.0 / hp_var + ss.N / cluster_var)
    posterior_mu = numerator / denominator
    posterior_var = 1.0 / denominator

    return stats.norm(posterior_mu, np.sqrt(posterior_var)).rvs()

def update_cluster_means(state):
    state['cluster_means'] = [sample_cluster_mean(cid, state)
                              for cid in state['cluster_ids_']]
```

Doing each of these three updates in sequence makes a complete _Gibbs step_ for our mixture model. Here is a function to do that:

```python
def gibbs_step(state):
    update_assignment(state)
    update_mixture_weights(state)
    update_cluster_means(state)
```

Initially, we assigned each data point to a random cluster. We can see this by plotting a histogram of each cluster.

```python
def plot_clusters(state):
    gby = pd.DataFrame({
            'data': state['data_'],
            'assignment': state['assignment']}
        ).groupby(by='assignment')['data']
    hist_data = [gby.get_group(cid).tolist()
                 for cid in gby.groups.keys()]
    plt.hist(hist_data,
             bins=20,
             histtype='stepfilled', alpha=.5 )

plot_clusters(state)
```

Each time we run `gibbs_step`, our `state` is updated with newly sampled assignments. Look what happens to our histogram after 5 steps:

```python
for _ in range(5):
    gibbs_step(state)
plot_clusters(state)
```

Suddenly, we are seeing clusters that appear very similar to what we would intuitively expect: three Gaussian clusters.

Another way to see the progress made by the Gibbs sampler is to plot the change in the model's log-likelihood after each step. The log likehlihood is given by:

$$
\log p(\mathbf{x} \,|\, \pi, \theta_1, \theta_2, \theta_3)
\propto \sum_x \log \left(
    \sum_{k=1}^3 \pi_k \exp
    \left\{
        -(x-\theta_k)^2 / (2\sigma^2)
    \right\}
\right)
$$

We can define this as a function of our `state` object:

```python
def log_likelihood(state):
    """Data log-likeliehood

    Equation 2.153 in Sudderth
    """

    ll = 0
    for x in state['data_']:
        pi = state['pi']
        mean = state['cluster_means']
        sd = np.sqrt(state['cluster_variance_'])
        ll += np.log(np.dot(pi, stats.norm(mean, sd).pdf(x)))
    return ll
```

```python
state = initial_state()
ll = [log_likelihood(state)]
for _ in range(20):
    gibbs_step(state)
    ll.append(log_likelihood(state))
pd.Series(ll).plot()
```

See that the log likelihood improves with iterations of the Gibbs sampler. This is what we should expect: the Gibbs sampler finds state configurations that make the data we have seem "likely". However, the likelihood isn't strictly monotonic: it jitters up and down. Though it behaves similarly, the Gibbs sampler isn't optimizing the likelihood function. In its steady state, it is sampling from the posterior distribution. The `state` after each step of the Gibbs sampler is a sample from the posterior.

```python
pd.Series(ll).plot(ylim=[-150, -100])
```

[In another post](/collapsed-gibbs/), I show how we can "collapse" the Gibbs sampler and sampling the assignment parameter without sampling the $\pi$ and $\theta$ values. This collapsed sampler can also be extended to the model with a Dirichet process prior that allows the number of clusters to be a parameter fit by the model.

## Notation Helper

* $N_k$, `state['suffstat'][k].N`: Number of points in cluster $k$.

* $\theta_k$, `state['suffstat'][k].theta`: Mean of cluster $k$.
* $\lambda_1$, `state['hyperparameters_']['mean']`: Mean of prior distribution over cluster means.
* $\lambda_2^2$, `state['hyperparameters_']['variance']` Variance of prior distribution over cluster means.
* $\sigma^2$, `state[cluster_variance_]`: Known, fixed variance of clusters.

The superscript $(t)$ on $\theta_k$, $pi_k$, and $z_i$ indicates the value of that variable at step $t$ of the Gibbs sampler.


# Conditional Probability - Regions & Wealth

> This document is written in *R*.
>
> ***GitHub***: https://github.com/czs108

## Background

> The following table gives the *mean wealth* and *percentage population* in various *regions* of a fictitious country.
>
> |   Region   | Ashire | Bshire | Cshire | West Toptown | East Toptown | South Toptown | North Toptown |
> | :--------: | :----: | :----: | :----: | :----------: | :----------: | :-----------: | :-----------: |
> | **Wealth** |   80   |  110   |  110   |      70      |     120      |      90       |      110      |
> |  **Pop**   |  15%   |  20%   |  20%   |     10%      |     10%      |      10%      |      15%      |

```R
wealth.all <- c(80, 110, 110, 70, 120, 90, 110)
probs.all <- c(0.15, 0.20, 0.20, 0.10, 0.10, 0.10, 0.15)
```

## Question A

> What are the *mean* and *variance* of the *wealth* for the *whole* country?

```R
mean.all <- sum(wealth.all * probs.all)
var.all <- sum(((wealth.all - mean.all) ^ 2) * probs.all)

print(sprintf("Mean = %.1f", mean.all), quote=FALSE)
print(sprintf("Variance = %.1f", var.all), quote=FALSE)
```

    [1] Mean = 100.5
    [1] Variance = 254.8

## Question B

> What are the *mean* and *variance* of the *wealth* for those who live in *Toptown*?

Make the probabilities add up to **1**.

```R
probs.tp <- probs.all[4:7]
probs.tp <- probs.tp / sum(probs.tp)

sum(probs.tp)
```

1

```R
wealth.tp <- wealth.all[4:7]

mean.tp <- sum(wealth.tp * probs.tp)
var.tp <- sum(((wealth.tp - mean.tp) ^ 2) * probs.tp)

print(sprintf("Mean = %.1f", mean.tp), quote=FALSE)
print(sprintf("Variance = %.1f", var.tp), quote=FALSE)
```

    [1] Mean = 98.9
    [1] Variance = 343.2

## Question C

> What is the probability of living in an area with $\text{mean wealth} > 100$ if you live in *Toptown*?

\begin{equation}
\begin{split}
P &= \frac{P(live\ in\ Toptown\, \cap\, wealth > 100)}{P(live\ in\ Toptown)} \\
    &= \frac{10\% + 15\%}{10\% + 10\% + 10\% + 15\%} \\
    &= \frac{25\%}{45\%}
\end{split}
\end{equation}

## Question D

> What is the probability that you live in *Toptown* given that you live in an area with $\text{mean wealth} > 100$?

\begin{equation}
\begin{split}
P &= \frac{P(live\ in\ Toptown\, \cap\, wealth > 100)}{P(wealth > 100)} \\
    &= \frac{10\% + 15\%}{20\% + 20\% + 10\% + 15\%} \\
    &= \frac{25\%}{65\%}
\end{split}
\end{equation}

exercises/Conditional Probability - Regions & Wealth.ipynb


# COURSE: Master math by coding in Python
## SECTION: Graphing

#### https://www.udemy.com/course/math-with-python/?couponCode=MXC-DISC4ALL
#### INSTRUCTOR: sincxpress.com

Note about this code: Each video in this section of the course corresponds to a section of code below. Please note that this code roughly matches the code shown in the live recording, but is not exactly the same -- the variable names, order of lines, and parameters may be slightly different.

```python
# import required packages at the top of the script!
import sympy as sym
import numpy as np
from IPython.display import display, Math
import matplotlib.pyplot as plt
```

# VIDEO: Plotting coordinates on a plane

```python
x = 3
y = 5

# basic plotting a red dot
plt.plot(x,y,'ro')

# set axis limits
plt.axis('square') # order matters
plt.axis([-6,6,-6,6])
plt.grid()

plt.show()
```

```python
# a set of coordinates

x = [-4,2,5,6,2,-5]
y = [5,2,10,-5,4,0]

for i in range(0,len(x)):
    plt.plot(x[i],y[i],'o',label='point %s'%i)

plt.legend()
plt.axis('square')
plt.grid()
plt.show()
```

```python
# getting information from axes

plt.plot(4,3,'rs')

# get an object for the current axis
axis = plt.gca()
ylim = axis.get_ylim()
print(ylim)

# now change only the upper y-axis limit
axis.set_ylim([ ylim[0],6 ])

plt.xlabel('X axis')
plt.ylabel('F(x)')

plt.show()
```

### Exercise

```python
# define a function and then subs
import sympy as sym

x = sym.symbols('x')
y = x**2 - 3*x

xrange = range(-10,11)

for i in range(0,len(xrange)):
    plt.plot(xrange[i],y.subs({x:xrange[i]}),'o')

plt.xlabel('x')
plt.ylabel('$f(x) = %s$' %sym.latex(y))
plt.show()
```

# VIDEO: Graphing lines

```python
# drawing lines

p1 = [-3,-1]
p2 = [4,4]

# nice try, but wrong code :(
plt.plot(p1,p2)
plt.plot([p1[0],p2[0]],[p1[1],p2[1]],color=[.7,.3,.8],linewidth=5)

plt.axis('square')
plt.axis([-6,6,-6,6])
plt.show()
```

```python
x = 3
y = 5

# basic plotting a red dot
plt.plot(x,y,'ro')
plt.plot([0,x],[0,y],'r')

# set axis limits
plt.axis('square') # order matters
plt.axis([-6,6,-6,6])
plt.grid()

# now add lines
plt.plot([-6,6],[0,0],'k')
plt.plot([0,0],[-6,6],'k')

plt.show()
```

### Exercises

```python
x = range(-20,20)

for i in range(0,len(x)):
    plt.plot([0,x[i]],[0,abs(x[i])**(1/2)])

plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

```python
# draw a square
plt.plot([0,2],[2,2],'r')
plt.plot([0,2],[0,0],'k')
plt.plot([0,0],[0,2],'g')
plt.plot([2,2],[0,2],'m')

plt.axis('square')
plt.axis([-3,5,-3,5])
plt.show()

```

# VIDEO: Linear equations in slope-intercept form

```python
# y = mx + b

x = [-5,5]
m = 2
b = 1

# next line doesn't work; solution comes later!
#y = m*x+b

# for now, this way
y = [0,0]
for i in range(0,len(x)):
    y[i] = m*x[i] + b

plt.plot(x,y,label='y=%sx+%s' %(m,b))
plt.axis('square')
plt.xlim(x)
plt.ylim(x)
plt.grid()
axis = plt.gca()
plt.plot(axis.get_xlim(),[0,0],'k--')
plt.plot([0,0],axis.get_ylim(),'k--')
plt.legend()
plt.title('The plot.')

plt.show()

```

```python
import numpy as np

# converting x into a numpy array
y = m*np.array(x) + b

plt.plot(x,y,label='y=%sx+%s' %(m,b))
plt.axis('square')
plt.xlim(x)
plt.ylim(x)
plt.grid()
axis = plt.gca()
plt.plot(axis.get_xlim(),[0,0],'k--')
plt.plot([0,0],axis.get_ylim(),'k--')
plt.legend()
plt.title('The plot.')

plt.show()
```

```python
print(type(x))
print(type(np.array(x)))
```

### Exercise

```python
# plot these two lines
import numpy as np

x = [-5,5]
m = [.7,-5/4]
b = [-2,3/4]

for i in range(0,len(x)):
    y = m[i]*np.array(x) + b[i]
    plt.plot(x,y,label='y=%sx+%s' %(m[i],b[i]))

plt.axis('square')
plt.xlim(x)
plt.ylim(x)
plt.grid()
plt.xlabel('x')
plt.ylabel('y')
axis = plt.gca()
plt.plot(axis.get_xlim(),[0,0],'k--')
plt.plot([0,0],axis.get_ylim(),'k--')
plt.legend(prop={'size':15})
plt.title('The plot.')

plt.show()

```

# VIDEO: Graphing rational functions

```python
import numpy as np

x = range(-3,4)
y = np.zeros(len(x))

for i in range(0,len(x)):
    y[i] = 2 - x[i]**2

plt.plot(x,y,'s-')
plt.xlabel('x'), plt.ylabel('y')
plt.show()
```

```python
# what if you want more spacing?

x = np.linspace(-3,4,14)
y = 2 + np.sqrt(abs(x))

plt.plot(x,y,'s-')
plt.show()
```

### Exercise

```python
import numpy as np

e = range(-1,4)
x = np.linspace(-4,4,300)

for i in e:
    y = x**i
    plt.plot(x,y,label='$y=x^{%s}$'%i,linewidth=4)

plt.legend()
plt.ylim([-20,20])
plt.xlim([x[0],x[-1]])
plt.xlabel('x')
plt.ylabel('y')
plt.show()
```

# VIDEO: Plotting functions with sympy

```python
# create symbolic variables
from sympy.abc import x

# define function
y = x**2

# plotting function in sympy
p = sym.plotting.plot(y) #(x,y)

# trying to adjust the y-axis limits
p.ylim = [0,50] # ...but it doesn't work :(
```

```python
# to set features of the plot, turn the plotting off, then make adjustments, then show the plot

# create a plot object
p = sym.plotting.plot(y,show=False)

# change the y-axis of the entire plot
p.xlim = (0,50)

# change a feature of only the first plot object (the line, in this case there is only one)
p[0].line_color = 'm'
p.title = 'This is a nice-looking plot!'

# now show the line
p.show()
```

```python
# This code shows how to use expressions with parameters
# and also how to plot multiple lines in the same plot

x,a = sym.symbols('x,a')

# a convenient way to import the plot module
import sympy.plotting.plot as symplot

# the basic expression with parameters
expr = a/x

# generate the first plot
p = symplot(expr.subs(a,1),(x,-5,5),show=False)
p[0].label = 'y = %s'%expr.subs(a,1) # create a label for the legend

# extend to show the second plot as well
p.extend( symplot(expr.subs(a,3),show=False) )
p[1].label = 'y = %s'%expr.subs(a,3)

# some plotting adjustments
p.ylim = [-5,5]
p[0].line_color = 'r'
p.legend = True # activate the legend

# and show the plot
p.show()

```

### Exercise

```python
# create variables
x,a = sym.symbols('x,a')

# define function
y = a/(x**2-a)

# reset and initialize the plot function
p = None
p = sym.plotting.plot(y.subs(a,1),(x,-5,5),show=False )
p[0].label = '$%s$'%sym.latex(y.subs(a,1))

# loop over values of a
for i in range(2,5):
    p.extend( sym.plotting.plot(y.subs(a,i),(x,-5,5),show=False ) )
    p[i-1].line_color = list(np.random.rand(3))
    p[i-1].label = '$%s$'%sym.latex(y.subs(a,i))

# a bit of touching up and show the plot
p.ylim = [-10,10]
p.legend = True
p.show()
```

# VIDEO: Making pictures from matrices

```python
# create a matrix
A = [ [1,2],[1,4] ]

# show it (yikes! many functions!)
display(Math(sym.latex(sym.sympify(np.array(A)))))

# now image it
plt.imshow(A)

plt.xticks([0,1])
plt.yticks([.85,1])

plt.show()
```

```python
A = np.zeros((10,14))

print( np.shape(A) )

for i in range(0,np.shape(A)[0]):
    for j in range(0,np.shape(A)[1]):

        # populate the matrix
        A[i,j] = 3*i-4*j

print(A)
plt.imshow(A)
plt.plot([0,3],[8,2],'r',linewidth=4)
plt.set_cmap('Purples')

for i in range(0,np.shape(A)[0]):
    for j in range(0,np.shape(A)[1]):
        plt.text(j,i,int(A[i,j]),horizontalalignment='center',verticalalignment='center')

plt.show()
```

### Exercise

```python
# make a checkerboard

C = np.zeros((10,10))

for i in range(0,10):
    for j in range(0,10):
        C[i,j] = (-1)**(i+j)

plt.imshow(C)
plt.set_cmap('gray')
plt.tick_params(labelleft=False,labelbottom=False)
plt.show()
```

# VIDEO: Drawing patches with polygons

```python
from matplotlib.patches import Polygon

x = np.linspace(0,1,100)
y = np.array([ [1,1],[2,3],[3,1] ])
p = Polygon(y,facecolor='m',alpha=.3)

# extend with two polygons
y1 = np.array([ [2,2],[2.5,4],[3.5,1] ])
p1 = Polygon(y1,alpha=.2,edgecolor='k')

fig, ax = plt.subplots()
ax.add_patch(p1)
ax.add_patch(p)
ax.set_ylim([0,4])
ax.set_xlim([0,4])
plt.show()
```

```python
x = np.linspace(-2,2,101)
f = -x**2

y = np.vstack((x,f)).T
p = Polygon(y,facecolor='g',alpha=.2,edgecolor='k')

p1 = Polygon(np.array([ [-.5,-4],[-.5,-2.5],[.5,-2.5],[.5,-4] ]),facecolor='k')

fig, ax = plt.subplots()
ax.add_patch(p)
ax.add_patch(p1)

plt.plot(x,f,'k')
plt.plot(x[[0,-1]],[-4,-4],'k')
plt.show()
```

# VIDEO: Exporting graphics as pictures

```python
C = np.zeros((10,10))

for i in range(0,10):
    for j in range(0,10):
        C[i,j] = (-1)**(i+j)

plt.imshow(C)
plt.set_cmap('gray')
plt.tick_params(axis='both',labelleft=False,labelbottom=False)

# save the figure!
plt.savefig('NiceFigure.png')
plt.show() # make sure this line comes after, not before, the savefig function call

```

# VIDEO: Graphing bug hunt!

```python
plt.plot(3,2,'ro')

# set axis limits
plt.axis('square')
plt.axis([-6,6,-6,6])
plt.show()
```

```python
# plot a line
plt.plot([0,3],[0,5])
plt.show()
```

```python
import numpy as np

x = range(-3,4)
y = np.zeros(len(x))

for i in range(0,len(x)):
    y[i] = 2 - x[i]**2

plt.plot(x,y,'s-')
plt.show()
```

```python
# plot two lines
plt.plot([-2,3],[4,0],'b',label='line 1')
plt.plot([0,3],[-3,3],'r',label='line 2')

plt.legend()
plt.show()
```

```python
randmat = np.random.randn(5,9)

# draw a line from lower-left corner to upper-right corner
plt.plot([8,0],[0,4],color=(.4,.1,.9),linewidth=5)

plt.imshow(randmat)
plt.set_cmap('Purples')
plt.show()
```

```python
# plot two lines
plt.plot([-2,3],[4,0],'b',label='line1')
plt.plot([0,3],[-3,3],'r',label='line2')

plt.legend(['line 1','line 2'])
plt.show()
```

```python
x = np.linspace(1,4,20)
y = x**2/(x-2)

plt.plot(x,y)

# adjust the x-axis limits according to the first and last points in x
plt.xlim(x[[0,-1]])

plt.show()
```

```python
x = sym.symbols('x')
y = x**2 - 3*x

xrange = range(-10,11)

for i in range(0,len(xrange)):
    plt.plot(xrange[i],y.subs(x,xrange[i]),'o')

plt.xlabel('x')
plt.ylabel('$f(x) = %s$' %sym.latex(y))
plt.show()
```

```python
x = [-5,5]
m = 2
b = 1

y = m*np.array(x)+b

plt.plot(x,y)
plt.show()
```

```python
x = range(-20,21)

for i in range(0,len(x)):
    plt.plot([0,x[i]],[0,abs(x[i])**(1/2)],color=(i/len(x),i/len(x),i/len(x)))

plt.axis('off')
plt.show()
```

```python
# draw a checkerboard with purple numbers on top

m = 8
n = 4

# initialize matrix
C = np.zeros((m,n))

# populate the matrix
for i in range(0,m):
    for j in range(0,n):
        C[i,j] = (-1)**(i+j)

# display some numbers
for i in range(0,m):
    for j in range(0,n):
        plt.text(j,i,i+j,\
                 horizontalalignment='center',verticalalignment='center',\
                 fontdict=dict(color='m'))

plt.imshow(C)
plt.set_cmap('gray')
plt.show()
```


# Marginal Likelihood in Python

```python
import sympy

sympy.init_printing()

x = sympy.Symbol('x', positive=True)
s = sympy.Symbol('s', positive=True)

cauchy = 2 / (sympy.pi * (1 + x ** 2))
normal = 2 / sympy.sqrt(2*sympy.pi) * sympy.exp(-x**2 / 2)

hs = sympy.integrate(normal * cauchy.subs({x: s / x}) / x, (x, 0, sympy.oo))
hs
```

```python
beta = sympy.Symbol('beta', positive=True)
gamma = sympy.Symbol('gamma', positive=True)
theta = sympy.Symbol('theta')
z1 = sympy.integrate(sympy.cos(theta)**(2*((gamma/(beta - 2)) - 3/2) + 3),
                    (theta, -sympy.pi, sympy.pi))
z1
```

```python
beta = sympy.Symbol('beta', positive=True)
gamma = sympy.Symbol('gamma', constant = True)
theta = sympy.Symbol('theta')
z1 = sympy.integrate(sympy.cos(theta)**(2*(gamma/(beta - 2))),
                    (theta, -sympy.pi/2, sympy.pi/2))
z1
```

```python
from sympy.stats import Beta, Binomial, density, E, variance
from sympy import Symbol, simplify, pprint, expand_func

alpha = Symbol("alpha", positive=True)
beta = Symbol("beta", positive=True)
z = Symbol("z")
theta = Beta("theta", alpha, beta)
D = density(theta)(z)
pprint(D, use_unicode=False)
expand_func(simplify(E(theta, meijerg=True)))
```

     alpha - 1         beta - 1
    z         *(-z + 1)
    ---------------------------
         beta(alpha, beta)

    alpha/(alpha + beta)

```python

```

```python
normal = 1 / sympy.sqrt(2*sympy.pi) * sympy.exp(-x**2 / 2)
```

```python
x = sympy.Symbol('x')
normal = 1 / sympy.sqrt(2*sympy.pi) * sympy.exp(-x**2 / 2)
sympy.integrate(normal, (x, -sympy.oo, sympy.oo))
```

```python
n = sympy.Symbol('n', integer=True, positive=True)
k = sympy.Symbol('k', integer=True, positive=True)
a = b = 1.
p = .25
pr = sympy.binomial(n, k)*(p**k)*(1-p)**(n-k)
pr
```

```python
sympy.summation(pr*k, (k, 0, 20), (n, 20, 20))
```

```python
a = .5
b = 1.
p = x**(a-1)*(1-x)**(b-1)/sympy.beta(a, b)
n = 10
k = sympy.Symbol('k', positive=True)
pr = sympy.binomial(n, k)*(p**k)*(1-p)**(n-k)
sympy.integrate(pr.subs({x: k}), (x, 0, n))
```

```python
a = 1.
b = 3.
n = 20
k = sympy.Symbol('k', integer=True, positive=True)
p = sympy.binomial(n, k)*sympy.beta(k+a, n-k+b)/sympy.beta(a, b)
sympy.summation(p*k, (k, 0, 20))
```

```python
x = sympy.Symbol('x', positive=True)
a = 1.#sympy.Symbol('alpha', positive=True)
b = 1.#sympy.Symbol('beta', positive=True)
beta = (x**(a-1))*((1-x)**(b-1))/sympy.beta(a, b)
sympy.integrate(beta*x, (x, 0, 1))
```

```python
sympy.beta(1, 1)
```


# Monte Carlo

```julia
using Plots
```

## Tetrahedron volume

$$
\begin{align}
V &= \int_0^1{\int_0^{1-x}{\int_0^{1-x-y}{1 \quad dz}dy}dx} \\
&= \int_0^1{\int_0^{1-x}{1-x-y \quad dy}dx} \\
&= \int_0^1{ \left( y(1-x)-\frac{y^2}{2} \right)_{y=0}^{y=1-x} \quad dx} \\
&= \int_0^1{ \frac{(1-x)^2}{2} \quad dx} \\
&= \int_0^1{ \frac{1-2x+x^2}{2} \quad dx} \\
&= \left( \frac{x}{2} - \frac{x^2}{2} + \frac{x^3}{6} \right)_{x=0}^{x=1} \\
&= \frac{1}{6}
\end{align}
$$

## Standard Monte Carlo integration

```julia
function montecarlo(iterations)
    integral = 0
    for i=1:iterations
        x, y, z = rand(3)
        if (x+y+z<1)
            integral += 1
        end
    end
    volume = integral/iterations
end

montecarlo(100)
```

    0.16

# Test

```julia
function montecarlotest(iterations)
    err = log10(abs(montecarlo(iterations)-1/6))
end
```

    montecarlotest (generic function with 1 method)

```julia
p = plot()
for i=1:10
    x = 1:7
    y = montecarlotest.(10 .^x)
    p = scatter!(p, x, y, label=:false)
end

display(p)
```

## Adjourn

```julia
using Dates
println("mahdiar")
Dates.format(now(), "Y/U/d HH:MM")
```

    mahdiar

    "2021/April/17 16:17"