# Enhancing performance using compiled code

We first look at native code compilation. Here we show 3 common methods for doing this using

- `numba` JIT compilation
- `cython` AOT compilation,
- Direct wrapping of C++ code using `cffi`.

In general, `numba` is the simplest to use, while you have the most flexibility with `cffi`. Which approach gives the best performance generally requires some experimentation.

## Using `numba` Just-in-time compilation (JIT)
For programmer productivity, it often makes sense to code the majority of your application in a high-level language such as Python and only optimize code bottlenecks identified by profiling. One way to speed up these bottlenecks is to compile the code to machine executables, often via an intermediate C or C-like stage. There are two common approaches to compiling Python code - using a Just-In-Time (JIT) compiler and using `Cython` for Ahead of Time (AOT) compilation.

LLVM (Low Level Virtual Machine) library already implements a compiler backend. It is used to construct, optimize and produce intermediate and/or binary machine code.
- A compiler framework, where you provide the "front end" (parser and lexer) and the "back end" (code that converts LLVM's representation to actual machine code).
- Multi platform.
- LLVM optimizations (inlining, loop unrolling, SIMD vectorization etc).

 source: https://stackoverflow.com/questions/2354725/what-exactly-is-llvm

### `Numba`
- From the types of the function arguments, `numba` can often generate a specialized, fast, machine code implementation at runtime.
- Designed to work best with numerical code and `numpy` arrays.
- Does not require any C/C++ compiler.
- Does not replace the standard Python interpreter (all Python libraries are still available).

 source: https://nbviewer.jupyter.org/github/akittas/presentations/blob/master/pythess/numba/numba.ipynb?utm_source=newsletter_mailer&utm_medium=email&utm_campaign=weekly

- Object mode - Compiled code operates on Python objects. Supports nearly all of Python, but may not speed up code by a large factor.
- nopython mode - Compiled code operates on native machine data. Supports a subset of Python, but runs close to C/C++/FORTRAN speed.

```python
from numba import jit, njit
import numpy as np

x = np.arange(256).reshape(16, 16)

def go(a): # Function is compiled to machine code when called the first time
    trace = 0.0
    for i in range(a.shape[0]):
        trace += np.tanh(a[i, i])
    return a + trace

@jit
def go_fast(a): # Function is compiled to machine code when called the first time
    trace = 0.0
    for i in range(a.shape[0]):   # Numba likes loops
        trace += np.tanh(a[i, i]) # Numba likes NumPy functions
    return a + trace              # Numba likes NumPy broadcasting

@jit(nopython=True)# Set "nopython" mode for best performance, equivalent to @njit
def go_fast2(a): # Function is compiled to machine code when called the first time
    trace = 0.0
    for i in range(a.shape[0]):   # Numba likes loops
        trace += np.tanh(a[i, i]) # Numba likes NumPy functions
    return a + trace              # Numba likes NumPy broadcasting
```

```python
%timeit go(x)
%timeit go_fast(x)
%timeit go_fast2(x)
```

    The slowest run took 9.42 times longer than the fastest. This could mean that an intermediate result is being cached.
    10000 loops, best of 5: 36.7 µs per loop
    The slowest run took 426357.55 times longer than the fastest. This could mean that an intermediate result is being cached.
    1 loop, best of 5: 1.23 µs per loop
    The slowest run took 178838.64 times longer than the fastest. This could mean that an intermediate result is being cached.
    1 loop, best of 5: 1.39 µs per loop

### MoteCarlo PI

 source: https://upload.wikimedia.org/wikipedia/commons/2/20/MonteCarloIntegrationCircle.svg

If we sample uniformly N points in $[-1,1] \times [-1,1]$ and find that M points are inside circle. Then, we get $\frac{M}{N} \sim \frac{\pi}{4}$ which can help us to estimate $\pi$

```python
def monte_carlo_pi(n):
    x = np.random.uniform(-1, 1, (n,2))
    return 4*np.sum((x**2).sum(1) < 1)/n
```

```python
n = int(1e7)
monte_carlo_pi(n)
```

    3.1410448

```python
%timeit monte_carlo_pi(n)
```

    1 loop, best of 5: 446 ms per loop

```python
@jit(nopython=True)
def monte_carlo_pi_numba(n):
    x = np.random.uniform(-1, 1, (n,2))
    return 4*np.sum((x**2).sum(1) < 1)/n
```

```python
%timeit monte_carlo_pi_numba(n)
```

    The slowest run took 7.33 times longer than the fastest. This could mean that an intermediate result is being cached.
    1 loop, best of 5: 225 ms per loop

While NumPy has developed a strong idiom around the use of vector operations, Numba is perfectly happy with loops too. For users familiar with C or Fortran, writing Python in this style will work fine in Numba (after all, LLVM gets a lot of use in compiling C lineage languages).

```python
@jit(nopython=True, cache=True)
def monte_carlo_pi_numba_unrolled(nsamples):
    acc = 0
    for i in range(nsamples):
        x = np.random.uniform(-1,1)
        y = np.random.uniform(-1,1)
        if (x*x + y*y) < 1:
            acc += 1
    return 4 * acc / nsamples
```

```python
%timeit monte_carlo_pi_numba_unrolled(n)
```

    1 loop, best of 5: 144 ms per loop

- Check "cache" flag in the `Numba` decorector: https://numba.readthedocs.io/en/stable/user/jit.html

### Vectorization
Numba’s vectorize allows Python functions taking scalar input arguments to be used as NumPy `ufuncs`. Creating a traditional NumPy ufunc is not the most straightforward process and involves writing some C code. Numba makes this easy. Using the `vectorize()` decorator, Numba can compile a pure Python function into a `ufunc` that operates over NumPy arrays as fast as traditional ufuncs written in C.

**Using `vectorize()`, you write your function as operating over input scalars, rather than arrays. Numba will generate the surrounding loop (or kernel) allowing efficient iteration over the actual inputs.**

```python
def Theta_for(x):
    a = []
    for i in x:
        if i >= 0:
            a.append(1)
        else:
            a.append(0)
    return a
```

```python
from numba import vectorize
@vectorize
def Theta_numba(x):
    """
    Scalar implemenation of the Heaviside step function.
    """
    if x >= 0:
        return 1
    else:
        return 0
```

```python
def Theta(x):
    """
    Scalar implemenation of the Heaviside step function.
    """
    if x >= 0:
        return 1
    else:
        return 0

Theta_vec = np.vectorize(Theta)
```

```python
%timeit Theta_for(np.random.randn(500))
%timeit Theta_numba(np.random.randn(500))
%timeit Theta_vec(np.random.randn(500))
```

    1000 loops, best of 5: 242 µs per loop
    The slowest run took 2666.77 times longer than the fastest. This could mean that an intermediate result is being cached.
    10000 loops, best of 5: 26.6 µs per loop
    The slowest run took 12.81 times longer than the fastest. This could mean that an intermediate result is being cached.
    10000 loops, best of 5: 122 µs per loop

### FastMath
In certain classes of applications strict IEEE 754 compliance is less important. As a result, it is possible to relax some numerical rigor with the view of gaining additional performance.
Use with care. Some functions may be less accurate.

```python
@njit
def do_sum(A):
    acc = 0.
    # without fastmath, this loop must accumulate in strict order
    for x in A:
        acc += np.sqrt(x)
    return acc

@njit(fastmath=True)
def do_sum_fast(A):
    acc = 0.
    # with fastmath, the reduction can be vectorized as floating point
    # reassociation is permitted. (Out-of-order)
    for x in A:
        acc += np.sqrt(x)
    return acc

A = np.arange(1000)
```

```python
%timeit do_sum(A)
%timeit do_sum_fast(A)
```

    The slowest run took 45555.89 times longer than the fastest. This could mean that an intermediate result is being cached.
    100000 loops, best of 5: 2.66 µs per loop
    The slowest run took 34396.92 times longer than the fastest. This could mean that an intermediate result is being cached.
    100000 loops, best of 5: 2.64 µs per loop

```python
np.allclose(do_sum(A), do_sum_fast(A))
```

    True

## Using `cython`
`Cython` is an “optimizing static compiler ” that combines Python with C to generate optimized code. Since `Cython` is a superset of Python, all valid Python programs are also valid `Cython` programs. However, by providing hints and static typing, we can get much faster programs. In this sense one could also call it a Python with types.

Note that while `numba` often provides similar speedups with less work, an advantage of `Cython` is that it works for general python language while `numba` is target for numpy.

In addition to the basic use case of wrapping native code, `Cython` supports an additional use-case, namely interactive optimization. Basically, one starts out with a pure-Python script and incrementally adds `Cython` types to the bottleneck code to optimize only those code paths that really matter.
The code can be autogenerated and the wrapping code can (almost) be written in Python.

- How to build Cython modules

Using Cython consists of these steps:

* Write a `.pyx` source file
* Run the Cython compiler to generate a C file
* Run a C compiler to generate a compiled library
* Run the Python interpreter and ask it to import the module
* In the `Jupyter notebook`, we can use the `%%cython` cell magic to automate these steps.

```python
%load_ext cython
```

```python
def matrix_multiply(u, v, res):
    m, n = u.shape
    n, p = v.shape
    for i in range(m):
        for j in range(p):
            res[i,j] = 0
            for k in range(n):
                res[i,j] += u[i,k] * v[k,j]
    return res
```

```python
u = np.random.random((10,20))
v = np.random.random((20,5))
```

```python
res = np.zeros((u.shape[0], v.shape[1]))
matrix_multiply(u, v, res)
```

    array([[3.46135596, 3.27110322, 4.16821885, 2.12434671, 3.06437438],
           [5.65805401, 4.95464457, 5.29249904, 3.57914102, 4.85371357],
           [5.46485866, 4.90452363, 6.37657047, 3.54256309, 4.75101659],
           [3.95024825, 4.11892064, 5.68919065, 3.14419891, 3.7029538 ],
           [4.87341185, 4.8921277 , 4.24987716, 3.31412014, 4.33666217],
           [3.49624858, 3.79519057, 4.28709505, 2.79202349, 3.18632487],
           [4.26899041, 4.13147792, 4.51677833, 3.011796  , 4.32194033],
           [6.4585535 , 5.98440406, 5.92163001, 3.73743771, 5.09868502],
           [5.46751236, 5.17874013, 5.5222365 , 3.47967041, 5.61399437],
           [3.47652206, 3.62730312, 4.41391252, 1.91206498, 3.4786661 ]])

```python
res = np.zeros((u.shape[0], v.shape[1]))
%timeit -r3 -n3 matrix_multiply(u, v, res)
```

    3 loops, best of 3: 912 µs per loop

```cython
%%cython -a

import numpy as np

def matrix_multiply1(u, v, res):
    m, n = u.shape
    n, p = v.shape
    for i in range(m):
        for j in range(p):
            res[i,j] = 0
            for k in range(n):
                res[i,j] += u[i,k] * v[k,j]
    return res
```

<!DOCTYPE html>

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Cython: _cython_magic_537ab4dc69a4da26c742992eab5394b9.pyx</title>

Generated by Cython 0.29.23

    Yellow lines hint at Python interaction.

    Click on a line that starts with a "+" to see the C code that Cython generated for it.

 01:

+02: import numpy as np

  __pyx_t_1 = __Pyx_Import(__pyx_n_s_numpy, 0, 0); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 2, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_np, __pyx_t_1) < 0) __PYX_ERR(0, 2, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

 03:

+04: def matrix_multiply1(u, v, res):

/* Python wrapper */
static PyObject *__pyx_pw_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_1matrix_multiply1(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/
static PyMethodDef __pyx_mdef_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_1matrix_multiply1 = {"matrix_multiply1", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_1matrix_multiply1, METH_VARARGS|METH_KEYWORDS, 0};
static PyObject *__pyx_pw_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_1matrix_multiply1(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {
  PyObject *__pyx_v_u = 0;
  PyObject *__pyx_v_v = 0;
  PyObject *__pyx_v_res = 0;
  PyObject *__pyx_r = 0;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("matrix_multiply1 (wrapper)", 0);
  {
    static PyObject **__pyx_pyargnames[] = {&__pyx_n_s_u,&__pyx_n_s_v,&__pyx_n_s_res,0};
    PyObject* values[3] = {0,0,0};
    if (unlikely(__pyx_kwds)) {
      Py_ssize_t kw_args;
      const Py_ssize_t pos_args = PyTuple_GET_SIZE(__pyx_args);
      switch (pos_args) {
        case  3: values[2] = PyTuple_GET_ITEM(__pyx_args, 2);
        CYTHON_FALLTHROUGH;
        case  2: values[1] = PyTuple_GET_ITEM(__pyx_args, 1);
        CYTHON_FALLTHROUGH;
        case  1: values[0] = PyTuple_GET_ITEM(__pyx_args, 0);
        CYTHON_FALLTHROUGH;
        case  0: break;
        default: goto __pyx_L5_argtuple_error;
      }
      kw_args = PyDict_Size(__pyx_kwds);
      switch (pos_args) {
        case  0:
        if (likely((values[0] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_u)) != 0)) kw_args--;
        else goto __pyx_L5_argtuple_error;
        CYTHON_FALLTHROUGH;
        case  1:
        if (likely((values[1] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_v)) != 0)) kw_args--;
        else {
          __Pyx_RaiseArgtupleInvalid("matrix_multiply1", 1, 3, 3, 1); __PYX_ERR(0, 4, __pyx_L3_error)
        }
        CYTHON_FALLTHROUGH;
        case  2:
        if (likely((values[2] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_res)) != 0)) kw_args--;
        else {
          __Pyx_RaiseArgtupleInvalid("matrix_multiply1", 1, 3, 3, 2); __PYX_ERR(0, 4, __pyx_L3_error)
        }
      }
      if (unlikely(kw_args > 0)) {
        if (unlikely(__Pyx_ParseOptionalKeywords(__pyx_kwds, __pyx_pyargnames, 0, values, pos_args, "matrix_multiply1") < 0)) __PYX_ERR(0, 4, __pyx_L3_error)
      }
    } else if (PyTuple_GET_SIZE(__pyx_args) != 3) {
      goto __pyx_L5_argtuple_error;
    } else {
      values[0] = PyTuple_GET_ITEM(__pyx_args, 0);
      values[1] = PyTuple_GET_ITEM(__pyx_args, 1);
      values[2] = PyTuple_GET_ITEM(__pyx_args, 2);
    }
    __pyx_v_u = values[0];
    __pyx_v_v = values[1];
    __pyx_v_res = values[2];
  }
  goto __pyx_L4_argument_unpacking_done;
  __pyx_L5_argtuple_error:;
  __Pyx_RaiseArgtupleInvalid("matrix_multiply1", 1, 3, 3, PyTuple_GET_SIZE(__pyx_args)); __PYX_ERR(0, 4, __pyx_L3_error)
  __pyx_L3_error:;
  __Pyx_AddTraceback("_cython_magic_537ab4dc69a4da26c742992eab5394b9.matrix_multiply1", __pyx_clineno, __pyx_lineno, __pyx_filename);
  __Pyx_RefNannyFinishContext();
  return NULL;
  __pyx_L4_argument_unpacking_done:;
  __pyx_r = __pyx_pf_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_matrix_multiply1(__pyx_self, __pyx_v_u, __pyx_v_v, __pyx_v_res);
  int __pyx_lineno = 0;
  const char *__pyx_filename = NULL;
  int __pyx_clineno = 0;

  /* function exit code */
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

static PyObject *__pyx_pf_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_matrix_multiply1(CYTHON_UNUSED PyObject *__pyx_self, PyObject *__pyx_v_u, PyObject *__pyx_v_v, PyObject *__pyx_v_res) {
  PyObject *__pyx_v_m = NULL;
  PyObject *__pyx_v_n = NULL;
  PyObject *__pyx_v_p = NULL;
  PyObject *__pyx_v_i = NULL;
  PyObject *__pyx_v_j = NULL;
  PyObject *__pyx_v_k = NULL;
  PyObject *__pyx_r = NULL;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("matrix_multiply1", 0);
/* … */
  /* function exit code */
  __pyx_L1_error:;
  __Pyx_XDECREF(__pyx_t_1);
  __Pyx_XDECREF(__pyx_t_2);
  __Pyx_XDECREF(__pyx_t_3);
  __Pyx_XDECREF(__pyx_t_4);
  __Pyx_XDECREF(__pyx_t_12);
  __Pyx_XDECREF(__pyx_t_13);
  __Pyx_XDECREF(__pyx_t_14);
  __Pyx_XDECREF(__pyx_t_15);
  __Pyx_AddTraceback("_cython_magic_537ab4dc69a4da26c742992eab5394b9.matrix_multiply1", __pyx_clineno, __pyx_lineno, __pyx_filename);
  __pyx_r = NULL;
  __pyx_L0:;
  __Pyx_XDECREF(__pyx_v_m);
  __Pyx_XDECREF(__pyx_v_n);
  __Pyx_XDECREF(__pyx_v_p);
  __Pyx_XDECREF(__pyx_v_i);
  __Pyx_XDECREF(__pyx_v_j);
  __Pyx_XDECREF(__pyx_v_k);
  __Pyx_XGIVEREF(__pyx_r);
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}
/* … */
  __pyx_tuple_ = PyTuple_Pack(9, __pyx_n_s_u, __pyx_n_s_v, __pyx_n_s_res, __pyx_n_s_m, __pyx_n_s_n, __pyx_n_s_p, __pyx_n_s_i, __pyx_n_s_j, __pyx_n_s_k); if (unlikely(!__pyx_tuple_)) __PYX_ERR(0, 4, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_tuple_);
  __Pyx_GIVEREF(__pyx_tuple_);
/* … */
  __pyx_t_1 = PyCFunction_NewEx(&__pyx_mdef_46_cython_magic_537ab4dc69a4da26c742992eab5394b9_1matrix_multiply1, NULL, __pyx_n_s_cython_magic_537ab4dc69a4da26c7); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 4, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_matrix_multiply1, __pyx_t_1) < 0) __PYX_ERR(0, 4, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

+05:     m, n = u.shape

  __pyx_t_1 = __Pyx_PyObject_GetAttrStr(__pyx_v_u, __pyx_n_s_shape); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 5, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if ((likely(PyTuple_CheckExact(__pyx_t_1))) || (PyList_CheckExact(__pyx_t_1))) {
    PyObject* sequence = __pyx_t_1;
    Py_ssize_t size = __Pyx_PySequence_SIZE(sequence);
    if (unlikely(size != 2)) {
      if (size > 2) __Pyx_RaiseTooManyValuesError(2);
      else if (size >= 0) __Pyx_RaiseNeedMoreValuesError(size);
      __PYX_ERR(0, 5, __pyx_L1_error)
    }
    #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
    if (likely(PyTuple_CheckExact(sequence))) {
      __pyx_t_2 = PyTuple_GET_ITEM(sequence, 0);
      __pyx_t_3 = PyTuple_GET_ITEM(sequence, 1);
    } else {
      __pyx_t_2 = PyList_GET_ITEM(sequence, 0);
      __pyx_t_3 = PyList_GET_ITEM(sequence, 1);
    }
    __Pyx_INCREF(__pyx_t_2);
    __Pyx_INCREF(__pyx_t_3);
    #else
    __pyx_t_2 = PySequence_ITEM(sequence, 0); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 5, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_2);
    __pyx_t_3 = PySequence_ITEM(sequence, 1); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 5, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_3);
    #endif
    __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  } else {
    Py_ssize_t index = -1;
    __pyx_t_4 = PyObject_GetIter(__pyx_t_1); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 5, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_4);
    __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
    __pyx_t_5 = Py_TYPE(__pyx_t_4)->tp_iternext;
    index = 0; __pyx_t_2 = __pyx_t_5(__pyx_t_4); if (unlikely(!__pyx_t_2)) goto __pyx_L3_unpacking_failed;
    __Pyx_GOTREF(__pyx_t_2);
    index = 1; __pyx_t_3 = __pyx_t_5(__pyx_t_4); if (unlikely(!__pyx_t_3)) goto __pyx_L3_unpacking_failed;
    __Pyx_GOTREF(__pyx_t_3);
    if (__Pyx_IternextUnpackEndCheck(__pyx_t_5(__pyx_t_4), 2) < 0) __PYX_ERR(0, 5, __pyx_L1_error)
    __pyx_t_5 = NULL;
    __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
    goto __pyx_L4_unpacking_done;
    __pyx_L3_unpacking_failed:;
    __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
    __pyx_t_5 = NULL;
    if (__Pyx_IterFinish() == 0) __Pyx_RaiseNeedMoreValuesError(index);
    __PYX_ERR(0, 5, __pyx_L1_error)
    __pyx_L4_unpacking_done:;
  }
  __pyx_v_m = __pyx_t_2;
  __pyx_t_2 = 0;
  __pyx_v_n = __pyx_t_3;
  __pyx_t_3 = 0;

+06:     n, p = v.shape

  __pyx_t_1 = __Pyx_PyObject_GetAttrStr(__pyx_v_v, __pyx_n_s_shape); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 6, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if ((likely(PyTuple_CheckExact(__pyx_t_1))) || (PyList_CheckExact(__pyx_t_1))) {
    PyObject* sequence = __pyx_t_1;
    Py_ssize_t size = __Pyx_PySequence_SIZE(sequence);
    if (unlikely(size != 2)) {
      if (size > 2) __Pyx_RaiseTooManyValuesError(2);
      else if (size >= 0) __Pyx_RaiseNeedMoreValuesError(size);
      __PYX_ERR(0, 6, __pyx_L1_error)
    }
    #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
    if (likely(PyTuple_CheckExact(sequence))) {
      __pyx_t_3 = PyTuple_GET_ITEM(sequence, 0);
      __pyx_t_2 = PyTuple_GET_ITEM(sequence, 1);
    } else {
      __pyx_t_3 = PyList_GET_ITEM(sequence, 0);
      __pyx_t_2 = PyList_GET_ITEM(sequence, 1);
    }
    __Pyx_INCREF(__pyx_t_3);
    __Pyx_INCREF(__pyx_t_2);
    #else
    __pyx_t_3 = PySequence_ITEM(sequence, 0); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 6, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_3);
    __pyx_t_2 = PySequence_ITEM(sequence, 1); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 6, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_2);
    #endif
    __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  } else {
    Py_ssize_t index = -1;
    __pyx_t_4 = PyObject_GetIter(__pyx_t_1); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 6, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_4);
    __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
    __pyx_t_5 = Py_TYPE(__pyx_t_4)->tp_iternext;
    index = 0; __pyx_t_3 = __pyx_t_5(__pyx_t_4); if (unlikely(!__pyx_t_3)) goto __pyx_L5_unpacking_failed;
    __Pyx_GOTREF(__pyx_t_3);
    index = 1; __pyx_t_2 = __pyx_t_5(__pyx_t_4); if (unlikely(!__pyx_t_2)) goto __pyx_L5_unpacking_failed;
    __Pyx_GOTREF(__pyx_t_2);
    if (__Pyx_IternextUnpackEndCheck(__pyx_t_5(__pyx_t_4), 2) < 0) __PYX_ERR(0, 6, __pyx_L1_error)
    __pyx_t_5 = NULL;
    __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
    goto __pyx_L6_unpacking_done;
    __pyx_L5_unpacking_failed:;
    __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;
    __pyx_t_5 = NULL;
    if (__Pyx_IterFinish() == 0) __Pyx_RaiseNeedMoreValuesError(index);
    __PYX_ERR(0, 6, __pyx_L1_error)
    __pyx_L6_unpacking_done:;
  }
  __Pyx_DECREF_SET(__pyx_v_n, __pyx_t_3);
  __pyx_t_3 = 0;
  __pyx_v_p = __pyx_t_2;
  __pyx_t_2 = 0;

+07:     for i in range(m):

  __pyx_t_1 = __Pyx_PyObject_CallOneArg(__pyx_builtin_range, __pyx_v_m); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 7, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (likely(PyList_CheckExact(__pyx_t_1)) || PyTuple_CheckExact(__pyx_t_1)) {
    __pyx_t_2 = __pyx_t_1; __Pyx_INCREF(__pyx_t_2); __pyx_t_6 = 0;
    __pyx_t_7 = NULL;
  } else {
    __pyx_t_6 = -1; __pyx_t_2 = PyObject_GetIter(__pyx_t_1); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 7, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_2);
    __pyx_t_7 = Py_TYPE(__pyx_t_2)->tp_iternext; if (unlikely(!__pyx_t_7)) __PYX_ERR(0, 7, __pyx_L1_error)
  }
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  for (;;) {
    if (likely(!__pyx_t_7)) {
      if (likely(PyList_CheckExact(__pyx_t_2))) {
        if (__pyx_t_6 >= PyList_GET_SIZE(__pyx_t_2)) break;
        #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
        __pyx_t_1 = PyList_GET_ITEM(__pyx_t_2, __pyx_t_6); __Pyx_INCREF(__pyx_t_1); __pyx_t_6++; if (unlikely(0 < 0)) __PYX_ERR(0, 7, __pyx_L1_error)
        #else
        __pyx_t_1 = PySequence_ITEM(__pyx_t_2, __pyx_t_6); __pyx_t_6++; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 7, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_1);
        #endif
      } else {
        if (__pyx_t_6 >= PyTuple_GET_SIZE(__pyx_t_2)) break;
        #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
        __pyx_t_1 = PyTuple_GET_ITEM(__pyx_t_2, __pyx_t_6); __Pyx_INCREF(__pyx_t_1); __pyx_t_6++; if (unlikely(0 < 0)) __PYX_ERR(0, 7, __pyx_L1_error)
        #else
        __pyx_t_1 = PySequence_ITEM(__pyx_t_2, __pyx_t_6); __pyx_t_6++; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 7, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_1);
        #endif
      }
    } else {
      __pyx_t_1 = __pyx_t_7(__pyx_t_2);
      if (unlikely(!__pyx_t_1)) {
        PyObject* exc_type = PyErr_Occurred();
        if (exc_type) {
          if (likely(__Pyx_PyErr_GivenExceptionMatches(exc_type, PyExc_StopIteration))) PyErr_Clear();
          else __PYX_ERR(0, 7, __pyx_L1_error)
        }
        break;
      }
      __Pyx_GOTREF(__pyx_t_1);
    }
    __Pyx_XDECREF_SET(__pyx_v_i, __pyx_t_1);
    __pyx_t_1 = 0;
/* … */
  }
  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;

+08:         for j in range(p):

    __pyx_t_1 = __Pyx_PyObject_CallOneArg(__pyx_builtin_range, __pyx_v_p); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 8, __pyx_L1_error)
    __Pyx_GOTREF(__pyx_t_1);
    if (likely(PyList_CheckExact(__pyx_t_1)) || PyTuple_CheckExact(__pyx_t_1)) {
      __pyx_t_3 = __pyx_t_1; __Pyx_INCREF(__pyx_t_3); __pyx_t_8 = 0;
      __pyx_t_9 = NULL;
    } else {
      __pyx_t_8 = -1; __pyx_t_3 = PyObject_GetIter(__pyx_t_1); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 8, __pyx_L1_error)
      __Pyx_GOTREF(__pyx_t_3);
      __pyx_t_9 = Py_TYPE(__pyx_t_3)->tp_iternext; if (unlikely(!__pyx_t_9)) __PYX_ERR(0, 8, __pyx_L1_error)
    }
    __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
    for (;;) {
      if (likely(!__pyx_t_9)) {
        if (likely(PyList_CheckExact(__pyx_t_3))) {
          if (__pyx_t_8 >= PyList_GET_SIZE(__pyx_t_3)) break;
          #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
          __pyx_t_1 = PyList_GET_ITEM(__pyx_t_3, __pyx_t_8); __Pyx_INCREF(__pyx_t_1); __pyx_t_8++; if (unlikely(0 < 0)) __PYX_ERR(0, 8, __pyx_L1_error)
          #else
          __pyx_t_1 = PySequence_ITEM(__pyx_t_3, __pyx_t_8); __pyx_t_8++; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 8, __pyx_L1_error)
          __Pyx_GOTREF(__pyx_t_1);
          #endif
        } else {
          if (__pyx_t_8 >= PyTuple_GET_SIZE(__pyx_t_3)) break;
          #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
          __pyx_t_1 = PyTuple_GET_ITEM(__pyx_t_3, __pyx_t_8); __Pyx_INCREF(__pyx_t_1); __pyx_t_8++; if (unlikely(0 < 0)) __PYX_ERR(0, 8, __pyx_L1_error)
          #else
          __pyx_t_1 = PySequence_ITEM(__pyx_t_3, __pyx_t_8); __pyx_t_8++; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 8, __pyx_L1_error)
          __Pyx_GOTREF(__pyx_t_1);
          #endif
        }
      } else {
        __pyx_t_1 = __pyx_t_9(__pyx_t_3);
        if (unlikely(!__pyx_t_1)) {
          PyObject* exc_type = PyErr_Occurred();
          if (exc_type) {
            if (likely(__Pyx_PyErr_GivenExceptionMatches(exc_type, PyExc_StopIteration))) PyErr_Clear();
            else __PYX_ERR(0, 8, __pyx_L1_error)
          }
          break;
        }
        __Pyx_GOTREF(__pyx_t_1);
      }
      __Pyx_XDECREF_SET(__pyx_v_j, __pyx_t_1);
      __pyx_t_1 = 0;
/* … */
    }
    __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;

+09:             res[i,j] = 0

      __pyx_t_1 = PyTuple_New(2); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 9, __pyx_L1_error)
      __Pyx_GOTREF(__pyx_t_1);
      __Pyx_INCREF(__pyx_v_i);
      __Pyx_GIVEREF(__pyx_v_i);
      PyTuple_SET_ITEM(__pyx_t_1, 0, __pyx_v_i);
      __Pyx_INCREF(__pyx_v_j);
      __Pyx_GIVEREF(__pyx_v_j);
      PyTuple_SET_ITEM(__pyx_t_1, 1, __pyx_v_j);
      if (unlikely(PyObject_SetItem(__pyx_v_res, __pyx_t_1, __pyx_int_0) < 0)) __PYX_ERR(0, 9, __pyx_L1_error)
      __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

+10:             for k in range(n):

      __pyx_t_1 = __Pyx_PyObject_CallOneArg(__pyx_builtin_range, __pyx_v_n); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 10, __pyx_L1_error)
      __Pyx_GOTREF(__pyx_t_1);
      if (likely(PyList_CheckExact(__pyx_t_1)) || PyTuple_CheckExact(__pyx_t_1)) {
        __pyx_t_4 = __pyx_t_1; __Pyx_INCREF(__pyx_t_4); __pyx_t_10 = 0;
        __pyx_t_11 = NULL;
      } else {
        __pyx_t_10 = -1; __pyx_t_4 = PyObject_GetIter(__pyx_t_1); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 10, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_4);
        __pyx_t_11 = Py_TYPE(__pyx_t_4)->tp_iternext; if (unlikely(!__pyx_t_11)) __PYX_ERR(0, 10, __pyx_L1_error)
      }
      __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
      for (;;) {
        if (likely(!__pyx_t_11)) {
          if (likely(PyList_CheckExact(__pyx_t_4))) {
            if (__pyx_t_10 >= PyList_GET_SIZE(__pyx_t_4)) break;
            #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
            __pyx_t_1 = PyList_GET_ITEM(__pyx_t_4, __pyx_t_10); __Pyx_INCREF(__pyx_t_1); __pyx_t_10++; if (unlikely(0 < 0)) __PYX_ERR(0, 10, __pyx_L1_error)
            #else
            __pyx_t_1 = PySequence_ITEM(__pyx_t_4, __pyx_t_10); __pyx_t_10++; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 10, __pyx_L1_error)
            __Pyx_GOTREF(__pyx_t_1);
            #endif
          } else {
            if (__pyx_t_10 >= PyTuple_GET_SIZE(__pyx_t_4)) break;
            #if CYTHON_ASSUME_SAFE_MACROS && !CYTHON_AVOID_BORROWED_REFS
            __pyx_t_1 = PyTuple_GET_ITEM(__pyx_t_4, __pyx_t_10); __Pyx_INCREF(__pyx_t_1); __pyx_t_10++; if (unlikely(0 < 0)) __PYX_ERR(0, 10, __pyx_L1_error)
            #else
            __pyx_t_1 = PySequence_ITEM(__pyx_t_4, __pyx_t_10); __pyx_t_10++; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 10, __pyx_L1_error)
            __Pyx_GOTREF(__pyx_t_1);
            #endif
          }
        } else {
          __pyx_t_1 = __pyx_t_11(__pyx_t_4);
          if (unlikely(!__pyx_t_1)) {
            PyObject* exc_type = PyErr_Occurred();
            if (exc_type) {
              if (likely(__Pyx_PyErr_GivenExceptionMatches(exc_type, PyExc_StopIteration))) PyErr_Clear();
              else __PYX_ERR(0, 10, __pyx_L1_error)
            }
            break;
          }
          __Pyx_GOTREF(__pyx_t_1);
        }
        __Pyx_XDECREF_SET(__pyx_v_k, __pyx_t_1);
        __pyx_t_1 = 0;
/* … */
      }
      __Pyx_DECREF(__pyx_t_4); __pyx_t_4 = 0;

+11:                 res[i,j] += u[i,k] * v[k,j]

        __pyx_t_1 = PyTuple_New(2); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_1);
        __Pyx_INCREF(__pyx_v_i);
        __Pyx_GIVEREF(__pyx_v_i);
        PyTuple_SET_ITEM(__pyx_t_1, 0, __pyx_v_i);
        __Pyx_INCREF(__pyx_v_j);
        __Pyx_GIVEREF(__pyx_v_j);
        PyTuple_SET_ITEM(__pyx_t_1, 1, __pyx_v_j);
        __pyx_t_12 = __Pyx_PyObject_GetItem(__pyx_v_res, __pyx_t_1); if (unlikely(!__pyx_t_12)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_12);
        __pyx_t_13 = PyTuple_New(2); if (unlikely(!__pyx_t_13)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_13);
        __Pyx_INCREF(__pyx_v_i);
        __Pyx_GIVEREF(__pyx_v_i);
        PyTuple_SET_ITEM(__pyx_t_13, 0, __pyx_v_i);
        __Pyx_INCREF(__pyx_v_k);
        __Pyx_GIVEREF(__pyx_v_k);
        PyTuple_SET_ITEM(__pyx_t_13, 1, __pyx_v_k);
        __pyx_t_14 = __Pyx_PyObject_GetItem(__pyx_v_u, __pyx_t_13); if (unlikely(!__pyx_t_14)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_14);
        __Pyx_DECREF(__pyx_t_13); __pyx_t_13 = 0;
        __pyx_t_13 = PyTuple_New(2); if (unlikely(!__pyx_t_13)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_13);
        __Pyx_INCREF(__pyx_v_k);
        __Pyx_GIVEREF(__pyx_v_k);
        PyTuple_SET_ITEM(__pyx_t_13, 0, __pyx_v_k);
        __Pyx_INCREF(__pyx_v_j);
        __Pyx_GIVEREF(__pyx_v_j);
        PyTuple_SET_ITEM(__pyx_t_13, 1, __pyx_v_j);
        __pyx_t_15 = __Pyx_PyObject_GetItem(__pyx_v_v, __pyx_t_13); if (unlikely(!__pyx_t_15)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_15);
        __Pyx_DECREF(__pyx_t_13); __pyx_t_13 = 0;
        __pyx_t_13 = PyNumber_Multiply(__pyx_t_14, __pyx_t_15); if (unlikely(!__pyx_t_13)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_13);
        __Pyx_DECREF(__pyx_t_14); __pyx_t_14 = 0;
        __Pyx_DECREF(__pyx_t_15); __pyx_t_15 = 0;
        __pyx_t_15 = PyNumber_InPlaceAdd(__pyx_t_12, __pyx_t_13); if (unlikely(!__pyx_t_15)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_GOTREF(__pyx_t_15);
        __Pyx_DECREF(__pyx_t_12); __pyx_t_12 = 0;
        __Pyx_DECREF(__pyx_t_13); __pyx_t_13 = 0;
        if (unlikely(PyObject_SetItem(__pyx_v_res, __pyx_t_1, __pyx_t_15) < 0)) __PYX_ERR(0, 11, __pyx_L1_error)
        __Pyx_DECREF(__pyx_t_15); __pyx_t_15 = 0;
        __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

+12:     return res

  __Pyx_XDECREF(__pyx_r);
  __Pyx_INCREF(__pyx_v_res);
  __pyx_r = __pyx_v_res;
  goto __pyx_L0;

The `nogil` keyword tells Cython that a particular function or code section should be executed without the GIL. When the GIL is released, it is not possible to make any Python API calls, meaning that only C variables and C functions (declared with `cdef`) can be used.

```cython
%%cython -a

import cython

@cython.boundscheck(False)
@cython.wraparound(False) #non-negative
def matrix_multiply1(double[:,:] u, double[:, :] v, double[:, :] res):
    cdef int i, j, k
    cdef int m, n, p

    m = u.shape[0]
    n = u.shape[1]
    p = v.shape[1]

    with cython.nogil:
        for i in range(m):
            for j in range(p):
                res[i,j] = 0
                for k in range(n):
                    res[i,j] += u[i,k] * v[k,j]
```

<!DOCTYPE html>

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Cython: _cython_magic_231f6419b2e54af595f8becce81af957.pyx</title>

Generated by Cython 0.29.23

    Yellow lines hint at Python interaction.

    Click on a line that starts with a "+" to see the C code that Cython generated for it.

 01:

+02: import cython

  __pyx_t_1 = __Pyx_PyDict_NewPresized(0); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 2, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_test, __pyx_t_1) < 0) __PYX_ERR(0, 2, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

 03:

 04: @cython.boundscheck(False)

 05: @cython.wraparound(False) #non-negative

+06: def matrix_multiply1(double[:,:] u, double[:, :] v, double[:, :] res):

/* Python wrapper */
static PyObject *__pyx_pw_46_cython_magic_231f6419b2e54af595f8becce81af957_1matrix_multiply1(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/
static PyMethodDef __pyx_mdef_46_cython_magic_231f6419b2e54af595f8becce81af957_1matrix_multiply1 = {"matrix_multiply1", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_46_cython_magic_231f6419b2e54af595f8becce81af957_1matrix_multiply1, METH_VARARGS|METH_KEYWORDS, 0};
static PyObject *__pyx_pw_46_cython_magic_231f6419b2e54af595f8becce81af957_1matrix_multiply1(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {
  __Pyx_memviewslice __pyx_v_u = { 0, 0, { 0 }, { 0 }, { 0 } };
  __Pyx_memviewslice __pyx_v_v = { 0, 0, { 0 }, { 0 }, { 0 } };
  __Pyx_memviewslice __pyx_v_res = { 0, 0, { 0 }, { 0 }, { 0 } };
  PyObject *__pyx_r = 0;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("matrix_multiply1 (wrapper)", 0);
  {
    static PyObject **__pyx_pyargnames[] = {&__pyx_n_s_u,&__pyx_n_s_v,&__pyx_n_s_res,0};
    PyObject* values[3] = {0,0,0};
    if (unlikely(__pyx_kwds)) {
      Py_ssize_t kw_args;
      const Py_ssize_t pos_args = PyTuple_GET_SIZE(__pyx_args);
      switch (pos_args) {
        case  3: values[2] = PyTuple_GET_ITEM(__pyx_args, 2);
        CYTHON_FALLTHROUGH;
        case  2: values[1] = PyTuple_GET_ITEM(__pyx_args, 1);
        CYTHON_FALLTHROUGH;
        case  1: values[0] = PyTuple_GET_ITEM(__pyx_args, 0);
        CYTHON_FALLTHROUGH;
        case  0: break;
        default: goto __pyx_L5_argtuple_error;
      }
      kw_args = PyDict_Size(__pyx_kwds);
      switch (pos_args) {
        case  0:
        if (likely((values[0] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_u)) != 0)) kw_args--;
        else goto __pyx_L5_argtuple_error;
        CYTHON_FALLTHROUGH;
        case  1:
        if (likely((values[1] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_v)) != 0)) kw_args--;
        else {
          __Pyx_RaiseArgtupleInvalid("matrix_multiply1", 1, 3, 3, 1); __PYX_ERR(0, 6, __pyx_L3_error)
        }
        CYTHON_FALLTHROUGH;
        case  2:
        if (likely((values[2] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_res)) != 0)) kw_args--;
        else {
          __Pyx_RaiseArgtupleInvalid("matrix_multiply1", 1, 3, 3, 2); __PYX_ERR(0, 6, __pyx_L3_error)
        }
      }
      if (unlikely(kw_args > 0)) {
        if (unlikely(__Pyx_ParseOptionalKeywords(__pyx_kwds, __pyx_pyargnames, 0, values, pos_args, "matrix_multiply1") < 0)) __PYX_ERR(0, 6, __pyx_L3_error)
      }
    } else if (PyTuple_GET_SIZE(__pyx_args) != 3) {
      goto __pyx_L5_argtuple_error;
    } else {
      values[0] = PyTuple_GET_ITEM(__pyx_args, 0);
      values[1] = PyTuple_GET_ITEM(__pyx_args, 1);
      values[2] = PyTuple_GET_ITEM(__pyx_args, 2);
    }
    __pyx_v_u = __Pyx_PyObject_to_MemoryviewSlice_dsds_double(values[0], PyBUF_WRITABLE); if (unlikely(!__pyx_v_u.memview)) __PYX_ERR(0, 6, __pyx_L3_error)
    __pyx_v_v = __Pyx_PyObject_to_MemoryviewSlice_dsds_double(values[1], PyBUF_WRITABLE); if (unlikely(!__pyx_v_v.memview)) __PYX_ERR(0, 6, __pyx_L3_error)
    __pyx_v_res = __Pyx_PyObject_to_MemoryviewSlice_dsds_double(values[2], PyBUF_WRITABLE); if (unlikely(!__pyx_v_res.memview)) __PYX_ERR(0, 6, __pyx_L3_error)
  }
  goto __pyx_L4_argument_unpacking_done;
  __pyx_L5_argtuple_error:;
  __Pyx_RaiseArgtupleInvalid("matrix_multiply1", 1, 3, 3, PyTuple_GET_SIZE(__pyx_args)); __PYX_ERR(0, 6, __pyx_L3_error)
  __pyx_L3_error:;
  __Pyx_AddTraceback("_cython_magic_231f6419b2e54af595f8becce81af957.matrix_multiply1", __pyx_clineno, __pyx_lineno, __pyx_filename);
  __Pyx_RefNannyFinishContext();
  return NULL;
  __pyx_L4_argument_unpacking_done:;
  __pyx_r = __pyx_pf_46_cython_magic_231f6419b2e54af595f8becce81af957_matrix_multiply1(__pyx_self, __pyx_v_u, __pyx_v_v, __pyx_v_res);
  int __pyx_lineno = 0;
  const char *__pyx_filename = NULL;
  int __pyx_clineno = 0;

  /* function exit code */
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

static PyObject *__pyx_pf_46_cython_magic_231f6419b2e54af595f8becce81af957_matrix_multiply1(CYTHON_UNUSED PyObject *__pyx_self, __Pyx_memviewslice __pyx_v_u, __Pyx_memviewslice __pyx_v_v, __Pyx_memviewslice __pyx_v_res) {
  int __pyx_v_i;
  int __pyx_v_j;
  int __pyx_v_k;
  int __pyx_v_m;
  int __pyx_v_n;
  int __pyx_v_p;
  PyObject *__pyx_r = NULL;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("matrix_multiply1", 0);
/* … */
  /* function exit code */
  __pyx_r = Py_None; __Pyx_INCREF(Py_None);
  __PYX_XDEC_MEMVIEW(&__pyx_v_u, 1);
  __PYX_XDEC_MEMVIEW(&__pyx_v_v, 1);
  __PYX_XDEC_MEMVIEW(&__pyx_v_res, 1);
  __Pyx_XGIVEREF(__pyx_r);
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}
/* … */
  __pyx_tuple__19 = PyTuple_Pack(9, __pyx_n_s_u, __pyx_n_s_v, __pyx_n_s_res, __pyx_n_s_i, __pyx_n_s_j, __pyx_n_s_k, __pyx_n_s_m, __pyx_n_s_n, __pyx_n_s_p); if (unlikely(!__pyx_tuple__19)) __PYX_ERR(0, 6, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_tuple__19);
  __Pyx_GIVEREF(__pyx_tuple__19);
/* … */
  __pyx_t_1 = PyCFunction_NewEx(&__pyx_mdef_46_cython_magic_231f6419b2e54af595f8becce81af957_1matrix_multiply1, NULL, __pyx_n_s_cython_magic_231f6419b2e54af595); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 6, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_matrix_multiply1, __pyx_t_1) < 0) __PYX_ERR(0, 6, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  __pyx_codeobj__20 = (PyObject*)__Pyx_PyCode_New(3, 0, 9, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__19, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_root_cache_ipython_cython__cyth, __pyx_n_s_matrix_multiply1, 6, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__20)) __PYX_ERR(0, 6, __pyx_L1_error)

 07:     cdef int i, j, k

 08:     cdef int m, n, p

 09:

+10:     m = u.shape[0]

  __pyx_v_m = (__pyx_v_u.shape[0]);

+11:     n = u.shape[1]

  __pyx_v_n = (__pyx_v_u.shape[1]);

+12:     p = v.shape[1]

  __pyx_v_p = (__pyx_v_v.shape[1]);

 13:

+14:     with cython.nogil:

  {
      #ifdef WITH_THREAD
      PyThreadState *_save;
      Py_UNBLOCK_THREADS
      __Pyx_FastGIL_Remember();
      #endif
      /*try:*/ {
/* … */
      /*finally:*/ {
        /*normal exit:*/{
          #ifdef WITH_THREAD
          __Pyx_FastGIL_Forget();
          Py_BLOCK_THREADS
          #endif
          goto __pyx_L5;
        }
        __pyx_L5:;
      }
  }

+15:         for i in range(m):

        __pyx_t_1 = __pyx_v_m;
        __pyx_t_2 = __pyx_t_1;
        for (__pyx_t_3 = 0; __pyx_t_3 < __pyx_t_2; __pyx_t_3+=1) {
          __pyx_v_i = __pyx_t_3;

+16:             for j in range(p):

          __pyx_t_4 = __pyx_v_p;
          __pyx_t_5 = __pyx_t_4;
          for (__pyx_t_6 = 0; __pyx_t_6 < __pyx_t_5; __pyx_t_6+=1) {
            __pyx_v_j = __pyx_t_6;

+17:                 res[i,j] = 0

            __pyx_t_7 = __pyx_v_i;
            __pyx_t_8 = __pyx_v_j;
            *((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_res.data + __pyx_t_7 * __pyx_v_res.strides[0]) ) + __pyx_t_8 * __pyx_v_res.strides[1]) )) = 0.0;

+18:                 for k in range(n):

            __pyx_t_9 = __pyx_v_n;
            __pyx_t_10 = __pyx_t_9;
            for (__pyx_t_11 = 0; __pyx_t_11 < __pyx_t_10; __pyx_t_11+=1) {
              __pyx_v_k = __pyx_t_11;

+19:                     res[i,j] += u[i,k] * v[k,j]

              __pyx_t_8 = __pyx_v_i;
              __pyx_t_7 = __pyx_v_k;
              __pyx_t_12 = __pyx_v_k;
              __pyx_t_13 = __pyx_v_j;
              __pyx_t_14 = __pyx_v_i;
              __pyx_t_15 = __pyx_v_j;
              *((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_res.data + __pyx_t_14 * __pyx_v_res.strides[0]) ) + __pyx_t_15 * __pyx_v_res.strides[1]) )) += ((*((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_u.data + __pyx_t_8 * __pyx_v_u.strides[0]) ) + __pyx_t_7 * __pyx_v_u.strides[1]) ))) * (*((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_v.data + __pyx_t_12 * __pyx_v_v.strides[0]) ) + __pyx_t_13 * __pyx_v_v.strides[1]) ))));
            }
          }
        }
      }

```python
res = np.zeros((u.shape[0], v.shape[1]))
%timeit -r3 -n3 matrix_multiply1(u, v, res)
```

    3 loops, best of 3: 17 µs per loop

### Fibonacci series
The Fibonacci Sequence is the series of numbers:

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, ...

The following number is obtained by summing up the two numbers before it.

```python
def fib(n):
    a, b = 0, 1
    for i in range(n):
        a, b = b, a + b
    return a

@njit
def fib_numba(n):
    a, b = 0, 1
    for i in range(n):
        a, b = b, a + b
    return a
```

```cython
%%cython

def cfib(int n):
    cdef int i, a, b
    a, b = 0, 1
    for i in range(n):
        a, b = b, a + b
    return a
```

```python
%timeit fib(1000)
%timeit fib_numba(1000)
%timeit cfib(1000)
```

    10000 loops, best of 5: 92.2 µs per loop
    The slowest run took 78692.60 times longer than the fastest. This could mean that an intermediate result is being cached.
    1000000 loops, best of 5: 1.03 µs per loop
    1000000 loops, best of 5: 840 ns per loop

## Numba+ CFFI

One of the limitations of the above approach is when need some specialized function that is available in `Numpy` or `Scipy`, but that function has not been re-implemented in the `Numba` core library so it can be called in the so-called **nopython** mode. Basically this means that if we want to call one of these functions, we have to go through Numba's **object mode**, which typically cannot generate nearly as efficient code.

We have other options:

- The first is to re-implement a function using `Numba`.

- A second technique that involves using `CFFI` to call external C code directly within Numba code.

We can find a library [RMath](https://github.com/JuliaLang/Rmath-julia) that implements the same funtions of `scipy.stats` in C.

Other techniques to bind C code is introduced [here](http://www.scipy-lectures.org/advanced/interfacing_with_c/interfacing_with_c.html)

### Compile

The following script from [here](http://nbviewer.jupyter.org/github/synapticarbors/rmath-cffi-example/blob/master/rmath-cffi-example.ipynb)

```python
%%writefile build_rmath.py

import glob
import os
import platform

from cffi import FFI

include_dirs = [os.path.join('Rmath-julia', 'src'),
                os.path.join('Rmath-julia', 'include')]

rmath_src = glob.glob(os.path.join('Rmath-julia', 'src', '*.c'))

rmath_src = [f for f in rmath_src if ('librandom.c' not in f) and ('randmtzig.c' not in f)]

extra_compile_args = ['-DMATHLIB_STANDALONE']
extra_compile_args.append('-std=c99')

ffi = FFI()
ffi.set_source('_rmath_ffi', '#include <Rmath.h>',
        include_dirs=include_dirs,
        sources=rmath_src,
        libraries=[],
        extra_compile_args=extra_compile_args)

# We can add more available functions in Rmath here
ffi.cdef('''\
// Gamma Distribution
double dgamma(double, double, double, int);
double pgamma(double, double, double, int, int);
''')

if __name__ == '__main__':
    ffi.compile(verbose=False)
```

    Writing build_rmath.py

```python
!git clone https://github.com/phonchi/Rmath-julia.git
```

    Cloning into 'Rmath-julia'...
    remote: Enumerating objects: 765, done.
    remote: Counting objects: 100% (137/137), done.
    remote: Compressing objects: 100% (132/132), done.
    remote: Total 765 (delta 52), reused 11 (delta 2), pack-reused 628
    Receiving objects: 100% (765/765), 524.90 KiB | 4.82 MiB/s, done.
    Resolving deltas: 100% (495/495), done.

```python
!python build_rmath.py
```

    Rmath-julia/src/qt.c: In function ‘qt’:
    Rmath-julia/src/qt.c:105:29: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
      is_neg_lower = (lower_tail == neg); /* both TRUE or FALSE == !xor */
                                 ^~

Now that we have built our module wrapping `Rmath` using `cffi`, we can write Numba versions of scipy stats functions that we can call without additional overhead

```python
# Import our Rmath module
import _rmath_ffi

dgamma = _rmath_ffi.lib.dgamma
pgamma = _rmath_ffi.lib.pgamma
```

In order for us to use these methods from within a function that we can use JIT with `Numba`, we need to import `cffi_support` and register the module:

```python
from numba.core.typing import cffi_utils as cffi_support
cffi_support.register_module(_rmath_ffi)
```

```python
def cffi_gamma(x,alpha,beta):
    y = np.empty_like(x)
    for k in range(x.shape[0]):
        y[k] = dgamma(x[k], alpha, 1./beta, 0)

    return y
```

```python
@jit(nopython=True)
def nb_gamma_jit(x,alpha,beta):
    y = np.empty_like(x)
    for k in range(x.shape[0]):
        y[k] = dgamma(x[k], alpha, 1./beta, 0)

    return y

@vectorize(nopython=True)
def nb_gamma_vec(x,alpha,beta):
    return dgamma(x, alpha, 1./beta, 0) ##double x, double shape, double scale, int give_log
```

```python
import scipy.stats as st

x = np.random.normal(size=(100,))

def gamma_pdf(x,alpha,beta):
    gamma = st.gamma(alpha, loc=0, scale=1./beta)
    return gamma.pdf(x)

y1 = gamma_pdf(x,1.0,1.0)
y2 = cffi_gamma(x, 1.0, 1.0)
y3 = nb_gamma_vec(x, 1.0, 1.0)
y4 = nb_gamma_jit(x, 1.0, 1.0)
# Check that they all give the same results
print (np.allclose(y1, y2))
print (np.allclose(y1, y3))
print (np.allclose(y1, y4))
```

    True
    True
    True

```python
%timeit gamma_pdf(x,1.0,1.0)
%timeit cffi_gamma(x,1.0,1.0)
%timeit nb_gamma_jit(x,1.0,1.0)
%timeit nb_gamma_vec(x, 1.0, 1.0)
```

    The slowest run took 9.20 times longer than the fastest. This could mean that an intermediate result is being cached.
    1000 loops, best of 5: 811 µs per loop
    10000 loops, best of 5: 58.3 µs per loop
    The slowest run took 12.50 times longer than the fastest. This could mean that an intermediate result is being cached.
    100000 loops, best of 5: 4.58 µs per loop
    The slowest run took 8.36 times longer than the fastest. This could mean that an intermediate result is being cached.
    100000 loops, best of 5: 5.64 µs per loop

## Laboratories
Given that we have two, 2D arrays. `x` has a shape of $(M, P)$ and `y` has a shape of $(N, P)$. We want to compute the Euclidean distance (a.k.a. the $L_2$-distance) between *each pair* of rows between the two arrays. That is, if a given row of `x` is represented by $P$ numbers $(x_0, x_1, \ldots, x_{P-1})$, and similarly, a row `y` is represented by $(y_0, y_1, \ldots, y_{P-1})$, and we want to compute the Euclidean distance between the two rows:

\begin{equation}
\sqrt{(x_{0} - y_{0})^2 + (x_{1} - y_{1})^2 + \ldots + (x_{P-1} - y_{P-1})^2} = \sqrt{\sum_{i=0}^{P-1}{(x_{i} - y_{i})^2}}
\end{equation}

```python
# Baseline
def cdist(xs, ys):
    """Returns pairwise distance between row vectors in xs and ys.

    xs has shape (m, p)
    ys has shape (n, p)

    Return value has shape (m, n)
    """

    m, p = xs.shape
    n, p = ys.shape

    res = np.empty((m, n))
    for i in range(m):
        for j in range(n):
            res[i, j] = np.sqrt(np.sum((ys[j] - xs[i])**2))
    return res
```

```python
xs = np.arange(6).reshape(3,2).astype('float')
ys = np.arange(4).reshape(2,2).astype('float')
zs = cdist(xs, ys)
```

```python
zs
```

    array([[0.        , 2.82842712],
           [2.82842712, 0.        ],
           [5.65685425, 2.82842712]])

```python
%timeit -r 3 -n 10 cdist(xs, ys)
```

    101 µs ± 3.43 µs per loop (mean ± std. dev. of 3 runs, 10 loops each)

```cython
%%cython -a

import cython
import numpy as np
from libc.math cimport sqrt, pow

@cython.boundscheck(False)
@cython.wraparound(False)
def cdist_cython(double[:, :] xs, double[:, :] ys):
    """Returns pairwise distance between row vectors in xs and ys.

    xs has shape (m, p)
    ys has shape (n, p)

    Return value has shape (m, n)
    """

    cdef int m, n, p

    m = xs.shape[0]
    n = ys.shape[0]
    p = xs.shape[1]

    cdef double[:, :] res = np.empty((m, n))

    cdef int i, j

    cdef double s
    for i in range(m):
        for j in range(n):
            s = 0.0
            for k in range(p):
                s += pow(ys[j,k] - xs[i,k], 2)
            res[i, j] = sqrt(s)
    return res
```

<!DOCTYPE html>

    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Cython: _cython_magic_2d01417ec369243d2699243e7c557b66.pyx</title>

Generated by Cython 0.29.21

    Yellow lines hint at Python interaction.

    Click on a line that starts with a "+" to see the C code that Cython generated for it.

 01:

+02: import cython

  __pyx_t_1 = __Pyx_PyDict_NewPresized(0); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 2, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_test, __pyx_t_1) < 0) __PYX_ERR(0, 2, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

+03: import numpy as np

  __pyx_t_1 = __Pyx_Import(__pyx_n_s_numpy, 0, 0); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 3, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_np, __pyx_t_1) < 0) __PYX_ERR(0, 3, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;

 04: from libc.math cimport sqrt, pow

 05:

 06: @cython.boundscheck(False)

 07: @cython.wraparound(False)

+08: def cdist_cython(double[:, :] xs, double[:, :] ys):

/* Python wrapper */
static PyObject *__pyx_pw_46_cython_magic_2d01417ec369243d2699243e7c557b66_1cdist_cython(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds); /*proto*/
static char __pyx_doc_46_cython_magic_2d01417ec369243d2699243e7c557b66_cdist_cython[] = "Returns pairwise distance between row vectors in xs and ys.\n    \n    xs has shape (m, p)\n    ys has shape (n, p)\n    \n    Return value has shape (m, n)    \n    ";
static PyMethodDef __pyx_mdef_46_cython_magic_2d01417ec369243d2699243e7c557b66_1cdist_cython = {"cdist_cython", (PyCFunction)(void*)(PyCFunctionWithKeywords)__pyx_pw_46_cython_magic_2d01417ec369243d2699243e7c557b66_1cdist_cython, METH_VARARGS|METH_KEYWORDS, __pyx_doc_46_cython_magic_2d01417ec369243d2699243e7c557b66_cdist_cython};
static PyObject *__pyx_pw_46_cython_magic_2d01417ec369243d2699243e7c557b66_1cdist_cython(PyObject *__pyx_self, PyObject *__pyx_args, PyObject *__pyx_kwds) {
  __Pyx_memviewslice __pyx_v_xs = { 0, 0, { 0 }, { 0 }, { 0 } };
  __Pyx_memviewslice __pyx_v_ys = { 0, 0, { 0 }, { 0 }, { 0 } };
  PyObject *__pyx_r = 0;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("cdist_cython (wrapper)", 0);
  {
    static PyObject **__pyx_pyargnames[] = {&__pyx_n_s_xs,&__pyx_n_s_ys,0};
    PyObject* values[2] = {0,0};
    if (unlikely(__pyx_kwds)) {
      Py_ssize_t kw_args;
      const Py_ssize_t pos_args = PyTuple_GET_SIZE(__pyx_args);
      switch (pos_args) {
        case  2: values[1] = PyTuple_GET_ITEM(__pyx_args, 1);
        CYTHON_FALLTHROUGH;
        case  1: values[0] = PyTuple_GET_ITEM(__pyx_args, 0);
        CYTHON_FALLTHROUGH;
        case  0: break;
        default: goto __pyx_L5_argtuple_error;
      }
      kw_args = PyDict_Size(__pyx_kwds);
      switch (pos_args) {
        case  0:
        if (likely((values[0] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_xs)) != 0)) kw_args--;
        else goto __pyx_L5_argtuple_error;
        CYTHON_FALLTHROUGH;
        case  1:
        if (likely((values[1] = __Pyx_PyDict_GetItemStr(__pyx_kwds, __pyx_n_s_ys)) != 0)) kw_args--;
        else {
          __Pyx_RaiseArgtupleInvalid("cdist_cython", 1, 2, 2, 1); __PYX_ERR(0, 8, __pyx_L3_error)
        }
      }
      if (unlikely(kw_args > 0)) {
        if (unlikely(__Pyx_ParseOptionalKeywords(__pyx_kwds, __pyx_pyargnames, 0, values, pos_args, "cdist_cython") < 0)) __PYX_ERR(0, 8, __pyx_L3_error)
      }
    } else if (PyTuple_GET_SIZE(__pyx_args) != 2) {
      goto __pyx_L5_argtuple_error;
    } else {
      values[0] = PyTuple_GET_ITEM(__pyx_args, 0);
      values[1] = PyTuple_GET_ITEM(__pyx_args, 1);
    }
    __pyx_v_xs = __Pyx_PyObject_to_MemoryviewSlice_dsds_double(values[0], PyBUF_WRITABLE); if (unlikely(!__pyx_v_xs.memview)) __PYX_ERR(0, 8, __pyx_L3_error)
    __pyx_v_ys = __Pyx_PyObject_to_MemoryviewSlice_dsds_double(values[1], PyBUF_WRITABLE); if (unlikely(!__pyx_v_ys.memview)) __PYX_ERR(0, 8, __pyx_L3_error)
  }
  goto __pyx_L4_argument_unpacking_done;
  __pyx_L5_argtuple_error:;
  __Pyx_RaiseArgtupleInvalid("cdist_cython", 1, 2, 2, PyTuple_GET_SIZE(__pyx_args)); __PYX_ERR(0, 8, __pyx_L3_error)
  __pyx_L3_error:;
  __Pyx_AddTraceback("_cython_magic_2d01417ec369243d2699243e7c557b66.cdist_cython", __pyx_clineno, __pyx_lineno, __pyx_filename);
  __Pyx_RefNannyFinishContext();
  return NULL;
  __pyx_L4_argument_unpacking_done:;
  __pyx_r = __pyx_pf_46_cython_magic_2d01417ec369243d2699243e7c557b66_cdist_cython(__pyx_self, __pyx_v_xs, __pyx_v_ys);
  int __pyx_lineno = 0;
  const char *__pyx_filename = NULL;
  int __pyx_clineno = 0;

  /* function exit code */
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}

static PyObject *__pyx_pf_46_cython_magic_2d01417ec369243d2699243e7c557b66_cdist_cython(CYTHON_UNUSED PyObject *__pyx_self, __Pyx_memviewslice __pyx_v_xs, __Pyx_memviewslice __pyx_v_ys) {
  int __pyx_v_m;
  int __pyx_v_n;
  int __pyx_v_p;
  __Pyx_memviewslice __pyx_v_res = { 0, 0, { 0 }, { 0 }, { 0 } };
  int __pyx_v_i;
  int __pyx_v_j;
  double __pyx_v_s;
  int __pyx_v_k;
  PyObject *__pyx_r = NULL;
  __Pyx_RefNannyDeclarations
  __Pyx_RefNannySetupContext("cdist_cython", 0);
/* … */
  /* function exit code */
  __pyx_L1_error:;
  __Pyx_XDECREF(__pyx_t_1);
  __Pyx_XDECREF(__pyx_t_2);
  __Pyx_XDECREF(__pyx_t_3);
  __Pyx_XDECREF(__pyx_t_4);
  __Pyx_XDECREF(__pyx_t_5);
  __PYX_XDEC_MEMVIEW(&__pyx_t_6, 1);
  __Pyx_AddTraceback("_cython_magic_2d01417ec369243d2699243e7c557b66.cdist_cython", __pyx_clineno, __pyx_lineno, __pyx_filename);
  __pyx_r = NULL;
  __pyx_L0:;
  __PYX_XDEC_MEMVIEW(&__pyx_v_res, 1);
  __PYX_XDEC_MEMVIEW(&__pyx_v_xs, 1);
  __PYX_XDEC_MEMVIEW(&__pyx_v_ys, 1);
  __Pyx_XGIVEREF(__pyx_r);
  __Pyx_RefNannyFinishContext();
  return __pyx_r;
}
/* … */
  __pyx_tuple__19 = PyTuple_Pack(10, __pyx_n_s_xs, __pyx_n_s_ys, __pyx_n_s_m, __pyx_n_s_n, __pyx_n_s_p, __pyx_n_s_res, __pyx_n_s_i, __pyx_n_s_j, __pyx_n_s_s, __pyx_n_s_k); if (unlikely(!__pyx_tuple__19)) __PYX_ERR(0, 8, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_tuple__19);
  __Pyx_GIVEREF(__pyx_tuple__19);
/* … */
  __pyx_t_1 = PyCFunction_NewEx(&__pyx_mdef_46_cython_magic_2d01417ec369243d2699243e7c557b66_1cdist_cython, NULL, __pyx_n_s_cython_magic_2d01417ec369243d26); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 8, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  if (PyDict_SetItem(__pyx_d, __pyx_n_s_cdist_cython, __pyx_t_1) < 0) __PYX_ERR(0, 8, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  __pyx_codeobj__20 = (PyObject*)__Pyx_PyCode_New(2, 0, 10, 0, CO_OPTIMIZED|CO_NEWLOCALS, __pyx_empty_bytes, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_tuple__19, __pyx_empty_tuple, __pyx_empty_tuple, __pyx_kp_s_C_Users_phonchi_ipython_cython, __pyx_n_s_cdist_cython, 8, __pyx_empty_bytes); if (unlikely(!__pyx_codeobj__20)) __PYX_ERR(0, 8, __pyx_L1_error)

 09:     """Returns pairwise distance between row vectors in xs and ys.

 10:

 11:     xs has shape (m, p)

 12:     ys has shape (n, p)

 13:

 14:     Return value has shape (m, n)

 15:     """

 16:

 17:     cdef int m, n, p

 18:

+19:     m = xs.shape[0]

  __pyx_v_m = (__pyx_v_xs.shape[0]);

+20:     n = ys.shape[0]

  __pyx_v_n = (__pyx_v_ys.shape[0]);

+21:     p = xs.shape[1]

  __pyx_v_p = (__pyx_v_xs.shape[1]);

 22:

+23:     cdef double[:, :] res = np.empty((m, n))

  __Pyx_GetModuleGlobalName(__pyx_t_2, __pyx_n_s_np); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __pyx_t_3 = __Pyx_PyObject_GetAttrStr(__pyx_t_2, __pyx_n_s_empty); if (unlikely(!__pyx_t_3)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_3);
  __Pyx_DECREF(__pyx_t_2); __pyx_t_2 = 0;
  __pyx_t_2 = __Pyx_PyInt_From_int(__pyx_v_m); if (unlikely(!__pyx_t_2)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_2);
  __pyx_t_4 = __Pyx_PyInt_From_int(__pyx_v_n); if (unlikely(!__pyx_t_4)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_4);
  __pyx_t_5 = PyTuple_New(2); if (unlikely(!__pyx_t_5)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_5);
  __Pyx_GIVEREF(__pyx_t_2);
  PyTuple_SET_ITEM(__pyx_t_5, 0, __pyx_t_2);
  __Pyx_GIVEREF(__pyx_t_4);
  PyTuple_SET_ITEM(__pyx_t_5, 1, __pyx_t_4);
  __pyx_t_2 = 0;
  __pyx_t_4 = 0;
  __pyx_t_4 = NULL;
  if (CYTHON_UNPACK_METHODS && unlikely(PyMethod_Check(__pyx_t_3))) {
    __pyx_t_4 = PyMethod_GET_SELF(__pyx_t_3);
    if (likely(__pyx_t_4)) {
      PyObject* function = PyMethod_GET_FUNCTION(__pyx_t_3);
      __Pyx_INCREF(__pyx_t_4);
      __Pyx_INCREF(function);
      __Pyx_DECREF_SET(__pyx_t_3, function);
    }
  }
  __pyx_t_1 = (__pyx_t_4) ? __Pyx_PyObject_Call2Args(__pyx_t_3, __pyx_t_4, __pyx_t_5) : __Pyx_PyObject_CallOneArg(__pyx_t_3, __pyx_t_5);
  __Pyx_XDECREF(__pyx_t_4); __pyx_t_4 = 0;
  __Pyx_DECREF(__pyx_t_5); __pyx_t_5 = 0;
  if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  __Pyx_DECREF(__pyx_t_3); __pyx_t_3 = 0;
  __pyx_t_6 = __Pyx_PyObject_to_MemoryviewSlice_dsds_double(__pyx_t_1, PyBUF_WRITABLE); if (unlikely(!__pyx_t_6.memview)) __PYX_ERR(0, 23, __pyx_L1_error)
  __Pyx_DECREF(__pyx_t_1); __pyx_t_1 = 0;
  __pyx_v_res = __pyx_t_6;
  __pyx_t_6.memview = NULL;
  __pyx_t_6.data = NULL;

 24:

 25:     cdef int i, j

 26:

 27:     cdef double s

+28:     for i in range(m):

  __pyx_t_7 = __pyx_v_m;
  __pyx_t_8 = __pyx_t_7;
  for (__pyx_t_9 = 0; __pyx_t_9 < __pyx_t_8; __pyx_t_9+=1) {
    __pyx_v_i = __pyx_t_9;

+29:         for j in range(n):

    __pyx_t_10 = __pyx_v_n;
    __pyx_t_11 = __pyx_t_10;
    for (__pyx_t_12 = 0; __pyx_t_12 < __pyx_t_11; __pyx_t_12+=1) {
      __pyx_v_j = __pyx_t_12;

+30:             s = 0.0

      __pyx_v_s = 0.0;

+31:             for k in range(p):

      __pyx_t_13 = __pyx_v_p;
      __pyx_t_14 = __pyx_t_13;
      for (__pyx_t_15 = 0; __pyx_t_15 < __pyx_t_14; __pyx_t_15+=1) {
        __pyx_v_k = __pyx_t_15;

+32:                 s += pow(ys[j,k] - xs[i,k], 2)

        __pyx_t_16 = __pyx_v_j;
        __pyx_t_17 = __pyx_v_k;
        __pyx_t_18 = __pyx_v_i;
        __pyx_t_19 = __pyx_v_k;
        __pyx_v_s = (__pyx_v_s + pow(((*((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_ys.data + __pyx_t_16 * __pyx_v_ys.strides[0]) ) + __pyx_t_17 * __pyx_v_ys.strides[1]) ))) - (*((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_xs.data + __pyx_t_18 * __pyx_v_xs.strides[0]) ) + __pyx_t_19 * __pyx_v_xs.strides[1]) )))), 2.0));
      }

+33:             res[i, j] = sqrt(s)

      __pyx_t_19 = __pyx_v_i;
      __pyx_t_18 = __pyx_v_j;
      *((double *) ( /* dim=1 */ (( /* dim=0 */ (__pyx_v_res.data + __pyx_t_19 * __pyx_v_res.strides[0]) ) + __pyx_t_18 * __pyx_v_res.strides[1]) )) = sqrt(__pyx_v_s);
    }
  }

+34:     return res

  __Pyx_XDECREF(__pyx_r);
  __pyx_t_1 = __pyx_memoryview_fromslice(__pyx_v_res, 2, (PyObject *(*)(char *)) __pyx_memview_get_double, (int (*)(char *, PyObject *)) __pyx_memview_set_double, 0);; if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 34, __pyx_L1_error)
  __Pyx_GOTREF(__pyx_t_1);
  __pyx_r = __pyx_t_1;
  __pyx_t_1 = 0;
  goto __pyx_L0;

```python
np.allclose(cdist(xs, ys), cdist_cython(xs, ys))
```

    True

```python
%timeit  cdist_cython(xs, ys)
```

    2.77 µs ± 75.6 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

```python
m = 1000
n = 1000
p = 100

X = np.random.random((m, p))
Y = np.random.random((n, p))
```

```python
%timeit cdist(X, Y)
%timeit cdist_cython(X, Y)
```

    7.06 s ± 246 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    3.23 s ± 59.4 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

## Exercise 2:
- Calculate the pairwise euclidean distance between two matrices X and Y using `Numba` and report the speedup over baseline
- Hint: to get better performance, you can try writing the code with loops

```python
## Solution here
# Write the Numba acceleration code for cdist(xs, ys)
@njit
def cdist_numba(xs, ys):
    """Returns pairwise distance between row vectors in xs and ys.

    xs has shape (m, p)
    ys has shape (n, p)

    Return value has shape (m, n)
    """

    m, p = xs.shape
    n, p = ys.shape

    res = np.empty((m, n))
    for i in range(m):
        for j in range(n):
            res[i, j] = np.sqrt(np.sum((ys[j] - xs[i])**2))
    return res
```

```python
np.allclose(cdist(X, Y), cdist_numba(X, Y))
```

    True

```python
## Timing profiling for the above Numba code and report the speedup
%timeit cdist_numba(X, Y)
```

    314 ms ± 11.3 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

The speedup is 7.06/0.314~ 22.5 times faster

```python
### Loop unrolling
```

```python
@njit
def cdist_numba1(xs, ys):
    """Returns pairwise distance between row vectors in xs and ys.

    xs has shape (m, p)
    ys has shape (n, p)

    Return value has shape (m, n)
    """

    m, p = xs.shape
    n, p = ys.shape

    res = np.empty((m, n))
    for i in range(m):
        for j in range(n):
            s = 0
            for k in range(p):
                s += (ys[j,k] - xs[i,k])**2
            res[i, j] = np.sqrt(s)
    return res
```

```python
np.allclose(cdist(X, Y), cdist_numba1(X, Y))
```

    True

```python
%timeit cdist_numba1(X, Y)
```

    85.3 ms ± 1.82 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

## References
- https://numba.readthedocs.io/en/stable/user/5minguide.html# - A great introduction for `Numba` from official site
- https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S13A_Numba.html# - A series of great introduction for HPC
- https://nbviewer.jupyter.org/github/akittas/presentations/blob/master/pythess/numba/numba.ipynb?utm_source=newsletter_mailer&utm_medium=email&utm_campaign=weekly - An indepth discusion about `Numba`
- https://ipython-books.github.io/55-accelerating-python-code-with-cython/ - A gentle introduction to `Cython`
- https://people.duke.edu/~ccc14/sta-663-2018/notebooks/S13B_Cython.html - A series of great introduction for HPC
- https://people.duke.edu/~ccc14/bios-823-2020/notebooks/E05_Code_Optimization.html?highlight=performance - A series of great introduction for HPC


# Дружественные числа. Исследование

### #Занимательная Математика

#### Весь код на Github, ссылка в конце статьи!

Импорт библиотек

```python
from IPython.display import Image
from IPython.core.display import HTML
from IPython.core.interactiveshell import InteractiveShell
from scipy.ndimage.filters import gaussian_filter1d
from scipy.signal import savgol_filter
import numpy as np
import sympy as sp
import pandas as pd
import random as r
import time
import matplotlib.pyplot as plt
import ipyturtle as turtle
InteractiveShell.ast_node_interactivity = "all"

def drawPlot(ss,title="Скорости",y="Секунд",x="Номер итерации"):
    fig,ax=plt.subplots(figsize=(6,6))
    ax.set_facecolor("#F2F2F2")
    ax.grid()
    ax.set_title(title)
    ax.set_ylabel(y)
    ax.set_xlabel(x)
    ax.plot(ss)
```

```python
Image(url="https://sun9-46.userapi.com/c858036/v858036072/1e3bba/s6JwXFoOgLM.jpg", width=400)
```

Хммм. Что же такое дружественные числа?

Посмотрим на примере:
Есть число 220. И есть число 284. Так вот. Они - друзья.

Почему? Как определить дружественные числа?
Дружественные числа - это такая пара, у которой сумма собственных делителей каждого числа равна другому числу.

Посмотрим на практике. Создадим функцию поиска делителей, а потом просуммируем.

```python
def Divisors(num):
    from math import sqrt as mmsq
    s=set([1])
    i=1
    a=int(mmsq(num)+1)
    while i<=a:
        if(num//i==num):
            i+=1
            continue
        if (num%i==0):
            if (num//i!=i):
                s.add(num//i)
            s.add(i)
        i+=1
    return s
```

```python
Divisors(220)," #Делители числа 220"
Divisors(284)," #Делители числа 284"
```

    ({1, 2, 4, 5, 10, 11, 20, 22, 44, 55, 110}, ' #Делители числа 220')

    ({1, 2, 4, 71, 142}, ' #Делители числа 284')

1+2+4 = 6 Верно? Продолжим прибавлять..
1+2+4+5+10+11+20+22+44+55+110

Ну, или зачем вручную. Для подсчёта суммы любого множества(списка) есть готовая функция **sum()**

```python
sum(Divisors(220))
sum(Divisors(284))
```

    284

    220

И что мы наблюдаем? Сумма собственных делителей у 220 равна 284, а у числа 284 сумма - 220
Поэтому эти числа и являются парой друзей :)

```python
Image(url="https://wikimedia.org/api/rest_v1/media/math/render/svg/f5f9a3fcecf20f39301b0f651492c3fbe712262d")
```

```python
Image(url="https://wikimedia.org/api/rest_v1/media/math/render/svg/4c5a454d5e4adb6fb2e7e3e467e0804e28d01824")
```

А ещё кстати говоря, эта пара является первой из всего списка дружественных чисел!

Кстати интересный факт, но хоть нам и известно уже огромное количество таких пар, мы не знаем есть ли такая пара, у которой числа были разной чётности. Вот например 220 и 284 чётные...Есть числа, где первое и второе нечётные, но чтобы один чётный, другой нечётный, такого мы не знаем ещё.

А вот возникает наверное вопрос. Как находить эти пары?
Конкретной формулы пока ещё нет для этих чисел, поэтому я создал итерационную фунцию.

Для этого я создал функцию **AmicableNumber()**, которая на вход принимает какое-либо число. В самой функции алгоритм находит друга, если он есть на ближайшей дистанции по числовой оси.

```python
def AmicableNumber(k,returni=False):
    allDels = dict()
    from itertools import chain
    concatenated = chain( range(k, int(k*1.6)+1 ),range(k, int(k/1.6)+1 ,-1) )
    for i in concatenated:
        if(str(i) not in allDels):
            allDels[str(i)] = Divisors(i)
        if(i != k and sum(allDels[str(i)]) == k and sum(allDels[str(k)]) == i):
            if(returni):
                return (k,i)
            else:
                print(k,"->",i)
```

```python
AmicableNumber(1) #Например тут ответа нет
AmicableNumber(219) #Например тут ответа нет
AmicableNumber(220) #А тут уже есть
```

    220 -> 284

Если не понятно, что я имел в виду под "дистанциями", то объясню.  Для начала нам нужно найти хотя-бы 5 пар дружественных чисел.

```python
AmicableNumbers=[]
i=0
while len(AmicableNumbers)!=10:
    i+=1
    amn=AmicableNumber(i,True)
    if(amn is not None):
        AmicableNumbers.append(amn)

AmicableNumbers=[{tuple(sorted(amn)) for amn in AmicableNumbers}]
print(AmicableNumbers)
```

    [{(1184, 1210), (220, 284), (5020, 5564), (6232, 6368), (2620, 2924)}]

Решил найти только первые 5 пар, потому что таким алгоритмом всё это вычисляется примерно 10 минут.
А как он работает?

Мы просто перебираем относительно заданного числа x в интервале x/1.6 -> 1.6x
Это значительно ускоряет программу, но всё ещё недостаточно.
Вопрос, почему именно такие цифры?

Показываю. Возьмём первые 10 пар дружественных чисел, а потом составим вектор из отношений чисел в парах.

```python
df=pd.DataFrame([(220,284),(1184,1210),(2620,2924),(5020,5564),(6232,6368),(10744,10856),(12285,14595),(17296,18416),(63020,76084),(66928,66992)])
df
```

      0
      1

      0
      220
      284

      1
      1184
      1210

      2
      2620
      2924

      3
      5020
      5564

      4
      6232
      6368

      5
      10744
      10856

      6
      12285
      14595

      7
      17296
      18416

      8
      63020
      76084

      9
      66928
      66992

```python
ratio=[i[1]/i[0] for i in df.to_numpy()]
drawPlot(ratio,title="Отношения пар",x="Номер пары",y="Отношение")
```

На данном графике мы видим по оси Х номер пары, а на оси У отношение большего числа в паре к меньшему.
Это своего рода расстояние от 1 до другого числа.
То, есть 220 умножить на 1.29, мы получим 284. Нет смысла искать пару этому числу, если умножать больше чем на 1.29

У другого числа расстояние до его пары вообще близкое.
В целом среднее соотношение пар

```python
np.mean(ratio)
```

    1.1030557457540984

Кстати, раз уж мы заговорили про соотношения, попробуем составить числовую последовательность из дружественных чисел
А затем найдём соотношения чисел к предыдущим в таком ряду.

```python
seq=df.to_numpy().flatten()
seq #Собственно первые 20 чисел последовательности (10 пар)
```

    array([  220,   284,  1184,  1210,  2620,  2924,  5020,  5564,  6232,
            6368, 10744, 10856, 12285, 14595, 17296, 18416, 63020, 76084,
           66928, 66992])

```python
drawPlot(seq,title="Последовательность",x="Номер числа",y="Число")
```

Для начала я показал график, где мы видим как изменяется число со временем во всей последовательности.

```python
ratio = [seq[i]/seq[i-1] for i in range(1,len(seq))]
drawPlot(ratio,title="Отношение чисел",x="Номер числа",y="Отношение")
```

Вот пожалуйста, мы видим, что отношения чисел во всей последовательности отличаются от отношений чисел внутри пар!

Найдём производную графика. Да, ряд не сходится, но мы можем узнать скорость изменения отношений.

```python
dratio = [ratio[i]-ratio[i-1] for i in range(1,len(ratio))]
drawPlot(dratio,title="Скорость отношений чисел",x="Отношение",y="Скорость")
```

А ускорение?)

```python
ddratio = [dratio[i]-dratio[i-1] for i in range(1,len(dratio))]
drawPlot(ddratio,title="Ускорение отношений чисел",x="Скорость",y="Ускорение")
```

В целом не изменилось особо. Ах..мы это исследовали графики соотношений чисел!
Сделаем тоже самое с самой последовательностью.
Так мы узнаем, похожи ли графики.

Кстати, скорость ряда -
Другими словами это будет расстояние в единицах от 1 элемента ряда до другого.

```python
dseq = [seq[i]-seq[i-1] for i in range(1,len(seq))]
drawPlot(dseq,title="Дистанции между числами",x="Номер числа",y="Отношение")
```

Ого, скорость вроде нарастает, но что же интересно со следующими парами в дружественных числах?
Узнаем ускорение

```python
ddseq = [dseq[i]-dseq[i-1] for i in range(1,len(dseq))]
drawPlot(ddseq,title="Ускорение дружественных чисел",x="Номер числа",y="Ускорение")
```

Графики скорости и ускорения чисел не похожи на то же у их соотношений, но мы замечаем другое!
Не зная все следующие пары чисел после 11, мы можем предполагать, что скорость и ускорение начинают увеличиваться.

В принципе неудивительно.
Если мы вспомним график самого ряда, то заметим, что числа очень сильно увеличиваются со временем.

#### Плотность
Если вы читали предыдущую мою статью про числа Фибоначчи, то вы уже знаете про плотность последовательности.

Предлагаю исследовать и ряд дружественных чисел на это свойство! +Скорость изменения плотности!)

```python
densities=[(len(list(filter(lambda x: x < i, sorted(seq))))-1)/i for i in range(200,10000)]
drawPlot(densities,"Изменение плотности","Плотность","Дружественное число")
```

Ого! Неожиданно. Плотность вроде увеличивается, а потом уменьшается.
Причём, у вас, дорогие читатели, есть возможность поучаствовать в моём исследовании.
Сможете вычислить тангенс угла между проекциями на оХ скачков и осью оХ ?

```python
ddensities = [densities[i]-densities[i-1] for i in range(1,len(densities))]
drawPlot(ddensities,title="Скорость плотности",x="Дружественное число",y="Скорсоть")
```

Ну, значит в целом стремится к нулю, что неудивительно. Ведь числа значительно увеличиваются со временем, а также значительно увеличивается их дистанция от одной пары до другой.

## Алгоритмы

Ну хорошо, а можно ли придумать другой алгоритм нахождения дружественных чисел?
Конечно да

Я даже предлагаю использовать библиотеку **sympy**. Взять функцию от туда и сравнить по скорости обе.

Единственный недостаток. У библиотеки есть функция не нахождение дружественных чисел, а их проверка! Поэтому, самостоятельно создадим двухмерную итерацию.

```python
from sympy.ntheory.factor_ import is_amicable

for i in range(1,1000):
    for j in range(1,1000):
        if(is_amicable(i, j)):
            print(i,j)
```

    220 284
    284 220

Попробуйте воссоздать это на с++

Ведь на Python, циклы работают медленно, поэтому даже замерить в сравнении будет тяжело. Не думаю, что числа получатся годными для сравнения, но попробуем.

```python
%time is_amicable(1184,1210)
```

    CPU times: user 824 µs, sys: 0 ns, total: 824 µs
    Wall time: 832 µs

    True

```python
%time AmicableNumber(1184)
```

    1184 -> 1210
    CPU times: user 11.7 ms, sys: 11 µs, total: 11.7 ms
    Wall time: 11.2 ms

Так, мы видим, что нахождение пары для ЧИСЛА намного дольше, чем проверка уже готовой пары.
Однако при переборе пар чисел при проверках уходит намного больше времени!

## Другие способы построения ряда

Так или иначе, если хорошо поинтересоваться данной темой, можно найти ещё два способа построения последовательности дружественных чисел.

Представляю вашему вниманию: "Формула Сабита ибн Курра" и "Метод Вальтера Боро"

Начну с первого.

Если для натурального числа n>1 все три числа p,q,r простые, то будет пара дружественных чисел по формуле: (2^{n} * pq, 2^{n} * r),
где:
p = (3 * 2^{n-1}) - 1
q = (3 * 2^n) - 1
r = (9 * 2^{2*n-1})-1

```python
def _isPrime(n):
    if n%2==0:
        return n==2
    d=3
    while d*d<=n and n%d!=0:
        d+=2
    return d*d>n
```

```python
def Formula_Sabita_ibn_Kurra(n):
    p = (3 * 2**(n-1)) - 1
    q = (3 * 2**n) - 1
    r = (9 * 2**(2*n-1))-1

    if(_isPrime(p) and _isPrime(q) and _isPrime(r)):
        return ( 2**n * p * q, 2**n * r )
```

```python
Formula_Sabita_ibn_Kurra(2) # Нашло!
```

    (220, 284)

Эта формула даёт пары (220, 284), (17 296, 18 416) и (9 363 584, 9 437 056) соответственно для n=2,4,7 ,
но больше никаких пар дружественных чисел, которые могли бы быть получены по этой формуле для n < 20000 не существует.

Кроме того, многие пары дружественных чисел, например, (6232, 6368), не могут быть получены по этой формуле.

Так, что эта формула вышла именно для целей Занимательной Математики 😀

---

Так, а что по Методу Вальтера ?

Если для пары дружественных чисел вида A=a*u и B=a*s числа s и p=u+s+1 являются простыми, причём a не делится на p, то при всех тех натуральных n, при которых оба числа q_{1}=(u+1)p^{{n+1}}-1 и q_{2}=(u+1)(s+1)p^{n}-1 просты, числа
B_{1}=Ap^{n}q_{1} и B_{2}=ap^{n}q_{2} — дружественные.

Сложно? Понимаю. Сам долго пытался понять алгоритм

```python
def Walter_Borough(n,A=220,a=4,u=55,s=71):
    p = s+u+1
    q1 = (u+1)*p**n - 1
    if(_isPrime(q1)):
        q2 = (u+1)*(s+1)*p**n - 1
        if(_isPrime(q2)):
            return ( A*p**n *q1, a*p**n *q2 )
```

```python
Walter_Borough(2)
```

    (3204978428740, 4195612705532)

Но и тут найти весь ряд не выйдет. То есть можно лишь найти огромные числа!
Попробуем, но это будут лоскутики от всей последовательности дружественных чисел

```python
for i in range(1,50):
    if(Walter_Borough(i)):
        print(i,end=",")
```

    2,

Решил проверить первые 50 чисел по Вальтеру, но в итоге ушло 26 секунд...так ещё и найдена была только одна пара :)

Почему так долго? В принципе, давайте спроектируем график скорости, чтобы разобраться

```python
#Создаём массив, в котором будем хранить тайминги
s = np.array([])

#Сделаем итерации такие, что будем подавать на вход индексы от 1 до 50
for i in range(1,50):

    #Включаем "таймер"
    start_time = time.time()

    Walter_Borough(i)

    #Стоп таймер (время зафиксируется как только закончится выполнения рекурсии)
    end_time = (time.time() - start_time)
    s=np.append(s,end_time)
    #print("--- %s seconds ---" % end_time)
```

    (3204978428740, 4195612705532)

```python
drawPlot(s,title="Скорость Вальтера Боро",x="Номер итерации",y="сек.")
```

Дело в том, что мы каждый раз проверяем число на простоту, а это в свою очередь разбор числа на делимости.
Поэтому, как попадётся огромное простое число уходит много времени на обработку!

---

В целом, я готов предоставить первые дружественные 42 пары (84 числа):

220,284,1184,1210,2620,2924,5020,5564,6232,6368,10744,10856,12285,14595,17296,18416,63020,76084,66928,66992,67095,71145,69615,87633,79750,88730,100485,124155,122265,139815,122368,123152,141664,153176,142310,168730,171856,176336,176272,180848,185368,203432,196724,202444,280540,365084,308620,389924,319550,430402,356408,399592,437456,455344,469028,486178,503056,514736,522405,525915,600392,669688,609928,686072,624184,691256,635624,712216,643336,652664,667964,783556,726104,796696,802725,863835,879712,901424,898216,980984,947835,1125765,998104,1043096

```python
seq=[220,284,1184,1210,2620,2924,5020,5564,6232,6368,10744,10856,12285,14595,17296,18416,63020,76084,66928,66992,67095,71145,69615,87633,79750,88730,100485,124155,122265,139815,122368,123152,141664,153176,142310,168730,171856,176336,176272,180848,185368,203432,196724,202444,280540,365084,308620,389924,319550,430402,356408,399592,437456,455344,469028,486178,503056,514736,522405,525915,600392,669688,609928,686072,624184,691256,635624,712216,643336,652664,667964,783556,726104,796696,802725,863835,879712,901424,898216,980984,947835,1125765,998104,1043096]
```

### Статистика

Узнаем сколько чётных и нечётных чисел в последовательности?
Это также покажет, сколько чётных и нечётных пар!

```python
InteractiveShell.ast_node_interactivity = "last"

EvenAmicable=[i for i in seq if(i%2==0)]
len(EvenAmicable)
```

    68

```python
plt.pie([len(EvenAmicable),84-len(EvenAmicable)],
        colors=["#A7FF5B","#E7AFFF"],
        labels=["Чётные","Нечётные"],
        autopct='%1.1f%%',
        shadow=True,
        textprops={'color':"black"})
plt.legend([len(EvenAmicable),84-len(EvenAmicable)])
;
```

То есть 68 чисел из 84 - чётные.
И 32 пары из 42 - чётные.

Исследуем теперь на простоту!

```python
PrimeAmicable=[i for i in seq if(_isPrime(i))]
plt.pie([len(PrimeAmicable),84-len(PrimeAmicable)],
        colors=["#A7FF5B","#E7AFFF"],
        labels=["Составные","Простые"],
        autopct='%1.1f%%',
        shadow=True,
        textprops={'color':"black"})
plt.legend([len(PrimeAmicable),84-len(PrimeAmicable)])
;
```

ШОК Контент! В первых 84 числах ни одно не является простым!

### Фракталы

В завершение моего доклада по дружественным числам, я бы хотел поиграться с самой интересной частью.
Рисование фракталов.

Хотя в текущем случае выйдет, что мы будем просто рисовать спирали согласну тому или иному углу последовательности

```python
ratio=[seq[i]/seq[i-1] for i in range(1,len(seq))]
```

В общем, я нашёл отношения чисел в первых 42 дружественных парах.
Попробуем нарисовать два рисунка. Первый будет рисовать по среднему значению, а второй по мере перечисления списка отношений.

Но вы можете спросить. Как знать насколько поворачивать в спирали? Угол просчитать очень просто.
Мы берём число x. И узнаём угол по формуле angle=2pi/x^2

$ угол={{2 \pi }\over{x^2}} $

```python
np.mean(ratio)
```

    1.1533085895428854

Таким образом, мы знаем среднее отношение дружественных чисел в первых 42 парах.
Значит угол равен:

```python
angle=sp.N(360/np.mean(ratio)**2)
angle
```

$\displaystyle 270.652128987702$

```python
#Создаём картинку 300х300

N=300
t = turtle.Turtle(fixed=False, width=N, height=N)
t.hideturtle()
t
```

    Turtle()

```python
for i in range(15):
    for j in range(int(angle)):
        t.forward(i/10)
        t.left(1)
        if(i>10):
            time.sleep(0.01) # Чтобы не терять рисунок
```

Получилось что-то примерно типа розочки )))

А теперь посмотрим на динамический угол!

```python
t = turtle.Turtle(fixed=False, width=N, height=N)
t.hideturtle()
t
```

    Turtle()

```python
for i in range(15):
    angle=sp.N(360/ratio[i]**2)
    for j in range(int(angle)):
        t.forward(i/10)
        t.left(1)
        if(i>10):
            time.sleep(0.04) # Чтобы не терять рисунок
```

В целом криво получается при динамике, но уже более менее красивый рисунок при среднем значении отношений дружественных чисел.
Хотя мы представляем эти пары как друзей, так или иначе в древности они вызывали ассоциацию и с любовной парой. Поэтому, забавно, что рисунок спирали похож на розу. (Мне так кажется)

### Подведём итог

Если говорить об открытых проблемах в математике дружественных чисел:

Неизвестно, конечно или бесконечно количество пар дружественных чисел. На апрель 2016 года известно более 1 000 000 000 пар дружественных чисел. Все они состоят из чисел одинаковой чётности.

Неизвестно, существует ли чётно-нечётная пара дружественных чисел.

Также неизвестно, существуют ли взаимно простые дружественные числа, но если такая пара дружественных чисел существует, то их произведение должно быть больше 1067.

# Конец

На этом всё, дорогие читатели!
Надеюсь вам было интересно узнать что-то вроде такого из занимательной математики 😍

Ожидайте следующей статьи, в которой будет исследование компанейских чисел.
Если у вас есть мысли или пожелания, обязательно напишите...Может найду что подправить или добавить.

---

Здесь ссылка на датасеты дружественных чисел
[Дружественные числа](https://github.com/lonagi/pysasha/tree/master/datasets/Number%20Theory/Amicable%20Numbers)

Датасет хранит 42 пары, а так есть отдельные, для последовательности из 84 чисел.

Если есть желание прочитать статью на другом источнике, то пожалуйста, вот ниже ссылки:

[ВКонтакте](vk.com/@lonagi-druzhestvennye-chisla-issledovanie)
[Instagram](https://www.instagram.com/p/B__8gjzDa1r/)
[Facebook](https://www.facebook.com/%D0%97%D0%B0%D0%BD%D0%B8%D0%BC%D0%B0%D1%82%D0%B5%D0%BB%D1%8C%D0%BD%D0%B0%D1%8F-%D0%BC%D0%B0%D1%82%D0%B5%D0%BC%D0%B0%D1%82%D0%B8%D0%BA%D0%B0-%D0%BE%D1%82-Lonagi-112410007105730)
[Github](https://github.com/lonagi/pysasha/blob/master/docs/Number%20Theory/Amicable%20Numbers/AmicableNumbers.md)

```python

```


# Quantum Entanglement and Separability

Date : December 15, 2021

This notebook contains material supporting a paper, currently titled *Five Starter Pieces: Quantum Information Science via Semi-definite Programs*, by Vikesh Siddhu (vsiddhu@protonmail.com) and Sridhar Tayur (stayur@cmu.edu). The paper is available on this **[arXiv](http://arxiv.org/abs/2112.08276)** link. The arXiv paper is released there is under the **[arXiv.org perpetual, non-exclusive license](https://arxiv.org/licenses/nonexclusive-distrib/1.0/license.html)**, and this code is released under the **[MIT license](https://opensource.org/licenses/MIT)**.

This notebook depends upon various packages including [numpy](https://numpy.org/) >= 1.19.5, [picos](https://picos-api.gitlab.io/picos/index.html) >= 2.2.55, and [cvxopt](http://cvxopt.org/) >= 1.2.5.

[](https://colab.research.google.com/github/vsiddhu/SDP-Quantum-OR/blob/master/Notebook%204%20-%20Quantum%20Entanglement%20and%20Separability.ipynb)

## Introduction

By definition, a quantum state $\rho_{ab}$ on $\mathcal{H}_a \otimes \mathcal{H}_{b}$ is separable if the state can be written as a convex combination of pure states; in other words,
$$
\rho_{ab1} = \sum_i p_i|\psi_i \rangle \langle \psi_i| \otimes |\phi_i \rangle \langle \phi_i|,
$$
where $p_i>0$ and $\sum_i p_i = 1$. A state is entangled if it is not separable.

A state $\rho_{ab}$ is separable if and only if, for each $k \geq 1$ there is a positive (under) partial transpose (PPT) symmetric extension $\rho_{aB}$, where
$$\mathcal{H}_B := \bigotimes_{j=1}^k \mathcal{H}_{bj},$$
and all $\mathcal{H}_{bi}$ have same dimension as $\mathcal{H}_b$. For each $k$, the existence of a PPT symmetric extension can be formulated as an SDP. Here we discuss SDPs coming from the cases $k=1$ and $k=2$.

#### Case 1: SDP for $k=1$

\begin{align}
\begin{aligned}
\text{minimize} \; & \mu \\
\text{subject to} \; & \mu I_{ab} + T_{b1}(\rho_{ab}) \succeq 0
\end{aligned}
\end{align}

If the optimal value of the SDP above is strictly positive, then $\rho_{ab}$ does not have a PPT symmetric extension for $k=1$ and thus $\rho_{ab}$ is entangled.

#### Case 2: SDP for $k=2$

To check for a PPT symmetric extension of $\rho_{ab}$ at $k=2$, we let $\rho_{ab1} = \rho_{ab}$ and solve an SDP

\begin{align}
\begin{aligned}
\text{minimize} \; & \mu \\
\text{subject to} \; & \rho_{ab1} = \rm Tr_{b2}(\rho_{aB}) & \\
& \rho_{aB} = \Pi_{b1b2} \rho_{aB} \Pi_{b1b2}, & \\
& \rm Tr(\rho_{aB}) = 1, & \\
& \rho_{aB} + \mu I_{aB} \succeq 0, &\\
& \mathcal{T}_{b1}(\rho_{aB}) + \mu I_{aB} \succeq 0, & \\
& \mathcal{T}_{b1 b2}(\rho_{aB}) + \mu I_{aB} \succeq 0, & \\
\end{aligned}
\end{align}

where $\Pi_{b1b2} = I_a \otimes S_{b1b2}$, and the swap operator $S_{b1b2}|i \rangle_{b1} |j \rangle_{b2} = |j \rangle_{b1} |i \rangle_{b2}$. If the optimal value of the SDP above is strictly positive, then $\rho_{ab}$ does not have a PPT symmetric extension for $k=2$ and thus $\rho_{ab}$ is entangled.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
# For Google Colab use, commands installing packages
try:
  import google.colab
  IN_COLAB = True
except:
  IN_COLAB = False

# Install PICOS and CVXOPT in Google Colab
if IN_COLAB:
    !pip install -q picos
    !pip install -q cvxopt
```

```python
import picos as pic
import cvxopt as cvx
#Import functions to help formulate the SDP
from HelperFunctionsforNotebook4 import swapLastTwoOfThreePartyState
```

```python
print('Solvers supported on this installation of picos:', pic.solvers.all_solvers().keys())

```

    Solvers supported on this installation of picos: dict_keys(['cplex', 'cvxopt', 'ecos', 'glpk', 'gurobi', 'mosek', 'mskfsn', 'scip', 'smcp'])

```python
print('Solvers available to picos on this machine :', pic.solvers.available_solvers())
```

    Solvers available to picos on this machine : ['cvxopt', 'mosek', 'mskfsn']

### Example 1
Given the bell state $| \phi \rangle = \frac{1}{\sqrt{2}}(|00 \rangle + |11 \rangle)$, check whether $\rho_{ab} = | \phi \rangle \langle \phi |$ has a $k=1$ PPT symmetric extension.

```python
#State to check separability
phiMat = np.array([[1.,0.,0.,1.],
                [0.,0.,0.,0.],
                [0.,0.,0.,0.],
                [1.,0.,0.,1.]])/2

da = 2
db1 = 2
dB = db1
```

```python
#Constants
#----------
pMPic = pic.Constant("phiMat", phiMat)
iMatAB1 = pic.Constant('Iab1', np.eye(da*dB))
shpAB1 = (da*dB,da*dB)

#Variables
#----------
mu = pic.RealVariable("mu")

```

```python
prob1P = pic.Problem()

#Constraint
#----------
prob1P.add_constraint(pic.partial_transpose(pMPic,subsystems=(1), dimensions=(da,dB)) + mu*iMatAB1 >> 0)

```

    <4×4 LMI Constraint: phiMat.{[2×2]⊗[2×2]ᵀ} + mu·Iab1 ≽ 0>

```python
#Objective
#----------
prob1P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob1P)
```

    ---------------------------------------
    Semidefinite Program
      minimize mu
      over
        1×1 real variable mu
      subject to
        phiMat.{[2×2]⊗[2×2]ᵀ} + mu·Iab1 ≽ 0
    ---------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob1P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
print('Status at the end of solving the problem:', prob1P.status)

mu1P =  prob1P.value
print('Least mu obtained from solving the SDP above', mu1P)
print('Since this SDP value is positive, the state is entangled')

```

    Status at the end of solving the problem: optimal
    Least mu obtained from solving the SDP above 0.49999999985774324
    Since this SDP value is positive, the state is entangled

### Example 2
Check existence of PPT symmetric extension with $k=1$ for a two-qutrit state,
\begin{equation}
\rho_{ab} = \frac{2}{7}|\psi_{+}\rangle \langle\psi_{+}|
+ \frac{\alpha}{7} \sigma_{+} + \frac{5 - \alpha}{7} S_{ab} \sigma_{+} S_{ab},
\end{equation}
where $|\psi_+ \rangle = \frac{1}{\sqrt{3}}( |00\rangle + |11\rangle + |22 \rangle )$,
$\sigma_{+} = \frac{1}{3} (|01 \rangle \langle 01| + |12 \rangle \langle 12|
+ |20 \rangle \langle 20|)$, $S_{ab}$ is the swap operator, and $0 \leq \alpha \leq 5/2$,

```python
da = 3
db1 = 3
dB = db1

#Construct input state
idMat = np.eye(da*db1)
psiP = idMat[0] + idMat[4] + idMat[8]
psiP = np.outer(psiP,psiP)
psiP = psiP/np.trace(psiP)

sigPlus = np.outer(idMat[1],idMat[1]) + np.outer(idMat[5],idMat[5]) + np.outer(idMat[6],idMat[6])
sigPlus = sigPlus/np.trace(sigPlus)
sigPlusEx = np.outer(idMat[3],idMat[3]) + np.outer(idMat[7],idMat[7]) + np.outer(idMat[2],idMat[2])
sigPlusEx = sigPlusEx/np.trace(sigPlusEx)

al = 0.75
rhoMat = (2/7)*psiP + (al/7)*sigPlus + ((5-al)/7)*sigPlusEx
```

```python
#Constants
#----------
pMPic = pic.Constant("rhoAl", rhoMat)
iMatAB = pic.Constant('Iab1', np.eye(da*dB))
shpAB = (da*dB,da*dB)

#Variables
#----------
mu = pic.RealVariable("mu")
```

```python
prob2P = pic.Problem()

#Constraint
#----------
prob2P.add_constraint(pic.partial_transpose(pMPic,subsystems=(1), dimensions=(da,dB)) + mu*iMatAB >> 0)

```

    <9×9 LMI Constraint: rhoAl.{[3×3]⊗[3×3]ᵀ} + mu·Iab1 ≽ 0>

```python
#Objective
#----------
prob2P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob2P)
```

    --------------------------------------
    Semidefinite Program
      minimize mu
      over
        1×1 real variable mu
      subject to
        rhoAl.{[3×3]⊗[3×3]ᵀ} + mu·Iab1 ≽ 0
    --------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob2P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
print('Status at the end of solving the problem:', prob2P.status)
mu2P =  prob2P.value

print('The input state paramter alpha is', al)
print('Least mu obtained from solving the SDP above is', mu2P)
```

    Status at the end of solving the problem: optimal
    The input state paramter alpha is 0.75
    Least mu obtained from solving the SDP above is 0.007501735821370089

The value of the SDP above depends on the parameter $\alpha$. When $\alpha < 1$, $\rho_{ab}$ has no PPT symmetric extension for $k \geq 1$, and hence the SDP value is strictly positive.

### Example 3
Check for existence of $k=2$ PPT symmetric extension for the state defined in Example 1.

```python
#State to check separability
phiMat = np.array([[1.,0.,0.,1.],
                [0.,0.,0.,0.],
                [0.,0.,0.,0.],
                [1.,0.,0.,1.]])/2

da = 2
db = 2
db1 = db
db2 = db
dB = db1*db2
```

```python
#Constants
#----------
pMPic = pic.Constant("phiMat", phiMat)
iMatAB = pic.Constant('IaB', np.eye(da*dB))
shpAB = (da*dB,da*dB)
shpAbis = (da,db1,db2)

#Variables
#----------
mu = pic.RealVariable("mu")
ZPic = pic.HermitianVariable("ZaB", shpAB)
```

```python
prob3P = pic.Problem()

#Constraint
#----------
#Partial Trace returns the original state
prob3P.add_constraint(pic.partial_trace(ZPic, subsystems=(2),dimensions=shpAbis) == pMPic)
#Swapping b1 and b2 spaces leaves the state invariant
prob3P.add_constraint(swapLastTwoOfThreePartyState(ZPic,da,db) == ZPic)

prob3P.add_constraint(pic.trace(ZPic) == 1)
prob3P.add_constraint(ZPic + mu*iMatAB >> 0)

#Partial Transpose constraints for transpose on b1 and b2 spaces
prob3P.add_constraint(pic.partial_transpose(ZPic,subsystems=(1), dimensions=shpAbis)  + mu*iMatAB >> 0)
prob3P.add_constraint(pic.partial_transpose(ZPic,subsystems=(1,2), dimensions=shpAbis) + mu*iMatAB >> 0)

```

    <8×8 Complex LMI Constraint: ZaB.{[2×2]⊗[2×2]ᵀ⊗[2×2]ᵀ} + mu·IaB ≽ 0>

```python
#Objective
#----------
prob3P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob3P)
```

    ----------------------------------------------
    Complex Semidefinite Program
      minimize mu
      over
        8×8 hermitian variable ZaB
        1×1 real variable mu
      subject to
        ZaB.{[2×2]⊗[2×2]⊗tr([2×2])} = phiMat
        ZaB = shuffled(ZaB,ikjlnm,(2,2,2,2,2,2),C)
        tr(ZaB) = 1
        ZaB + mu·IaB ≽ 0
        ZaB.{[2×2]⊗[2×2]ᵀ⊗[2×2]} + mu·IaB ≽ 0
        ZaB.{[2×2]⊗[2×2]ᵀ⊗[2×2]ᵀ} + mu·IaB ≽ 0
    ----------------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob3P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
print('Status at the end of solving the problem:', prob3P.status)

mu3P =  prob3P.value

print('Least mu obtained from solving the SDP above is', mu3P)

```

    Status at the end of solving the problem: optimal
    Least mu obtained from solving the SDP above is 0.24999999989117092

The optimal value is found to be strictly positive. This is expected from results of Example 1, where no PPT  symmetric extension was found for $k=1$. If there is no symmetric extension for some $k$, then for $k' \geq k$, there is no symmetric extension either.

### Example 4
Check for existence of $k=2$ PPT symmetric extension for the state $\rho_{ab}$ defined in Example 2

```python
da = 3
db = 3
db1 = db
db2 = db
dB = db1*db2
```

```python
#Re-construct input state from Example 2
idMat = np.eye(da*db)
psiP = idMat[0] + idMat[4] + idMat[8]
psiP = np.outer(psiP,psiP)
psiP = psiP/np.trace(psiP)
```

```python
sigPlus = np.outer(idMat[1],idMat[1]) + np.outer(idMat[5],idMat[5]) + np.outer(idMat[6],idMat[6])
sigPlus = sigPlus/np.trace(sigPlus)
sigPlusEx = np.outer(idMat[3],idMat[3]) + np.outer(idMat[7],idMat[7]) + np.outer(idMat[2],idMat[2])
sigPlusEx = sigPlusEx/np.trace(sigPlusEx)
```

```python
al = 1.95
rhoMat = (2/7)*psiP + (al/7)*sigPlus + ((5-al)/7)*sigPlusEx
```

```python
#Constants
#----------
pMPic = pic.Constant("phiMat", rhoMat)
iMatAB = pic.Constant('IaB', np.eye(da*dB))
shpAB = (da*dB,da*dB)
shpAbi = (da,db1,db2)

#Variables
#----------
mu = pic.RealVariable("mu")
ZPic = pic.HermitianVariable("ZaB", shpAB)
```

```python
prob4P = pic.Problem()

#Constraint
#----------
#Partial Trace returns the original state
prob4P.add_constraint(pic.partial_trace(ZPic, subsystems=(2),dimensions=shpAbi) == pMPic)
prob4P.add_constraint(swapLastTwoOfThreePartyState(ZPic,da,db) == ZPic)

prob4P.add_constraint(pic.trace(ZPic) == 1)
prob4P.add_constraint(ZPic + mu*iMatAB >> 0)

# #Partial Transpose constraints for transpose on b1 and b2 spaces
prob4P.add_constraint(pic.partial_transpose(ZPic,subsystems=(1), dimensions=shpAbi)  + mu*iMatAB >> 0)
prob4P.add_constraint(pic.partial_transpose(ZPic,subsystems=(1,2), dimensions=shpAbi) + mu*iMatAB >> 0)
```

    <27×27 Complex LMI Constraint: ZaB.{[3×3]⊗[3×3]ᵀ⊗[3×3]ᵀ} + mu·IaB ≽ 0>

```python
#Objective
#----------
prob4P.set_objective('min',mu)

#User readable view of the problem being composed in PICOS'
print(prob4P)
```

    ----------------------------------------------
    Complex Semidefinite Program
      minimize mu
      over
        27×27 hermitian variable ZaB
        1×1 real variable mu
      subject to
        ZaB.{[3×3]⊗[3×3]⊗tr([3×3])} = phiMat
        ZaB = shuffled(ZaB,ikjlnm,(3,3,3,3,3,3),C)
        tr(ZaB) = 1
        ZaB + mu·IaB ≽ 0
        ZaB.{[3×3]⊗[3×3]ᵀ⊗[3×3]} + mu·IaB ≽ 0
        ZaB.{[3×3]⊗[3×3]ᵀ⊗[3×3]ᵀ} + mu·IaB ≽ 0
    ----------------------------------------------

```python
#Solve the problem using cvxopt as a solver
prob4P.solve(verbosity=False,solver='cvxopt')
```

    <primal feasible solution pair (claimed optimal) from cvxopt>

```python
print('Status at the end of solving the problem:', prob4P.status)
mu4P =  prob4P.value

print('The input state paramter alpha is', al)
print('Least mu obtained from solving the SDP above is', mu4P)

```

    Status at the end of solving the problem: optimal
    The input state paramter alpha is 1.95
    Least mu obtained from solving the SDP above is 0.0003607212345043077

The value of the SDP above depends on the parameter $\alpha$.

* In Example 2 we found when $\alpha < 1$, $\rho_{\alpha}$ has no PPT symmetric extension for $k = 1$. Here $k=2$ and we also find no symmetric extension. This absense of a PPT symmetric extension is shown here by an optimal $k=2$ SDP value which is strictly positive.

* In Example 2 we also found that $k=1$ PPT symmetric extension exists when $1 \leq \alpha < 2$. However, in the current case we find no PPT symmetric extension for $k=2$; that is, a postive optimal value is returned by the SDP solver. This positive value indicates that the $k=2$ SDP finds entanglement for $1 \leq \alpha <2$, but the $k=1$ SDP does not.

* In Example 2 we found when $2 \leq \alpha \leq 5/2$, there is a symmetric extension for $k=1$. In the current case we find a symmetric extension for $k=2$ as well. This symmetric extension is expected because the state $\rho_{\alpha}$ is separable for $2 \leq \alpha \leq 5/2$.

```python

```

Notebook 4 - Quantum Entanglement and Separability.ipynb