Advanced Expression Manipulation

In this section, we discuss some ways that we can perform advanced manipulation of expressions.

Understanding Expression Trees

Quick Tip

To play with the srepr form of expressions in the SymPy Live shell,    change the output format to Repr in the settings.

Before we can do this, we need to understand how expressions are represented in SymPy.  A mathematical expression is represented as a tree.  Let us take the expression 2^x + xy, i.e., 2**x + x*y.  We can see what this expression looks like internally by using srepr

```julia
    >>> from sympy import *
    >>> x, y, z = symbols('x y z')

    >>> expr = 2**x + x*y
    >>> srepr(expr)
    "Add(Pow(Integer(2), Symbol('x')), Mul(Symbol('x'), Symbol('y')))"
```

In Julia:

We replace the import command with a using command, as this will import functions (not Classes though) from sympy

```julia
using SymPy
x, y, z = symbols("x y z")

expr = 2^x + x*y
srepr(expr)
```

Add(Pow(Integer(2), Symbol('x')), Mul(Symbol('x'), Symbol('y')))

The easiest way to tear this apart is to look at a diagrm of the expression tree. Here is a diagram.

Note

This comes from dotprint(2**x + x*y, labelfunc=srepr). But we don't render digraph objects here

First, let's look at the leaves of this tree.  Symbols are instances of the class Symbol.  While we have been doing

```julia
    >>> x = symbols('x')
```

In Julia:

```julia
x = symbols("x")
```

\begin{equation*}x\end{equation*}

we could have also done

```julia
    >>> x = Symbol('x')
```

In Julia:

this can be done, but @vars would be suggested:

```julia
x = sympy.Symbol("x")
```

\begin{equation*}x\end{equation*}

```julia
@vars x
```

    (x,)

Either way, we get a Symbol with the name "x" [#symbols-fn]_.  For the number in the expression, 2, we got Integer(2).  Integer is the SymPy class for integers.  It is similar to the Python built-in type int, except that Integer plays nicely with other SymPy types.

When we write 2**x, this creates a Pow object.  Pow is short for "power".

```julia
    >>> srepr(2**x)
    "Pow(Integer(2), Symbol('x'))"
```

In Julia:

we replace ** by ^

```julia
srepr(2^x)
```

Pow(Integer(2), Symbol('x'))

We could have created the same object by calling Pow(2, x)

```julia
    >>> Pow(2, x)
    2**x
```

In Julia:

Pow is not a function, rather a managed property, so we must qualify it, as it wasn't brought in when loading the package

```julia
sympy.Pow(2, x)
```

\begin{equation*}2^{x}\end{equation*}

Note that in the srepr output, we see Integer(2), the SymPy version of integers, even though technically, we input 2, a Python int.  In general, whenever you combine a SymPy object with a non-SymPy object via some function or operation, the non-SymPy object will be converted into a SymPy object.  The function that does this is sympify [#sympify-fn]_.

```julia
    >>> type(2)
    <... 'int'>
    >>> type(sympify(2))
    <class 'sympy.core.numbers.Integer'>
```

In Julia:

```julia
typeof(2)
```

    Int64

```julia
typeof(sympify(2))
```

    SymPy.Sym

We have seen that 2**x is represented as Pow(2, x).  What about x*y?  As we might expect, this is the multiplication of x and y. The SymPy class for multiplication is Mul.

```julia
    >>> srepr(x*y)
    "Mul(Symbol('x'), Symbol('y'))"
```

In Julia:

```julia
srepr(x*y)
```

Mul(Symbol('x'), Symbol('y'))

Thus, we could have created the same object by writing Mul(x, y).

```julia
    >>> Mul(x, y)
    x*y
```

In Julia:

Again, Mul is not a function, so it must be qualified

```julia
sympy.Mul(x, y)
```

\begin{equation*}x y\end{equation*}

Now we get to our final expression, 2**x + x*y.  This is the addition of our last two objects, Pow(2, x), and Mul(x, y).  The SymPy class for addition is Add, so, as you might expect, to create this object, we use Add(Pow(2, x), Mul(x, y)).

```julia
    >>> Add(Pow(2, x), Mul(x, y))
    2**x + x*y
```

In Julia:

We can import these operations to avoid qualifying them as done here:

```julia
import_from(sympy, (:Add, :Mul, :Pow), typ=:Any)
Add(Pow(2, x), Mul(x, y))
```

\begin{equation*}2^{x} + x y\end{equation*}

SymPy expression trees can have many branches, and can be quite deep or quite broad.  Here is a more complicated example

```julia
    >>> expr = sin(x*y)/2 - x**2 + 1/y
    >>> srepr(expr)
    "Add(Mul(Integer(-1), Pow(Symbol('x'), Integer(2))), Mul(Rational(1, 2),
    sin(Mul(Symbol('x'), Symbol('y')))), Pow(Symbol('y'), Integer(-1)))"
```

In Julia:

```julia
expr = sin(x*y)/2 - x^2 + 1/y
srepr(expr)
```

Add(Mul(Integer(-1), Pow(Symbol('x'), Integer(2))), Mul(Rational(1, 2), sin(Mul(Symbol('x'), Symbol('y')))), Pow(Symbol('y'), Integer(-1)))

Here is a diagram

Note

produced with dotprint(sin(x*y)/2 - x**2 + 1/y, labelfunc=srepr), but not rendered here

This expression reveals some interesting things about SymPy expression trees. Let's go through them one by one.

Let's first look at the term x**2.  As we expected, we see Pow(x, 2). One level up, we see we have Mul(-1, Pow(x, 2)).  There is no subtraction class in SymPy.  x - y is represented as x + -y, or, more completely, x + -1*y, i.e., Add(x, Mul(-1, y)).

```julia
    >>> srepr(x - y)
    "Add(Symbol('x'), Mul(Integer(-1), Symbol('y')))"
```

In Julia:

```julia
srepr(x - y)
```

Add(Symbol('x'), Mul(Integer(-1), Symbol('y')))

Next, look at 1/y.  We might expect to see something like Div(1, y), but similar to subtraction, there is no class in SymPy for division.  Rather, division is represented by a power of -1.  Hence, we have Pow(y, -1). What if we had divided something other than 1 by y, like x/y?  Let's see.

```julia
    >>> expr = x/y
    >>> srepr(expr)
    "Mul(Symbol('x'), Pow(Symbol('y'), Integer(-1)))"
```

In Julia:

```julia
expr = x/y
srepr(expr)
```

Mul(Symbol('x'), Pow(Symbol('y'), Integer(-1)))

We see that x/y is represented as x*y**-1, i.e., Mul(x, Pow(y, -1)).

Finally, let's look at the sin(x*y)/2 term.  Following the pattern of the previous example, we might expect to see Mul(sin(x*y), Pow(Integer(2), -1)).  But instead, we have Mul(Rational(1, 2), sin(x*y)).  Rational numbers are always combined into a single term in a multiplication, so that when we divide by 2, it is represented as multiplying by 1/2.

Finally, one last note.  You may have noticed that the order we entered our expression and the order that it came out from srepr or in the graph were different.  You may have also noticed this phenomenon earlier in the tutorial.  For example

```julia
     >>> 1 + x
     x + 1
```

In Julia:

```julia
1 + x
```

\begin{equation*}x + 1\end{equation*}

This because in SymPy, the arguments of the commutative operations Add and Mul are stored in an arbitrary (but consistent!) order, which is independent of the order inputted (if you're worried about noncommutative multiplication, don't be.  In SymPy, you can create noncommutative Symbols using Symbol('A', commutative=False), and the order of multiplication for noncommutative Symbols is kept the same as the input).  Furthermore, as we shall see in the next section, the printing order and the order in which things are stored internally need not be the same either.

Quick Tip

The way an expression is represented internally and the way it is printed    are often not the same.

In general, an important thing to keep in mind when working with SymPy expression trees is this:  the internal representation of an expression and the way it is printed need not be the same.  The same is true for the input form.   If some expression manipulation algorithm is not working in the way you expected it to, chances are, the internal representation of the object is different from what you thought it was.

Recursing through an Expression Tree

Now that you know how expression trees work in SymPy, let's look at how to dig our way through an expression tree.  Every object in SymPy has two very important attributes, func, and args.

func

func is the head of the object. For example, (x*y).func is Mul. Usually it is the same as the class of the object (though there are exceptions to this rule).

Two notes about func.  First, the class of an object need not be the same as the one used to create it.  For example

```julia
    >>> expr = Add(x, x)
    >>> expr.func
    <class 'sympy.core.mul.Mul'>
```

In Julia:

```julia
expr = Add(x, x)
expr.func
```

    PyObject <class 'sympy.core.mul.Mul'>

The output isn't as desired, as PyObjects don't show nicely here. We can ask for the name, which does display as desired:

```julia
expr.func.__name__
```

Mul

In SymPy the func and args properties are exported as functions in the module SymPy.Introspection.

We created Add(x, x), so we might expect expr.func to be Add, but instead we got Mul.  Why is that?  Let's take a closer look at expr.

```julia
    >>> expr
    2*x
```

In Julia:

```julia
expr
```

\begin{equation*}2 x\end{equation*}

Add(x, x), i.e., x + x, was automatically converted into Mul(2, x), i.e., 2*x, which is a Mul.   SymPy classes make heavy use of the __new__ class constructor, which, unlike __init__, allows a different class to be returned from the constructor.

Second, some classes are special-cased, usually for efficiency reasons [#singleton-fn]_.

```julia
    >>> Integer(2).func
    <class 'sympy.core.numbers.Integer'>
    >>> Integer(0).func
    <class 'sympy.core.numbers.Zero'>
    >>> Integer(-1).func
    <class 'sympy.core.numbers.NegativeOne'>
```

In Julia:

```julia
sympy.Integer(2).func.__name__
```

Integer

```julia
sympy.Integer(0).func.__name__
```

Zero

```julia
sympy.Integer(-1).func.__name__
```

NegativeOne

For the most part, these issues will not bother us.  The special classes Zero, One, NegativeOne, and so on are subclasses of Integer, so as long as you use isinstance, it will not be an issue.

args

args are the top-level arguments of the object.  (x*y).args would be (x, y).  Let's look at some examples

```julia
    >>> expr = 3*y**2*x
    >>> expr.func
    <class 'sympy.core.mul.Mul'>
    >>> expr.args
    (3, x, y**2)
```

In Julia:

The args property can be accessed exactly as func

```julia
expr = 3*y^2*x
expr.func.__name__
```

Mul

```julia
expr.args
```

    (3, x, y^2)

From this, we can see that expr == Mul(3, y**2, x).  In fact, we can see that we can completely reconstruct expr from its func and its args.

```julia
    >>> expr.func(*expr.args)
    3*x*y**2
    >>> expr == expr.func(*expr.args)
    True
```

In Julia:

```julia
expr.func(expr.args...)
```

\begin{equation*}3 x y^{2}\end{equation*}

```julia
expr == expr.func(expr.args...)
```

    true

Note that although we entered 3*y**2*x, the args are (3, x, y**2). In a Mul, the Rational coefficient will come first in the args, but other than that, the order of everything else follows no special pattern.  To be sure, though, there is an order.

```julia
    >>> expr = y**2*3*x
    >>> expr.args
    (3, x, y**2)
```

In Julia:

```julia
expr = y^2*3*x
expr.args
```

    (3, x, y^2)

Mul's args are sorted, so that the same Mul will have the same args.  But the sorting is based on some criteria designed to make the sorting unique and efficient that has no mathematical significance.

The srepr form of our expr is Mul(3, x, Pow(y, 2)).  What if we want to get at the args of Pow(y, 2).  Notice that the y**2 is in the third slot of expr.args, i.e., expr.args[2].

```julia
    >>> expr.args[2]
    y**2
```

In Julia:

```julia
expr.args[2]
```

\begin{equation*}x\end{equation*}

So to get the args of this, we call expr.args[2].args.

```julia
    >>> expr.args[2].args
    (y, 2)
```

In Julia:

Python uses 0-based indexing, so we bump the index by 1

```julia
expr.args[3].args
```

    (y, 2)

Now what if we try to go deeper.  What are the args of y.  Or 2. Let's see.

```julia
    >>> y.args
    ()
    >>> Integer(2).args
    ()
```

In Julia:

```julia
y.args
```

    ()

```julia
sympy.Integer(2).args
```

    ()

They both have empty args.  In SymPy, empty args signal that we have hit a leaf of the expression tree.

So there are two possibilities for a SymPy expression. Either it has empty args, in which case it is a leaf node in any expression tree, or it has args, in which case, it is a branch node of any expression tree.  When it has args, it can be completely rebuilt from its func and its args. This is expressed in the key invariant.

Key Invariant

Every well-formed SymPy expression must either have empty args or    satisfy expr == expr.func(expr.args...).

(Recall that in Python if a is a tuple, then f(*a) means to call f with arguments from the elements of a, e.g., f(*(1, 2, 3)) is the same as f(1, 2, 3).)

This key invariant allows us to write simple algorithms that walk expression trees, change them, and rebuild them into new expressions.

In Julia:

Splatting replaces the *a term above, or f(a...).

Walking the Tree

With this knowledge, let's look at how we can recurse through an expression tree.  The nested nature of args is a perfect fit for recursive functions. The base case will be empty args.  Let's write a simple function that goes through an expression and prints all the args at each level.

```julia
    >>> def pre(expr):
    ...     print(expr)
    ...     for arg in expr.args:
    ...         pre(arg)
```

In Julia:

```julia
function pre(expr)
    @show expr
    for arg in expr.args
    	pre(arg)
	end
end
```

    pre (generic function with 1 method)

See how nice it is that () signals leaves in the expression tree.  We don't even have to write a base case for our recursion; it is handled automatically by the for loop.

Let's test our function.

```julia
    >>> expr = x*y + 1
    >>> pre(expr)
    x*y + 1
    1
    x*y
    x
    y
```

In Julia:

@show does not work in these notes; this would need copy-and-pasting to be verified

```julia
expr = x*y + 1
pre(expr)
```

Can you guess why we called our function pre?  We just wrote a pre-order traversal function for our expression tree.   See if you can write a post-order traversal function.

Such traversals are so common in SymPy that the generator functions preorder_traversal and postorder_traversal are provided to make such traversals easy.  We could have also written our algorithm as

```julia
    >>> for arg in preorder_traversal(expr):
    ...     print(arg)
    x*y + 1
    1
    x*y
    x
    y
```

In Julia:

The preorder_traversal function is not a function, so needs to be qualified:

```julia
for arg in sympy.preorder_traversal(expr)
   @show arg
end
```

Prevent expression evaluation

There are generally two ways to prevent the evaluation, either pass an evaluate=False parameter while constructing the expression, or create an evaluation stopper by wrapping the expression with UnevaluatedExpr.

For example:

```julia
    >>> from sympy import Add
    >>> from sympy.abc import x, y, z
    >>> x + x
    2*x
    >>> Add(x, x)
    2*x
    >>> Add(x, x, evaluate=False)
    x + x
```

In Julia:

```julia
@vars x y z
x + x
```

\begin{equation*}2 x\end{equation*}

```julia
Add(x, x)
```

\begin{equation*}2 x\end{equation*}

```julia
Add(x, x, evaluate=False)
```

\begin{equation*}x + x\end{equation*}

If you don't remember the class corresponding to the expression you want to build (operator overloading usually assumes evaluate=True), just use sympify and pass a string:

```julia
    >>> from sympy import sympify
    >>> sympify("x + x", evaluate=False)
    x + x
```

In Julia:

```julia
sympify("x + x", evaluate=false)
    x + x
```

\begin{equation*}2 x\end{equation*}

Note that evaluate=False won't prevent future evaluation in later usages of the expression:

```julia
    >>> expr = Add(x, x, evaluate=False)
    >>> expr
    x + x
    >>> expr + x
    3*x
```

In Julia:

```julia
expr = Add(x, x, evaluate=false)
expr
```

\begin{equation*}x + x\end{equation*}

```julia
expr + x
```

\begin{equation*}3 x\end{equation*}

That's why the class UnevaluatedExpr comes handy. UnevaluatedExpr is a method provided by SymPy which lets the user keep an expression unevaluated. By unevaluated it is meant that the value inside of it will not interact with the expressions outside of it to give simplified outputs. For example:

```julia
    >>> from sympy import UnevaluatedExpr
    >>> expr = x + UnevaluatedExpr(x)
    >>> expr
    x + x
    >>> x + expr
    2*x + x
```

In Julia:

```julia
import_from(sympy, (:UnevaluatedExpr,))
expr = x + UnevaluatedExpr(x)
expr
```

\begin{equation*}x + x\end{equation*}

```julia
x + expr
```

\begin{equation*}2 x + x\end{equation*}

The x remaining alone is the x wrapped by UnevaluatedExpr. To release it:

```julia
    >>> (x + expr).doit()
    3*x
```

In Julia:

```julia
(x + expr).doit()
```

\begin{equation*}3 x\end{equation*}

Other examples:

```julia
    >>> from sympy import *
    >>> from sympy.abc import x, y, z
    >>> uexpr = UnevaluatedExpr(S.One*5/7)*UnevaluatedExpr(S.One*3/4)
    >>> uexpr
    (5/7)*(3/4)
    >>> x*UnevaluatedExpr(1/x)
    x*1/x
```

In Julia:

```julia
@vars x y z
const S = sympy.S
uexpr = UnevaluatedExpr(S.One * 5/7) * UnevaluatedExpr(S.One * 3/4)
uexpr
```

\begin{equation*}\frac{5}{7} \frac{3}{4}\end{equation*}

```julia
x * UnevaluatedExpr(1/x)
```

\begin{equation*}x \frac{1}{x}\end{equation*}

A point to be noted is that  UnevaluatedExpr cannot prevent the evaluation of an expression which is given as argument. For example:

```julia
    >>> expr1 = UnevaluatedExpr(x + x)
    >>> expr1
    2*x
    >>> expr2 = sympify('x + x', evaluate=False)
    >>> expr2
    x + x
```

In Julia:

```julia
expr1 = UnevaluatedExpr(x + x)
expr1
```

\begin{equation*}2 x\end{equation*}

```julia
expr2 = sympify("x + x", evaluate=False)
expr2
```

\begin{equation*}x + x\end{equation*}

Remember that expr2 will be evaluated if included into another expression. Combine both of the methods to prevent both inside and outside evaluations:

```julia
    >>> UnevaluatedExpr(sympify("x + x", evaluate=False)) + y
    y + x + x
```

In Julia:

```julia
UnevaluatedExpr(sympify("x + x", evaluate=False)) + y
```

\begin{equation*}y + x + x\end{equation*}

UnevalutedExpr is supported by SymPy printers and can be used to print the result in different output forms. For example

```julia
    >>> from sympy import latex
    >>> uexpr = UnevaluatedExpr(S.One*5/7)*UnevaluatedExpr(S.One*3/4)
    >>> print(latex(uexpr))
    \frac{5}{7} \frac{3}{4}
```

In Julia:

The printing support is through show, but we can use SymPy's:

```julia
uexpr = UnevaluatedExpr(S.One*5/7)*UnevaluatedExpr(S.One*3/4)
sympy.latex(uexpr)
```

\frac{5}{7} \frac{3}{4}

In order to release the expression and get the evaluated LaTeX form, just use .doit():

```julia
    >>> print(latex(uexpr.doit()))
    \frac{15}{28}
```

In Julia:

```julia
sympy.latex(uexpr.doit())
```

\frac{15}{28}

Footnotes

[#symbols-fn] We have been using symbols instead of Symbol because it automatically splits apart strings into multiple Symbol\ s. symbols('x y z') returns a tuple of three Symbol\ s.  Symbol('x y z') returns a single Symbol called x y z.

[#sympify-fn] Technically, it is an internal function called _sympify, which differs from sympify in that it does not convert strings.  x + '2' is not allowed.

[#singleton-fn] Classes like One and Zero are singletonized, meaning that only one object is ever created, no matter how many times the class is called.  This is done for space efficiency, as these classes are very common.  For example, Zero might occur very often in a sparse matrix represented densely.  As we have seen, NegativeOne occurs any time we have -x or 1/x.  It is also done for speed efficiency because singletonized objects can be compared by is.  The unique objects for each singletonized class can be accessed from the S object.

return to index


# Multicomplex Mathematics

```python
# Standard libraries
import sys

# Package from conda
import numpy as np
import sympy as sy

# The MultiComplex library
import pymcx
```

## Getting started

The function
$$ f= x\sin(x) $$
has the derivative
$$ f' = x\cos(x)+\sin(x) $$

In order to evaluate this derivative with MultiComplex algebra, we define a function that takes multicomplex arguments, in this case the lambda function ``lambda z: z*np.sin(z)``, and we compare the derivative obtained with the exact solution.

The function ``diff_mcx1`` takes as first argument the callable function of a single variable to be differentiated numerically, followed by the real value at which the derivative(s) are to be taken, and finally the number of derivatives that are to be taken.  All derivatives from order 1 to nderiv are returned in a list

```python
print(pymcx.diff_mcx1.__doc__)
```

    diff_mcx1(*args, **kwargs)
    Overloaded function.

    1. diff_mcx1(f: Callable[[MultiComplex<double>], MultiComplex<double>], x: float, numderiv: int, and_val: bool = False) -> List[float]

    2. diff_mcx1(f: Callable[[MultiComplex<double>], Tuple[MultiComplex<double>, MultiComplex<double>]], x: float, numderiv: int, and_val: bool = False) -> Tuple[List[float], List[float]]

```python
x = 0.1234
exact = x*np.cos(x) + np.sin(x)
nderiv = 4
and_val = False
mcx = pymcx.diff_mcx1(lambda z: z*np.sin(z), x, nderiv, and_val)[0]
error = mcx - exact
error
```

    0.0

## Derivatives of a function of one variable

Here we test some more interesting functions, and we show that the relative errors of these functions are all small, close to the numerical precision

```python
def deriver(var, func, nderiv):
    # The python translation of the sympy function
    # that takes numpy arguments (which allows for direct
    # use of pymcx arguments)
    py_func = sy.lambdify(var, func, 'numpy')

    # Calculate "exact" derivatives from sympy and mpmath
    exacts = []
    for N in range(1, nderiv+1):
        f = sy.lambdify(var,sy.diff(func,var,N),'mpmath')
        exacts.append(f(0.1234))
    exacts = np.array(exacts)

    # Calculate MultiComplex derivatives for derivatives of order
    # 1 to nderiv (inclusive) in one shot
    mcx = pymcx.diff_mcx1(py_func, 0.1234, nderiv)
    error = np.array(mcx - exacts)/np.array(exacts)
    return error

# Calculate some derivatives of functions to test out our implementation
y = sy.symbols('y')
display(deriver(y, 1.0/sy.log(y), 3))
display(deriver(y, sy.cos(y)*sy.sin(y)*sy.exp(y), 3))
display(deriver(y, 1/(4+sy.cos(y)*sy.sin(y)*sy.exp(y)/sy.cosh(y))-sy.log(y), 3))
```

    array([mpf('2.3990722300687498e-16'), mpf('5.7028372834229652e-15'),
           mpf('-2.440695170029012e-15')], dtype=object)

    array([mpf('1.7975675189132607e-16'), mpf('1.247770572193306e-16'),
           mpf('0.0')], dtype=object)

    array([mpf('2.1730817635703391e-16'), mpf('2.1651710041894285e-16'),
           mpf('-2.1372587912639966e-16')], dtype=object)

## Derivatives of multivariate function

In the multivariate case, the analysis is very similar.  We define a function that takes a _vector_ of multicomplex arguments, and in the diff_mcxN function specify the real values at which the derivatives are to be taken, and the number of times to take derivatives with respect to each of the independent variables

```python
def func(zs):
    x, y, z = zs
    return np.cos(x)*np.sin(y)*np.exp(z)
xs = [0.1234, 20.1234, -4.1234]
orders = [1, 1, 2]
def exact(zs):
    x, y, z=zs
    return -np.sin(x)*np.cos(y)*np.exp(z)

pymcx.diff_mcxN(func, xs, orders), exact(xs)
```

    (-0.0005830791997394103, -0.0005830791997394103)

```python
# A bad set of inputs (order too short), and an error returned
pymcx.diff_mcxN(func, xs, [1])
```

```python
def func(zs):
    x, y, z = zs
    return np.cos(x)*np.sin(y)*np.exp(z)

def der_sympy(xs):
    """ "Exact" solution in higher precision mathematics with sympy """
    x, y, z = sy.symbols('x,y,z')
    func = sy.cos(x)*sy.sin(y)*sy.exp(z)
    f = sy.diff(sy.diff(sy.diff(func,x),z,2),y)
    func = sy.lambdify((x,y,z),f,'mpmath')
    return func(*xs)

zs = []
h = 1e-50
L = 2
xs = [0.1234, 20.1234, -4.1234]
orders = [1, 1, 2]
numderiv = sum(orders)

# This block here is the implementation of the code that goes into
# diff_mcxN, and a simplified variant of it goes into diff_mcx1
for i in range(3):
    c = np.zeros((2**numderiv,))
    c[0] = xs[i]
    assert(orders==[1,1,2]) # The orders are hard-coded here..
    # <magic>
    if i == 0:
        c[1] = h
    elif i == 1:
        c[2] = h
    elif i == 2:
        c[4] = h
        c[8] = h
    # </magic>
    zs.append(pymcx.MultiComplex(c))

# Three values are the derivatives from multicomplex, derivative from symbolic math,
# and derivative from N-dimensional derivative function
func(zs).get_coef()[2**L-1]/h**L, float(der_sympy(xs)), pymcx.diff_mcxN(func, xs, orders)
```

    (-0.0005830791997394104, -0.0005830791997394103, -0.0005830791997394103)

## Time Profiling

```python
x = pymcx.MultiComplex([1,2])
%timeit x*x
x = pymcx.MultiComplex([1,2,3,4])
%timeit x*x
nderiv = 4
%timeit pymcx.diff_mcx1(lambda x: x*np.sin(x), 0.1234, nderiv)
%timeit pymcx.diff_mcx1(lambda x: x*np.sin(x)*np.cos(x), 0.1234, nderiv)
```

    1.34 µs ± 6.95 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
    1.43 µs ± 10.1 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
    18 µs ± 55.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
    30.8 µs ± 90.7 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)


```python
from sympy import *
x, y, z, t = symbols('x y z t')
```

## Vectors

A vector $\vec{v} \in \mathbb{R}^n$ is an $n$-tuple of real numbers.
For example, consider a vector that has three components:

$$
 \vec{v} = (v_1,v_2,v_3) \  \in \  (\mathbb{R},\mathbb{R},\mathbb{R}) \equiv \mathbb{R}^3.
$$

To specify the vector $\vec{v}$,
we specify the values for its three components $v_1$, $v_2$, and $v_3$.

A matrix $A \in \mathbb{R}^{m\times n}$ is a rectangular array of real numbers with $m$ rows and $n$ columns.
A vector is a special type of matrix; we can think of a vector $\vec{v}\in \mathbb{R}^n$
either as a row vector ($1\times n$ matrix) or a column vector ($n \times 1$ matrix).
Because of this equivalence between vectors and matrices,
there is no need for a special vector object in `SymPy`,
and `Matrix` objects are used for vectors as well.

This is how we define vectors
and compute their properties:

```python
u = Matrix([[4,5,6]])  # a row vector = 1x3 matrix
v = Matrix([[7],
            [8],       # a col vector = 3x1 matrix
            [9]])
```

```python
v.T                    # use the transpose operation to convert a col vec to a row vec
```

$\displaystyle \left[\begin{matrix}7 & 8 & 9\end{matrix}\right]$

```python
u[0]                   # 0-based indexing for entries
```

$\displaystyle 4$

```python
u.norm()               # length of u
```

$\displaystyle \sqrt{77}$

```python
uhat = u/u.norm()      # unit-length vec in same dir as u
uhat
```

$\displaystyle \left[\begin{matrix}\frac{4 \sqrt{77}}{77} & \frac{5 \sqrt{77}}{77} & \frac{6 \sqrt{77}}{77}\end{matrix}\right]$

```python
uhat.norm()
```

$\displaystyle 1$

### Dot product

The dot product of the 3-vectors $\vec{u}$ and $\vec{v}$ can be defined two ways:

$$
  \vec{u}\cdot\vec{v}
  	\equiv
	\underbrace{u_xv_x+u_yv_y+u_zv_z}_{\textrm{algebraic def.}}
	\equiv
	\underbrace{\|\vec{u}\|\|\vec{v}\|\cos(\varphi)}_{\textrm{geometric def.}}
	\quad \in \mathbb{R},
$$

where $\varphi$ is the angle between the vectors $\vec{u}$ and $\vec{v}$.
In `SymPy`,

```python
u = Matrix([ 4,5,6])
v = Matrix([-1,1,2])
u.dot(v)
```

$\displaystyle 13$

We can combine the algebraic and geometric formulas for the dot product
to obtain the cosine of the angle between the vectors

$$
    \cos(\varphi)
        = \frac{ \vec{u}\cdot\vec{v} }{  \|\vec{u}\|\|\vec{v}\| }
        = \frac{ u_xv_x+u_yv_y+u_zv_z  }{  \|\vec{u}\|\|\vec{v}\| },
$$

and use the `acos` function to find the angle measure:

```python
acos(u.dot(v)/(u.norm()*v.norm())).evalf()  # in radians = 52.76 degrees
```

$\displaystyle 0.921263115666387$

Just by looking at the coordinates of the vectors $\vec{u}$ and $\vec{v}$,
it's difficult to determine their relative direction.
Thanks to the dot product, however,
we know the angle between the vectors is $52.76^\circ$,
which means they *kind of* point in the same direction.
Vectors that are at an angle $\varphi=90^\circ$ are called *orthogonal*, meaning at right angles with each other.
The dot product of vectors for which $\varphi > 90^\circ$ is negative because they point *mostly* in opposite directions.

The notion of the “angle between vectors” applies more generally to vectors with any number of dimensions.
The dot product for $n$-dimensional vectors is $\vec{u}\cdot\vec{v}=\sum_{i=1}^n u_iv_i$.
This means we can talk about “the angle between” 1000-dimensional vectors.
That's pretty crazy if you think about it—there is no way we could possibly “visualize” 1000-dimensional vectors,
yet given two such vectors we can tell if they point mostly in the same direction,
in perpendicular directions, or mostly in opposite directions.

The dot product is a commutative operation $\vec{u}\cdot\vec{v} = \vec{v}\cdot\vec{u}$:

```python
u.dot(v) == v.dot(u)
```

    True

### Projections

Dot products are used for computing projections.
Assume you're given two vectors $\vec{u}$ and $\vec{n}$ and you want to find the component
of $\vec{u}$ that points in the $\vec{n}$ direction.
The following formula based on the dot product will give you the answer:

$$
 \Pi_{\vec{n}}( \vec{u} ) \equiv \frac{  \vec{u} \cdot \vec{n}  }{ \| \vec{n} \|^2 } \vec{n}.
$$

This is how to implement this formula in `SymPy`:

```python
u = Matrix([4,5,6])
n = Matrix([1,1,1])
(u.dot(n) / n.norm()**2)*n  # projection of v in the n dir
```

$\displaystyle \left[\begin{matrix}5\\5\\5\end{matrix}\right]$

In the case where the direction vector $\hat{n}$ is of unit length $\|\hat{n}\| = 1$,
the projection formula simplifies to $\Pi_{\hat{n}}( \vec{u} ) \equiv (\vec{u}\cdot\hat{n})\hat{n}$.

Consider now the plane $P$ defined by $(1,1,1)\cdot[(x,y,z)-(0,0,0)]=0$.
A plane is a two dimensional subspace of $\mathbb{R}^3$.
We can decompose any vector $\vec{u} \in \mathbb{R}^3$ into two parts $\vec{u}=\vec{v} + \vec{w}$
such that $\vec{v}$ lies inside the plane and $\vec{w}$ is perpendicular to the plane (parallel to $\vec{n}=(1,1,1)$).

To obtain the perpendicular-to-$P$ component of $\vec{u}$,
compute the projection of $\vec{u}$ in the direction $\vec{n}$:

```python
w = (u.dot(n) / n.norm()**2)*n
w
```

$\displaystyle \left[\begin{matrix}5\\5\\5\end{matrix}\right]$

To obtain the in-the-plane-$P$ component of $\vec{u}$,
start with $\vec{u}$ and subtract the perpendicular-to-$P$ part:

```python
v = u - (u.dot(n)/n.norm()**2)*n  # same as u - w
v
```

$\displaystyle \left[\begin{matrix}-1\\0\\1\end{matrix}\right]$

You should check on your own that $\vec{v}+\vec{w}=\vec{u}$ as claimed.

### Cross product

The *cross product*, denoted $\times$, takes two vectors as inputs and produces a vector as output.
The cross products of individual basis elements are defined as follows:

$$
 \hat{\imath}\times\hat{\jmath} =\hat{k}, \qquad
 \hat{\jmath}\times\hat{k} =\hat{\imath}, \qquad
 \hat{k}\times \hat{\imath}= \hat{\jmath}.
$$

Here is how to compute the cross product of two vectors in `SymPy`:

```python
u = Matrix([ 4,5,6])
v = Matrix([-1,1,2])
u.cross(v)
```

$\displaystyle \left[\begin{matrix}4\\-14\\9\end{matrix}\right]$

The vector $\vec{u}\times \vec{v}$ is orthogonal to both $\vec{u}$ and $\vec{v}$.
The norm of the cross product  $\|\vec{u}\times \vec{v}\|$ is proportional to the lengths of the vectors
and the sine of the angle between them:

```python
(u.cross(v).norm()/(u.norm()*v.norm())).n()
```

$\displaystyle 0.796366206088088$

The name “cross product” is well-suited for this operation since it is
calculated by “cross-multiplying” the coefficients of the vectors:

$$
   \vec{u}\times\vec{v}=
   \left(
     u_yv_z-u_zv_y, \ u_zv_x-u_xv_z, \ u_xv_y-u_yv_x
    \right).
$$

By defining individual symbols for the entries of two vectors,
we can make `SymPy` show us the cross-product formula:

```python
u1,u2,u3 = symbols('u1:4')
v1,v2,v3 = symbols('v1:4')
Matrix([u1,u2,u3]).cross(Matrix([v1,v2,v3]))
```

$\displaystyle \left[\begin{matrix}u_{2} v_{3} - u_{3} v_{2}\\- u_{1} v_{3} + u_{3} v_{1}\\u_{1} v_{2} - u_{2} v_{1}\end{matrix}\right]$

The dot product is anti-commutative $\vec{u}\times\vec{v} = -\vec{v}\times\vec{u}$:

```python
u.cross(v)
```

$\displaystyle \left[\begin{matrix}4\\-14\\9\end{matrix}\right]$

```python
v.cross(u)
```

$\displaystyle \left[\begin{matrix}-4\\14\\-9\end{matrix}\right]$

The product of two numbers and the dot product of two vectors are commutative operations.
The cross product, however, is not commutative: $\vec{u}\times\vec{v} \neq \vec{v}\times\vec{u}$.


# Amortized Neural Variational Inference for a toy probabilistic model

Consider a certain number of sensors  placed at known locations, $\mathbf{s}_1,\mathbf{s}_2,\ldots,\mathbf{s}_L$. There is a target at an unknown position $\mathbf{z}\in\mathbb{R}^2$ that is emitting a certain signal that is received at the $i$-th sensor with a signal strength distributed as follows:

\begin{align}
x_i \sim \mathcal{N}\Big(- A \log\left(||\mathbf{s}_i-\mathbf{z} ||^2\right), \sigma^2\Big),
\end{align}

where $A$ is a constant related to how fast signal strength degrades with distance. We assume a Gaussian prior for the unknown position $\mathcal{N}(\mathbf{0},\mathbf{I})$. Given a set of $N$ i.i.d. samples for each sensor, $\mathbf{X}\in\mathbb{R}^{L\times N}$, we will use a Amortized Neural Variational Inference to find a Gaussian approximation to

\begin{align}
p(\mathbf{z}|\mathbf{X}) \propto  p(\mathbf{X}|\mathbf{z}) p(\mathbf{z})
\end{align}

Our approximation to $p(\mathbf{z}|\mathbf{X})$ is of the form
\begin{align}
p(\mathbf{z}|\mathbf{X}) \approx q(\mathbf{z}|\mathbf{X})=\mathcal{N}\Big(\mu(\mathbf{X}),\Sigma(\mathbf{X})\Big),
\end{align}
where

- $\mu(\mathbf{X})$ --> Given by a Neural Network with parameter vector $\theta$ and input $\mathbf{X}$

- $\Sigma(\mathbf{X})$ --> Diagonal covariance matrix, where the log of the main diagonal is constructed by a Neural Network with parameter vector $\gamma$ and input $\mathbf{X}$

## ELBO lower-bound to $p(\mathbf{X})$

We will optimize $q(\mathbf{z}|\mathbf{X})$ w.r.t. $\theta,\gamma$ by optimizing the Evidence-Lower-Bound (ELBO):

\begin{align}
p(\mathbf{X}) &= \int p(\mathbf{X}|\mathbf{z}) p(\mathbf{z}) d\mathbf{z}\\
&\geq \int q(\mathbf{X}|\mathbf{z}) \log \left(\frac{p(\mathbf{X},\mathbf{z})}{q(\mathbf{X}|\mathbf{z})}\right)d\mathbf{z}\\
& = \mathbb{E}_{q}\left[\log  p(\mathbf{X}|\mathbf{z})\right] - D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})\triangleq \mathcal{L}(\mathbf{X},\theta,\gamma),
\end{align}
where $D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})$ is known in closed form since it is the KL divergence between two Gaussian pdfs:

\begin{align}
D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})) = \frac{1}{2} \left[\text{tr}\left(\Sigma(\mathbf{X})\right)+\left(\mu(\mathbf{X})^T\mu(\mathbf{X})\right)-2-\log\det \left(\Sigma(\mathbf{X})\right) \right]
\end{align}

## SGD optimization

- Sample $\mathbf{\epsilon}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$
- Sample from $q(\mathbf{z}|\mathbf{X})$:
\begin{align}
\mathbf{z}^0 = \mu(\mathbf{X}) + \sqrt{\text{diag}(\Sigma(\mathbf{X}))} \circ \mathbf{\epsilon}
\end{align}
- Compute gradients of
\begin{align}
\hat{\mathcal{L}}(\mathbf{X},\theta,\gamma) =\log  p(\mathbf{X}|\mathbf{z}^0) - D_{KL}(q(\mathbf{z}|\mathbf{X})||p(\mathbf{z})
\end{align}
w.r.t. $\theta,\gamma$

```python
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
%matplotlib inline

# use seaborn plotting defaults
import seaborn as sns; sns.set()
```

    /Users/olmos/anaconda3/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

### Probabilistic model definition and generating samples

```python
############## Elements of the true probabilistic model ####################

loc_info = {}

loc_info['S'] = 3 # Number o sensors

loc_info['pos_s'] = np.array([[0.5,1], [3.5,1], [2,3]]) #Position of sensors

#loc_info['target'] = np.random.uniform(-3,3,[2,])  #(Unknown target position)

loc_info['target'] = np.array([-1,2])  #(Unknown target position)

loc_info['var_s'] = 5.*np.ones(loc_info['S']).reshape([loc_info['S'],1]) #Variance of sensors

loc_info['A'] = np.ones(loc_info['S'],dtype=np.float32) * 10.0 #Attenuation mean factor per sensor

loc_info['N'] = 5 # Number of measurements per sensor

def sample_X(S,M,z,pos_s,A,var_s):

    means = -1*A*np.log(np.sum((pos_s-z)**2,1))

    X = means.reshape([S,1]) + np.random.randn(S,M) * np.sqrt(var_s)

    return X

```

```python
# Sampling from model for the right target

X = sample_X(loc_info['S'],loc_info['N'], loc_info['target'],loc_info['pos_s'],loc_info['A'],loc_info['var_s'])
```

```python
plt.plot(loc_info['pos_s'][:,0],loc_info['pos_s'][:,1],'b>',label='Sensors',ms=15)
plt.plot(loc_info['target'][0],loc_info['target'][1],'ro',label='Target',ms=15)
plt.legend()
```

### TensorFlow Computation Graph and Loss Function

```python
z_dim = 2       #Latent Space

model_name = 'model1'   #In 'model1.py' we define the variational family

learning_rate = 1e-2
num_samples_avg = 1     #Number of samples to approximate the expectation in the ELBO
num_samples = 10        #Number of samples from the posterior (for testing)
num_it = int(1e4)       #SGD iterations
period_plot = int(1000) #Show resuts every period_plot iterations
dims = X.shape #X.shape

```

```python
sess_VAE = tf.Graph()

with sess_VAE.as_default():

    print('[*] Importing model: ' + model_name)
    model = __import__(model_name)

    print('[*] Defining placeholders')

    inputX = tf.placeholder(tf.float32, shape=dims, name='x-input')

    print('[*] Defining the encoder')
    log_var, mean, samples_z, KL = model.encoder(inputX,dims,z_dim,num_samples_avg)

    print('[*] Defining the log_likelyhood')

    loglik = model.decoder(loc_info,inputX,samples_z,num_samples_avg)

    loss = -(loglik-KL)

    optim = tf.train.AdamOptimizer(learning_rate).minimize(loss)

    # Output dictionary -> Useful if computation graph is defined in a separate .py file

    tf_nodes = {}

    tf_nodes['X'] = inputX

    tf_nodes['mean'] = mean

    tf_nodes['logvar'] = log_var

    tf_nodes['KL'] = KL

    tf_nodes['loglik'] = loglik

    tf_nodes['optim'] = optim

    tf_nodes['samples'] = samples_z

```

    [*] Importing model: model1
    [*] Defining placeholders
    [*] Defining the encoder
    [*] Defining the log_likelyhood

## SGD optimization

```python

############ SGD Inference #####################################

mean_list = []

with tf.Session(graph=sess_VAE) as session:

    # Add ops to save and restore all the variables.
    saver = tf.train.Saver()

    tf.global_variables_initializer().run()

    print('Training the VAE ...')

    for it in range(num_it):

        feedDict = {tf_nodes['X'] : X}

        _= session.run(tf_nodes['optim'],feedDict)

        if(it % period_plot ==0):

            mean, logvar,loglik,KL = session.run([tf_nodes['mean'],tf_nodes['logvar'],tf_nodes['loglik'],tf_nodes['KL']],feedDict)

            print("It = %d, loglik = %.5f, KL = %.5f" %(it,loglik,KL))

            mean_list.append(mean)

    samples = session.run(tf_nodes['samples'],feedDict)

```

    Training the VAE ...
    It = 0, loglik = -50926.40625, KL = 1.63167
    It = 1000, loglik = -113.08937, KL = 7.00147
    It = 2000, loglik = -115.41471, KL = 7.60987
    It = 3000, loglik = -115.16046, KL = 7.83914
    It = 4000, loglik = -112.18390, KL = 8.10726
    It = 5000, loglik = -113.60258, KL = 8.09174
    It = 6000, loglik = -134.30858, KL = 8.19208
    It = 7000, loglik = -113.80139, KL = 8.34016
    It = 8000, loglik = -323.55902, KL = 7.21795
    It = 9000, loglik = -118.62814, KL = 8.40690

```python
#Samples from q(z|x)
m_evol = np.vstack(mean_list)
nsamples = 50

samples = mean + np.sqrt(np.exp(logvar)) * np.random.randn(nsamples,2)

plt.plot(loc_info['pos_s'][:,0],loc_info['pos_s'][:,1],'b>',label='Sensors',ms=15)
plt.plot(loc_info['target'][0],loc_info['target'][1],'ro',label='Target',ms=15)
plt.plot(m_evol[:,0],m_evol[:,1],'g>',label='Post Mean')
plt.scatter(samples[:,0],samples[:,1],label='Post Samples')
plt.rcParams["figure.figsize"] = [8,8]
plt.legend()
```

```python

```

Notebooks/Part_7/Vartional Neural Inference Localization/Neural Variational Inference Localization Problem.ipynb


# Classification Problem  (Assignment 5, TAO, Spring 2019)
### Instructor: Dr. Pawan Kumar

## Classification Problem
### Given a set of input vectors corresponding to objects (or featues) decide which of the N classes the object belogs to.

### Reference (some figures for illustration below are taken from this):
1. SVM without Tears, https://med.nyu.edu/chibi/sites/default/files/chibi/Final.pdf

## Possible Methods for Classification Problem
1. Perceptron
2. SVM
3. Neural Networks

## SVM: Support Vector Machines

We will briefly describe the idea behind support vector machines for classification problems. We first describe linear SVM used to classify linearly separable data, and then we describe how we can use these algorithm for non-linearly separable data by so called kernels. The kernels are functions that map non-linearly separable data to a space usually higher dimensional space where the data becomes linearly separable. Let us quickly start with linear SVM.

### Linear SVM for two class classification
We recall the separating hyperpplane theorem: If there are two non-intersecting convex set, then there exists a hyperplane that separates the two convex sets. This is the assumption we will make: we assume that the convex hull of the given data leads to two convex sets for two classes, such that a hyperplane exists that separates the convex hulls.

### Main idea of SVM:
Not just find a hyperplane (as in perceptrons), but find one that keeps a good (largest possible) gap from the the data samples of each class. This gap is popularly called margins.

### Illustration of problem, and kewords
Consider the dataset of cancer and normal patients, hence it is a two class problem. Let us visualize the data:

Let us notice the following about the given data:
0. There are two classes: blue shaded stars and red shaded circles.
2. The input vector is two dimensional, hence it is of the form $(x_1^1, x_2^1).$
2. Here $x_1^1, x_2^2$ are values of the features corresponding to two gene features: Gene X, Gene Y.
3. Here red line is the linear classifier or hyperplane that separates the given input data.
4. There are two dotted lines: one passes through a blue star point, and another dotted line passes through two red shaded circle points.
5. The distance between the two dotted lines is called gap or margin that we mentioned before.
6. Goal of SVM compared to perceptrons is to maximize this margin.

## Formulation of Optimization Model for Linear SVM
We now assume that the red hyperplane above with maximum margin is given by $$w \cdot x + b,$$
We further assume that the dotted lines above are given by $$w \cdot x + b = -1, \quad w \cdot x + b = +1.$$

For reasons, on why we can choose such hyperplane is shown in slides Lecture 16 of TAO. Since we want to maximize the margin the distance between the dotted lines, we recall the formula for diatance between planes. Let $D$ denote the distance, then
$$D = 2/ \| w \|.$$
So, to maximize the margin $D,$ we need to minimize $\| w \|.$ For convenience of writing algorithm (for differentiable function), we can say that minimizing $\| w \|$ is equivalent to minimizing $1/2 \| w \|^2.$ Hence
### Objective function: $\dfrac{1}{2} \| w \|^2$
For our hyperplane to classify correctly, we need points of one class on one side of dotted line, more concretely
$$w \cdot x + b \leq -1,$$
and the we want the samples of another class (red ones) be on the other side of other dotted lines, i.e.,
$$ w \cdot x + b \geq +1.$$
Let us now look what constraints mean in figure:

With this we are all set to write the constraints for our optimization model for SVM.
### Constraints:
$$
\begin{align}
&w \cdot x_i + b \leq -1, \quad \text{if}~y_i = -1\\
&w \cdot x_i + b \geq +1, \quad \text{if}~y_i = +1
\end{align}
$$
Hence, objective function with constraints, gives us the full model. The data for which the label $y_i$ is $-1$ satisfies $w \cdot x + b \leq -1,$ and the data for which the lable $y_i$ is $+1$ satisfies $w \cdot x + b \geq +1.$ Hence both these conditions can be combined to get
$$
\begin{align}
y_i (w \cdot x_i + b) \geq 1
\end{align}
$$

## Optimization Model (Primal Form):
$$
\begin{align}
\text{minimize} \quad & \dfrac{1}{2} \| w \|^2 \\
\text{subject to} \quad &w \cdot x_i + b \geq 1, \quad i=1,\dots,m,
\end{align}
$$
where $m$ is the number of samples $x_i,$ and $w \in \mathbb{R}^n.$

$\color{red}{Question:}$ Prove that the primal objective is convex.

$\color{blue}{Answer}:$

$$
\begin{align}
\text{First, we need to prove that} \quad \dfrac{1}{2} \|w \|^2 \quad \text{is a convex function} \\
\text{Let x and y be two vectors} \quad \quad \quad \quad  \\ \\
Now, \quad \theta ||x||^2 + (1- \theta)||y||^2 - || \theta .x + (1- \theta).y ||^2 \\ \\
=> \theta ||x||^2 + (1- \theta)||y||^2 - \theta^2.||x||^2 -\theta^2||y||^2 -2\theta(1-\theta)<x|y> \\ \\
\text{using the property that } <x|y> \leq |x|.|y| \quad \quad \quad \\ \\
\text{we can write,} \quad \quad \quad \quad \quad \quad\\
\theta ||x||^2 + (1- \theta)||y||^2 - \theta^2.||x||^2 -\theta^2||y||^2 -2\theta(1-\theta)<x|y>\quad \\ \\ \geq  \theta(1-\theta)||x||^2 + \theta(1-\theta)||y||^2 -2\theta(1-\theta)||x|||y|| \\ \\
= \theta(1-\theta)(||x||-||y||)^2 \geq=0 \qquad \qquad \qquad \\ \\ \\
=> \theta ||x||^2 + (1- \theta)||y||^2 \geq  || \theta .x + (1- \theta).y ||^2 \\
\text{Hence proved that the objective function which is} \quad \dfrac{1}{2}||w||^2 \quad \text{is a convex function}
\end{align}
$$

$\color{red}{Question}:$ Write the primal problem in standard form.

$\color{blue}{Answer}:$
$$
\begin{align}
\text{The standard form of a primal problem is given by:}\\
\text{minimise} \quad f(x) \qquad \qquad \qquad \\
\text{subject to} \quad g_i(x) \leq 0 \quad i=1.......m\\
h_j(x) = 0 \quad j=1.........p \\
\text{Now,}\quad y_i(wx_i+b) \geq 1, \quad i=1.....m \quad \text{can be written as}\\
1-y_i(wx_i+b) \leq 0 \quad for \quad  i=1.......m \\ \\
\text{Thus, the optimisation problem can be formulated as:} \\ \\
\text{minimise} \quad \dfrac{1}{2}||w||^2 \qquad \qquad \qquad \qquad \qquad \\
\text{subject to} \quad 1-y_i(wx_i+b) \leq 0 \quad i=1.......m\\
\end{align}
$$

## Optimization Model (Dual Form)
The dual form was derived in lecture 16:
$$
\begin{align*}
&\text{maximize} \quad \sum_{i=1}^m{\lambda_i} - \dfrac{1}{2} \sum_{i=1}^m \lambda_i \lambda_j y_i y_j (x_i \cdot x_j) \\
&\text{subject to} \quad \lambda_i \geq 0, \quad \sum_{i=1}^m{\lambda_i y_i} = 0, \quad i = 1, \dots, m
\end{align*},
$$
where $\lambda_i$ is the Lagrange multiplier. We claim that strong duality holds.

$\color{red}{Question:}$ Show the derivation of dual.

$\color{blue}{Answer:}$

$$
\begin{align*}
\text{The Lagrangian can be written as:} \quad \quad \quad \quad \quad\\
\text{minimise} L(w,b,\lambda) = \dfrac{1}{2} ||w||^2 + \sum_{i=1}^m\lambda_i(1-y_i(w^Tx_i+b)) \\
\text{subject to} \quad \lambda_i \geq 0 \quad \text{for all i=1,2.....m} \quad \quad \quad \quad \quad \\
\text{Now, to minimise we will have to differentiate with respect to w as well as with respect to b}\\
=> \dfrac{dL}{dw}=0= w- \sum_i^m(\lambda_i*y_i*x_i) \quad \quad \quad \quad\\
\text{Also,} \dfrac{dL}{db}=0= - \sum_{i=1}^m \lambda_i y_i=0 \quad \quad \quad \quad\\
\text{Substituting the value of } w=\sum_{i=1}^m(\lambda_i*y_i*x_i) and \sum_{i=1}^m(\lambda_i*y_i)=0, we get \\
\text{L}(\lambda,b)=\dfrac{1}{2}w^T\sum_{i=1}^m\lambda_i*y_i*x_i-\sum_{i=1}^m\lambda_i[y_i(w^TX_i+b)-1]\\
\text{L}(\lambda,b)=\dfrac{1}{2}w^T\sum_{i=1}^m\lambda_i*y_i*x_i-w^T\sum_{i=1}^m\lambda_i*y_i*x_i-b\sum_{i=1}^m\lambda_i*y_i+\sum_{i=1}^m \lambda_i \\
\text{L}(\lambda,b)=\sum_{i=1}^m\lambda_i-b\sum_{i=1}^m\lambda_i*y_i-\dfrac{1}{2}\sum_{i=1}^m\sum_{j=1}^m\lambda_i*\lambda_j*y_i*y_j*(X_i^T*X_j) \\
But, \sum_{i=1}^m\lambda_i*y_i=0 \quad \quad \quad \quad \quad\\
\text{So, the final equation will be:} \quad \quad \quad \quad \\
\text{maximise}\quad \text{L}(\lambda)=\sum_{i=1}^m\lambda_i-\dfrac{1}{2}\sum_{i=1}^m\sum_{j=1}^m\lambda_i*\lambda_j*y_i*y_j*(X_i^T*X_j) \\
\text{subject to}\quad \lambda_i \geq 0 \quad \text{for all i=1,2......m} \quad \text{and} \sum_{i=1}^m \lambda_i*y_i=0
\end{align*}
$$

$\color{red}{Question:}$ Prove that strong duality holds.

$\color{blue}{Answer:}$

$$
\begin{align*}
\text{our original problem was to minimise} \quad \dfrac{1}{2}||x||^2 \quad \quad \quad\\
\text{subject to} \quad y_i(w^T*x + b) \geq 1 \quad \text{for i=1........m} \quad \quad \quad\\
\text{Now, to prove the strong duality, we would use slater conditions according to which,given}\quad \quad \quad \\
\text{minimise} f_0(x) \quad \quad \quad \quad \quad \quad\quad \quad \quad \\
\text{subject to} \quad f_i(x) \leq 0 \quad  i=1.....m  \quad \quad \quad\quad \quad \quad\\
Ax=b \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad\\
\text{where all the functions are convex and strict inequality holds for a feasible point such that} \quad \quad \quad\\
f_i(x)  <  0 \quad \text{for i =1.......m} \quad \quad \quad \quad \quad \quad\\ \\
\text{Then, strong duality holds} \quad \quad \quad \quad\quad \quad \quad\\ \\
\text{in case of svm, the support vectors are the ones which follow equality whereas all the other points follow strict inequality} \quad \quad \quad\\ \\
\text{this means that there exists a point such that} \quad \quad \quad y_i(wx_i+b) > 1 \quad \quad \quad \quad \quad \quad\\ \\
\text{also, all the objective and constraint functions(affine) are convex} \quad \quad \quad \quad \quad \quad\\ \\
\text{thus, slater conditions and strong duality hold} \quad \quad \quad \quad \quad \quad \quad \quad \quad\\
\end{align*}
$$

$\color{red}{Question:}$ Prove that the dual objective is concave.

$\color{blue}{Answer}:$
$$
\begin{align}
\text{the dual objective is given by:}\\
\sum_{i=1}^m{\lambda_i} - \dfrac{1}{2} \sum_{i=1}^m \lambda_i \lambda_j y_i y_j (x_i \cdot x_j)\\
\text{this can be written in the form of a quadratic equation like this:}\\
\lambda^T.1 - \dfrac{1}{2}\lambda^T A \lambda \\
=>-(\dfrac{1}{2}\lambda^T A \lambda - \lambda^T.1) \\ \\
\text{Here,}A_{ij}=y_i.y_j.x_i^T.x_j \quad \quad\\
\text{This is basically a negative of convex quadratic function which is given by} \dfrac{1}{2}X^TPX+q^TX \\
\text{We Know that the negative of a convex function is a concave function}\\
\text{Hence Proved} \\
\end{align}
$$

$\color{red}{Question}:$ Write the dual problem in standard form.

$\color{blue}{Answer}:$
$$
\begin{align}
\text{The standard form is given by:}\\
\text{minimise} \quad f(x) \\
\text{subject to} \quad g_i(x) \leq 0 \quad i=1.......m\\ \\ \\
\text{Now, the dual problem is,}\\
\text{maximize} \quad \sum_{i=1}^m{\lambda_i} - \dfrac{1}{2} \sum_{i=1}^m \lambda_i \lambda_j y_i y_j (x_i \cdot x_j) \\
\text{subject to} \quad \lambda_i \geq 0, \quad \sum_{i=1}^m{\lambda_i y_i} = 0, \quad i = 1, \dots, m\\ \\ \\
\text{This can be written as} \\
\text{minimise} \quad \dfrac{1}{2} \sum_{i=1}^m \lambda_i \lambda_j y_i y_j (x_i \cdot x_j) -\sum_{i=1}^m{\lambda_i}\\
\text{subject to} \quad \quad -\lambda_i \leq 0 \quad \sum_{i=1}^m{\lambda_i y_i} = 0, \quad i = 1, \dots, m\\
\end{align}
$$

## Soft Margin SVM
In a variant of soft margin SVM, we assume that some data samples may be outliers or noise, and this prevents the data from being linearly separable. For example, see the figure below

In the figure, we see that

- We believe that two red and one blue sample is noisy or outliers.
- We now want to take into account that real life data is noisy, we decide to allow for some of the noisy data in the margin.
- Let $\xi_i$ denotes how far a data sample is from the middle plane (margin is the area between dotted line).
- For example, one of the noisy red data point in 0.6 away from middle red plane.
- We introduce this slack variable $\xi_i \geq 0$ for each data sample $x_i.$

## Optimization Model: Primal Soft-Margin
We can then write the primal soft-margin optimization model as follows:
$$
\begin{align*}
&\text{minimize} \quad \dfrac{1}{2} \| w \|^2 + C \sum_{i=1}^m \xi_i \\
&\text{subject to} \quad y_i (w \cdot x_i + b) \geq 1 - \xi_i, \quad \xi_i \geq 0, \quad i = 1, \dots, m.
\end{align*}
$$

## Optimization Model: Dual Soft-Margin
We can also write the dual form of soft-margin SVM as follows:
$$
\begin{align*}
\text{Maximize} \quad &\sum_{i=1}^m \lambda_i - \dfrac{1}{2} \sum_{i,j=1}^m \lambda_i \lambda_j \: y_i y_j \: x_i \cdot x_j \\
\text{subject to} \quad &0 \leq \lambda_i \leq C, \quad  i=1, \dots, m, \\
&\sum_{i=1}^m \lambda_i y_i = 0.
\end{align*}
$$

$\color{red}{Question:}$ Show the derivation of dual.

$\color{blue}{Answer:}$
$$
\begin{align}
\text{We would have to first convert it into lagrangian} \\
=>L(w,b,\lambda,\mu)= \dfrac{1}{2}||w||^2+C\sum_{i=1}^m\xi_i -\sum_{i=1}^m\lambda_i[(y_i(wx_i+b)-1+\xi_i]-\sum_{i=1}^m\mu_i\xi_i\\
=> \text{Now, to find the minimimum we would equate the gradient as zero with respect to w,b,}\xi \\
=> \dfrac{dL}{dw}=w-\sum_{i=1}^m \lambda_i*y_i*x_i=0 \\
=> w=\sum_{i=1}^m \lambda_i*y_i*x_i \\
\text{also,} \quad \quad \quad \quad \quad \qquad\\
=>\dfrac{dL}{db}=\sum_{i=1}^m \lambda_i*y_i=0 \\
\text{also,} \quad \quad \quad \quad \quad \qquad\\
=>\dfrac{dL}{d\xi_i}=C-\lambda_i-\mu_i=0 \quad \quad\\
\text{as the Lagrangian can be expected in the form} \quad L= ......\sum_{i=1}^mC\xi_i-\sum_{i=1}^m\lambda_i\xi_i-\sum_{i=1}^m\mu_i\xi_i \\
=> C= \lambda_i + \mu_i \quad \quad\\
\text{Thus,} \lambda_i \leq C \quad as \quad \mu_i\geq 0\\ \\ \\
\text{Substituting the value of } w=\sum_{i=1}^m(\lambda_i*y_i*x_i) and \sum_{i=1}^m(\lambda_i*y_i)=0, we get \\
\text{L}(\lambda,b)=\dfrac{1}{2}w^T\sum_{i=1}^m\lambda_i*y_i*x_i-\sum_{i=1}^m\lambda_i[y_i(w^TX_i+b)-1]+ \sum_{i=1}^mC\xi_i-\sum_{i=1}^m\lambda_i\xi_i-\sum_{i=1}^m\mu_i\xi_i\\
\text{But we know that}\sum_{i=1}^mC\xi_i=\sum_{i=1}^m\lambda_i\xi_i+\sum_{i=1}^m\mu_i\xi_i\\
\text{L}(\lambda,b)=\dfrac{1}{2}w^T\sum_{i=1}^m\lambda_i*y_i*x_i-w^T\sum_{i=1}^m\lambda_i*y_i*x_i-b\sum_{i=1}^m\lambda_i*y_i+\sum_{i=1}^m \lambda_i \\ \\ \\
\text{L}(\lambda,b)=\sum_{i=1}^m\lambda_i-b\sum_{i=1}^m\lambda_i*y_i-\dfrac{1}{2}\sum_{i=1}^m\sum_{j=1}^m\lambda_i*\lambda_j*y_i*y_j*(X_i^T*X_j) \\
But, \sum_{i=1}^m\lambda_i*y_i=0 \quad \quad \quad \quad \quad\\
\text{So, the final equation will be:} \quad \quad \quad \quad \\
\text{maximise}\quad \text{L}(\lambda)=\sum_{i=1}^m\lambda_i-\dfrac{1}{2}\sum_{i=1}^m\sum_{j=1}^m\lambda_i*\lambda_j*y_i*y_j*(X_i^T*X_j) \\
\text{subject to}\quad \lambda_i \geq 0 ,\quad \lambda_i \leq C\quad \text{for all i=1,2......m} \quad \text{and} \sum_{i=1}^m \lambda_i*y_i=0
\end{align}
$$

$\color{red}{Question:}$ List advantages of dual over primal.

$\color{blue}{Answer:}$
$$
\begin{align}
1.\quad \quad \quad\text{ The most significant benefit from solving the dual comes when you are using} \\
\text{the "Kernel Trick" to classify data that is not linearly separable in the original feature space.} \\ \\
2.\text{If D(number of features) is too large, then computing this mapping (and solving for W using the} \\
\text{primal formulation) becomes too expensive. The "trick" part of the the kernel trick comes}\\
\text{here:  we can avoid computing this mapping altogether. Suppose we require only the dot-product between }\\
\text{two data points in the mapped feature space. Now suppose you have a function (unsurprisingly named }\\
\text{the "Kernel Function") which directly computes this dot-product between points in R^D from}\\
\text{their corresponding points in R^d.This is almost the same as the original dual formulation, except}\\
\text{that you compute the kernel function instead of the ordinary dot-product. Note that this is not}\\
\text{possible in the primal formulation, where it would be necessary to explicitly compute}\\
\text{the mapping for each data point.}\quad \quad \quad \quad\\ \\
3.\quad \quad \quad \text{for Data with a large number of features, say, P, the time complexity of the computation}\\
\text{would be of the order of NP where N is the number of samples and P is the number of features}\\
\text{But if we solve it using the dual formulation, then it would be of the order of}N^2.\\
\text{As only the dot products of samples with each other are needed, it would only be of the order of N^2}\\
\text{This is efficient for those problems where P>>>N, i.e. the number of features are way more than the number of samples.}
\end{align}
$$

# Kernels in SVM
## Non-Linear Classifiers
- For nonlinear data, we may map the data to a higher dimensional feature space where it is separable. See the figure below:

Such non-linear transformation can be implemented more effectively using the dual formulation.
- If we solve the dual form of linear SVM, then the predictions is given by
$$
\begin{align*}
f(x) &= \text{sign}(w \cdot x + b) \\
w &= \sum_{i=1}^m \alpha_i y_i x_i
\end{align*}
$$
If we assume that we did some transform $\Phi,$ then the classifier is given by
$$
\begin{align*}
f(x) &= \text{sign} (w \cdot \Phi(x) + b) \\
w &= \sum_{i=1}^m \alpha_i y_i \Phi(x_i)
\end{align*}
$$
If we substitute $w$ in $f(x),$ we observe that
$$
\begin{align*}
f(x) = \text{sign} \left ( \sum_{i=1}^m \alpha_i y_i \, \Phi(x_i) \cdot \Phi(x) + b \right) = \text{sign} \left( \sum_{i=1}^m \alpha_i y_i \, K(x_i, x) + b \right)
\end{align*}
$$
Note that doing dot products such as $\Phi(x_i) \cdot \Phi(x),$ if $\Phi(x)$ is a long vector! An important observation is to define this dot product or $K(x,z)$ such that dot products happen in input space rather than the feature space. We can see this with following example:
$$
\begin{align*}
K(x \cdot z) &= (x \cdot z)^2 = \left( \begin{bmatrix}
x_{(1)} \\ x_{(2)}
\end{bmatrix} \cdot \begin{bmatrix}
z_{(1)} \\ z_{(2)}
\end{bmatrix} \right)^2 = (x_{(1)} z_{(1)} + x_{(2)} z_{(2)})^2 \\
&= x_{(1)}^2 z_{(1)}^2 + 2x_{(1)} z_{(1)} x_{(2)} z_{(2)} + x_{(2)}^2 z_{(2)}^2 = \begin{bmatrix}
x_{(1)}^2 \\ \sqrt{2} x_{(1)} x_{(2)} \\ x_{(2)}^2
\end{bmatrix} \cdot \begin{bmatrix}
z_{(1)}^2 \\ \sqrt{2} z_{(1)} z_{(2)} \\ z_{(2)}^2
\end{bmatrix}  \\
&= \Phi(x) \cdot \Phi(z)
\end{align*}
$$

$\color{red}{Question:}$ Let the kernel be defined by $K(x,z) = (x \cdot z)^3.$ Define $\Phi(x).$ Assuming that one multiplications is 1 FLOP, and one addition is 1 FLOP, then how many flops you need to compute $K(x \cdot z)$ in input space versus feature space?

$\color{blue}{Answer:}$
$$
\begin{align}
K(x.z)=(x.z)^3 \text{can be expaned as} \\
=>(x_1*z_1+x_2*z_2)^3= x_1^3*z_1^3+x_2^3*z_2^3+3*x_1^2*z_1^2*x_2*z_2+3*x_1*z_1*x_2^2*z_2^2 \\ \\
\begin{bmatrix}
\ x_1^3 \\
\ x_2^3 \\
\sqrt[2]{3}*x_1^2*x_2 \\
\sqrt[2]{3}*x_1*x_2^2 \\
\end{bmatrix}
\begin{bmatrix}
\ z_1^3 \\
\ z_2^3 \\
\sqrt[2]{3}*z_1^2*z_2 \\
\sqrt[2]{3}*z_1*z_2^2 \\
\end{bmatrix}
\text{So, in the feature space there are 4 multiplications and 3 additions involved which is equal to 7 flops}\\
\text{the number of multiplications are 4 as it is a 4 dimensional vector now and to add those 4, 3 additions are required}\\
\text{whereas, in the input space the number of multiplications are 4 and one addition is involved, so, overall 5}\\
\text{in the input space, the number of multiplications are 4 to first calculate the dot dproduct which requires 2}\\
\text{and to raise the power by three, two more multiplications are required}\\ \\ \\
\text{So, answer is 7 Flops in feature space and 5 in input space}\\
\end{align}
$$

## Optimization Model: Dual Soft Margin Kernel SVM
We can now write the dual form of soft-margin Kernel SVM as follows:
$$
\begin{align*}
\text{Maximize} \quad &\sum_{i=1}^m \lambda_i - \dfrac{1}{2} \sum_{i, \, j=1}^m \lambda_i \lambda_j \: y_i y_j \: \Phi(x_i) \cdot \Phi(x_j) \\
\text{subject to} \quad &0 \leq \lambda_i \leq C, \quad  i=1, \dots, m, \\
&\sum_{i=1}^m \lambda_i y_i = 0.
\end{align*}
$$

## Solver for Optimization Problem: Quadratic Programming
We aspire to solve the above optimization problem using existing quadraric programming library. But we have a problem: the standard libraries use the standard form of the quadratic optimization problem that looks like the following:
$$
\begin{align*}
\text{minimize} \quad &\dfrac{1}{2} x^T P x + q^T x, \\
\text{subject to} \quad &Gx \leq h, \\
&Ax = b
\end{align*}
$$

# Dual Soft-Margin Kernel SVM in Standard QP: Assemble Matrices Vectors
To put the dual Kernel SVM in standard form, we need to set
- matrix $P$
- vector $x$
- vector $q$
- vector $h$
- vector $b$
- matrix $G$
- matrix $A$

### Matrix $P$
Let $$K(x_i, x_j) = \Phi(x_i) \cdot \Phi(x_j),$$ and set $(i,j)$ entry of matrix $P$ as $$P_{ij} = y_iy_j K(x_i,x_j)$$

## Vector $x$
Set $$x = \begin{bmatrix}
\lambda_1 \\
\lambda_2 \\
\vdots \\
\lambda_m
\end{bmatrix}
$$

## Vector $q$
Set $q \in \mathbb{R}^m$
$$ q =
\begin{bmatrix}
-1 \\ -1 \\ \vdots \\ -1
\end{bmatrix}
$$

## Matrix $A$
Set the matrix (in fact vector) $A$ as
$$
A = [y_1, y_2, \dots, y_m]
$$

## Vector $b$
In fact vector $b$ is a scalar here: $$b = 0$$

## Matrix $G$
$$
\begin{align*}
G = \begin{bmatrix}
1 & 0 & \dots & 0 \\
0 & 1 & \dots & 0 \\
\vdots & \ddots & \dots & \vdots \\
0 & 0 & \dots & 1 \\ \hline
-1 & 0 & \dots & 0 \\
\vdots & \ddots & \dots & \vdots \\
0 & 0 & \dots& -1
\end{bmatrix}
\end{align*}
$$

## Vector $h$
Set $h$ as
$$
h = \begin{bmatrix}
C \\
C \\
\vdots \\
C \\
0 \\
\vdots \\
0
\end{bmatrix}
$$

# Implementation of Kernel SVM
We are all set to try out coding the classifier using Kernel SVM. We will first import some libraries. Some of these libraries may not be available in your system. You may install them as follows:
- conda install numpy
- conda install -c conda-forge cvxopt
- sudo apt-get install python-scipy python-matplotlib

Try google search, if these does not work.

```python
import pylab as pl
import cvxopt as cvxopt
from cvxopt import solvers
import numpy as np
```

We will now define a class: svm
This class will have the following functions:
- __init__: where we will define initial default parameters
- *construct_kernel*: here we will define some kernels such as polynomial and RBF (radial basis or Gaussian kernel)
- *train_kernel_svm*: Here we will train, i.e, we will call a quadratic programming solver from cvxopt
- *classify*: Here we will test our classifier

$\color{red}{Question:}$ Fill the TODO below.

```python
class svm:

    def __init__(self, kernel='linear', C=None, sigma=1., degree=1., threshold=1e-5):
        self.kernel = kernel
        if self.kernel == 'linear':
            self.degree = 1.
            self.kernel = 'poly'

        self.C = C
        self.sigma = sigma
        self.threshold = threshold
        self.degree = degree

    def construct_kernel(self, X):
        self.K = np.dot(X, X.T)
        print(np.diag(self.K)*np.ones((1, self.N)))

        if self.kernel == 'poly':
            self.K = (1. + 1./self.sigma*self.K)**self.degree

        elif self.kernel == 'rbf':
            self.xsquared = (np.diag(self.K)*np.ones((1, self.N))).T
            b = np.ones((self.N, 1))
            self.K -= 0.5*(np.dot(self.xsquared, b.T) +
                           np.dot(b, self.xsquared.T))
            self.K = np.exp(self.K/(2.*self.sigma**2))

    def train_kernel_svm(self, X, targets):
        self.N = np.shape(X)[0]
        self.construct_kernel(X)
        print(self.N)

        # Assemble the matrices for the constraints
        P=np.zeros((self.N,self.N))
        for i in range(self.N):
            for j in range(self.N):
                P[i][j]=targets[i]*targets[j]*self.K[i][j]

        q = -np.ones((self.N,1))
        G=np.zeros((2*self.N,self.N))
        for i in range(self.N):
            for j in range(self.N):
                G[i][j]=1
        for i in range(self.N):
            for j in range(self.N):
                G[self.N+i][j]=-1
        h=np.zeros((2*self.N,1))
        for i in range(self.N):
            if self.C==None:
                h[i]=0
            else:
                h[i]=self.C

        A = np.zeros((1,self.N))
        for i in range(self.N):
            A[0][i]=targets[i]
        b = np.zeros(1)

        # Call the the quadratic solver of cvxopt library.
        sol = cvxopt.solvers.qp(cvxopt.matrix(P), cvxopt.matrix(q), cvxopt.matrix(
            G), cvxopt.matrix(h), cvxopt.matrix(A), cvxopt.matrix(b))

        # Get the Lagrange multipliers out of the solution dictionary
        lambdas = np.array(sol['x'])

        # Find the (indices of the) support vectors, which are the vectors with non-zero Lagrange multipliers
        self.sv = np.where(lambdas > self.threshold)[0]
        self.nsupport = len(self.sv)
        print ("Number of support vectors = ", self.nsupport)

        # Keep the data corresponding to the support vectors
        self.X = X[self.sv, :]
        self.lambdas = lambdas[self.sv]
        self.targets = targets[self.sv]

        self.b = np.sum(self.targets)
        for n in range(self.nsupport):
            self.b -= np.sum(self.lambdas*self.targets *
                             np.reshape(self.K[self.sv[n], self.sv], (self.nsupport, 1)))
        self.b /= len(self.lambdas)

        if self.kernel == 'poly':
            def classify(Y, soft=False):
                K = (1. + 1./self.sigma*np.dot(Y, self.X.T))**self.degree

                self.y = np.zeros((np.shape(Y)[0], 1))
                for j in range(np.shape(Y)[0]):
                    for i in range(self.nsupport):
                        self.y[j] += self.lambdas[i]*self.targets[i]*K[j, i]
                    self.y[j] += self.b

                if soft:
                    return self.y
                else:
                    return np.sign(self.y)

        elif self.kernel == 'rbf':
            def classify(Y, soft=False):
                K = np.dot(Y, self.X.T)
                c = (1./self.sigma * np.sum(Y**2, axis=1)
                     * np.ones((1, np.shape(Y)[0]))).T
                c = np.dot(c, np.ones((1, np.shape(K)[1])))
                aa = np.dot(self.xsquared[self.sv],
                            np.ones((1, np.shape(K)[0]))).T
                K = K - 0.5*c - 0.5*aa
                K = np.exp(K/(2.*self.sigma**2))

                self.y = np.zeros((np.shape(Y)[0], 1))
                for j in range(np.shape(Y)[0]):
                    for i in range(self.nsupport):
                        self.y[j] += self.lambdas[i]*self.targets[i]*K[j, i]
                    self.y[j] += self.b

                if soft:
                    return self.y
                else:
                    return np.sign(self.y)
        else:
            print ("Error: Invalid kernel")
            return

        self.classify = classify

```

$\color{red}{Question:}$ How $b$ was computed?

$\color{blue}{Answer:}$
$$
\begin{align}
\text{only the support vectors which lie on the separating hyperplanes are considered in both hard margin}\\
\text{and soft margin classifiers. the condition for support vector lying on separating hyperplane is same}\\
=>1= y_i*(w^T*x_i+b)=y_i(\sum_{j\in SV}^m \lambda_j*y_j*x_j^T*x_i+b) \\
\text{After putting}\quad  w =\sum_{j\in SV}^m \lambda_j*y_j*x_j^T \\
\text{Now, if we multiply by}\quad y_i\quad \text{on both sides, we get}\\
b= y_i-\sum_{j\in SV}^m \lambda_j*y_j*x_j^T*x_i \quad \text{after putting}\quad y_i^2=1 \\
\text{Here, Sv is the set of the support vectors, ie. all the vectors which lie on the separating hyperplane}\\
\text{All these vectors have a value of } \lambda_i > 0 \\
\end{align}
$$

# Test the Classifier
In the following, we will now test our classifier.

```python
from importlib import reload
import pylab as pl
import numpy as np

iris = np.loadtxt('iris_proc.data', delimiter=',')
imax = np.concatenate((iris.max(axis=0)*np.ones((1, 5)),
                       iris.min(axis=0)*np.ones((1, 5))), axis=0).max(axis=0)
target = -np.ones((np.shape(iris)[0], 3), dtype=float)
indices = np.where(iris[:, 4] == 0)
target[indices, 0] = 1.
indices = np.where(iris[:, 4] == 1)
target[indices, 1] = 1.
indices = np.where(iris[:, 4] == 2)
target[indices, 2] = 1.

train = iris[::2, 0:4]
traint = target[::2]
test = iris[1::2, 0:4]
testt = target[1::2]

```

```python
# Training the machines
output = np.zeros((np.shape(test)[0], 3))

# Train for the first set of train data
#svm0 = svm(kernel='linear')
#svm0 = svm(kernel='linear')
#svm0 = svm.svm(kernel='poly',C=0.1,degree=1)
svm0 = svm(kernel='rbf',C=0.2)
svm0.train_kernel_svm(train, np.reshape(traint[:, 0], (np.shape(train[:, :2])[0], 1)))
output[:, 0] = svm0.classify(test, soft=True).T

# Train for the second set of train data
#svm1 = svm(kernel='linear')
#svm1 = svm(kernel='linear')
#svm1 = svm(kernel='poly',degree=3)
svm1 = svm(kernel='rbf',C=0.2)
svm1.train_kernel_svm(train, np.reshape(traint[:, 1], (np.shape(train[:, :2])[0], 1)))
output[:, 1] = svm1.classify(test, soft=True).T

# Train for the third set of train data
#svm2 = svm(kernel='linear')
#svm2 = svm(kernel='linear')
#svm2 = svm(kernel='poly',C=0.1,degree=1)
svm2 = svm(kernel='rbf',C=0.2)
svm2.train_kernel_svm(train, np.reshape(traint[:, 2], (np.shape(train[:, :2])[0], 1)))
output[:, 2] = svm2.classify(test, soft=True).T
```

    [[ 40.26  34.06  39.96  34.77  29.77  45.14  34.01  51.12  46.22  49.91
       43.65  35.16  38.25  39.28  40.6   35.25  46.11  35.88  44.23  30.09
       39.03  31.33  44.22  43.05  44.07  83.29  83.48  73.5   75.23  74.82
       42.25  57.84  54.42  62.86  65.78  71.33  72.2   69.55  79.08  66.91
       51.66  57.58  60.66  78.84  58.86  57.81  57.84  57.98  60.23  42.47
       92.83  98.63  89.73  53.4   88.02  82.5   89.9   73.25  84.74 118.95
       95.63 116.02  92.68  72.56  84.57 103.42  85.    77.29  88.37  71.28
       91.62  70.55  94.52  74.55  84.45]]
    75
         pcost       dcost       gap    pres   dres
     0: -1.0648e-01 -1.5200e+01  2e+01  4e-15  2e-13
     1: -1.0777e-01 -3.4955e-01  2e-01  4e-15  7e-14
     2: -1.6785e-01 -2.0094e-01  3e-02  9e-16  2e-14
     3: -1.9911e-01 -1.9957e-01  5e-04  5e-15  1e-14
     4: -1.9943e-01 -1.9943e-01  5e-06  9e-15  9e-15
     5: -1.9943e-01 -1.9943e-01  5e-08  2e-15  4e-14
    Optimal solution found.
    Number of support vectors =  41
    [[ 40.26  34.06  39.96  34.77  29.77  45.14  34.01  51.12  46.22  49.91
       43.65  35.16  38.25  39.28  40.6   35.25  46.11  35.88  44.23  30.09
       39.03  31.33  44.22  43.05  44.07  83.29  83.48  73.5   75.23  74.82
       42.25  57.84  54.42  62.86  65.78  71.33  72.2   69.55  79.08  66.91
       51.66  57.58  60.66  78.84  58.86  57.81  57.84  57.98  60.23  42.47
       92.83  98.63  89.73  53.4   88.02  82.5   89.9   73.25  84.74 118.95
       95.63 116.02  92.68  72.56  84.57 103.42  85.    77.29  88.37  71.28
       91.62  70.55  94.52  74.55  84.45]]
    75
         pcost       dcost       gap    pres   dres
     0: -1.0667e-01 -1.5200e+01  2e+01  3e-16  3e-14
     1: -1.0796e-01 -3.5000e-01  2e-01  3e-16  6e-14
     2: -1.6844e-01 -2.0150e-01  3e-02  5e-16  8e-15
     3: -1.9968e-01 -2.0014e-01  5e-04  2e-16  7e-16
     4: -2.0000e-01 -2.0000e-01  5e-06  5e-16  2e-15
     5: -2.0000e-01 -2.0000e-01  5e-08  5e-16  2e-15
    Optimal solution found.
    Number of support vectors =  41
    [[ 40.26  34.06  39.96  34.77  29.77  45.14  34.01  51.12  46.22  49.91
       43.65  35.16  38.25  39.28  40.6   35.25  46.11  35.88  44.23  30.09
       39.03  31.33  44.22  43.05  44.07  83.29  83.48  73.5   75.23  74.82
       42.25  57.84  54.42  62.86  65.78  71.33  72.2   69.55  79.08  66.91
       51.66  57.58  60.66  78.84  58.86  57.81  57.84  57.98  60.23  42.47
       92.83  98.63  89.73  53.4   88.02  82.5   89.9   73.25  84.74 118.95
       95.63 116.02  92.68  72.56  84.57 103.42  85.    77.29  88.37  71.28
       91.62  70.55  94.52  74.55  84.45]]
    75
         pcost       dcost       gap    pres   dres
     0: -1.0667e-01 -1.5200e+01  2e+01  5e-16  2e-14
     1: -1.0796e-01 -3.5000e-01  2e-01  9e-16  3e-14
     2: -1.6844e-01 -2.0150e-01  3e-02  4e-16  3e-15
     3: -1.9968e-01 -2.0014e-01  5e-04  1e-16  2e-15
     4: -2.0000e-01 -2.0000e-01  5e-06  1e-15  5e-15
     5: -2.0000e-01 -2.0000e-01  5e-08  6e-16  6e-15
    Optimal solution found.
    Number of support vectors =  36

```python
# Make a decision about which class
# Pick the one with the largest margin
# print(np.shape(output))
print(output)
bestclass = np.argmax(output, axis=1)
print (bestclass)
print (iris[1::2, 4])
print("Misclassified locations:")
err = np.where(bestclass != iris[1::2, 4])[0]
print (err)
print (float(np.shape(testt)[0] - len(err)) /
       (np.shape(testt)[0]), "test accuracy")
```

    [[ 7.64303951 -0.46552986 -0.25061284]
     [ 7.48693593 -0.46488703 -0.24813203]
     [ 7.67708623 -0.45432574 -0.25545977]
     [ 8.1743781  -0.46696161 -0.25565272]
     [ 7.76839503 -0.46511391 -0.2530793 ]
     [ 7.95981174 -0.46462939 -0.25534217]
     [ 6.87078897 -0.46968904 -0.22783511]
     [ 6.94896151 -0.45629642 -0.22962122]
     [ 8.27977622 -0.46880181 -0.25344547]
     [ 8.23512701 -0.46707207 -0.25121927]
     [ 8.17294454 -0.46517706 -0.25405172]
     [ 7.49153053 -0.44991041 -0.26724505]
     [ 7.46865702 -0.4572944  -0.25877561]
     [ 8.2313703  -0.46588391 -0.2561192 ]
     [ 7.64379742 -0.46265932 -0.25380329]
     [ 7.86538329 -0.45861076 -0.25906144]
     [ 7.70640681 -0.46504815 -0.23550261]
     [ 8.04541424 -0.47270719 -0.2463286 ]
     [ 7.76839503 -0.46511391 -0.2530793 ]
     [ 8.1807916  -0.46604668 -0.25660821]
     [ 5.65368955 -0.45416122 -0.22616312]
     [ 7.71306536 -0.4565502  -0.26009382]
     [ 7.53754139 -0.46467014 -0.24992285]
     [ 7.69558131 -0.46897349 -0.24573056]
     [ 8.14660407 -0.46909753 -0.25298448]
     [-6.54884928 -0.3337532  -0.26253334]
     [-4.9465713  -0.26484916 -0.33963469]
     [-6.55158023 -0.28670356 -0.31214133]
     [-1.01225988 -0.30207626 -0.32574881]
     [-4.42702905 -0.26829406 -0.34080633]
     [-6.43772957 -0.28420675 -0.31729282]
     [-6.99566848 -0.32402663 -0.2715709 ]
     [-5.95746076 -0.33510489 -0.26109994]
     [-5.49785455 -0.25752072 -0.34845727]
     [-4.79960332 -0.25493147 -0.35424097]
     [-5.88868462 -0.26914773 -0.33519777]
     [-6.73337845 -0.31225386 -0.28378721]
     [-6.24093049 -0.32645904 -0.27004825]
     [-6.20220868 -0.40553471 -0.18683062]
     [-3.26618242 -0.26193575 -0.35909514]
     [-3.72506837 -0.25998992 -0.35440965]
     [-6.6430455  -0.37346987 -0.21942633]
     [-6.39576807 -0.32917847 -0.26817574]
     [-6.02061222 -0.30237439 -0.29413809]
     [-5.19557434 -0.26052274 -0.34536979]
     [-6.94327207 -0.31640611 -0.28015998]
     [-1.21295795 -0.29840027 -0.32769048]
     [-5.9187223  -0.26792369 -0.33657864]
     [-6.53515956 -0.28965678 -0.30999093]
     [-5.91103596 -0.26283135 -0.34204693]
     [-6.33072335 -0.38781704 -0.20481698]
     [-5.53458038 -0.45109787 -0.14027958]
     [-0.87472304 -0.48644853 -0.10473664]
     [-2.10602575 -0.48854247 -0.10253606]
     [-1.96382451 -0.50194287 -0.08911034]
     [-6.03787289 -0.43198339 -0.1596656 ]
     [-5.97229251 -0.37989815 -0.21287827]
     [-5.41119459 -0.46379517 -0.1276423 ]
     [-0.17846919 -0.47624144 -0.11504338]
     [-5.89522627 -0.35579222 -0.23722866]
     [-6.19055144 -0.36956166 -0.22391251]
     [-6.77327667 -0.37958797 -0.21334217]
     [-2.99894501 -0.48680924 -0.10423938]
     [-6.91193825 -0.37513357 -0.21828157]
     [-3.50909892 -0.47034506 -0.12078213]
     [-0.35432803 -0.47252709 -0.11875164]
     [-6.62190302 -0.38068478 -0.212053  ]
     [-1.50003462 -0.48992119 -0.10119726]
     [-5.69114221 -0.4488064  -0.14267131]
     [-4.87030309 -0.47149253 -0.11972763]
     [-4.93797036 -0.45966456 -0.13178586]
     [-3.65877783 -0.50361732 -0.08734993]
     [-5.28166672 -0.46202677 -0.12936965]
     [-6.08885147 -0.43756384 -0.15417067]
     [-6.58656055 -0.38849529 -0.2043457 ]]
    [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2 1 1 1 1 1 2 2 1 1 1 2
     2 2 1 1 2 2 2 1 2 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
     2]
    [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
     0. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1. 1.
     1. 1. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2. 2.
     2. 2. 2.]
    Misclassified locations:
    [25 31 32 36 37 38 41 42 43 45]
    0.8666666666666667 test accuracy

## Further Questions

$\color{red}{Question}:$ The IRIS dataset has three classes. Explain by observing the code above how the two class SVM was modified for multiclass classification.

$\color{red}{Question}:$ Write mathematical expressions for the kernels defined above.

$\color{red}{Question}:$ Play with different Kernels. Which kernels (polynomial, RBF, or polynomial) give the best test accuracy?

Question1 from further questions:
$$
\begin{align}
\text{For this the output column in the training data was made into a 3 column form}\\
\text{So, the columns were created for three classes and all the samples were checked}\\
\text{for these 3 classes, the columns contained a one for true class label and -1 otherwise}\\
\text{So, binary classification was done 3 times using the training data and an N*3 size matrix was formed} \\
\text{After this, for each sample, the class corresponding to highest value was given as output}\\ \\ \\
\text{So, basically, in this case one to rest classification was done where the class with the highest value }\\
\text{is given as output for each sample of the data}\\
\end{align}
$$

Question2 from further questions
$$
\begin{align}
\text{Mathematical expressions for kernels} \\ \\
\text{The RBF kernel is given by:}\\
K(x,x')=exp (-\dfrac{||x-x'||^2}{2}) \\
=>exp(-\dfrac{1}{2}||x-x'||^2)=exp(X^TX'-\dfrac{1}{2}||X||^2-\dfrac{1}{2}||X'||^2)\\ \\
=> exp(X^TX')exp(-\dfrac{1}{2}||X||^2)exp(-\dfrac{1}{2}||X'||^2) \\ \\ \\
\text{the Polynomial kernel is given by}\\
=> K(x,y)=(x^Ty+c)^d \text{for a d degree polynomial} \\
\end{align}
$$

Question3 further questions:
$$
\begin{align}
\text{The RBF kernel gave an accuracy of 86.666 %} \\
\end{align}
$$

```python

```


# Sampling from a uniform distribution over a polytope
We consider a vector of variables $x \in \mathbb{R}^n$ subject to systems of linear inequality constraints, $\mathbf{A_1} x \leq b_1$.
These constraints form the faces of a convex polytope in $\mathbb{R}^n$, and the solutions satisfying the constraints fall inside this polytope.
Additionally we might have a system of equality constraints, $\mathbf{A_2} x = b_2$, effectively reducing the dimensionality of the space in which the solution lie.

For several applications we want to sample uniformly under these constraints.
For example a mixture can be described as $x \in [0, 1]^n$ with $\sum x_i = 1$, i.e. a unit-simplex.
Sampling uniformly from this simplex can be done, e.g. using the exponential distribution (and normalizing to 1) or using the Dirichlet distribution.
However, this is not possible anymore, when the individiual mixture components are individually bounded $x_{i,min} \leq x_i \leq x_{i,max}$.

For a general solution to this problem we need to employ Markov Chain Monte Carlo sampling methods.
A review of such sampling methods for the specific case of convex polytopes is given in https://arxiv.org/abs/1710.08165.

```python
import numpy as np
import matplotlib.pyplot as plt
import diversipy
```

## Sampling under equality constraints

Equality constraints $Ax = b$ reduce the dimensionality of the polytope.
Solutions can be found via the nullspace of A, i.e. the subspace containing the homogeneous solutions of $Ax = 0$, and adding a particular solution of $Ax = b$.

```python
# Example: x1 + x2 + x3 = 1
A = np.array([[1, 1, 1]])
b = np.array([1])

N, xp = diversipy.polytope.solve_equality(A, b)
print(f"nullspace:\n{N}")
print(f"particular solution: {xp}")
```

    nullspace:
    [[-0.57735027 -0.57735027]
     [ 0.78867513 -0.21132487]
     [-0.21132487  0.78867513]]
    particular solution: [0.33333333 0.33333333 0.33333333]

The nullspace basis and particular solution can be used to project to the subspace $z \in \mathbb{R}^m$ and back.
Note that $z = N \cdot (x - x_p) = N \cdot x - N \cdot x_p = N \cdot x$ so we don't need to substract the particular solution when projecting to the subspace.

```python
X1 = np.array([
    [1, 0, 0],
    [0, 9, -8],
    [2, -1, 0],
    [0.1, 0.4, 0.5]
])
X1.round(3)
```

    array([[ 1. ,  0. ,  0. ],
           [ 0. ,  9. , -8. ],
           [ 2. , -1. ,  0. ],
           [ 0.1,  0.4,  0.5]])

```python
Z = X1 @ N
X2 = Z @ N.T + xp
X2.round(3)
```

    array([[ 1. ,  0. ,  0. ],
           [-0. ,  9. , -8. ],
           [ 2. , -1. ,  0. ],
           [ 0.1,  0.4,  0.5]])

We can sample solutions in the unbounded projected space.

```python
Z = np.random.uniform(low=-1, high=1, size=(5, 2))
X = Z @ N.T + xp
print(f"some random solutions:\n{X}")
print("do they satisfy Ax=b?", np.allclose(X @ A.T, b))
```

    some random solutions:
    [[ 0.08467589  0.11441739  0.80090672]
     [ 0.38901666 -0.58817925  1.19916259]
     [ 0.42649104 -0.45568571  1.02919467]
     [ 0.41255803  0.68093568 -0.0934937 ]
     [-0.58205599  0.89382612  0.68822986]]
    do they satisfy Ax=b? True

The projection to the nullspace and back can be used to make any point satisfy $Ax = b$.-

```python
X1 = np.array([
    [1, 2, 2],
    [1, 0, 0],
    [0, 9, 9]
])

Z = X1 @ N
X2 = Z @ N.T + xp

print(f"X1 (all valid? {np.allclose(X1 @ A.T, b)}):\n{X1}")
print(f"X2 (all valid? {np.allclose(X2 @ A.T, b)}):\n{X2.round(2)}")
```

    X1 (all valid? False):
    [[1 2 2]
     [1 0 0]
     [0 9 9]]
    X2 (all valid? True):
    [[-0.33  0.67  0.67]
     [ 1.    0.    0.  ]
     [-5.67  3.33  3.33]]

## Sampling under inequality constraints

Let's consider the example of sampling $x \in \mathbb{R}^2$ subject to
\begin{align}
0.1 \leq x_1 &\leq 0.9 \\
0.2 \leq x_2 &\leq 1 \\
1/2 x_1 + x_2 &\leq 1 \\
0.2 \leq -2/3 x_1 + x_2
\end{align}

```python
# bounds
lower = [0.1, 0.2]
upper = [0.7, 0.9]

# inequality constraints
A1 = np.array([[1 / 2, 1], [2 / 3, -1]])
b1 = np.array([1, -0.2])

points = diversipy.polytope.sample(n_points=1000, lower=lower, upper=upper, A1=A1, b1=b1)

fig, ax = plt.subplots(figsize=(6, 6))
x1lim = np.array([-0.05, 1.05])
x2lim = np.array([-0.05, 1.05])
ax.scatter(*points.T, s=5)
for ai, bi in zip(A1, b1):
    x1 = x1lim
    x2 = (bi - x1 * ai[0]) / ai[1]  # a1 x1 + a2 x2 = b --> x2 = (b - a1 x1) / a2
    ax.plot(x1, x2, 'k', lw=1)

ax.axvline(lower[0], color='k', lw=1)
ax.axhline(lower[1], color='k', lw=1)
ax.axvline(upper[0], color='k', lw=1)
ax.axhline(upper[1], color='k', lw=1)

ax.grid()
ax.set(xlim=x1lim, ylim=x2lim, xlabel="x1", ylabel="x2");
fig.savefig("polytope.png")
```

## Sampling under both equality and inequality constraints

As example we consider sampling from the unit-simplex in R³.

```python
# bounds
lower = [0, 0, 0]
upper = [1, 1, 1]

# equality constraints
A2 = np.array([[1, 1, 1]])
b2 = np.array([1])

X = diversipy.polytope.sample(n_points=10000, lower=lower, upper=upper, A2=A2, b2=b2)

fig, ax = plt.subplots(figsize=(6, 6))
x1lim = np.array([-0.05, 1.05])
x2lim = np.array([-0.05, 1.05])
ax.scatter(X[:, 0], X[:, 1], s=5)
for ai, bi in zip(A2, b2):
    x1 = x1lim
    x2 = (bi - x1 * ai[0]) / ai[1]  # a1 x1 + a2 x2 = b --> x2 = (b - a1 x1) / a2
    ax.plot(x1, x2, 'k', lw=1)

ax.axvline(lower[0], color='k', lw=1)
ax.axhline(lower[1], color='k', lw=1)
ax.axvline(upper[0], color='k', lw=1)
ax.axhline(upper[1], color='k', lw=1)

ax.grid()
ax.set(xlim=x1lim, ylim=x2lim, xlabel="x1", ylabel="x2");
```

We can compare with `simplex.sample` function which is specific to the unit-simplex and see agreement.

```python
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 5), sharey=True)
ax1.hist(X, histtype="step", density=True)

Xs = diversipy.simplex.sample(dimension=3, n_points=10000)
ax2.hist(Xs, histtype="step", bins=np.linspace(0, 1, 11), density=True)

for ax in (ax1, ax2):
    ax.legend([f"x{i+1}" for i in range(3)])
    ax.set(xlabel="$x$")

ax1.set(title="polytope.sample")
ax2.set(title="simplex.sample");
```

```python

```


Computers deal in fixed-size bit sequences. For instance, a dedicated machine instruction on a so-called "64-bit machine" may take two 64-bit sequences as input, and produce a 64-bit sequence as output. If you need more than 64 bits for your intermediate results, you must resort to issueing multiple such (or similar) instructions.

Intuitively, this bears resemblance to a common grade school mantra like:

> Doing arithmetic with up to 3-4 decimal places should be sufficient for all practical intents and purposes.

The purpose of this notebook is to showcase some practical examples where 64 bits proves insufficient. We take 64 bits since this is default in R. In particular, R uses double-precision binary floating-point numbers. We begin with a brief recap of what that means, more conretely.

## Binary Approximation

Computers deal in bit sequences, and so all data must be represented in binary notation, in one way or another.

The reader is humbly assumed to be familiar with the classical binary representation of the natural numbers i.e., the numbers 0, 1, 2, 3, ... This is a positional notation, where the symbols `0` or `1` (bits) are put in a sequence, with each position signifying the inclusion of a power of 2, with the most-signficant bit on the left end, and the least-signifcant bit to the right end of the sequence.

For instance, the 6-bit sequence `101010` signifies the number

\begin{align}
1\cdot2^5 + 0\cdot2^4 + 1\cdot2^3 + 0\cdot2^2 + 1\cdot 2^1 + 0 \cdot 2^0
&= 1\cdot2^5 + 1\cdot2^3 + 1\cdot 2^1 \\
&= 32 + 8 + 2 \\
&= 42
\end{align}

This notation bears structural resemblance to the classical decimal notation we learn in grade school. This too, is a positional notation, except that the symbols are `0`, `1`, `2`, ... , `9` (digits), and a position signifies the magnitude of an inclusion of a power of 10.

For instance, the 4-digit sequence `1337` signifies the number

\begin{align}
1\cdot{10}^{3} + 3\cdot{10}^{2} + 3\cdot{10}^{1} + 7\cdot{10}^{0}
&= 1000 + 300 + 30 + 7 \\
&= 1337
\end{align}

Classical binary represenation is elegant because there is a one-to-one mapping between natural numbers sequences of bits.

This elegance, much like one's spirit in grade school, breaks down with the introduction of fractions. Some fractions have a finite, decimal representation, compossed of a "decimal" and a "fractional" part, separated by a "radix point".

Numerically, we extend the powers of 10, right of 0, into the negative.

For instance, the 4-digit sequence `13.37` signifies the number

\begin{align}
1\cdot{10}^{1} + 3\cdot{10}^{0} + 3\cdot{10}^{-1} + 7\cdot{10}^{-2}
&= 10 + 3 + 0.3 + 0.07 \\
&= 13.37
\end{align}

This representation is useful for numbers where it applies since many standard algorithms, which worked for natural numbers, work for this representation, if we disregard the radix point until the very last.

There is no one-to-one mapping between the binary and decimal numbers, as soon as there is a radix point involved.

The following sections discuss two classical arithmetics, which are commonly supported directly in computer hardware. "Direct support" means that there typically exists a single machine instruction for every single operation we discuss, on most modern CPUs and GPUs.

Which (few) numbers can be represented exactly?

## Scientific Notation

## Combinators

### Sum

Here's a simple implementation of a summation function. This example is interesting also because you can imagine a summation intertwined with another operation.

```R
fpa_sum0 <- function (xs) {
  sum <- 0.0
  for (x in xs) {
    sum <- sum + x;
  }
  return(sum)
}
```

This seems good and dandy for small, and even some fairly large data sets:

```R
lapply(list(
  c(0.25, 0.25, 0.25),
  rep(0.25,100),
  rep(0.25,100000)
), fpa_sum0)
```

Yet, it yields some odd results for some fairly small data sets:

```R
lapply(list(
  rep(0.1,80),
  rep(0.2,120),
  rep(0.3,150)
), fpa_sum0)
```

Fortunately, if you use the built-in `sum`, you avoid such problems.

```R
lapply(list(
  c(0.25, 0.25, 0.25),
  rep(0.25,100),
  rep(0.25,100000)
), sum)
```

```R
lapply(list(
  rep(0.1,100),
  rep(0.2,150),
  rep(0.3,200)
), sum)
```

# Conclusion

The take-away from this is two-fold:

1. Use built-in, or carefully constructed combinators to gain a greater numerical stability.
2. Avoid a word-at-a-time style of programming. [(Backus, 1978)](#cite-backus-1978)

# References

1. John Backus. 1978. *Can programming be liberated from the von Neumann style? A functional style and its algebra of programs.* Commun. ACM 21, 8 (August 1978), 613-641.


```python
from scipy import stats
from scipy import integrate
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
from matplotlib import rc,rcParams
rc('text', usetex=True)
rcParams['figure.figsize'] = (9, 5.5)
rcParams['ytick.labelsize'],rcParams['xtick.labelsize'] = 17.,17.
rcParams['axes.labelsize']=19.
rcParams['legend.fontsize']=17.
rcParams['axes.titlesize']=20.
import seaborn as sb
sb.despine()
sb.set_style('white', {'xes.linewidth': 0.5, 'axes.edgecolor':'black'})
sb.despine(left=True)
```

    <Figure size 648x396 with 0 Axes>

### Κατανομή Poisson
Η κατανομή Poisson είναι μια διακριτή κατανομή του αριθμού γεγονότων σε ένα συγκεκριμένο χρονικό διάστημα δεδομένου του μέσου αριθμού γεγονότων $\mu$ για αυτό το διάστημα. Η συνάρτηση πυκνότητας πιθανότητας είναι:
$$
Pr(x;\mu)=\frac{\mu ^x e^{-\mu}}{x!}
$$

Η πιθανότερη τιμή της κατανομής καί η διακύμανση είναι:
\begin{align}
E[x]=\mu && var[x]=\sigma ^2=\mu
\end{align}

#### Παράδειγμα
Σε έναν αγώνα ποδοσφαίρου μπαίνουν κατα μέσο όρο $2.5$ γκόλ. Ποιά είναι η πιθανότητα να μπούν $x$ γκόλ?

```python
xx=np.linspace(0,8,9,dtype=int)
pr=stats.poisson.pmf(xx,mu=2.5)
plt.bar(xx,pr)
# for mu in np.linspace(0.5,2.5,4):
#     pr=stats.poisson(mu).pmf(xx)
#     plt.plot(xx,pr,label='$\mu = {:.2f}$'.format(mu))
plt.legend()
```

We will use the Poissonian Distribution to study the observed SN from earth.

From 185 until now (2019) 12 SN have been observed by eye (from wikipedia)

```python
T=2019-185
N=12
r=N/T
print(f'Rate of SN per year {r:.3}')
```

    Rate of SN per year 0.00654

Probabillity of seeing one, two (or zero) SN in one year given this rate?

```python
xx=np.arange(0,3,1,dtype=int)
pr=stats.poisson(r).pmf(xx)
plt.bar(xx,pr,label='$\mu = {:.4f}$'.format(r))
plt.yscale('log');plt.legend()
```

Seems that we haven't seen a SN from 1604, what is the probabillity of this happening?

```python
xx=np.arange(0,6,1,dtype=int)
pr=stats.poisson(r*(2019-1604)).pmf(xx)
plt.bar(xx,pr,label='$\mu = {:.4f}$'.format(r))
#plt.yscale('log');
plt.legend()
plt.annotate('No SN from 1604 \n unitl 2019',
            xy=(0, 0.07), xycoords='data',
            xytext=(0., 0.2),
            arrowprops=dict(facecolor='black', shrink=0.05),
            horizontalalignment='center', verticalalignment='top')
```

## Bayesian Inference

```python
snt= np.array([185,369,386,393,437,827,902,1006,1054,1181,1572,1604])
tt= np.arange(185,2020,1)
sn=np.zeros(tt.shape)
for i,t in enumerate(tt):
    if t in snt:
        sn[i]=1
plt.plot(tt,sn)
plt.xlabel('Year');plt.ylabel('SN')
```

```python
import emcee
import corner
from scipy.special import factorial
```

```python
def norm(x,x0,s):
    return np.exp((x-x0)**2/(2*s**2))/np.sqrt(2*np.pi*s**2)
def lnlike(theta, t, N):
    r=theta
    return np.sum(np.log(r)*N-r-factorial(N))
def lnprior(theta):
    r=theta
    if 0 < r < 1:
        return 0#np.log(norm(r,1/50/2,1/50/4))
    return -np.inf
def lnprob(theta, t, N):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, t, N)
ndim, nwalkers = 1, 256
r0=1e-2
pos = [[np.random.uniform(1e-7,1e-2)] for i in range(nwalkers)]
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(tt, sn))
res=sampler.run_mcmc(pos, 500)
samples = sampler.chain[:, 50:, :].reshape((-1, ndim))
fig = corner.corner(samples, labels=["r"],truths=[N/T])
```

Does something changed in the SN rate from 185 until now?

```python

def lnlike(theta, t, N):
    r1,r2,tau=theta
    return np.nansum(np.where(t<tau,N*np.log(r1)-r1-factorial(N),N*np.log(r2)-r2-factorial(N)))
def lnprior(theta):
    r1,r2,tau=theta
    if (0 < r1 < 0.06) and (0 < r2 < 0.06) and (185 < tau < 2019):
        return 0#np.log(norm(r,1/50/2,1/50/4))
    return -np.inf
def lnprob(theta, t, N):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, t, N)
ndim, nwalkers = 3, 512
p0=[N/T,N/T,1000]
p0mi=[0,0,400]
p0ma=[0.1,0.1,2010]
pos = [np.random.uniform(p0mi,p0ma) for i in range(nwalkers)]
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(tt, sn))
res=sampler.run_mcmc(pos, 5000)
samples = sampler.chain[:, 50:, :].reshape((-1, ndim))

fig = corner.corner(samples, labels=["r1",'r2','t'],truths=[12/T,12/T,1000])
```

```python
disaster_data = np.array([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                           3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                           2, 2, 3, 4, 2, 1, 3, np.nan, 2, 1, 1, 1, 1, 3, 0, 0,
                           1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                           0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                           3, 3, 1, np.nan, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                           0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])
years = np.arange(1851, 1962)
```

```python
plt.plot(years,disaster_data,'o')
```

```python
from scipy.special import factorial
```

```python
factorial([1,2,3])
```

    array([1., 2., 6.])

```python
lnlike([3,2,1940],years, disaster_data)
```

    nan

```python
theta=np.random.uniform(p0mi,p0ma)
r1,r2,tau=theta
t=years
N=disaster_data
print(t[t<tau],N[t<tau]*np.log(r1)-r1-factorial(N[t<tau]))
np.nansum(np.where(t<tau,N*np.log(r1)-r1-factorial(N),N*np.log(r2)-r2-factorial(N)))
```

    [1851 1852 1853 1854 1855 1856 1857 1858 1859 1860 1861 1862 1863 1864
     1865 1866 1867 1868 1869 1870 1871 1872 1873 1874 1875 1876 1877 1878
     1879 1880 1881 1882 1883 1884 1885 1886 1887 1888 1889 1890 1891 1892
     1893 1894 1895 1896 1897 1898 1899 1900 1901 1902 1903 1904 1905 1906
     1907 1908 1909 1910 1911 1912 1913 1914 1915 1916 1917 1918 1919 1920
     1921 1922 1923 1924 1925 1926 1927 1928 1929 1930 1931 1932 1933 1934
     1935 1936 1937 1938 1939 1940 1941 1942 1943 1944 1945 1946 1947 1948
     1949 1950 1951 1952 1953] [ -24.87547145 -120.83366656  -24.87547145   -2.04269102   -2.00088613
      -24.87547145   -6.91727634  -24.87547145   -2.04269102 -720.79186167
       -6.91727634   -6.91727634  -24.87547145   -2.04269102   -2.95908124
     -720.79186167   -6.91727634   -6.91727634 -120.83366656  -24.87547145
     -120.83366656   -6.91727634   -2.00088613  -24.87547145  -24.87547145
       -2.00088613 -120.83366656 -120.83366656   -6.91727634  -24.87547145
       -2.95908124 -120.83366656   -2.95908124   -2.95908124   -6.91727634
      -24.87547145   -2.95908124   -2.00088613   -6.91727634           nan
       -2.95908124   -2.00088613   -2.00088613   -2.00088613   -2.00088613
       -6.91727634   -2.04269102   -2.04269102   -2.00088613   -2.04269102
       -2.00088613   -2.00088613   -2.04269102   -2.04269102   -6.91727634
       -2.00088613   -2.04269102   -6.91727634   -2.95908124   -2.95908124
       -2.04269102   -2.00088613   -2.00088613   -2.00088613   -2.04269102
       -2.00088613   -2.04269102   -2.00088613   -2.04269102   -2.04269102
       -2.04269102   -2.95908124   -2.00088613   -2.04269102   -2.04269102
       -2.04269102   -2.00088613   -2.00088613   -2.04269102   -2.95908124
       -6.91727634   -6.91727634   -2.00088613           nan   -2.95908124
       -2.00088613   -2.00088613   -2.00088613   -2.00088613   -2.95908124
      -24.87547145   -2.95908124   -2.04269102   -2.04269102   -2.00088613
      -24.87547145   -2.04269102   -2.04269102   -2.04269102   -2.00088613
       -2.04269102   -2.04269102   -2.04269102]

    -2737.7917198998916

```python
lnlike([5.68818417e-02, 1.84081966e+00, 1.85987260e+03],years, disaster_data)
```

    nan

```python
def lnlike(theta, t, N):
    r1,r2,tau=theta
    #return np.sum(np.where(t<tau,np.log(r1**N*np.exp(-r1)/factorial(N)),np.log(r2**N*np.exp(-r2)/factorial(N))))
    return np.nansum(np.where(t<tau,N*np.log(r1)-r1-factorial(N),N*np.log(r2)-r2-factorial(N)))
def lnprior(theta):
    r1,r2,tau=theta
    if (0 < r1 < 10) and (0 < r2 < 10) and (1851 < tau < 1962):
        return 0
    return -np.inf
def lnprob(theta, t, N):
    lp = lnprior(theta)
    if not np.isfinite(lp):
        return -np.inf
    return lp + lnlike(theta, t, N)
ndim, nwalkers = 3, 1024
p0=[4,2,1890]
p0mi=[0,0,1855]
p0ma=[7,7,1960]
pos = [np.random.uniform(p0mi,p0ma) for i in range(nwalkers)]
sampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=(years, disaster_data))
sampler.run_mcmc(pos, 6000);
samples = sampler.chain[:, 50:, :].reshape((-1, ndim));

fig = corner.corner(samples, labels=["r1",'r2','t'],truths=p0)
```

```python
pars=np.quantile(samples[100:,:],[0.5],axis=0).T
```

```python
plt.plot(years,disaster_data,'o')
plt.plot(years[years<pars[2]],pars[0]*np.ones(years[years<pars[2]].shape[0]))
plt.plot(years[years>pars[2]],pars[1]*np.ones(years[years>pars[2]].shape[0]))
```

```python
pars
```

    array([[   3.95749071],
           [   2.01772077],
           [1890.01739492]])