```python
import numpy as np
import sympy as sy
sy.init_printing()
```

```python
s = sy.symbols('s', real=False)
T,t = sy.symbols('T,t', real=True, positive=True)
T1,T2 = sy.symbols('T1,T2', real=True, positive=True)
```

```python
G = 1/(s*T+1)**2
G
```

```python
Y = sy.apart(G*1/s, s)
Y
```

```python
y = sy.inverse_laplace_transform(Y,s,t)
y
```

```python
G2 = 1/((s*T1+1)*(s*T2+1))
G2
```

```python
Y2 = sy.apart(G2*1/s, s)
Y2
```

```python
y2 = sy.inverse_laplace_transform(Y2,s,t)
y2
```

```python
sy.simplify(sy.expand(y2))
```


# M01. Exponential Growth

It is more or less obligatory to begin by mentioning *Moore's Law*, which, if it has somewhat lost its conviction over the years, nonetheless remains a robust icebreaker. The Law is usually stated as follows: *the density of components in an integrated circuit doubles about once every two years*.

In truth, the force of the Law comes not so much from it saying that transistor density doubles every *two* years as from it saying that it doubles *every* two years—it says that the growth of transistor density is an exponential process. It would certainly be a category-error to make this sort of remark about something linear ('I double in age every quarter-century') or logarithmic ('Usain Bolt doubles in speed every decade'). Exponential growth is quite special, and always transient. If you are an optimist, Moore's Law is really a law of human heroism. If you are slightly more disillusioned, or Japanese, it is what we might call a *seishin-ron*, an 'argument from willpower.'

Formally, a continuous signal $x(t)$ exhibits exponential growth if its rate of change is proportional to itself,

$$\frac{dx}{dt} \propto x(t).$$

More conveniently, $x(t)$ is an exponential function of time:

\begin{align}
&\Rightarrow \int \frac{1}{x(t)}dx(t) \propto \int dt\\
&\Rightarrow \ln{|x(t)|} \propto t+c\\
&\ldots\\
&\Rightarrow x(t) = x(0)e^{kt}.
\end{align}

When the exponential process is defined over discrete time, we may write

$$x(t) =  x(0)(1+\frac{r}{100})^t,$$

where we have written $x(t)$ in the familiar form of an initial value multiplied by a compounding interest rate $r$.

The doubling time $t_2$ of such an exponential process may be estimated by the so-called Rule of 72,

$$t_2 \approx \frac{72}{r},$$

which follows from

\begin{align}
&2x(0) = x(0)(1+\frac{r}{100})^{t_2} \\
&\Rightarrow 2 = (1+\frac{r}{100})^{t_2} \\
&\Rightarrow \log_{1+\frac{r}{100}}2 = t_2 \\
&\Rightarrow \frac{\ln2}{\ln{(1+\frac{r}{100})}} = t_2.
\end{align}

With a second-order Taylor-expansion of the denominator,

\begin{align}
t_2 &\approx \frac{\ln2}{\frac{r}{100}-\frac{\frac{r}{100}^2}{2}}\\
&\approx \frac{\ln2}{\frac{r}{100}}\\
&\approx \frac{69}{r}\\
&\approx \frac{72}{r}.
\end{align}

The last approximation may appear slightly cheeky but we do it for the acceptable reason that $69$ has four factors while $72$ has twelve.

An example is maybe in order: a quantity that increases at a rate of $r=8$ percent per unit time (around the average annual return of the S&P 500) will double in roughly $\frac{72}{8} = 9$ time-units.

Finally, we note that the Rule of 72 works best (has minimum error) for *small* values of $r$ (incidentally, around 8 percent). Perhaps what best illustrates the might of Moore's Law is that it corresponds to an annualized rate of *40 percent*.


Simplification

From

To make this document easier to read, we are going to enable pretty printing.

```julia
    >>> from sympy import *
    >>> x, y, z = symbols('x y z')
    >>> init_printing(use_unicode=True)
```

In Julia:

" not ' are used for strings:

pretty printing in enable by default

we input extra functions from sympy such as powsimp, ...

```julia
using SymPy
import_from(sympy)
x, y, z = symbols("x y z")
```

    (x, y, z)

simplify

Now let's jump in and do some interesting mathematics.  One of the most useful features of a symbolic manipulation system is the ability to simplify mathematical expressions.  SymPy has dozens of functions to perform various kinds of simplification.  There is also one general function called simplify() that attempts to apply all of these functions in an intelligent way to arrive at the simplest form of an expression.  Here are some examples

```julia
    >>> simplify(sin(x)**2 + cos(x)**2)
    1
    >>> simplify((x**3 + x**2 - x - 1)/(x**2 + 2*x + 1))
    x - 1
    >>> simplify(gamma(x)/gamma(x - 2))
    (x - 2)⋅(x - 1)
```

In Julia:

we need to load in SpecialFunctions to have access to gamma:

```julia
simplify(sin(x)^2 + cos(x)^2)
```

\begin{equation*}1\end{equation*}

```julia
simplify((x^3 + x^2 - x - 1)/(x^2 + 2*x + 1))
```

\begin{equation*}x - 1\end{equation*}

```julia
using SpecialFunctions
simplify(gamma(x)/gamma(x - 2))
```

\begin{equation*}\left(x - 2\right) \left(x - 1\right)\end{equation*}

Here, gamma(x) is $\Gamma(x)$, the gamma function <http://en.wikipedia.org/wiki/Gamma_function>_.  We see that simplify() is capable of handling a large class of expressions.

But simplify() has a pitfall.  It just applies all the major simplification operations in SymPy, and uses heuristics to determine the simplest result. But "simplest" is not a well-defined term.  For example, say we wanted to "simplify" x^2 + 2x + 1 into (x + 1)^2:

```julia
    >>> simplify(x**2 + 2*x + 1)
     2
    x  + 2⋅x + 1
```

In Julia:

```julia
simplify(x^2 + 2*x + 1)
```

\begin{equation*}x^{2} + 2 x + 1\end{equation*}

We did not get what we want.  There is a function to perform this simplification, called factor(), which will be discussed below.

Another pitfall to simplify() is that it can be unnecessarily slow, since it tries many kinds of simplifications before picking the best one.  If you already know exactly what kind of simplification you are after, it is better to apply the specific simplification function(s) that apply those simplifications.

Applying specific simplification functions instead of simplify() also has the advantage that specific functions have certain guarantees about the form of their output.  These will be discussed with each function below.  For example, factor(), when called on a polynomial with rational coefficients, is guaranteed to factor the polynomial into irreducible factors. simplify() has no guarantees.  It is entirely heuristical, and, as we saw above, it may even miss a possible type of simplification that SymPy is capable of doing.

simplify() is best when used interactively, when you just want to whittle down an expression to a simpler form.  You may then choose to apply specific functions once you see what simplify() returns, to get a more precise result.  It is also useful when you have no idea what form an expression will take, and you need a catchall function to simplify it.

Polynomial/Rational Function Simplification

expand

expand() is one of the most common simplification functions in SymPy. Although it has a lot of scopes, for now, we will consider its function in expanding polynomial expressions. For example:

```julia
    >>> expand((x + 1)**2)
     2
    x  + 2⋅x + 1
    >>> expand((x + 2)*(x - 3))
     2
    x  - x - 6
```

In Julia:

```julia
expand((x + 1)^2)
```

\begin{equation*}x^{2} + 2 x + 1\end{equation*}

```julia
expand((x + 2)*(x - 3))
```

\begin{equation*}x^{2} - x - 6\end{equation*}

Given a polynomial, expand() will put it into a canonical form of a sum of monomials.

expand() may not sound like a simplification function.  After all, by its very name, it makes expressions bigger, not smaller.  Usually this is the case, but often an expression will become smaller upon calling expand() on it due to cancellation.

```julia
    >>> expand((x + 1)*(x - 2) - (x - 1)*x)
    -2
```

In Julia:

```julia
expand((x + 1)*(x - 2) - (x - 1)*x)
```

\begin{equation*}-2\end{equation*}

factor

factor() takes a polynomial and factors it into irreducible factors over the rational numbers.  For example:

```julia
    >>> factor(x**3 - x**2 + x - 1)
            ⎛ 2    ⎞
    (x - 1)⋅⎝x  + 1⎠
    >>> factor(x**2*z + 4*x*y*z + 4*y**2*z)
               2
    z⋅(x + 2⋅y)
```

In Julia:

```julia
factor(x^3 - x^2 + x - 1)
```

\begin{equation*}\left(x - 1\right) \left(x^{2} + 1\right)\end{equation*}

```julia
factor(x^2*z + 4*x*y*z + 4*y^2*z)
```

\begin{equation*}z \left(x + 2 y\right)^{2}\end{equation*}

For polynomials, factor() is the opposite of expand().  factor() uses a complete multivariate factorization algorithm over the rational numbers, which means that each of the factors returned by factor() is guaranteed to be irreducible.

If you are interested in the factors themselves, factor_list returns a more structured output.

```julia
    >>> factor_list(x**2*z + 4*x*y*z + 4*y**2*z)
    (1, [(z, 1), (x + 2⋅y, 2)])
```

In Julia:

```julia
factor_list(x^2*z + 4*x*y*z + 4*y^2*z)
```

    (1, Tuple{SymPy.Sym,Int64}[(z, 1), (x + 2*y, 2)])

Note that the input to factor and expand need not be polynomials in the strict sense.  They will intelligently factor or expand any kind of expression (though note that the factors may not be irreducible if the input is no longer a polynomial over the rationals).

```julia
    >>> expand((cos(x) + sin(x))**2)
       2                           2
    sin (x) + 2⋅sin(x)⋅cos(x) + cos (x)
    >>> factor(cos(x)**2 + 2*cos(x)*sin(x) + sin(x)**2)
                     2
    (sin(x) + cos(x))
```

In Julia:

```julia
expand((cos(x) + sin(x))^2)
factor(cos(x)^2 + 2*cos(x)*sin(x) + sin(x)^2)
```

\begin{equation*}\left(\sin{\left (x \right )} + \cos{\left (x \right )}\right)^{2}\end{equation*}

collect

collect() collects common powers of a term in an expression.  For example

```julia
    >>> expr = x*y + x - 3 + 2*x**2 - z*x**2 + x**3
    >>> expr
     3    2        2
    x  - x ⋅z + 2⋅x  + x⋅y + x - 3
    >>> collected_expr = collect(expr, x)
    >>> collected_expr
     3    2
    x  + x ⋅(-z + 2) + x⋅(y + 1) - 3
```

In Julia:

```julia
expr = x*y + x - 3 + 2*x^2 - z*x^2 + x^3
expr
```

\begin{equation*}x^{3} - x^{2} z + 2 x^{2} + x y + x - 3\end{equation*}

```julia
collected_expr = collect(expr, x)
collected_expr
```

\begin{equation*}x^{3} + x^{2} \left(- z + 2\right) + x \left(y + 1\right) - 3\end{equation*}

collect() is particularly useful in conjunction with the .coeff() method.  expr.coeff(x, n) gives the coefficient of x**n in expr:

```julia
    >>> collected_expr.coeff(x, 2)
    -z + 2
```

In Julia:

```julia
collected_expr.coeff(x, 2)
```

\begin{equation*}- z + 2\end{equation*}

TODO

Discuss coeff method in more detail in some other section (maybe basic expression manipulation tools)

cancel

cancel() will take any rational function and put it into the standard canonical form, $\frac{p}{q}$, where $p$ and $q$ are expanded polynomials with no common factors, and the leading coefficients of $p$ and $q$ do not have denominators (i.e., are integers).

```julia
    >>> cancel((x**2 + 2*x + 1)/(x**2 + x))
    x + 1
    ─────
      x

    >>> expr = 1/x + (3*x/2 - 2)/(x - 4)
    >>> expr
    3⋅x
    ─── - 2
     2        1
    ─────── + ─
     x - 4    x
    >>> cancel(expr)
       2
    3⋅x  - 2⋅x - 8
    ──────────────
         2
      2⋅x  - 8⋅x

    >>> expr = (x*y**2 - 2*x*y*z + x*z**2 + y**2 - 2*y*z + z**2)/(x**2 - 1)
    >>> expr
       2                2    2            2
    x⋅y  - 2⋅x⋅y⋅z + x⋅z  + y  - 2⋅y⋅z + z
    ───────────────────────────────────────
                      2
                     x  - 1
    >>> cancel(expr)
     2            2
    y  - 2⋅y⋅z + z
    ───────────────
         x - 1
```

In Julia:

```julia
cancel((x^2 + 2*x + 1)/(x^2 + x))
```

\begin{equation*}\frac{x + 1}{x}\end{equation*}

```julia
expr = 1/x + (3*x/2 - 2)/(x - 4)
expr
```

\begin{equation*}\frac{\frac{3 x}{2} - 2}{x - 4} + \frac{1}{x}\end{equation*}

```julia
cancel(expr)
```

\begin{equation*}\frac{3 x^{2} - 2 x - 8}{2 x^{2} - 8 x}\end{equation*}

```julia
expr = (x*y^2 - 2*x*y*z + x*z^2 + y^2 - 2*y*z + z^2)/(x^2 - 1)
expr
cancel(expr)
```

\begin{equation*}\frac{y^{2} - 2 y z + z^{2}}{x - 1}\end{equation*}

Note that since factor() will completely factorize both the numerator and the denominator of an expression, it can also be used to do the same thing:

```julia
    >>> factor(expr)
           2
    (y - z)
    ────────
     x - 1
```

In Julia:

```julia
factor(expr)
```

\begin{equation*}\frac{\left(y - z\right)^{2}}{x - 1}\end{equation*}

However, if you are only interested in making sure that the expression is in canceled form, cancel() is more efficient than factor().

apart

apart() performs a partial fraction decomposition <http://en.wikipedia.org/wiki/Partial_fraction_decomposition>_ on a rational function.

```julia
    >>> expr = (4*x**3 + 21*x**2 + 10*x + 12)/(x**4 + 5*x**3 + 5*x**2 + 4*x)
    >>> expr
       3       2
    4⋅x  + 21⋅x  + 10⋅x + 12
    ────────────────────────
      4      3      2
     x  + 5⋅x  + 5⋅x  + 4⋅x
    >>> apart(expr)
     2⋅x - 1       1     3
    ────────── - ───── + ─
     2           x + 4   x
    x  + x + 1
```

In Julia:

```julia
expr = (4*x^3 + 21*x^2 + 10*x + 12)/(x^4 + 5*x^3 + 5*x^2 + 4*x)
expr
```

\begin{equation*}\frac{4 x^{3} + 21 x^{2} + 10 x + 12}{x^{4} + 5 x^{3} + 5 x^{2} + 4 x}\end{equation*}

```julia
apart(expr)
```

\begin{equation*}\frac{2 x - 1}{x^{2} + x + 1} - \frac{1}{x + 4} + \frac{3}{x}\end{equation*}

Trigonometric Simplification

Note

SymPy follows Python's naming conventions for inverse trigonometric functions, which is to append an a to the front of the function's name.  For example, the inverse cosine, or arc cosine, is called acos().

```julia
   >>> acos(x)
   acos(x)
   >>> cos(acos(x))
   x
   >>> asin(1)
   π
   ─
   2
```

In Julia:

```julia
acos(x)
```

\begin{equation*}\operatorname{acos}{\left (x \right )}\end{equation*}

```julia
cos(acos(x))
```

\begin{equation*}x\end{equation*}

```julia
asin(1)
```

    1.5707963267948966

TODO

Can we actually do anything with inverse trig functions, simplification wise?

trigsimp

To simplify expressions using trigonometric identities, use trigsimp().

```julia
    >>> trigsimp(sin(x)^2 + cos(x)**2)
    1
    >>> trigsimp(sin(x)**4 - 2*cos(x)**2*sin(x)**2 + cos(x)**4)
    cos(4⋅x)   1
    ──────── + ─
       2       2
    >>> trigsimp(sin(x)*tan(x)/sec(x))
       2
    sin (x)
```

In Julia:

```julia
trigsimp(sin(x)^2 + cos(x)^2)
```

\begin{equation*}1\end{equation*}

```julia
trigsimp(sin(x)^4 - 2*cos(x)^2*sin(x)^2 + cos(x)^4)
```

\begin{equation*}\frac{\cos{\left (4 x \right )}}{2} + \frac{1}{2}\end{equation*}

```julia
trigsimp(sin(x)*tan(x)/sec(x))
```

\begin{equation*}\sin^{2}{\left (x \right )}\end{equation*}

trigsimp() also works with hyperbolic trig functions.

```julia
    >>> trigsimp(cosh(x)**2 + sinh(x)**2)
    cosh(2⋅x)
    >>> trigsimp(sinh(x)/tanh(x))
    cosh(x)
```

In Julia:

```julia
trigsimp(cosh(x)^2 + sinh(x)^2)
```

\begin{equation*}\cosh{\left (2 x \right )}\end{equation*}

```julia
trigsimp(sinh(x)/tanh(x))
```

\begin{equation*}\cosh{\left (x \right )}\end{equation*}

Much like simplify(), trigsimp() applies various trigonometric identities to the input expression, and then uses a heuristic to return the "best" one.

expand_trig

To expand trigonometric functions, that is, apply the sum or double angle identities, use expand_trig().

```julia
    >>> expand_trig(sin(x + y))
    sin(x)⋅cos(y) + sin(y)⋅cos(x)
    >>> expand_trig(tan(2*x))
       2⋅tan(x)
    ─────────────
         2
    - tan (x) + 1
```

In Julia:

```julia
expand_trig(sin(x + y))
```

\begin{equation*}\sin{\left (x \right )} \cos{\left (y \right )} + \sin{\left (y \right )} \cos{\left (x \right )}\end{equation*}

```julia
expand_trig(tan(2*x))
```

\begin{equation*}\frac{2 \tan{\left (x \right )}}{- \tan^{2}{\left (x \right )} + 1}\end{equation*}

Because expand_trig() tends to make trigonometric expressions larger, and trigsimp() tends to make them smaller, these identities can be applied in reverse using trigsimp()

```julia
    >>> trigsimp(sin(x)*cos(y) + sin(y)*cos(x))
    sin(x + y)
```

In Julia:

```julia
trigsimp(sin(x)*cos(y) + sin(y)*cos(x))
```

\begin{equation*}\sin{\left (x + y \right )}\end{equation*}

TODO

It would be much better to teach individual trig rewriting functions here, but they don't exist yet.  See https://github.com/sympy/sympy/issues/3456.

Powers

Before we introduce the power simplification functions, a mathematical discussion on the identities held by powers is in order.  There are three kinds of identities satisfied by exponents

x^ax^b = x^{a + b}

x^ay^a = (xy)^a

(x^a)^b = x^{ab}

Identity 1 is always true.

Identity 2 is not always true.  For example, if $x = y = -1$ and $a = \frac{1}{2}$, then $x^ay^a = \sqrt{-1}\sqrt{-1} = i\cdot i = -1$, whereas $(xy)^a = \sqrt{-1\cdot-1} = \sqrt{1} = 1$.  However, identity 2 is true at least if $x$ and $y$ are nonnegative and $a$ is real (it may also be true under other conditions as well).  A common consequence of the failure of identity 2 is that $\sqrt{x}\sqrt{y} \neq \sqrt{xy}$.

Identity 3 is not always true.  For example, if $x = -1$, $a = 2$, and $b = \frac{1}{2}$, then $(x^a)^b = {\left ((-1)^2\right )}^{1/2} = \sqrt{1} = 1$ and $x^{ab} = (-1)^{2\cdot1/2} = (-1)^1 = -1$.  However, identity 3 is true when $b$ is an integer (again, it may also hold in other cases as well).  Two common consequences of the failure of identity 3 are that $\sqrt{x^2}\neq x$ and that $\sqrt{\frac{1}{x}} \neq \frac{1}{\sqrt{x}}$.

To summarize

This: $x^ax^b = x^{a + b}$ is always true

This: $x^ay^a = (xy)^a$ is true when  $x, y \geq 0$ and $a \in \mathbb{R}$; but note $(-1)^{1/2}(-1)^{1/2} \neq (-1\cdot-1)^{1/2}$       and $\sqrt{x}\sqrt{y} \neq \sqrt{xy}$ in general

This: $(x^a)^b = x^{ab}$  when $b \in \mathbb{Z}$;                  but note ${\left((-1)^2\right )}^{1/2} \neq (-1)^{2\cdot1/2}$ and $\sqrt{x^2}\neq x$ and $\sqrt{\frac{1}{x}}\neq\frac{1}{\sqrt{x}}$ in general

This is important to remember, because by default, SymPy will not perform simplifications if they are not true in general.

In order to make SymPy perform simplifications involving identities that are only true under certain assumptions, we need to put assumptions on our Symbols.  We will undertake a full discussion of the assumptions system later, but for now, all we need to know are the following.

By default, SymPy Symbols are assumed to be complex (elements of $\mathbb{C}$).  That is, a simplification will not be applied to an expression with a given Symbol unless it holds for all complex numbers.

Symbols can be given different assumptions by passing the assumption to symbols().  For the rest of this section, we will be assuming that x and y are positive, and that a and b are real.  We will leave z, t, and c as arbitrary complex Symbols to demonstrate what happens in that case.

```julia
    >>> x, y = symbols('x y', positive=True)
    >>> a, b = symbols('a b', real=True)
    >>> z, t, c = symbols('z t c')
```

In Julia:

```julia
x, y = symbols("x y", positive=true)
a, b = symbols("a b", real=true)
z, t, c = symbols("z t c")
```

    (z, t, c)

TODO:

Rewrite this using the new assumptions

Note

In SymPy, sqrt(x) is just a shortcut to x**Rational(1, 2).  They are exactly the same object.

```julia
     >>> sqrt(x) == x**Rational(1, 2)
     True
```

In Julia:

we can construction rational numbers with //

```julia
sqrt(x) == x^(1//2)
```

    true

powsimp

powsimp() applies identities 1 and 2 from above, from left to right.

```julia
   >>> powsimp(x**a*x**b)
     a + b
    x
   >>> powsimp(x**a*y**a)
        a
   (x⋅y)
```

In Julia:

```julia
powsimp(x^a*x^b)
powsimp(x^a*y^a)
```

\begin{equation*}\left(x y\right)^{a}\end{equation*}

Notice that powsimp() refuses to do the simplification if it is not valid.

```julia
    >>> powsimp(t**c*z**c)
     c  c
    t ⋅z
```

In Julia:

```julia
powsimp(t^c*z^c)
```

\begin{equation*}t^{c} z^{c}\end{equation*}

If you know that you want to apply this simplification, but you don't want to mess with assumptions, you can pass the force=True flag.  This will force the simplification to take place, regardless of assumptions.

```julia
    >>> powsimp(t**c*z**c, force=True)
         c
    (t⋅z)
```

In Julia:

```julia
powsimp(t^c*z^c, force=true)
```

\begin{equation*}\left(t z\right)^{c}\end{equation*}

Note that in some instances, in particular, when the exponents are integers or rational numbers, and identity 2 holds, it will be applied automatically.

```julia
   >>> (z*t)**2
     2  2
    t ⋅z
   >>> sqrt(x*y)
    √x⋅√y
```

In Julia:

```julia
(z*t)^2
```

\begin{equation*}t^{2} z^{2}\end{equation*}

```julia
sqrt(x*y)
```

\begin{equation*}\sqrt{x} \sqrt{y}\end{equation*}

This means that it will be impossible to undo this identity with powsimp(), because even if powsimp() were to put the bases together, they would be automatically split apart again.

```julia
   >>> powsimp(z**2*t**2)
     2  2
    t ⋅z
   >>> powsimp(sqrt(x)*sqrt(y))
    √x⋅√y
```

In Julia:

```julia
powsimp(z^2*t^2)
```

\begin{equation*}t^{2} z^{2}\end{equation*}

```julia
powsimp(sqrt(x)*sqrt(y))
```

\begin{equation*}\sqrt{x} \sqrt{y}\end{equation*}

expand_power_exp / expand_power_base

expand_power_exp() and expand_power_base() apply identities 1 and 2 from right to left, respectively.

```julia
    >>> expand_power_exp(x**(a + b))
     a  b
    x ⋅x

    >>> expand_power_base((x*y)**a)
     a  a
    x ⋅y
```

In Julia:

```julia
expand_power_exp(x^(a + b))
expand_power_base((x*y)^a)
```

\begin{equation*}x^{a} y^{a}\end{equation*}

As with powsimp(), identity 2 is not applied if it is not valid.

```julia
    >>> expand_power_base((z*t)**c)
         c
    (t⋅z)
```

In Julia:

```julia
expand_power_base((z*t)^c)
```

\begin{equation*}\left(t z\right)^{c}\end{equation*}

And as with powsimp(), you can force the expansion to happen without fiddling with assumptions by using force=True.

```julia
   >>> expand_power_base((z*t)**c, force=True)
     c  c
    t ⋅z
```

In Julia:

```julia
expand_power_base((z*t)^c, force=true)
```

\begin{equation*}t^{c} z^{c}\end{equation*}

As with identity 2, identity 1 is applied automatically if the power is a number, and hence cannot be undone with expand_power_exp().

```julia
   >>> x**2*x**3
     5
    x
   >>> expand_power_exp(x**5)
     5
    x
```

In Julia:

```julia
x^2*x^3
expand_power_exp(x^5)
```

\begin{equation*}x^{5}\end{equation*}

powdenest

powdenest() applies identity 3, from left to right.

```julia
    >>> powdenest((x**a)**b)
     a⋅b
    x
```

In Julia:

```julia
powdenest((x^a)^b)
```

\begin{equation*}x^{a b}\end{equation*}

As before, the identity is not applied if it is not true under the given assumptions.

```julia
    >>> powdenest((z**a)**b)
        b
    ⎛ a⎞
    ⎝z ⎠
```

In Julia:

```julia
powdenest((z^a)^b)
```

\begin{equation*}\left(z^{a}\right)^{b}\end{equation*}

And as before, this can be manually overridden with force=True.

```julia
    >>> powdenest((z**a)**b, force=True)
     a⋅b
    z
```

In Julia:

```julia
powdenest((z^a)^b, force=true)
```

\begin{equation*}z^{a b}\end{equation*}

Exponentials and logarithms

Note

In SymPy, as in Python and most programming languages, log is the natural logarithm, also known as ln.  SymPy automatically provides an alias ln = log in case you forget this.

```julia
    >>> ln(x)
    log(x)
```

In Julia:

ln is exported

```julia
ln(x)
```

\begin{equation*}\log{\left (x \right )}\end{equation*}

Logarithms have similar issues as powers.  There are two main identities

$\log{(xy)} = \log{(x)} + \log{(y)}$

$\log{(x^n)} = n\log{(x)}$

Neither identity is true for arbitrary complex $x$ and $y$, due to the branch cut in the complex plane for the complex logarithm.  However, sufficient conditions for the identities to hold are if $x$ and $y$ are positive and $n$ is real.

```julia
    >>> x, y = symbols('x y', positive=True)
    >>> n = symbols('n', real=True)
```

In Julia:

```julia
x, y = symbols("x y", positive=true)
n = symbols("n", real=true)
```

\begin{equation*}n\end{equation*}

As before, z and t will be Symbols with no additional assumptions.

Note that the identity $\log{\left (\frac{x}{y}\right )} = \log(x) - \log(y)$ is a special case of identities 1 and 2 by $\log{\left (\frac{x}{y}\right )} =$ $\log{\left (x\cdot\frac{1}{y}\right )} =$ $\log(x) + \log{\left( y^{-1}\right )} =$ $\log(x) - \log(y)$, and thus it also holds if x and y are positive, but may not hold in general.

We also see that $\log{\left( e^x \right)} = x$ comes from $\log{\left ( e^x \right)} = x\log(e) = x$, and thus holds when $x$ is real (and it can be verified that it does not hold in general for arbitrary complex $x$, for example, $\log{\left (e^{x + 2\pi i}\right)} = \log{\left (e^x\right )} = x \neq x + 2\pi i$).

expand_log

To apply identities 1 and 2 from left to right, use expand_log().  As always, the identities will not be applied unless they are valid.

```julia
    >>> expand_log(log(x*y))
    log(x) + log(y)
    >>> expand_log(log(x/y))
    log(x) - log(y)
    >>> expand_log(log(x**2))
    2⋅log(x)
    >>> expand_log(log(x**n))
    n⋅log(x)
    >>> expand_log(log(z*t))
    log(t⋅z)
```

In Julia:

```julia
expand_log(log(x*y))
```

\begin{equation*}\log{\left (x \right )} + \log{\left (y \right )}\end{equation*}

```julia
expand_log(log(x/y))
```

\begin{equation*}\log{\left (x \right )} - \log{\left (y \right )}\end{equation*}

```julia
expand_log(log(x^2))
```

\begin{equation*}2 \log{\left (x \right )}\end{equation*}

```julia
expand_log(log(x^n))
```

\begin{equation*}n \log{\left (x \right )}\end{equation*}

```julia
expand_log(log(z*t))
```

\begin{equation*}\log{\left (t z \right )}\end{equation*}

As with powsimp() and powdenest(), expand_log() has a force option that can be used to ignore assumptions.

```julia
    >>> expand_log(log(z**2))
       ⎛ 2⎞
    log⎝z ⎠
    >>> expand_log(log(z**2), force=True)
    2⋅log(z)
```

In Julia:

```julia
expand_log(log(z^2))
```

\begin{equation*}\log{\left (z^{2} \right )}\end{equation*}

```julia
expand_log(log(z^2), force=true)
```

\begin{equation*}2 \log{\left (z \right )}\end{equation*}

logcombine

To apply identities 1 and 2 from right to left, use logcombine().

```julia
    >>> logcombine(log(x) + log(y))
    log(x⋅y)
    >>> logcombine(n*log(x))
       ⎛ n⎞
    log⎝x ⎠
    >>> logcombine(n*log(z))
    n⋅log(z)
```

In Julia:

```julia
logcombine(log(x) + log(y))
logcombine(n*log(x))
logcombine(n*log(z))
```

\begin{equation*}n \log{\left (z \right )}\end{equation*}

logcombine() also has a force option that can be used to ignore assumptions.

```julia
    >>> logcombine(n*log(z), force=True)
       ⎛ n⎞
    log⎝z ⎠
```

In Julia:

```julia
logcombine(n*log(z), force=true)
```

\begin{equation*}\log{\left (z^{n} \right )}\end{equation*}

Special Functions

SymPy implements dozens of special functions, ranging from functions in combinatorics to mathematical physics.

An extensive list of the special functions included with SymPy and their documentation is at the :ref:Functions Module <functions-contents> page.

For the purposes of this tutorial, let's introduce a few special functions in SymPy.

Let's define x, y, and z as regular, complex Symbols, removing any assumptions we put on them in the previous section.  We will also define k, m, and n.

```julia
    >>> x, y, z = symbols('x y z')
    >>> k, m, n = symbols('k m n')
```

In Julia:

```julia
x, y, z = symbols("x y z")
k, m, n = symbols("k m n")
```

    (k, m, n)

The factorial <http://en.wikipedia.org/wiki/Factorial>_ function is factorial.  factorial(n) represents $n!= 1\cdot2\cdots(n - 1)\cdot n$. n! represents the number of permutations of n distinct items.

```julia
    >>> factorial(n)
    n!
```

In Julia:

```julia
factorial(n)
```

\begin{equation*}n!\end{equation*}

The binomial coefficient <http://en.wikipedia.org/wiki/Binomial_coefficient>_ function is binomial.  binomial(n, k) represents $\binom{n}{k}$, the number of ways to choose k items from a set of n distinct items.  It is also often written as nCk, and is pronounced "n choose k".

```julia
    >>> binomial(n, k)
    ⎛n⎞
    ⎜ ⎟
    ⎝k⎠
```

In Julia:

```julia
binomial(n, k)
```

\begin{equation*}{\binom{n}{k}}\end{equation*}

The factorial function is closely related to the gamma function <http://en.wikipedia.org/wiki/Gamma_function>, gamma.  gamma(z) represents \Gamma(z) = \int0^\infty t^{z - 1}e^{-t}\,dt$, which for positive integer z is the same as (z - 1)!.

```julia
    >>> gamma(z)
    Γ(z)
```

In Julia:

recall, we need to load SpecialFunctions for gamma to be available

```julia
gamma(z)
```

\begin{equation*}\Gamma\left(z\right)\end{equation*}

The generalized hypergeometric function <http://en.wikipedia.org/wiki/Generalized_hypergeometric_function> is hyper.  hyper([a_1, ..., a_p], [b_1, ..., b_q], z) represents ${}_pF_q\left(\begin{matrix} a_1, \cdots, a_p \\ b_1, \cdots, b_q \end{matrix} \middle| z \right)$.  The most common case is ${}_2F_1$, which is often referred to as the ordinary hypergeometric function <http://en.wikipedia.org/wiki/Hypergeometric_function>.

```julia
    >>> hyper([1, 2], [3], z)
     ┌─  ⎛1, 2 │  ⎞
     ├─  ⎜     │ z⎟
    2╵ 1 ⎝ 3   │  ⎠
```

In Julia:

as [1,2] is not symbolic, we qualify hyper

```julia
sympy.hyper([1, 2], [3], z)
```

\begin{equation*}{{}_{2}F_{1}\left(\begin{matrix} 1, 2 \\ 3 \end{matrix}\middle| {z} \right)}\end{equation*}

rewrite

A common way to deal with special functions is to rewrite them in terms of one another.  This works for any function in SymPy, not just special functions. To rewrite an expression in terms of a function, use expr.rewrite(function).  For example,

```julia
    >>> tan(x).rewrite(sin)
         2
    2⋅sin (x)
    ─────────
     sin(2⋅x)
    >>> factorial(x).rewrite(gamma)
    Γ(x + 1)
```

In Julia:

```julia
tan(x).rewrite(sin)
```

\begin{equation*}\frac{2 \sin^{2}{\left (x \right )}}{\sin{\left (2 x \right )}}\end{equation*}

```julia
factorial(x).rewrite(gamma)
```

\begin{equation*}x!\end{equation*}

For some tips on applying more targeted rewriting, see the :ref:tutorial-manipulation section.

expand_func

To expand special functions in terms of some identities, use expand_func().  For example

```julia
    >>> expand_func(gamma(x + 3))
    x⋅(x + 1)⋅(x + 2)⋅Γ(x)
```

In Julia:

```julia
expand_func(gamma(x + 3))
```

\begin{equation*}x \left(x + 1\right) \left(x + 2\right) \Gamma\left(x\right)\end{equation*}

hyperexpand

To rewrite hyper in terms of more standard functions, use hyperexpand().

```julia
    >>> hyperexpand(hyper([1, 1], [2], z))
    -log(-z + 1)
    ─────────────
         z
```

In Julia:

As [1,1] is not symbolic, we qualify hyperexpand:

```julia
sympy.hyperexpand(hyper([1, 1], [2], z))
```

    MethodError(SymPy.hyper, ([1, 1], [2], z), 0x000000000000827e)

hyperexpand() also works on the more general Meijer G-function (see :py:meth:its documentation <sympy.functions.special.hyper.meijerg> for more information).

```julia
    >>> expr = meijerg([[1],[1]], [[1],[]], -z)
    >>> expr
    ╭─╮1, 1 ⎛1  1 │   ⎞
    │╶┐     ⎜     │ -z⎟
    ╰─╯2, 1 ⎝1    │   ⎠
    >>> hyperexpand(expr)
     1
     ─
     z
    ℯ
```

In Julia:

again, we qualify meijerg

```julia
expr = sympy.meijerg([[1],[1]], [[1],[]], -z)
expr
```

\begin{equation*}{G_{2, 1}^{1, 1}\left(\begin{matrix} 1 & 1 \\1 &  \end{matrix} \middle| {- z} \right)}\end{equation*}

```julia
hyperexpand(expr)
```

\begin{equation*}e^{\frac{1}{z}}\end{equation*}

combsimp

To simplify combinatorial expressions, use combsimp().

```julia
    >>> n, k = symbols('n k', integer = True)
    >>> combsimp(factorial(n)/factorial(n - 3))
    n⋅(n - 2)⋅(n - 1)
    >>> combsimp(binomial(n+1, k+1)/binomial(n, k))
    n + 1
    ─────
    k + 1
```

In Julia:

```julia
n, k = symbols("n k", integer = true)
```

    (n, k)

```julia
combsimp(factorial(n)/factorial(n - 3))
```

\begin{equation*}n \left(n - 2\right) \left(n - 1\right)\end{equation*}

```julia
combsimp(binomial(n+1, k+1)/binomial(n, k))
```

\begin{equation*}\frac{n + 1}{k + 1}\end{equation*}

gammasimp

To simplify expressions with gamma functions or combinatorial functions with non-integer argument, use gammasimp().

```julia
    >>> gammasimp(gamma(x)*gamma(1 - x))
       π
    ────────
    sin(π⋅x)
```

In Julia:

```julia
gammasimp(gamma(x)*gamma(1 - x))
```

\begin{equation*}\frac{\pi}{\sin{\left (\pi x \right )}}\end{equation*}

Example: Continued Fractions

Let's use SymPy to explore continued fractions.  A continued fraction <http://en.wikipedia.org/wiki/Continued_fraction>_ is an expression of the form

$$
   a_0 + \cfrac{1}{a_1 + \cfrac{1}{a_2 + \cfrac{1}{ \ddots + \cfrac{1}{a_n}
   }}}
   $$

where $a_0, \ldots, a_n$ are integers, and $a_1, \ldots, a_n$ are positive. Acontinued fraction can also be infinite, but infinite objects are more difficult to represent in computers, so we will only examine the finite case here.

A continued fraction of the above form is often represented as a list $[a_0; a_1, \ldots, a_n]$.  Let's write a simple function that converts such a list to its continued fraction form.  The easiest way to construct a continued fraction from a list is to work backwards.  Note that despite the apparent symmetry of the definition, the first element, a_0, must usually be handled differently from the rest.

```julia
    >>> def list_to_frac(l):
    ...     expr = Integer(0)
    ...     for i in reversed(l[1:]):
    ...         expr += i
    ...         expr = 1/expr
    ...     return l[0] + expr
    >>> list_to_frac([x, y, z])
          1
    x + ─────
            1
        y + ─
            z
```

In Julia:

```julia
function list_to_frac(l)
   expr = sympy.Integer(0)
   for i in reverse(l[1:end])
      expr += i
      expr = 1/expr
   end
   return l[1] + expr
end
list_to_frac([x, y, z])
```

\begin{equation*}x + \frac{1}{x + \frac{1}{y + \frac{1}{z}}}\end{equation*}

We use Integer(0) in list_to_frac so that the result will always be a SymPy object, even if we only pass in Python ints.

```julia
    >>> list_to_frac([1, 2, 3, 4])
    43
    ──
    30
```

In Julia:

```julia
list_to_frac([1, 2, 3, 4])
```

\begin{equation*}\frac{73}{43}\end{equation*}

Every finite continued fraction is a rational number, but we are interested in symbolics here, so let's create a symbolic continued fraction.  The symbols() function that we have been using has a shortcut to create numbered symbols.  symbols('a0:5') will create the symbols a0, a1, ..., a4.

```julia
    >>> syms = symbols('a0:5')
    >>> syms
    (a₀, a₁, a₂, a₃, a₄)
    >>> a0, a1, a2, a3, a4 = syms
    >>> frac = list_to_frac(syms)
    >>> frac
                 1
    a₀ + ─────────────────
                   1
         a₁ + ────────────
                      1
              a₂ + ───────
                        1
                   a₃ + ──
                        a₄
```

In Julia:

```julia
syms = symbols("a0:5")
syms
```

    (a0, a1, a2, a3, a4)

```julia
a0, a1, a2, a3, a4 = syms
```

    (a0, a1, a2, a3, a4)

```julia
frac = list_to_frac(syms)
frac
```

\begin{equation*}a_{0} + \frac{1}{a_{0} + \frac{1}{a_{1} + \frac{1}{a_{2} + \frac{1}{a_{3} + \frac{1}{a_{4}}}}}}\end{equation*}

This form is useful for understanding continued fractions, but lets put it into standard rational function form using cancel().

```julia
    >>> frac = cancel(frac)
    >>> frac
    a₀⋅a₁⋅a₂⋅a₃⋅a₄ + a₀⋅a₁⋅a₂ + a₀⋅a₁⋅a₄ + a₀⋅a₃⋅a₄ + a₀ + a₂⋅a₃⋅a₄ + a₂ + a₄
    ─────────────────────────────────────────────────────────────────────────
                     a₁⋅a₂⋅a₃⋅a₄ + a₁⋅a₂ + a₁⋅a₄ + a₃⋅a₄ + 1

```

In Julia:

```julia
frac = cancel(frac)
frac
```

\begin{equation*}\frac{a_{0}^{2} a_{1} a_{2} a_{3} a_{4} + a_{0}^{2} a_{1} a_{2} + a_{0}^{2} a_{1} a_{4} + a_{0}^{2} a_{3} a_{4} + a_{0}^{2} + a_{0} a_{2} a_{3} a_{4} + a_{0} a_{2} + a_{0} a_{4} + a_{1} a_{2} a_{3} a_{4} + a_{1} a_{2} + a_{1} a_{4} + a_{3} a_{4} + 1}{a_{0} a_{1} a_{2} a_{3} a_{4} + a_{0} a_{1} a_{2} + a_{0} a_{1} a_{4} + a_{0} a_{3} a_{4} + a_{0} + a_{2} a_{3} a_{4} + a_{2} + a_{4}}\end{equation*}

Now suppose we were given frac in the above canceled form. In fact, we might be given the fraction in any form, but we can always put it into the above canonical form with cancel().  Suppose that we knew that it could be rewritten as a continued fraction.  How could we do this with SymPy?  A continued fraction is recursively $c + \frac{1}{f}$, where $c$ is an integer and $f$ is a (smaller) continued fraction.  If we could write the expression in this form, we could pull out each $c$ recursively and add it to a list.  We could then get a continued fraction with our list_to_frac() function.

The key observation here is that we can convert an expression to the form c + \frac{1}{f} by doing a partial fraction decomposition with respect to c. This is because f does not contain c.  This means we need to use the apart() function.  We use apart() to pull the term out, then subtract it from the expression, and take the reciprocal to get the f part.

```julia
    >>> l = []
    >>> frac = apart(frac, a0)
    >>> frac
                    a₂⋅a₃⋅a₄ + a₂ + a₄
    a₀ + ───────────────────────────────────────
         a₁⋅a₂⋅a₃⋅a₄ + a₁⋅a₂ + a₁⋅a₄ + a₃⋅a₄ + 1
    >>> l.append(a0)
    >>> frac = 1/(frac - a0)
    >>> frac
    a₁⋅a₂⋅a₃⋅a₄ + a₁⋅a₂ + a₁⋅a₄ + a₃⋅a₄ + 1
    ───────────────────────────────────────
               a₂⋅a₃⋅a₄ + a₂ + a₄
```

In Julia:

```julia
l = []
frac = apart(frac, a0)
frac
push!(l, append(a0))
frac = 1/(frac - a0)
frac
```

\begin{equation*}\frac{a_{0} a_{1} a_{2} a_{3} a_{4} + a_{0} a_{1} a_{2} + a_{0} a_{1} a_{4} + a_{0} a_{3} a_{4} + a_{0} + a_{2} a_{3} a_{4} + a_{2} + a_{4}}{a_{1} a_{2} a_{3} a_{4} + a_{1} a_{2} + a_{1} a_{4} + a_{3} a_{4} + 1}\end{equation*}

Now we repeat this process

```julia
    >>> frac = apart(frac, a1)
    >>> frac
             a₃⋅a₄ + 1
    a₁ + ──────────────────
         a₂⋅a₃⋅a₄ + a₂ + a₄
    >>> l.append(a1)
    >>> frac = 1/(frac - a1)
    >>> frac = apart(frac, a2)
    >>> frac
             a₄
    a₂ + ─────────
         a₃⋅a₄ + 1
    >>> l.append(a2)
    >>> frac = 1/(frac - a2)
    >>> frac = apart(frac, a3)
    >>> frac
         1
    a₃ + ──
         a₄
    >>> l.append(a3)
    >>> frac = 1/(frac - a3)
    >>> frac = apart(frac, a4)
    >>> frac
    a₄
    >>> l.append(a4)
    >>> list_to_frac(l)
                 1
    a₀ + ─────────────────
                   1
         a₁ + ────────────
                      1
              a₂ + ───────
                        1
                   a₃ + ──
                        a₄
```

In Julia:

```julia
frac = apart(frac, a1)
frac
```

\begin{equation*}a_{0} + \frac{a_{2} a_{3} a_{4} + a_{2} + a_{4}}{a_{1} a_{2} a_{3} a_{4} + a_{1} a_{2} + a_{1} a_{4} + a_{3} a_{4} + 1}\end{equation*}

```julia
push!(l, a1)
frac = 1/(frac - a1)
```

\begin{equation*}\frac{1}{a_{0} - a_{1} + \frac{a_{2} a_{3} a_{4} + a_{2} + a_{4}}{a_{1} a_{2} a_{3} a_{4} + a_{1} a_{2} + a_{1} a_{4} + a_{3} a_{4} + 1}}\end{equation*}

```julia
frac = apart(frac, a2)
frac
```

\begin{equation*}\frac{a_{1}}{a_{0} a_{1} - a_{1}^{2} + 1} + \frac{a_{3} a_{4} + 1}{\left(a_{0} a_{1} - a_{1}^{2} + 1\right) \left(a_{0} a_{1} a_{2} a_{3} a_{4} + a_{0} a_{1} a_{2} + a_{0} a_{1} a_{4} + a_{0} a_{3} a_{4} + a_{0} - a_{1}^{2} a_{2} a_{3} a_{4} - a_{1}^{2} a_{2} - a_{1}^{2} a_{4} - a_{1} a_{3} a_{4} - a_{1} + a_{2} a_{3} a_{4} + a_{2} + a_{4}\right)}\end{equation*}

```julia
push!(l, a2)
frac = 1/(frac - a2)
```

\begin{equation*}\frac{1}{\frac{a_{1}}{a_{0} a_{1} - a_{1}^{2} + 1} - a_{2} + \frac{a_{3} a_{4} + 1}{\left(a_{0} a_{1} - a_{1}^{2} + 1\right) \left(a_{0} a_{1} a_{2} a_{3} a_{4} + a_{0} a_{1} a_{2} + a_{0} a_{1} a_{4} + a_{0} a_{3} a_{4} + a_{0} - a_{1}^{2} a_{2} a_{3} a_{4} - a_{1}^{2} a_{2} - a_{1}^{2} a_{4} - a_{1} a_{3} a_{4} - a_{1} + a_{2} a_{3} a_{4} + a_{2} + a_{4}\right)}}\end{equation*}

```julia
frac = apart(frac, a3)
frac
```

\begin{equation*}\frac{a_{4}}{\left(a_{0} a_{1} a_{2}^{2} + a_{0} a_{2} - a_{1}^{2} a_{2}^{2} - 2 a_{1} a_{2} + a_{2}^{2} - 1\right) \left(a_{0} a_{1} a_{2}^{2} a_{3} a_{4} + a_{0} a_{1} a_{2}^{2} + a_{0} a_{1} a_{2} a_{4} + a_{0} a_{2} a_{3} a_{4} + a_{0} a_{2} - a_{1}^{2} a_{2}^{2} a_{3} a_{4} - a_{1}^{2} a_{2}^{2} - a_{1}^{2} a_{2} a_{4} - 2 a_{1} a_{2} a_{3} a_{4} - 2 a_{1} a_{2} - a_{1} a_{4} + a_{2}^{2} a_{3} a_{4} + a_{2}^{2} + a_{2} a_{4} - a_{3} a_{4} - 1\right)} - \frac{a_{0} a_{1} a_{2} + a_{0} - a_{1}^{2} a_{2} - a_{1} + a_{2}}{a_{0} a_{1} a_{2}^{2} + a_{0} a_{2} - a_{1}^{2} a_{2}^{2} - 2 a_{1} a_{2} + a_{2}^{2} - 1}\end{equation*}

```julia
push!(l, a3)
frac = 1/(frac - a3)
```

\begin{equation*}\frac{1}{- a_{3} + \frac{a_{4}}{\left(a_{0} a_{1} a_{2}^{2} + a_{0} a_{2} - a_{1}^{2} a_{2}^{2} - 2 a_{1} a_{2} + a_{2}^{2} - 1\right) \left(a_{0} a_{1} a_{2}^{2} a_{3} a_{4} + a_{0} a_{1} a_{2}^{2} + a_{0} a_{1} a_{2} a_{4} + a_{0} a_{2} a_{3} a_{4} + a_{0} a_{2} - a_{1}^{2} a_{2}^{2} a_{3} a_{4} - a_{1}^{2} a_{2}^{2} - a_{1}^{2} a_{2} a_{4} - 2 a_{1} a_{2} a_{3} a_{4} - 2 a_{1} a_{2} - a_{1} a_{4} + a_{2}^{2} a_{3} a_{4} + a_{2}^{2} + a_{2} a_{4} - a_{3} a_{4} - 1\right)} - \frac{a_{0} a_{1} a_{2} + a_{0} - a_{1}^{2} a_{2} - a_{1} + a_{2}}{a_{0} a_{1} a_{2}^{2} + a_{0} a_{2} - a_{1}^{2} a_{2}^{2} - 2 a_{1} a_{2} + a_{2}^{2} - 1}}\end{equation*}

```julia
frac = apart(frac, a4)
frac
```

\begin{equation*}- \frac{a_{0} a_{1} a_{2}^{2} a_{3} + a_{0} a_{1} a_{2} + a_{0} a_{2} a_{3} - a_{1}^{2} a_{2}^{2} a_{3} - a_{1}^{2} a_{2} - 2 a_{1} a_{2} a_{3} - a_{1} + a_{2}^{2} a_{3} + a_{2} - a_{3}}{a_{0} a_{1} a_{2}^{2} a_{3}^{2} + 2 a_{0} a_{1} a_{2} a_{3} + a_{0} a_{1} + a_{0} a_{2} a_{3}^{2} + a_{0} a_{3} - a_{1}^{2} a_{2}^{2} a_{3}^{2} - 2 a_{1}^{2} a_{2} a_{3} - a_{1}^{2} - 2 a_{1} a_{2} a_{3}^{2} - 2 a_{1} a_{3} + a_{2}^{2} a_{3}^{2} + 2 a_{2} a_{3} - a_{3}^{2} + 1} + \frac{1}{\left(a_{0} a_{1} a_{2}^{2} a_{3}^{2} + 2 a_{0} a_{1} a_{2} a_{3} + a_{0} a_{1} + a_{0} a_{2} a_{3}^{2} + a_{0} a_{3} - a_{1}^{2} a_{2}^{2} a_{3}^{2} - 2 a_{1}^{2} a_{2} a_{3} - a_{1}^{2} - 2 a_{1} a_{2} a_{3}^{2} - 2 a_{1} a_{3} + a_{2}^{2} a_{3}^{2} + 2 a_{2} a_{3} - a_{3}^{2} + 1\right) \left(a_{0} a_{1} a_{2}^{2} a_{3}^{2} a_{4} + a_{0} a_{1} a_{2}^{2} a_{3} + 2 a_{0} a_{1} a_{2} a_{3} a_{4} + a_{0} a_{1} a_{2} + a_{0} a_{1} a_{4} + a_{0} a_{2} a_{3}^{2} a_{4} + a_{0} a_{2} a_{3} + a_{0} a_{3} a_{4} + a_{0} - a_{1}^{2} a_{2}^{2} a_{3}^{2} a_{4} - a_{1}^{2} a_{2}^{2} a_{3} - 2 a_{1}^{2} a_{2} a_{3} a_{4} - a_{1}^{2} a_{2} - a_{1}^{2} a_{4} - 2 a_{1} a_{2} a_{3}^{2} a_{4} - 2 a_{1} a_{2} a_{3} - 2 a_{1} a_{3} a_{4} - a_{1} + a_{2}^{2} a_{3}^{2} a_{4} + a_{2}^{2} a_{3} + 2 a_{2} a_{3} a_{4} + a_{2} - a_{3}^{2} a_{4} - a_{3} + a_{4}\right)}\end{equation*}

```julia
push!(l, a4)
list_to_frac(l)
```

\begin{equation*}a_{1} + \frac{1}{a_{1} + \frac{1}{a_{2} + \frac{1}{a_{3} + \frac{1}{a_{4}}}}}\end{equation*}

Quick tip

You can execute multiple lines at once in SymPy Live.  Typing Shift-Enter instead of Enter will enter a newline instead of executing.

Of course, this exercise seems pointless, because we already know that our frac is list_to_frac([a0, a1, a2, a3, a4]).  So try the following exercise.  Take a list of symbols and randomize them, and create the canceled continued fraction, and see if you can reproduce the original list.  For example

```julia
    >>> import random
    >>> l = list(symbols('a0:5'))
    >>> random.shuffle(l)
    >>> orig_frac = frac = cancel(list_to_frac(l))
    >>> del l
```

In Julia:

shuffle from Python is randperm in the Random module

```julia
using Random
l = symbols("a0:5")
l = l[randperm(length(l))]
orig_frac = frac = cancel(list_to_frac(l))
```

\begin{equation*}\frac{a_{0} a_{1}^{2} a_{2} a_{3} a_{4} + a_{0} a_{1}^{2} a_{2} + a_{0} a_{1}^{2} a_{3} + a_{0} a_{2} a_{3} a_{4} + a_{0} a_{2} + a_{0} a_{3} + a_{1}^{2} a_{3} a_{4} + a_{1}^{2} + a_{1} a_{2} a_{3} a_{4} + a_{1} a_{2} + a_{1} a_{3} + a_{3} a_{4} + 1}{a_{0} a_{1} a_{2} a_{3} a_{4} + a_{0} a_{1} a_{2} + a_{0} a_{1} a_{3} + a_{1} a_{3} a_{4} + a_{1} + a_{2} a_{3} a_{4} + a_{2} + a_{3}}\end{equation*}

Click on "Run code block in SymPy Live" on the definition of list_to_frac() above, and then on the above example, and try to reproduce l from frac.  I have deleted l at the end to remove the temptation for peeking (you can check your answer at the end by calling cancel(list_to_frac(l)) on the list that you generate at the end, and comparing it to orig_frac.

See if you can think of a way to figure out what symbol to pass to apart() at each stage (hint: think of what happens to $a_0$ in the formula $a_0 + \frac{1}{a_1 + \cdots}$ when it is canceled).

Note

Answer: a0 is the only symbol that does not appear in the denominator

return to index


# Multivariate Linear and Logistic Regression with Regularization

## Terminology & Symbols

 * Training size ($m$) - The number of samples we can use for learning
 * Dimensionality ($d$) - The number of dimensions in the input (feature) space
 * Feature set ($X$) - An $m \times d$ matrix where every row represents a single feature vector
 * Target ($y$) - An $m$-vector representing the value we are trying to predict
 * Training set ($(X,y)$) - The combined matrix of inputs and their associated known target values
 * Test set - An optional set of hold out data used for validation

 * Feature weights ($\theta$) - the free variables in our modeling
 * Hypothesis function ($h_{\theta}(x)$) - the function/model we are trying to learn by manipulating $\theta$
 * Loss Function ($J(\theta)$) - the "error" introduced by our method (What we want to minimize)

## Motivating Problems

### Regression

```python
import numpy as np
import matplotlib.pyplot as plt

rng = np.random.RandomState(7)

## Setup training data
def f(x):
    """ Magic ground truth function """
    return 5./7. *((np.pi)**2 - (np.pi)**2 + - 0 * 4 +  x * np.sin(x) + 2*(rng.rand(len(x))-0.5))

# generate full validation set of points
X = np.linspace(0, 10, 100)

# select a subset to act as the training data
rng.shuffle(X)
train_X = np.sort(X[:20])
train_y = f(train_X)
validation_y = f(X)

X = np.sort(X)[:, np.newaxis]
train_X = train_X[:, np.newaxis]
train_y = np.atleast_2d(train_y).T

## Plot the training data
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.legend(loc='lower left')

```

### Classification

```python
## Threshold the regression data
threshold = 3**2
radius_squared = (train_y)**2 + (train_X-5)**2
idxs1 = np.where(radius_squared > threshold)
idxs2 = np.where(radius_squared <= threshold)

## Plot the training data
plt.scatter(train_X[idxs1], train_y[idxs1], color='navy', s=30, marker='x', label="class 1")
plt.scatter(train_X[idxs2], train_y[idxs2], color='orange', s=30, marker='o', label="class 2")
plt.legend(loc='upper left')
plt.show()
```

## Linear Regression

### Univariate Linear Regression

```python
from sklearn.linear_model import LinearRegression

model = LinearRegression()
model.fit(train_X, train_y)
y = model.predict(X)
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.plot(X, y, color='teal', linewidth=lw, label="degree %d" % degree)
```

Given a training set consisting of one feature, $x$, and one target, $y$, we want to generate a hypothesis $h$ about the linear relationship between $x$ and $y$:

$y \approx h(x) = a + bx$

The bias ($a$) and feature weight ($b$) can be rolled into a single weight vector i.e., $\mathbf{\theta} = \begin{bmatrix} a \\ b\end{bmatrix}$:

$h_{\theta}(x) = \theta_0 + \theta_1 x$

In vector notation (we implicitly prepend a 1 to every x i.e., $\mathbf{x} = \begin{bmatrix} 1 \\ x\end{bmatrix}$):

$h_{\theta}(x) = \mathbf{\theta}^T\mathbf{x}$

One evaluation metric of the quality of fit is by looking at the **cumulative** squared error/residual:

$J(\theta) = \sum_{i=1}^{m}(h_{\theta}(x_i) - y_i)^2$

By minimizing $J$, we get the "best" model. I could make this an average by adding a $\frac{1}{m}$ out front, but it should be clear that this has no effect on where in weight space the minimum lies (it only squashes or stretches $J(\theta)$.

```python
def compute_cost(X, y, theta):
    m = y.shape[1]
    J = np.sum((X.dot(theta)-y.T)**2)/2/m
    return J

def gradient_descent(X, y, theta, alpha, iterations=100):
    X = np.atleast_2d(X)
    y = np.atleast_2d(y)
    theta = np.atleast_2d(theta)
    m = y.shape[1]
    J_history = np.zeros(iterations)

    for i in range(iterations):
        H = X.dot(theta)
        loss = H - y.T
        gradient = X.T.dot(loss) / m

        theta = theta - alpha * gradient
        J_history[i] = compute_cost(X, y, theta)

    return theta, J_history

##add bias
train_X_bias = np.hstack((np.ones((train_X.shape[0], 1)), train_X))
validation_X = np.hstack((np.ones((X.shape[0], 1)), X))

initial_theta = np.zeros((train_X_bias.shape[1],1))
final_theta, J_history = gradient_descent(train_X_bias, train_y, initial_theta, 0.01, 20)
print(final_theta)
y_predict = np.dot(validation_X,final_theta)
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.plot(X, y_predict, color='teal', linewidth=lw)
```

```python
plt.plot(range(len(J_history)), J_history)
```

### Multivariate Linear Regression

```python
from sklearn.preprocessing import PolynomialFeatures
degree = 25
model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
model.fit(train_X, train_y)
y = model.predict(X)
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.plot(X, y, color='yellowgreen', linewidth=lw, label="degree %d" % degree)
plt.gca().set_ylim(-10,10)
```

Note, these last two equations do not specify the dimensionality ($d$) of $\theta$ and $x$:

$h_{\theta}(x) = \mathbf{\theta}^T\mathbf{x}$

$J(\theta) = \sum_{i=1}^{m}(h_{\theta}(\mathbf{x_i}) - y_i)^2$

We can adapt our original test case just by adding new parameters: $\mathbf{x} = \begin{bmatrix} 1 \\ x \\ x^2 \\ x^3 \\ x^4 \end{bmatrix}$

The optimum lies where the gradient is zero, we can get there two ways:

 1. Solving the equation analytically
 2. Solving the equation numerically through iteration

#### Closed Form Solution of Ordinary Least Squares

$J(\theta) = \sum_{i=1}^{m}(h_{\theta}(x_i) - y_i)^2$

$J(\theta) = \sum_{i=1}^{m}(\theta^Tx_i - y_i)^2$

The optimum occurs where the gradient is zero:

\begin{align}
\frac{d J(\theta)}{d\mathbf{\theta}} & = 0 \\
2 \sum_{i=1}^{m}\left[\theta^Tx_i - y_i)x_i\right] & = 0 \\
\sum_{i=1}^{m}\left[(\theta^Tx_i)x_i - y_ix_i\right] & = 0 \\
\sum_{i=1}^{m}(\theta^Tx_i)x_i & = \sum_{i=1}^{m}y_ix_i \\
\sum_{i=1}^{m}(x_ix_i^T)\theta & = \sum_{i=1}^{m}y_ix_i \\
\end{align}

Let $A = \sum_{i=1}^{m}(x_ix_i^T) = \mathbf{X}^T \mathbf{X}$

$b=\sum_{i=1}^{m}y_ix_i$ = $\mathbf{X}^T\mathbf{y}$

\begin{align}
A\theta & = b \\
\theta & = A^{-1}b \\
\theta & = (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T\mathbf{y} \\
\end{align}

Note, we are inverting $\mathbf{X}^T \mathbf{X}$ which is $m \times m$, i.e., big with lots of data.

#### Gradient Descent

Recall:

$\frac{d J(\theta)}{d\theta_j} = 2 \sum_{i=1}^{m}\left[\theta^Tx_i - y_i)x_{i,j}\right]$

Represents the gradient of the cost function, so if we want to minimize this, we follow its negative direction.

Algorithm:

> Initialize $\theta_j = \mathbf{0}$

> Repeat {

>> $\theta_j = \theta_j - \alpha\left(\sum_{i=1}^{m}\left[(\theta^Tx_i - y_i)x_{i,j}\right]\right)$

> }

We saw last week the benefits of stochastic gradient descent where we don't have to use the whole dataset each time to do the update.

## Feature Scaling

### Why?

Varied dimension "widths" will pull the gradient descent algorithm in wider directions.

### How?

 * Range scaling $\left(\frac{x - x_{min}}{x_{max}- x_{min}}\right)$
 * Z-Score scaling $\left(\frac{x - \mu}{\sigma}\right)$

## Learning Rate

plot # of iterations vs. cost function

## Regularization

$J(\theta) = \sum_{i=1}^{m}(h_{\theta}(\mathbf{x_i}) - y_i)^2 + p(\theta)$

### Ridge Regularization

 * $L_2$ penalty term
 * Tikhonov Regularization

 $p(\theta) = \lambda \sum_{i=1}^{m}\theta_i^2$

```python
from sklearn.linear_model import Ridge
degree = 25
model = make_pipeline(PolynomialFeatures(degree), Ridge())
model.fit(train_X, train_y)
y = model.predict(X)
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.plot(X, y, color='gold', linewidth=lw, label="degree %d" % degree)
plt.gca().set_ylim(-10,10)
```

### LASSO Regularization

 * $L_1$ penalty term
 * Reduces number of dimensions
 * Good for feature selection

 $p(\theta) = \lambda \sum_{i=1}^{m}|\theta_i|$

```python
from sklearn.linear_model import Lasso
degree = 300
model = make_pipeline(PolynomialFeatures(degree), Lasso())
model.fit(train_X, train_y)
y = model.predict(X)
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.plot(X, y, color='darkorange', linewidth=lw, label="degree %d" % degree)
plt.gca().set_ylim(-10,10)
```

### Elastic Net Regularization

 * Combination $L_1$ and $L_2$ Regularization

 $p(\theta) = \lambda_1 \sum_{i=1}^{m}|\theta_i| + \lambda_2 \sum_{i=1}^{m}\theta_i^2$

```python
from sklearn.linear_model import ElasticNet
degree = 25
model = make_pipeline(PolynomialFeatures(degree), ElasticNet())
model.fit(train_X, train_y)
y = model.predict(X)
plt.scatter(train_X, train_y, color='navy', s=30, marker='o', label="training data")
plt.plot(X, y, color='firebrick', linewidth=lw, label="degree %d" % degree)
plt.gca().set_ylim(-10,10)
```

## Logistic Regression

We then formulate our problem such that we are using the sigmoid function to determine the probability that an input resides in a particular class of data. With multiple classes this turns into a "one-vs.-rest" scheme, thus if there are $k$ classes, there are $k$ instances of logistics regression. The highest prediction is taken as the class for that data point.

\begin{align}
h_\theta(x) & = g(\theta^Tx) \\
z & = \theta^Tx \\
g(z) & = \frac{1}{1+e^{-z}}\\
\end{align}

```python
def sigmoid(x):
  return 1. / (1 + np.exp(-x))

xx = np.linspace(-10,10,100)
plt.plot(xx, sigmoid(xx))
```

We use the sigmoid function to take the discrete output and make it differentiable.

Our $y$ values are discrete (0 or 1). How do we define a cost function for this?

Existing cost function:

$J(\theta) = \sum_{i=1}^{m}(h_{\theta}(x_i) - y_i)^2$

This will not work as it produces a non-convex $J$.

So, we need a convex cost function that has the following properties:
 * When the correct classification is 1, then a zero cost should be assigned to a value of $h(x)$ = 1
 * When the correct classification is 1, then a maximal cost should be assigned to a value of $h(x)$ = 0
 * When the correct classification is 0, then a zero cost should be assigned to a value of $h(x)$ = 0
 * When the correct classification is 0, then a maximal cost should be assigned to a value of $h(x)$ = 1

 Thus, we end up with:

$\text{Cost}(h_\theta(x), y) = $
\begin{cases}
    -\log(h_\theta(x)), & \text{if } y = 1\\
    -\log(1 - h_\theta(x)), &  \text{if } y = 0
\end{cases}

We can do this more compactly by using the fact that y is discrete:

$\text{Cost}(h, y) = -y\log(h - (1-y)\log(1 - h)$

This simplifies the gradient needed for gradient descent:

$
\frac{d}{d \theta}J(\theta) = \frac{1}{m} \left(-y^Tlog(h) - (1-y)^Tlog(1-h))\right)
$

### Regularized Logistic Regression

```python

```


# Illustration of the sampling theorem

## The Shannon sampling theorem
A signal $f(t)$ with Fourier transform that is zero outside $[-\omega_1, \omega_1]$ is completely described by equidistant points $f(kh)$ if the sampling frequency is higher than $2\omega_1$.

### Reconstruction
The reconstruction is given by
\begin{equation}
f(t) = \sum_{k=-\infty}^\infty f(kh) \frac{\sin (\omega_s(t-kh)/2)}{\omega_s (t-kh)/2} = \sum_{k=-\infty}^\infty f(kh) \mathrm{sinc} \frac{\omega_s(t-kh)}{2}
\end{equation}

## Example from class, Problem 7.2 in Åström & Wittenmark
A signal $y(t)$ that we want to sample for purpose of feedback control has frequency content within the range $(-\omega_0, \omega_2$. The signal is corrupted by a sinusoidal noise at the frequency $5\omega_0$, let's assume a cosine, since its spectrum is real:
\begin{equation}
y_m(t) = y(t) + a\cos 5\omega t
\end{equation}
What is the lowest sampling frequency we can use, and still separate the sampled sinusoid (possibly its alias frequency) from the frequency content of $y(t)$?
1. Solution in book: $\omega_s = 6\omega_0$.
2. Suggested in class: $\omega_s = 2\omega_0$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

# Generate signal of interest y(t). Let's use the sinc^2(t) function, which has a triangular fourier transform
# within (-2\pi, 2\pi)
def y_measurement(w0, t):
    y = np.sinc(w0*t/(2*np.pi))**2
    n = 0.1*np.cos(5*w0*t)
    return (y+n, y, n)

Nc = 4000 # Number of samples in "continuous" signal
T = 10.0 # Seconds to simulate
hc = T/Nc # Sampling frequency of "continuous" signal
wNc = np.pi/hc

t = np.linspace(0,10, Nc)
w0 = 2*np.pi

(ym, y, n) = y_measurement(w0, t)

# Fourier transforms
Yf = np.fft.fft(y)
Nf = np.fft.fft(n)
wpos = np.linspace(0,wNc, Nc/2)

# Plot signals and discrete Fourier transform
plt.figure(figsize=(16,4))
plt.plot(t,y)
plt.plot(t,n)
plt.plot(t, y+n)
plt.xlabel(r'$t$ [s]')
plt.xlim((-0.1,4))
plt.legend((r'$y(t)$', r'$n(t)$', r'$y(t)+n(t)$'))
plt.title('Time series')

plt.figure(figsize=(16,4))
plt.plot(np.hstack((-wpos[::-1]-wpos[1], wpos)),
         np.hstack((Yf[int(Nc/2):], Yf[:int(Nc/2)])))
plt.plot(np.hstack((-wpos[::-1]-wpos[1], wpos)),
         np.hstack((Nf[int(Nc/2):], Nf[:int(Nc/2)])))
plt.xlabel(r'$\omega$ [rad/s]')
plt.xlim((-6*w0, 6*w0))
plt.xticks((-5*w0, -w0, w0, 5*w0))
plt.ylim((-20, 220))
lbls=plt.gca().set_xticklabels([r'$-5\omega_0$', r'$-\omega_0$',  r'$\omega_0$', r'$5\omega_0$'])
plt.title('Spectrum (real part)')
```

```python
# Now let's sample at ws=6w0
N = 600 # Number of samples to take
ws1 = 6*w0
h1 = 2*np.pi/ws1
ts1 = np.arange(N)*h1
(ym1, y1, n1) = y_measurement(w0, ts1)

Ym1f = np.fft.fft(ym1)
wpos1 = np.linspace(0, ws1/2, N/2)

# Plot the sampled signal and its spectrum
plt.figure(figsize=(16,4))
plt.plot(t, ym, color=[0.7, 0.7, 1])
plt.stem(ts1,ym1, linefmt='r--', markerfmt='ro', basefmt = 'r-')
plt.xlabel(r'$t$ [s]')
plt.xlim((-0.1,4))
plt.title('Time series')
plt.figure(figsize=(10,4))
plt.plot(np.hstack((-wpos1[::-1]-wpos1[1], wpos1)),
         np.hstack((Ym1f[int(N/2):], Ym1f[:int(N/2)])))
plt.xlabel(r'$\omega$ [rad/s]')
plt.xlim((-6*w0, 6*w0))
plt.xticks((-5*w0, -w0, w0, 5*w0))
lbls=plt.gca().set_xticklabels([r'$-5\omega_0$', r'$-\omega_0$',  r'$\omega_0$', r'$5\omega_0$'])
plt.title('Spectrum (real part)')
```

```python
# And sampling at ws=2w0
N = 600 # Number of samples to take
ws2 = 2*w0
h2 = 2*np.pi/ws2
ts2 = np.arange(N)*h2
(ym2, y2, n2) = y_measurement(w0, ts2)

Ym2f = np.fft.fft(ym2)
Ym2fpos = Ym2f[:int(N/2)]
Ym2fpos[-1] = 0.5*Ym2f[int(N/2)] # Divide the energy at wN equally among the positive and negative part
Ym2fneg = Ym2f[int(N/2):]
Ym2fneg[0] /= 2.0

wpos2 = np.linspace(0, ws2/2, N/2)

# Plot the sampled signal and its spectrum
plt.figure(figsize=(16,4))
plt.plot(t, ym, color=[0.7, 0.7, 1])
plt.stem(ts2,ym2, linefmt='r--', markerfmt='ro', basefmt = 'r-')
plt.xlabel(r'$t$ [s]')
plt.xlim((-0.1,4))
plt.title('Time series')

plt.figure(figsize=(16,4))
plt.plot(np.hstack((-wpos2[::-1]-wpos2[1], wpos2)), np.hstack((Ym2fneg, Ym2fpos)))
plt.xlabel(r'$\omega$ [rad/s]')
plt.xlim((-6*w0, 6*w0))
plt.xticks((-5*w0, -w0, w0, 5*w0))
lbls=plt.gca().set_xticklabels([r'$-5\omega_0$', r'$-\omega_0$',  r'$\omega_0$', r'$5\omega_0$'])
plt.title('Spectrum (real part)')

```

## A digital notch filter to get rid of the alias of the sinusoid at $\omega_o$

A digital filter with two complex conjugated zeros at $\mathrm{e}^{\pm i \omega_n h}$ will filter out signals at the frequency $\omega_n$. In this case with $\omega_s = 2\omega_0$ we would want the zero at the Nyquist frequency, since for this case $\omega_N = \omega_0$. In order to not attenuate too much of the signal content near $\omega_0$, we combine the zero with a resonanse near the frequency, meaning two poles close to the unit circle at the frequency. How close the poles are is determined with a parameter $r < 1$. This gives the filter
\begin{equation}
H(z) = \frac{ z^2 -2\cos \omega_0 h z + 1}{z^2 - 2r\cos \omega_0 hz + r^2}
\end{equation}
With $r=0.9$, and $\omega_0 h = \pi$, this gives the filter
\begin{equation}
H(z) = \frac{(z+1)^2}{z^2 + 1.8z + 0.81}
\end{equation}

```python
# So, apply a digital notch filter at w0
import scipy.signal as ss

r = 0.9
bf1 = [1, -2*np.cos(w0*h1), 1]
af1 = [1, -2*np.cos(w0*h1), r**2]
bf2 = [1, 2, 1]
af2 = [1, 2*r, r**2]

yf1 = ss.lfilter(bf1, af1, ym1)
yf2 = ss.lfilter(bf2, af2, ym2)

# Fourier transform
Yf1f = np.fft.fft(yf1)*h1
Yf1fpos = Yf1f[:int(N/2)]
Yf1fpos[-1] = 0.5*Yf1f[int(N/2)] # Divide the energy at wN equally among the positive and negative part
Yf1fneg = Yf1f[int(N/2):]
Yf1fneg[0] /= 2.0

Yf2f = np.fft.fft(yf2)*h2
Yf2fpos = Yf2f[:int(N/2)]
Yf2fpos[-1] = 0.5*Yf2f[int(N/2)] # Divide the energy at wN equally among the positive and negative part
Yf2fneg = Yf2f[int(N/2):]
Yf2fneg[0] /= 2.0

wpos2 = np.linspace(0, ws2/2, N/2)
wpos1 = np.linspace(0, ws1/2, N/2)

# Plot the sampled signal and its spectrum
plt.figure(figsize=(16,4))
plt.plot(t, ym)
plt.plot(t, y)
plt.stem(ts2,ym2, linefmt='r--', markerfmt='ro', basefmt = 'r-')
plt.stem(ts2,yf2, linefmt='m--', markerfmt='mo', basefmt = 'm-')
plt.stem(ts1[::3],yf1[::3], linefmt='y--', markerfmt='yo', basefmt = 'y-')
plt.xlabel(r'$t$ [s]')
plt.xlim(-0.1,4)
plt.legend((r'$y(t)+n(t)$', r'$y(t)$', r'Sampled at $2\omega_0$', r'Sampled at $2\omega_0$ and filtered',
            r'Sampled at $6\omega_0$, filtered and resampled'))
plt.title('Time series')

plt.figure(figsize=(16,4))
plt.plot(np.hstack((-wpos2[::-1]-wpos2[1], wpos2)), np.hstack((Yf2fneg, Yf2fpos)))
plt.plot(np.hstack((-wpos1[::-1]-wpos1[1], wpos1)), np.hstack((Yf1fneg, Yf1fpos)))
plt.xlabel(r'$\omega$ [rad/s]')
plt.xlim((-3*w0, 3*w0))
plt.xticks((-2*w0, -w0, w0, 2*w0))
lbls=plt.gca().set_xticklabels([r'$-2\omega_0$', r'$-\omega_0$',  r'$\omega_0$', r'$2\omega_0$'])
plt.legend((r'Sampled at $2\omega_0$', r'Sampled at $6\omega$'))
plt.title('Spectrum (real part) of filtered signals')

```

```python

```


# Joint Probability Distribution

### Introduction

Before starting with *Joint probabilty distribution* we can have a look at Probability Density Function.

**Probability Density Function of a Discrete Random Variable**
Let X be a discrete rv taking distinct values x1, x2, ... , xn, .... Then the function

 $$ f (x) = P(X = x_{i})   \hspace{1cm}          for \hspace{0.1cm} i = 1, 2, ... , n, ...$$

 $$       = 0              \hspace{1cm}          for \hspace{0.1cm} x = x_{i}  $$

is called the **discrete probability density function** (PDF) of X, where P(X = $x_{i}$) means the probability that the discrete rv X takes the value of $x_{i}$ .

**Probability Density Function of a Continuous Random Variable**
Let X be a continuous rv. Then f(x) is said to be the PDF of X if the following conditions
are satisfied:

$$f(x) ≥ 0 $$

$$ \int_{−\infty}^{\infty} f(x) = 1  $$

$$ \int_a^b f(x) = P(a ≤ x ≤ b) $$

where f (x) dx is known as the probability element (the probability associated with a small interval of a continuous variable) and where P(a ≤ x ≤ b) means the probability that X lies in the interval a to b.
For a continuous rv, in contrast with a discrete rv, the probability that X takes a specific
value is zero;$x^{3}$ probability for such a variable is measurable only over a given range or interval, such as (a, b).

###  Joint Probability Distribution

The joint probability distribution can be expressed either in terms of a **joint cumulative distribution function** or in terms of a **joint probability density function** (in the case of continuous variables) or **joint probability mass function** (in the case of discrete variables).

These in turn can be used to find two other types of distributions: the **marginal distribution** giving the probabilities for any one of the variables with no reference to any specific ranges of values for the other variables, and the **conditional probability distribution** giving the probabilities for any subset of the variables conditional on particular values of the remaining variables.

**Joint Probability Density Function**

Let X and Y be two discrete random variables. Then the function

$$f (x, y) = P(X = x \hspace{0.1cm}and\hspace{0.1cm} Y = y)$$

$$ \hspace{3.5cm} = 0 \hspace{1cm}when \hspace{0.1cm}X \neq x  \hspace{0.2cm}and \hspace{0.2cm} Y \neq y$$

is known as the discrete joint probability density function and gives the (joint) probability that X takes the value of x and Y takes the value of y.

### Marginal Probability Distribution

In probability theory and statistics, the marginal distribution of a subset of a collection of [random variables](https://en.wikipedia.org/wiki/Random_variable) is the [probability distribution](https://en.wikipedia.org/wiki/Probability_distribution) of the variables contained in the subset. It gives the probabilities of various values of the variables in the subset without reference to the values of the other variables.

The **marginal probability** is the probability of a single event occurring, independent of other events .

**Marginal Probability Density Function**

In relation to *f (x, y), f (x) and f (y)* are called individual, or marginal, probability density functions. These marginal PDFs are derived as follows:

$$f (x) = \sum_{y}f (x, y)  \hspace{1cm} marginal\hspace{0.1cm} PDF\hspace{0.2cm} of\hspace{0.2cm} X$$

$$f (y) = \sum_{x}f (x, y)  \hspace{1cm} marginal\hspace{0.1cm} PDF \hspace{0.2cm}of\hspace{0.2cm} Y$$

where, for example $\sum_{y}$,  means the sum over all values of Y and $\sum_{x}$ means the sum over all values of X.

### Conditional Probability Distribution

A conditional probability, is the probability that an event occurs given that another specific event has already occurred. This means that the calculation for one variable is dependent on another variable.

In probability theory and statistics, given two jointly distributed random variables X and Y, the conditional probability distribution of Y given X is the probability distribution of Y when X is known to be a particular value; in some cases the conditional probabilities may be expressed as functions containing the unspecified value x of X as a parameter. When both X and Y are categorical variables, a conditional probability table is typically used to represent the conditional probability.

**Conditional PDF**

The function
$$f (x | y) = P(X = x | Y = y)$$

is known as the conditional PDF of X; it gives the probability that X takes on the value of x given that Y has assumed the value y. Similarly,
$$f (y | x) = P(Y = y | X = x)$$

which gives the conditional PDF of Y.
The conditional PDFs may be obtained as follows:

$$f(x | y) = \frac{f (x, y)}{f (y)} \hspace{1cm}conditional PDF\ of\ X$$
$$f (y | x) = \frac{f (x, y)}{f (x)} \hspace{1cm} conditional PDF\ of\ Y $$
As the preceding expressions show, the conditional PDF of one variable can be expressed
as the ratio of the joint PDF to the marginal PDF of another (conditioning) variable.

Now we will be looking at some problem examples on Joint Probability, Marginal Probability and Conditional Probability . So, before moving on let us take a simple example.

**Example 1.** A survey was taken to find the favourite food people . The data is as follows :

```python
import pandas as pd

df=pd.DataFrame({
    'Male' : [80,100,50,230],
    'Female': [120,25,125,270],
    'Total':[200,125,175,500]
},
index= ["Indian","Italian","Other",'Total'],)
df
```

      Male
      Female
      Total

      Indian
      80
      120
      200

      Italian
      100
      25
      125

      Other
      50
      125
      175

      Total
      230
      270
      500

Now we find **Joint Probability Distribution** of the above data. This can be done by dividing the overall data with the total i.e. **500**.

```python
# Joint Probability
ndf=pd.DataFrame()
ndf=df.div(500)
ndf
```

      Male
      Female
      Total

      Indian
      0.16
      0.24
      0.40

      Italian
      0.20
      0.05
      0.25

      Other
      0.10
      0.25
      0.35

      Total
      0.46
      0.54
      1.00

This is the Joint Probability Distribution of the given data.

|Male||Female||Total|
|----||------||-----|
|**0.16**||**0.24**||0.4|
|**0.20**||**0.05**||0.25|
|**0.10**||**0.25**||0.35|
|0.46||0.54||1|

Now **Marginal probability** distribution of the given data is containted by its two columns is :

```python
# Marginal Probability
m=pd.DataFrame()
m=ndf.loc["Total",['Male','Female']]
m
```

    Male      0.46
    Female    0.54
    Name: Total, dtype: float64

```python
# Marginal Probability
ndf.loc[['Indian','Italian','Other'],"Total"]
```

    Indian     0.40
    Italian    0.25
    Other      0.35
    Name: Total, dtype: float64

|Male||Female||Total|
|----||------||-----|
|0.16||0.24||0.4|
|0.20||0.05||0.25|
|0.10||0.25||0.35|
|**0.46**||**0.54**||1|

|Male||Female||Total|
|----||------||-----|
|0.16||0.24||**0.4**|
|0.20||0.05||**0.25**|
|0.10||0.25||**0.35**|
|0.46||0.54||1|

Now we find the **Conditional Probability**.
Here, like we need to find Conditional Probability of Indian food for Females.This can be found out by dividing probability of Indian Food for Probability of Females by Total Probability for Females i.e. **0.24/0.54**. In the same way we can find for Italian and Other also.

```python
cond=pd.DataFrame()
cond=ndf.loc[:,'Female']
cond
```

    Indian     0.24
    Italian    0.05
    Other      0.25
    Total      0.54
    Name: Female, dtype: float64

```python
# Conditional Probability
cond.div(cond.loc['Total'])
```

    Indian     0.444444
    Italian    0.092593
    Other      0.462963
    Total      1.000000
    Name: Female, dtype: float64

|Female||Female||Total|
|------||------||-----|
|0.24||**0.444**||0.4|
|0.05||**0.093**||0.25|
|0.25||**0.463**||0.35|
|0.54||1||1|

Similarly, we can find for Males also.

**Example 2.** Two balls are selected at random from a box containing three red, two green and four white. If X and Y are the number of red balls and green balls respectively included among two balls drawn from the box , find

1. Joint Probabilty of X and Y
2. Marginal Probabilty of X and Y
3. Conditional distribution of X given Y=1

As in the above example we were given the table so it was easy for us to find Joint Probability Distribution . In this example we will be making the Joint Probability Distribution.

The Joint Probability Distribution Table looks like this :

|X\Y||0||1||2||Total|
|---||-||-||-||-----|
|0|
|1|
|2|
|3|
|Total|

Since, there are three red balls so we will be taking X=0,1,2,3 and two green balls so we will take Y=0,1,2. Now we take out probabilities . We know that the probability of choosing two balls out of 3 red, 2 green and 4 white balls is given by choosing any two balls out of 9 balls i.e. $9 \choose 2$ .

Now , the block with row and column 0 represents P(0,0) means here we have 0 red and 0 green balls . It means we have only white balls with us so we can choose two balls from 4 white balls by $4\choose2$.

So, P(0,0)= $\frac{4\choose2}{9 \choose 2} \ =0.17$

Now,P(0,1) represents we can choose 0 red balls and 1 green ball. We know that we need to choose two balls in total according to the question so we will choose 1 ball from 2 green balls i.e. $2\choose1$ and 1 ball from 4 white balls i.e. $4\choose1$ and total probability of choosing 2 balls is $9\choose2$. Hence,

P(0,1) = ${2\choose1}\frac{4\choose1}{9\choose2} =0.22$

Similarly, if we look at P(0,2) so we can choose 2 green balls and we need to choose 2 balls in total. This can be done by choosing 2 balls from green itself and we need not take white balls here. This is done by$2\choose2$.

P(0,2)=$\frac{2\choose2}{9\choose2} = 0.03$

Now, we look at P(1,2). This means we can choose 1 red and 2 green balls but according to the question we have to choose only 2 balls so this is not possible. Hence,

P(1,2) = 0

Similarly, we can take out probabilities for all and this will complete our Joint Probability Distribution Table.

```python
# Joint Probability Distribution
df = pd.DataFrame({
             'X\Y' : [0, 1, 2, 3, 'Total'],
                 '0' : [ 1/6, 1/3, 1/12, 0, 21/36],
                 '1' : [2/9, 1/6, 0, 0, 14/36],
                 '2' :  [1/36, 0, 0, 0, 1/36],
           'Total' : [15/36, 18/36, 3/36, 0, 1],
                  }
    ,)
df=df.round(2)

df.set_index('X\Y')
print (df)

```

         X\Y     0     1     2  Total
    0      0  0.17  0.22  0.03   0.42
    1      1  0.33  0.17  0.00   0.50
    2      2  0.08  0.00  0.00   0.08
    3      3  0.00  0.00  0.00   0.00
    4  Total  0.58  0.39  0.03   1.00

|X\Y||0||1||2||Total|
|---||-||-||-||-----|
|0||0.17||0.22||0.03||0.42|
|1||0.33||0.17||0.00||0.50|
|2||0.08||0.00||0.00||0.08|
|3||0.00||0.00||0.00||0.00|
|Total||0.58||0.39||0.03||1.00|

Now, we have Joint Probability Distribution so we can easily find Marginal and Conditional Probability as seen in the previous example. You may find Marginal Probability of X and Y by yourself first and then check your answer here. Please choose an option   :
1. Marginal Probability
   To check Marginal Probability of X type x
   To check Marginal Probability of Y type y

2. Conditional Probability

```python
# Marginal Probability Distribution of X
mrx = pd.DataFrame()
mrx = df.loc[0:3,'Total']
mrx
```

    0    0.42
    1    0.50
    2    0.08
    3    0.00
    Name: Total, dtype: float64

```python
# Marginal Probability Distribution of Y
mry = pd.DataFrame()
mry = df.loc[4,['0','1','2']]
mry
```

    0    0.58
    1    0.39
    2    0.03
    Name: 4, dtype: object

```python
# Conditional Probability given Y=1
cond = pd.DataFrame()
cond =df.loc[0:3,'1'].div(df.loc[4,'1']).round(2)
cond
```

    0    0.56
    1    0.44
    2    0.00
    3    0.00
    Name: 1, dtype: float64

```python
# Marginal Probability Distribution
a = int(input("Enter a number: "))
if a == 1:
    b = input("Enter a x\y: ")
    if b == "x":
        print(mrx)
    elif b == 'y':
        print(mry)
elif a ==2:
    print('Conditional Probability when Y=1')
    print(cond)
else:
    print('Invalid input')
```

    Enter a number: 1
    Enter a x\y: x
    0    0.42
    1    0.50
    2    0.08
    3    0.00
    Name: Total, dtype: float64

# Joint Probability Density Function For Continuous Random Variable

For a **Continuous Random Variable** (X,Y) we define the ***Joint Probability Density Function*** f(x,y) , as
1. $f(x,y)\ge 0$

2. $ \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} f(x,y)dxdy = 1 \rightarrow$ **This states that total volume under the surface given by f(x,y) is 1.0**

 * The Joint pdf f(x,y) is not a probability.
 * For small$ \Delta x,\Delta y(+ve)$ , f(x,y) $\Delta x \Delta y$ is approximately equal to $P[x\leq X \leq x +\Delta x,y\leq Y \leq y +\Delta y] $

**Joint Cumulative Distribution Function (X,Y)**

The Joint Cumulative Distribution Function F(x,y) of two dimensional random variable x,y is defined as
$$ F(x,y)\ = P[X\leq x,\ Y\leq y] $$
$$ \\\\\\\\ = \int_{-\infty}^y\int_{-\infty}^x f(x,y)dxdy $$

It follows from the definition ,that

* F($-\infty,\infty$) = 1.0

* F($-\infty$,y) = F(x,$-\infty$) = 0

**Example** Flows in two adjacent streams are denoted as random vector (X,Y) with joint pdf

$$f(x,y) = c\ if\ 5\leq x\leq 10;\ 4\leq y\leq 9 $$
$$= 0 , \ elsewhere$$

1. Obtain c
2. Obtain P[X $\geq$ Y]

**Ans** To determine **c** we use the definition of joint pdf , $ \int_{-\infty}^\infty\int_{-\infty}^\infty f(x,y)dxdy = 1$

$ \int_{4}^9\int_{5}^{10} cf(x,y)dxdy = 1$

```python
# Integrating the function

import numpy as np
import matplotlib.pyplot as plt
from sympy import sin, cos, symbols, integrate
x, y = symbols('x y')
integrate(1, (x, 4, 9), (y, 5, 10))
from scipy.integrate import nquad

def f(x, y):
    return 1

def bounds_y():
    return [5, 10]

def bounds_x(y):
    return [4, 9]

y, err = nquad(f, [bounds_x, bounds_y])
y
```

    25.0

So, the value of integration is 25. The equation now becomes *25c = 1*

```python
# value of c
c = 1/25
c
```

    0.04

**Ans 2** For P[X $\geq$ Y]
P[X $\geq$ Y] = 1-P[X $\leq$ Y]

```python
# Plotting graph to visualise the area or region for finding probability

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
import numpy as np

a = plt.axhline(y=4,color='green',linestyle='dashed')
b = plt.axhline(y=9,color='red',linestyle='dashed')
c = plt.axvline(x=5,color='black',linestyle='dashed')
d =plt.axvline(x=10,color='brown',linestyle='dashed')

x = np.linspace(5,10,100)
y = x
y1 = 9

plt.title('Visualising the graph for finding Probability')
plt.plot(x,y,linestyle='solid')
plt.fill_between(x,y,y1,where=y1>y, facecolor='gold')
```

The dashed lines represent the limits or boundaries within which the function is positive. The shaded represents the area where x$\leq$y . So, we can focus on the shaded region and find the probability. We see that x varies from 5 to y and y varies from 5 to 9 and these become the limits for x and y.

We had,

P[X $\geq$ Y] = 1-P[X $\leq$ Y]

=1-$ \int_{5}^9\int_{5}^y f(x,y)dxdy $
=1-$\frac{1}{25} \int_{5}^9\int_{5}^y f(x,y)dxdy $

```python
# Integrating the function

import numpy as np
import matplotlib.pyplot as plt
from sympy import sin, cos, symbols, integrate
x, y = symbols('x y')
integrate(1, (x, 5, 9), (y, 5, y))
from scipy.integrate import nquad

def f(x, y):
    return 1

def bounds_y():
    return [5, 9]

def bounds_x(y):
    return [5, y]

y, err = nquad(f, [bounds_x, bounds_y])
y
```

    8.0

So, we have 1-{$\frac{1}{25}y$} after integration where y is the value of integration

```python
result = 1-1/25*y
result = round(result,2)
result
```

    0.68

Hence, P[X $\geq$ Y] = 0.68

So, before moving onto Marginal Density Functions for Continuous Random Variables we can have a little glance on **Monte Carlo Integration** which is used for fast integration of functions.

# Monte Carlo Integration

Monte Carlo methods are numerical techniques which rely on random sampling to approximate their results . Monte Carlo Integration applies this process to numerical estimation of integrals.Monte Carlo integration is a technique for numerical integration using random numbers. It is a particular Monte Carlo method that numerically computes a definite integral.Monte Carlo randomly chooses points at which the integrand is evaluated. This method is particularly useful for higher-dimensional integrals.

So, in simple words it can be said that Monte Carlo Integration is used for definite integration .

# How is Monte Carlo Integration Done ?

Here we will look at the method of doing Monte Carlo .

In order to integrate a function over a complicated domain D, Monte Carlo integration picks random points over some simple domain $D^{'}$ which is a superset of D, checks whether each point is within D, and estimates the area of D (volume, n-dimensional content, etc.) as the area of $D^{'}$ multiplied by the fraction of points falling within D.

Now, we pick up N randomly distributed points $x_{1},x_{2},x_{3},......x_{N},$ in multidimensional volume V to determine the integral of function f in this volume gives a result

$\int_ f dv \approx V<f>\pm V \sqrt \frac{<f^{2}>-<f>^{2}}{N}$

where,

$<f> \equiv \frac{1}{N}\sum_{i=1}^{N}f(x_{i})$

$<f^{2}>\equiv \frac{1}{N}\sum_{i=1}^{N}f^{2}(x_{i})$

Now , we will look at the implementation of Monte Carlo . We take $\sin x$ as the function for integration from 0 to $\pi$ .

```python
# Monte Carlo Integration

from scipy import random
import numpy as np

#Limits of integration
a = 0
b = np.pi

# No of random variables
N=1000

# Setting up random variable
xrand = random.uniform(a,b,N)

# Implementing the given function of integration
def func(x):
    return np.sin(x)
integral =0.0

for i in range(N):
    integral += func(xrand[i])

# Final answer of integration
answer = (b-a)/float(N)*integral
print('The integral from o to pi of sin(x):' ,answer)
```

    The integral from o to pi of sin(x): 2.0178989749080714

```python
# Visualising the answer using Monte Carlo Integration

from scipy import random
import numpy as np
import matplotlib.pyplot as plt

#Limits of integration
a = 0
b = np.pi

# No of random variables
N=1000

areas =[]
for i in range(N):

# Setting up random variable
    xrand = random.uniform(a,b,N)

# Implementing the given function of integration
    def func(x):
        return np.sin(x)
    integral =0.0

    for i in range(N):
        integral += func(xrand[i])

# Final answer of integration
    answer = (b-a)/float(N)*integral
    areas.append(answer)

plt.title('Distribution of Areas Calculated')
plt.hist(areas,bins=30, ec = 'black')
plt.xlabel('Areas')
```

So, we can see that we are getting approximately equivalent value using **Monte Carlo Integration**.

# Marginal Density Functions for Continuous Random Variable

* In continuous case, we proceed as follows:
- Let f(x,y) denote the joint pdf of (X,Y)
- We define g(x) and h(y) as marginal probability density functions of X and Y respectively as
$$g(x) = \int_{-\infty}^\infty f(x,y)dy , h(x) =\int_{-\infty}^\infty f(x,y)dx$$
These marginal pdfs which are infact derived from joint pdf f(x,y) correspond to original pdf of the one dimensional r.v.s X and Y

This may be seen from
P[c$\leq X\leq d$] = P[c$\leq X\leq d,-\infty\leq Y\leq \infty$]
 =$\int_{c}^d\underbrace{\int_{-\infty}^\infty f(x,y)dy}dx $
 =$\int_{c}^d g(x) dx$
 From the definitions of pdf it is thus seen that this is infact original pdf of r.v. X
 Thus ,$$g(x) = \int_{-\infty}^\infty f(x,y)dy\ and \ F(x) =\int_{-\infty}^\infty g(x,y)dx$$
 Similarly, for r.v. of Y
 That is, starting with joint pdf f(x,y), we are not able to get pdfs of X and Y respectively.

 Lets take an example.
**Example** Consider a joint pdf :
f(x,y) =$\frac{1}{25}\\\\\\\\$  {$5\leq X\leq 10 $; $4 \leq Y\leq 9$}
= 0, elsewhere

1. Obtain the marginal density g(x), h(y)
2. Obtain CDF G(x), H(y)
3. P[$x\geq 7$]
4. P[$5\leq Y \leq 8$]

**Ans** To obtain g(x),
g(x) = $\int_{-\infty}^\infty f(x,y)dy$ , $5\leq X \leq 10$
$\hspace {0.5cm}$ =$\int_{4}^9 \frac{1}{25}dy$

```python
# Using Monte Carlo for integration
from scipy import random
import numpy as np

#Limits of integration
a = 4
b = 9

# No of random variables
N=1000

# Setting up random variable
xrand = random.uniform(a,b,N)

# Implementing the given function of integration
def func(x):
    return 1/25
integral =0.0

for i in range(N):
    integral += func(xrand[i])

# Final answer of integration
answer = (b-a)/float(N)*integral
answer=round(answer,2)
answer
```

    0.2

Hence, g(x)= answer
g(x) = 5

**ii)** To obtain h(y)
h(y) = $\int_{-\infty}^{\infty} f(x,y)dx \hspace{0.5cm} 4 \leq Y\leq 9$
= $\int_{5}^{10} \frac{1}{25}dx$

```python
# Using Monte Carlo for integration
from scipy import random
import numpy as np

#Limits of integration
a = 5
b = 10

# No of random variables
N=1000

# Setting up random variable
xrand = random.uniform(a,b,N)

# Implementing the given function of integration
def func(x):
    return 1/25
integral =0.0

for i in range(N):
    integral += func(xrand[i])

# Final answer of integration
answer = (b-a)/float(N)*integral
answer=round(answer,2)
answer
```

    0.2

Similarly , h(y) = answer
h(y) = 0.2  $ \\\\$ $4 \leq Y\leq 9$

**2.i.)** To obtain G(x)
G(x) = $\int_{-\infty}^x g(x)dx$ $ \hspace{0.5cm} 5\leq x \leq 10$
= $\int_{5}^x \frac{1}{5}dx$
G(x) = $\frac{x-5}{5}  \hspace{0.5cm} 5\leq x \leq 10$

**ii.)** To obtain H(y) ,
H(y) = $\int_{-\infty}^y h(y)dy$ $ \hspace{0.5cm} 4\leq x \leq 9$
= $\int_{4}^y \frac{1}{5}dy$
H(y) = $\frac{y-4}{5}  \hspace{0.5cm} 4\leq x \leq 9$

**3.** P[$x\geq 7$] = 1- P[$x\leq 7$]
$\hspace{2cm} = 1 - G(7) $
$\hspace{2cm} = 1 -\frac{7-5}{5} = \frac{3}{5}$

**4.** P[$5\leq Y \leq 8$]  = H(8) - H(5)
$\hspace{2.7cm} = \frac{4}{5} - \frac {1}{5} = \frac{1}{5}$

# Conditional Distribution for Continuous variables

Definition : (X,Y) is a continuous two dimensional r.v. with joint pdf of f(x,y)
* Let g(x) and h(x) are marginal pdfs of X and Y respectively.
* The conditional pdf of X given Y=y is defined as
$$ g(x/y) = \frac{f(x,y)}{h(y)} \hspace{1cm}h(y)\geq 0$$
* The conditional pdf of Y given X=x is defined as
$$ g(y/x) = \frac{f(x,y)}{g(x)} \hspace{1cm}g(x)\geq 0$$
* The conditional pdf of X given Y$\in $ R and conditional pdf of Y given X$\in$R is defined as
$$ g(x/y\in R) =\frac{\int_{R}f(x,y)dy}{\int_{R}h(y)dy} \hspace{0.5cm},h(y/x\in R)= \frac{\int_{R}f(x,y)dx}{\int_{R}g(x)dx}$$

The conditional pdfs g(x\y) and h(y\x) satisfy all the conditions for a pdf.
For a given y, g(x/y)>0 as both f(x,y) and h(y) are positive .
$$ \int_{-\infty}^\infty g(x/y) =\int_{-\infty}^{\infty} \frac{f(x,y)}{h(y)}dx $$
$$= \frac{1}{h(y)}\int_{-\infty}^{\infty} f(x,y)dx$$
$$=\frac{h(y)}{h(y)}=1.0$$

**Cumulative Conditional Distributions**
$$G(x/y) = \int_{-\infty}^xg(x/y)dx, H(y/x)= \int_{-\infty}^y h(y/x)dy$$

Let us take a small example to understand this .

**Example** Consider the joint pdf
f(x,y) = $\frac{x(1+3y^2)}{4} \hspace{1cm} 0\leq x\leq 2 \hspace{1cm}  0\leq y\leq 1$
$\hspace{5.5cm}= 0,elsewhere$

* Obtain h(y/x)

**Ans** To obtain h(y/x) , first we obtain g(x)
g(x) = $\int_{-\infty}^\infty f(x,y)dy  \hspace{0.5cm} 0\leq x \leq 2  $

 = $\int_{0}^1\frac{x(1+3y^2)}{4}dy$

 = $\frac {(x+1)}{4}$

 h(y/x) =$\frac{f(x,y)}{g(x)}$

 = $\frac{x(1+3y^2)}{x+1}$

# Bivariate Normal Distribution

Suppose that $Z_{1} and Z_{2}$ are independent random variables, each of which has a standard normal distribution. Then the joint p.d.f. g($z_{1},z_{2}$) of $Z_{1} and Z_{2}$ is specified for all values of $z_{1} and z_{2}$ by the equation

g(z1,z2) = $\frac{1}{2\pi}exp[\frac{-1}{2}(z_{1}^2 + z_{2}^2)]$

where

For constants $\mu_{1}$,$\mu_{2},\sigma_{1},\sigma_{2}$ and $\rho$ such that $-\infty$ < $\mu_{i}$ < $\infty$ (i = 1, 2), $\sigma_{i} > 0$ (i = 1, 2) and -1 < $\rho$ < 1, we shall now define two new random variables $X_{1} and X_{2}$ as follows:

$X_{1}$ = $\sigma_{1}$ $Z_{1}$ + $\mu_{1}$

$X_{2} = \sigma_{2}[\rho Z_{1} +\sqrt{1-\rho^2} Z_{2}] + \mu_{2}$

The “regular” normal distribution has one random variable; A bivariate normal distribution is made up of two independent random variables. The two variables in a bivariate normal are both are normally distributed, and they have a normal distribution when both are added together. Visually, the bivariate normal distribution is a three-dimensional bell curve.

The bivariate distribution can be described in many different ways and as such, there isn’t a unified agreement for a succinct definition. Some of the more common ways to characterize it include:

* Random variables X & Y are bivariate normal if aX + bY has a normal distribution for all a,b$\in$R.

* X and Y are jointly normal if they can be expressed as X = aU + bV, and Y = cU + dV (Bertsekas & Tsitsiklis, 2002)

* If a and b are non-zero constants, aX + bY has a normal distribution (Johnson & Kotz, 1972).

* If X - aY and Y are independent and if Y - bx and X are independent for all a,b (such that ab $\neq$ 0 or 1), then (X,Y) has a   normal distribution (Rao, 1975).

The bivariate normal distribution arises directly and naturally in many practical problems. For example, for many populations the joint distribution of two physical characteristics such as the heights and the weights of the individuals in the population will be approximately a bivariate normal distribution. For other populations, the joint distribution of the scores of the individuals in the population on two related tests will be approximately a bivariate normal distribution.

# PDF of the Bivariate Normal Distribution

The bivariate normal distribution can be defined as the probability density function (PDF) of two variables X and Y that are linear functions of the same independent normal random variables (adapted from Wolfram):

P($x_{1},x_{2}) = \frac{1}{2\pi\sigma_{1}\sigma_{2}\sqrt{1-\rho^2}}exp[-\frac{z}{2(1-\rho^2)}]$

Where:

* μ = mean
* &sigma = standard deviation

$$ z\equiv \frac{(x_{1}-\mu_{1})^2}{\sigma_{1}^2}-\frac{2\rho(x_{1}-\mu_{1})(x_{2}-\mu_{2})}{\sigma_{1}\sigma_{2}}+\frac{(x_{1}-\mu_{2})^2}{\sigma_{2}^2}$$

* $$ρ\equiv cor(x_{1}x_{2}) =\frac{V_{12}}{\sigma_{1}\sigma_{2}}$$
where
* ρ correlation of x1 and x2.
* $V_{x12} = covariance\ of x1\ and\ x2$

If $\rho$ = 2, this is equal to the bivariate normal distribution.

```python
# Displaying Bell curve (shows Bivariate normal distribution)

import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import multivariate_normal
from mpl_toolkits.mplot3d import Axes3D

#Parameters to set
mu_x = 0
variance_x = 3

mu_y = 0
variance_y = 15

#Create grid and multivariate normal
x = np.linspace(-10,10,500)
y = np.linspace(-10,10,500)
X,Y = np.meshgrid(x,y)

pos = np.empty(X.shape +(2,))
pos[:, :, 0] = X; pos[:, :,1] = Y

rv = multivariate_normal([mu_x, mu_y], [[variance_x, 0], [0, variance_y]])

#Make a 3D plot
fig = plt.figure()
ax = fig.gca(projection='3d')
ax.plot_surface(X, Y, rv.pdf(pos),cmap='viridis',linewidth=0)
ax.set_xlabel = 'X-axis'
ax.set_ylabel = 'Y-axis'
ax.set_zlabel = 'Z-axis'
ax.set_title(label='Bell curve')
plt.show()
```

```python

```


The aim of this project is to study the following eigenvalue problem:

Find $\omega \in \mathbb{R}, ~ \omega \neq 0$, and $\mathbf{u} \in \mathbf{H}_0(\mbox{curl}, \Omega), ~ \mathbf{u} \neq \mathbf{0}$, such that

$
\begin{equation}
  \left( \mbox{curl} ~ \mathbf{u},  \mbox{curl}~ \mathbf{v} \right) = \omega^2 \left( \mathbf{u}, \mathbf{v} \right), \quad \forall ~ \mathbf{v} \in \mathbf{H}_0(\mbox{curl}, \Omega)
  \label{eq:weak-form}
\end{equation}
$

We shall restrict our study to the 2D case and will be interested in two geometries:

1. First we will consider the square $(0, 1)^2$,
2. Then we study the eigenvalues problem on an L-shape domain, using a $C^1$ description, as shown in the next plot.

## Questions

### Part I

The computational domain considered here is the square $(0, 1)^2$.

1. Show that the eigenvalues are given by $\omega^2 = \pi^2 ( m^2 + n^2 )$ for $m,n \in \mathbb{N}$ and their corresponding eigenfunctions are

$
\mathbf{u} =
\begin{pmatrix}
-n \cos (m x) \sin(n y) \\
 m \sin (m x) \cos(n y)
\end{pmatrix}
$

2. Write the matrix form associated to the eigenvalue problem.

3. The file **maxwell_eigen_exam_1.py** is an implementation of the eigenvalue problem on the unit square. There are two functions to be implemented:

    * **apply_bc**: the provided implementation uses all the B-Splines in the FEM space. We need to filter the assembled matrices in order to apply the associated boundary condition for $\mathbf{H}_0(\mbox{curl}, \Omega)$
    * **exact_eigen**: since the exact eigenvalues are infinite, we need to extract a set to compare our results with. This set is related the dimension of the discrete space.

4. Solve numericaly the eigenvalue problem and show the results for different degrees.

5. Show the number of zeros in each of the previous simulations and compare it to the dimension of the kernel of the curl operator.

6. (optional) Prove the previous result for a simply connected domain.

Note: We will restrict our study to small grids (<= 32 x 32), otherwise the computations will take time.

### Part II

We are now interested in introduction the mapping, and we will modify our code to solve the eigenvalue problem first on the square $(0, \pi)^2$.
We will be using the file **maxwell_eigen_exam_2.py** in addition to the functions **apply_bc** and **exact_eigen**.

1. Add the appropriate statements for the preseving transformations. Note that in the function **assemble_matrices**, the computations are given on the logical domain.

2. In order to validate the computations, use a square of length $\pi$ by calling

```python
mapping = create_mapping(V, kind='square', length=np.pi)
```

What are the exact eigenvalues in this case?

3. Add the L-Shape surface to **create_mapping** and compute the numerical eigenvalues for different grids and degrees.
For benchmarks, check the following [reference list](https://perso.univ-rennes1.fr/monique.dauge/benchmax.html)

### Part III (optional)

1. Change the implementations of the mappings so that we only have a $C^0$ continuity at a given knot. What do you observe?
2. Show the evolution of the relative error and give the convergence order for the mode $(3,3)$ using different mappings and regularities.

```python

```


## Quantum circuit for an exponential of pauli strings

For SUSY QM, the Hamiltonian, $H$ can be qubitized, which results in the Hamiltonian being written as a sum of terms, with each term containing a produce of pauli matrices acting on the qubits.  Given some initial state, we can apply the time evolution operator,
\begin{equation}
    e^{iHt}.
\end{equation}
To realize this on a quantum computer, we use the Suzuki-Trotter formula
\begin{equation}
    e^{i\sum_j H_j t}=\prod_j e^{i H_j \delta t} + \mathcal{O}()
\end{equation}
Since qubitizing the Hamiltonian results in an expression for $H$ in terms of pauli operators, we need to be able to write down the quantum circuit for an exponential of pauli matrices.  This is accomplished with the so-called "ladder" circuit, which we now detail.

First we go through some example cases showing that the exponential of the Hamiltonian is a quantum circuit.

```python
import numpy as np #we will use numpy's kron function for tensor products, and its matmul for matrix multiplication.

#definition of the identity, pauli X and Z matrices, and the two-qubit CNOT matrix.
ID=np.array([[1,0],[0,1]])
X=np.array([[0,1],[1,0]])
Z=np.array([[1,0],[0,-1]])
CNOT=np.array([[1,0,0,0],[0,1,0,0],[0,0,0,1],[0,0,1,0]])
```

```python
#A quick check that we are doing kronecker products correctly
#The CNOT gate is the identity, if the control qubit is zero, and a NOT(X) gate otherwise
assert(CNOT.all() == (np.kron([[1,0],[0,0]],ID) + np.kron([[0,0],[0,1]],X)).all())
```

```python
#To avoid using an algebraic library, like sci-py, I pick specific values for cos(t) and i sin(t)...
#cos(t) = A
#i sin(t) = B
A=0.2
B=0.3
RZ=A*ID + B*Z #a rotation around the z-axis with given values for t (that don't make sense)
```

Now we can check that the circuit for
\begin{equation}
    e^{-i(Z \otimes Z)t} = \text{CNOT}\times(\mathcal{1}\otimes R_z)\times \text{CNOT}
\end{equation}

```python
LHS=A*np.kron(ID,ID) + B*np.kron(Z,Z)
RHS=np.matmul(CNOT,np.matmul(np.kron(ID,RZ),CNOT))

#print(LHS)
#print(RHS)
assert(LHS.all()==RHS.all())
```

```python
#We now repeat this for a pauli Z applied to 3 qubits.
LHS = A*np.kron(ID,np.kron(ID,ID)) + B*np.kron(Z,np.kron(Z,Z))

CNOT1=np.kron(CNOT,ID)
CNOT2=np.kron(ID,CNOT)
RZ3=np.kron(ID,np.kron(ID,RZ))

RHS=np.matmul(CNOT1,np.matmul(CNOT2,np.matmul(RZ3,np.matmul(CNOT2,CNOT1))))

assert(LHS.all()==RHS.all())
```

QISKIT already contains a method for implementing Trotterization to a exponential written as a sum of pauli matrices.

```python
from qiskit.aqua.operators import I,X,Y,Z, PauliTrotterEvolution
from qiskit import QuantumCircuit, transpile
```

```python
operator = ((Z^Z).exp_i())
trotter_op = PauliTrotterEvolution(trotter_mode='suzuki').convert(operator)
print(operator)
print(trotter_op)
qc = QuantumCircuit(2,2)
qc.append(trotter_op, [0,1])
transpile(qc, basis_gates = ['cx', 'u1', 'u2', 'u3', 'H', 'X', 'Y', 'Z', 'id']).draw('mpl')
```

```python
transpile(qc, basis_gates = ['cx', 'u1', 'u2', 'u3', 'H', 'X', 'Y', 'Z', 'id'],optimization_level=3).draw('mpl')
```

```python
operator = ((X^Z).exp_i())
trotter_op = PauliTrotterEvolution(trotter_mode='suzuki').convert(operator)
print(operator)
print(trotter_op)
qc = QuantumCircuit(2,2)
qc.append(trotter_op, [0,1])
transpile(qc, basis_gates = ['cx', 'u1', 'u2', 'u3', 'H', 'X', 'Y', 'Z', 'id']).draw('mpl')
```

```python
operator = ((X^Y^Z).exp_i())
trotter_op = PauliTrotterEvolution(trotter_mode='suzuki').convert(operator)
print(operator)
print(trotter_op)
qc = QuantumCircuit(3,3)
qc.append(trotter_op, [0,1,2])
transpile(qc, basis_gates = ['cx', 'u1', 'u2', 'u3', 'H', 'X', 'Y', 'Z', 'id']).draw('mpl')
```