```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

[Python for Probability, Statistics, and Machine Learning](https://www.springer.com/fr/book/9783319307152)

This chapter takes a geometric view of probability theory and relates it to
familiar concepts in linear algebra and geometry.  This approach connects your
natural geometric intuition to the key abstractions in probability that can
help guide your reasoning.  This is particularly important in probability
because it is easy to be misled. We need a bit of rigor and some
intuition to guide us.

In grade school, you were introduced to the natural numbers (i.e., `1,2,3,..`)
and you learned how to manipulate them by operations like addition,
subtraction, and multiplication. Later, you were introduced to positive and
negative numbers and were again taught how to manipulate them. Ultimately, you
were introduced to the calculus of the real line, and learned how to
differentiate, take limits, and so on. This progression provided more
abstractions, but also widened the field of problems you could successfully
tackle. The same is true of probability. One way to think about probability is
as a  new number concept that allows you to tackle problems that have a special
kind of *uncertainty* built into them. Thus, the key idea is that there is some
number, say $x$, with a traveling companion, say, $f(x)$, and this companion
represents the uncertainties about the value of $x$ as if looking at the number
$x$ through a frosted window. The degree of opacity of the window is
represented by $f(x)$.  If we want to manipulate $x$, then we have to figure
out what to do with $f(x)$. For example if we want $y= 2 x $, then we have to
understand how $f(x)$ generates $f(y)$.

Where is the *random* part?   To conceptualize this, we need still another
analogy: think about a beehive with the swarm around it representing $f(x)$,
and the hive itself, which you can barely see through the swarm, as $x$.    The
random piece is you don't know *which* bee in particular is going to sting you!
Once this happens the uncertainty evaporates.
Up until that happens, all we have is a concept of a swarm (i.e., density of
bees) which represents a *potentiality* of which bee will ultimately sting.
In summary, one way to think about probability is as a way of carrying through
mathematical reasoning (e.g., adding, subtracting, taking
limits) with a notion of potentiality that is so-transformed by these
operations.

## Understanding Probability Density

In order to understand the heart of modern probability, which is built
on the Lesbesgue theory of integration, we need to extend the concept
of integration from basic calculus. To begin, let us consider the
following piecewise function

$$
f(x)  =  \begin{cases}
                1 & \mbox{if }  0 < x \leq 1 \\\
                2 & \mbox{if }  1 < x \leq 2 \\\
                0 & \mbox{otherwise }
            \end{cases}
$$

 as shown in [Figure](#fig:intro_001). In calculus, you learned
Riemann integration, which you can apply here  as

$$
\int_0^2 f(x) dx = 1 + 2 = 3
$$

 which has the usual interpretation as the area of the two rectangles
that make up $f(x)$.  So far, so good.

With Lesbesgue integration, the idea is very similar except that we
focus on the y-axis instead of moving along the x-axis. The question
is given $f(x) = 1$, what is the set of $x$ values for which this is
true? For our example, this is true whenever $x\in (0,1]$. So now we
have a correspondence between the values of the function (namely, `1`
and `2`) and the sets of $x$ values for which this is true, namely,
$\lbrace (0,1] \rbrace$ and $\lbrace (1,2] \rbrace$, respectively. To
compute the integral, we simply take the function values (i.e., `1,2`)
and some way of measuring the size of the corresponding interval
(i.e.,  $\mu$) as in the following:

$$
\int_0^2 f d\mu = 1 \mu(\lbrace (0,1] \rbrace) + 2 \mu(\lbrace (1,2] \rbrace)
$$

We have suppressed some of the notation above to emphasize generality. Note
that we obtain the same value of the integral as in the Riemann case when
$\mu((0,1]) = \mu((1,2]) = 1$. By introducing the $\mu$ function as  a way of
measuring the intervals above, we have introduced another degree of freedom in
our integration. This accommodates many weird functions that are not tractable
using the usual Riemann theory, but we refer you to a proper introduction to
Lesbesgue integration for further study [[jones2001lebesgue]](#jones2001lebesgue).  Nonetheless,
the key step in the above discussion is the introduction of the $\mu$ function,
which we will encounter again as the so-called probability density function.

## Random Variables

Most introductions to probability jump straight into *random variables* and
then explain how to compute complicated integrals. The problem with this
approach is that it skips over some of the important subtleties that we will now
consider. Unfortunately, the term *random variable* is not very descriptive. A
better term is *measurable function*.  To understand why this is a better term,
we have to dive into the formal constructions of probability by way of a simple
example.

Consider tossing a fair six-sided die. There are only six outcomes possible,

$$
\Omega=\lbrace 1,2,3,4,5,6 \rbrace
$$

As we know, if the die is fair, then the probability of each outcome is $1/6$.
To say this formally, the measure  of each set (i.e., $\lbrace 1 \rbrace,\lbrace
2 \rbrace,\ldots,\lbrace 6 \rbrace$) is $\mu(\lbrace 1 \rbrace ) =\mu(\lbrace 2
\rbrace ) \ldots = \mu(\lbrace 6 \rbrace ) = 1/6$. In this case, the $\mu$
function we discussed earlier is the usual *probability* mass function, denoted by
$\mathbb{P}$. The measurable function  maps a set into a
number on the real line. For example, $ \lbrace 1 \rbrace \mapsto 1 $ is
one such  uninteresting function.

Now, here's where things get interesting. Suppose you were asked to construct a
fair coin from the fair die. In other words, we want to throw the die and then
record the outcomes as if we had just tossed a fair coin. How could we do this?
One way would be to define a measurable function that says if the die comes up
`3` or less, then we declare *heads* and otherwise declare *tails*. This has
some strong intuition behind it, but let's articulate it in terms of formal
theory.  This strategy creates two different non-overlapping sets $\lbrace
1,2,3 \rbrace$ and $\lbrace 4,5,6 \rbrace$. Each set has the same probability
*measure*,

$$
\begin{eqnarray*}
\mathbb{P}(\lbrace 1,2,3 \rbrace)  & = &  1/2  \\\
\mathbb{P}(\lbrace 4,5,6 \rbrace) & = & 1/2
\end{eqnarray*}
$$

 And the problem is solved. Everytime the die comes up
$\lbrace 1,2,3 \rbrace$, we record heads and record tails otherwise.

Is this the only way to construct a fair coin experiment from a
fair die? Alternatively, we can define the sets as $\lbrace 1 \rbrace$,
$\lbrace 2 \rbrace$, $\lbrace 3,4,5,6 \rbrace$. If we define the corresponding
measure for each set as the following

$$
\begin{eqnarray*}
\mathbb{P}(\lbrace 1 \rbrace)  & = &  1/2  \\\
\mathbb{P}(\lbrace 2 \rbrace) & = & 1/2 \\\
\mathbb{P}(\lbrace 3,4,5,6 \rbrace) & = & 0
\end{eqnarray*}
$$

  then, we have another solution to the fair coin problem. To
implement this, all we do is ignore every time the die shows `3,4,5,6` and
throw again. This is wasteful, but it solves the problem.  Nonetheless,
we hope you can see how the interlocking pieces of the theory provide a
framework for carrying the notion of uncertainty/potentiality from one problem
to the next (e.g., from the fair die to the fair coin).

Let's consider a slightly more interesting problem where we toss two dice. We
assume that each throw is *independent*, meaning that the outcome of one does
not influence the other. What are the sets in this case? They are all pairs
of possible outcomes from two throws as shown below,

$$
\Omega = \lbrace (1,1),(1,2),\ldots,(5,6),(6,6) \rbrace
$$

  What are the measures of each of these sets? By virtue of the
independence claim, the measure of each is the product of the respective measures
of each element.  For instance,

$$
\mathbb{P}((1,2)) = \mathbb{P}(\lbrace 1 \rbrace) \mathbb{P}(\lbrace 2 \rbrace) = \frac{1}{6^2}
$$

 With all that established, we can ask the following
question: what is the probability that the sum of the dice equals
seven? As before, the first thing to do is characterize the
measurable function for this as $X:(a,b) \mapsto (a+b)$.  Next, we
associate all of the $(a,b)$ pairs with their sum. We can create a
Python dictionary for this as shown,

```python
d={(i,j):i+j for i in range(1,7) for j in range(1,7)}
```

 The next step is to collect all of the $(a,b)$ pairs that sum to
each of the possible values from two to twelve.

```python
from collections import defaultdict
dinv = defaultdict(list)
for i,j in d.iteritems():
    dinv[j].append(i)
```

**Programming Tip.**

The `defaultdict` object from the built-in collections module creates dictionaries with
default values when it encounters a new key. Otherwise, we would have had to
create default values manually for a regular dictionary.

 For example, `dinv[7]` contains the following list of pairs that
sum to seven,

```python
[(1, 6), (2, 5), (5, 2), (6, 1), (4, 3), (3, 4)]
```

    [(1, 6), (2, 5), (5, 2), (6, 1), (4, 3), (3, 4)]

The next step is to compute the probability measured for each of these items.
Using the independence assumption, this means we have to compute the sum of the
products of the individual item probabilities in `dinv`. Because we know that
each outcome is equally likely, every term in the sum equals $1/36$. Thus, all
we have to do is count the number of items in the corresponding list for each
key in `dinv` and divide by `36`. For example,  `dinv[11]` contains `[(5, 6),
(6, 5)]`.  The probability of `5+6=6+5=11` is the probability of this set which
is composed of the sum of the probabilities of the individual elements
`{(5,6),(6,5)}`. In this case, we have $\mathbb{P}(11) = \mathbb{P}(\lbrace
(5,6) \rbrace)+ \mathbb{P}(\lbrace (6,5) \rbrace) = 1/36 + 1/36 = 2/36$.
Repeating this procedure for all the elements, we derive the probability mass
function as shown below,

```python
X={i:len(j)/36. for i,j in dinv.iteritems() }
print X
{2: 0.027777777777777776,
 3: 0.05555555555555555,
 4: 0.08333333333333333,
 5: 0.1111111111111111,
 6: 0.1388888888888889,
 7: 0.16666666666666666,
 8: 0.1388888888888889,
 9: 0.1111111111111111,
 10: 0.08333333333333333,
 11: 0.05555555555555555,
 12: 0.027777777777777776}
```

    {2: 0.027777777777777776, 3: 0.05555555555555555, 4: 0.08333333333333333, 5: 0.1111111111111111, 6: 0.1388888888888889, 7: 0.16666666666666666, 8: 0.1388888888888889, 9: 0.1111111111111111, 10: 0.08333333333333333, 11: 0.05555555555555555, 12: 0.027777777777777776}

    {2: 0.027777777777777776,
     3: 0.05555555555555555,
     4: 0.08333333333333333,
     5: 0.1111111111111111,
     6: 0.1388888888888889,
     7: 0.16666666666666666,
     8: 0.1388888888888889,
     9: 0.1111111111111111,
     10: 0.08333333333333333,
     11: 0.05555555555555555,
     12: 0.027777777777777776}

**Programming Tip.**

In the preceding code note that `36.` is written with
the trailing decimal mark. This is a good habit to get into because division
in Python 2.x is integer division by default, which is not what we want here.
This can be fixed with a top-level `from __future__ import division`, but
that's easy to forget to do, especially when you are passing code
around and others may not reflexively do the future import.

The above example exposes the elements of probability theory that
are in play for this simple problem while deliberately suppressing some of the
gory technical details. With this framework, we can ask other questions like
what is the probability that half the product of three dice will exceed the
their sum? We can solve this using the same method as in the following. First,
let's create the first mapping,

```python
d={(i,j,k):((i*j*k)/2>i+j+k) for i in range(1,7)
                                for j in range(1,7)
                                   for k in range(1,7)}
```

 The keys of this dictionary are the triples and the values are the
logical values of whether or not half the product of three dice exceeds their sum.
Now, we do the inverse mapping to collect the corresponding lists,

```python
dinv = defaultdict(list)
for i,j in d.iteritems(): dinv[j].append(i)
```

 Note that `dinv` contains only two keys, `True` and `False`. Again,
because the dice are independent, the probability of any triple is $1/6^3$.
Finally, we collect this for each outcome as in the following,

```python
X={i:len(j)/6.0**3 for i,j in dinv.iteritems() }
print X
{False: 0.37037037037037035, True: 0.6296296296296297}
```

    {False: 0.37037037037037035, True: 0.6296296296296297}

    {False: 0.37037037037037035, True: 0.6296296296296297}

 Thus, the probability of half the product of three dice exceeding their sum is
`136/(6.0**3) = 0.63`. The set that is induced by the random variable has only
two elements in it, `True` and `False`, with $\mathbb{P}(\mbox{True})=136/216$
and $\mathbb{P}(\mbox{False})=1-136/216$.

As a final example to exercise another layer of generality, let is consider the
first problem with the two dice where we want the probability of a
seven, but this time one of the dice is no longer fair.  The distribution for
the unfair die is the following:

$$
\begin{eqnarray*}
\mathbb{P}(\lbrace 1\rbrace)=\mathbb{P}(\lbrace 2 \rbrace)=\mathbb{P}(\lbrace 3 \rbrace)  = \frac{1}{9} \\\
\mathbb{P}(\lbrace 4\rbrace)=\mathbb{P}(\lbrace 5 \rbrace)=\mathbb{P}(\lbrace 6 \rbrace)  = \frac{2}{9}
\end{eqnarray*}
$$

From our earlier work, we know the elements corresponding to the sum of seven
are the following:

$$
\lbrace (1,6),(2,5),(3,4),(4,3),(5,2),(6,1) \rbrace
$$

  Because we still have the independence assumption, all we need to
change is the probability computation of each of elements. For example, given
that the first die is the unfair one, we have

$$
\mathbb{P}((1,6)) = \mathbb{P}(1)\mathbb{P}(6) = \frac{1}{9} \times \frac{1}{6}
$$

  and likewise for $(2,5)$ we have the following:

$$
\mathbb{P}((2,5)) = \mathbb{P}(2)\mathbb{P}(5) = \frac{1}{9} \times \frac{1}{6}
$$

  and so forth. Summing all of these gives the following:

$$
\mathbb{P}_X(7) =  \frac{1}{9} \times \frac{1}{6}
                      +\frac{1}{9} \times \frac{1}{6}
                      +\frac{1}{9} \times \frac{1}{6}
                      +\frac{2}{9} \times \frac{1}{6}
                      +\frac{2}{9} \times \frac{1}{6}
                      +\frac{2}{9} \times \frac{1}{6}  = \frac{1}{6}
$$

 Let's try computing this using Pandas instead
of Python dictionaries. First, we construct
a `DataFrame` object with an index of tuples
consisting of all pairs of possible dice outcomes.

```python
from pandas import DataFrame
d=DataFrame(index=[(i,j) for i in range(1,7) for j in range(1,7)],
            columns=['sm','d1','d2','pd1','pd2','p'])
```

 Now, we can populate the columns that we set up above
where the outcome of the first die is the `d1` column and
the outcome of the second die is `d2`,

```python
d.d1=[i[0] for i in d.index]
d.d2=[i[1] for i in d.index]
```

 Next, we compute the sum of the dices in the `sm`
column,

```python
d.sm=map(sum,d.index)
```

 With that established, the DataFrame now looks like
the following:

```python
d.head(5) # show first five lines
```

      sm
      d1
      d2
      pd1
      pd2
      p

      (1, 1)
      2
      1
      1
      NaN
      NaN
      NaN

      (1, 2)
      3
      1
      2
      NaN
      NaN
      NaN

      (1, 3)
      4
      1
      3
      NaN
      NaN
      NaN

      (1, 4)
      5
      1
      4
      NaN
      NaN
      NaN

      (1, 5)
      6
      1
      5
      NaN
      NaN
      NaN

 Next, we fill out the probabilities for each face of the
unfair die (`d1`) and the fair die (`d2`),

```python
d.loc[d.d1<=3,'pd1']=1/9.
d.loc[d.d1 > 3,'pd1']=2/9.
d.pd2=1/6.
d.head(10)
```

      sm
      d1
      d2
      pd1
      pd2
      p

      (1, 1)
      2
      1
      1
      0.111111
      0.166667
      NaN

      (1, 2)
      3
      1
      2
      0.111111
      0.166667
      NaN

      (1, 3)
      4
      1
      3
      0.111111
      0.166667
      NaN

      (1, 4)
      5
      1
      4
      0.111111
      0.166667
      NaN

      (1, 5)
      6
      1
      5
      0.111111
      0.166667
      NaN

      (1, 6)
      7
      1
      6
      0.111111
      0.166667
      NaN

      (2, 1)
      3
      2
      1
      0.111111
      0.166667
      NaN

      (2, 2)
      4
      2
      2
      0.111111
      0.166667
      NaN

      (2, 3)
      5
      2
      3
      0.111111
      0.166667
      NaN

      (2, 4)
      6
      2
      4
      0.111111
      0.166667
      NaN

 Finally, we can compute the joint probabilities
for the sum of the shown faces as the following:

```python
d.p = d.pd1 * d.pd2
d.head(5)
```

      sm
      d1
      d2
      pd1
      pd2
      p

      (1, 1)
      2
      1
      1
      0.111111
      0.166667
      0.0185185

      (1, 2)
      3
      1
      2
      0.111111
      0.166667
      0.0185185

      (1, 3)
      4
      1
      3
      0.111111
      0.166667
      0.0185185

      (1, 4)
      5
      1
      4
      0.111111
      0.166667
      0.0185185

      (1, 5)
      6
      1
      5
      0.111111
      0.166667
      0.0185185

 With all that established, we can compute the
density of all the dice outcomes by using `groupby` as in the
following,

```python
d.groupby('sm')['p'].sum()
```

    sm
    2     0.018519
    3     0.037037
    4     0.055556
    5     0.092593
    6     0.129630
    7     0.166667
    8     0.148148
    9     0.129630
    10    0.111111
    11    0.074074
    12    0.037037
    Name: p, dtype: float64

  These examples have shown how the theory of probability
breaks down sets and measurements of  those sets and how these can be
combined to develop the probability mass functions for new random
variables.

## Continuous Random Variables

The same ideas work with continuous variables but managing the sets
becomes trickier because the real line, unlike discrete sets, has many
limiting properties already built into it that have to be handled
carefully.  Nonetheless, let's start with an example that should
illustrate the analogous ideas. Suppose a random variable $X$ is
uniformly distributed on the unit interval. What is the probability
that the variable takes on values less than 1/2?

In order to build intuition onto the discrete case, let's go back to our
dice-throwing experiment with the fair dice. The sum of the values of the dice
is a measurable function,

$$
Y \colon \lbrace 1,2,\dots,6 \rbrace^2 \mapsto \lbrace 2,3,\ldots, 12 \rbrace
$$

 That is, $Y$ is a mapping of the cartesian product of sets to a
discrete set of outcomes. In order to compute probabilities of the set of
outcomes, we need to derive the probability measure for $Y$, $\mathbb{P}_Y$,
from the corresponding probability measures for each die. Our previous discussion
went through the mechanics of that. This means that

$$
\mathbb{P}_Y \colon \lbrace 2,3,\ldots,12 \rbrace \mapsto [0,1]
$$

 Note there is a separation between the function definition and where the
target items of the function are measured in probability. More bluntly,

$$
Y \colon A \mapsto B
$$

 with,

$$
\mathbb{P}_Y \colon B \mapsto [0,1]
$$

 Thus, to compute $\mathbb{P}_Y$, which is derived
from other random variables, we have to express the equivalence classes
in $B$ in terms of their progenitor $A$ sets.

The situation for continuous variables follows the same pattern, but
with many more deep technicalities that we are going to skip. For the continuous
case, the random variable is now,

$$
X \colon \mathbb{R} \mapsto \mathbb{R}
$$

 with corresponding probability measure,

$$
\mathbb{P}_X \colon \mathbb{R} \mapsto [0,1]
$$

 But where are the corresponding sets here? Technically, these are the
*Borel* sets, but we can just think of them as intervals. Returning to our
question, what is the probability that a uniformly distributed random variable
on the unit interval takes values less than $1/2$? Rephrasing this question
according to the framework, we have the following:

$$
X \colon [0,1] \mapsto [0,1]
$$

 with corresponding,

$$
\mathbb{P}_X \colon [0,1] \mapsto [0,1]
$$

 To answer the question,  by the definition of the uniform random
variable on the unit interval, we compute the following integral,

$$
\mathbb{P}_X([0,1/2]) = \mathbb{P}_X(0 < X < 1/2) =  \int_0^{1/2} dx  = 1/2
$$

  where the above integral's $dx$ sweeps through intervals of the
$B$-type. The measure of any $dx$ interval (i.e., $A$-type set) is equal to
$dx$, by definition of the uniform random variable. To get all the moving parts
into one notationally rich integral, we can also  write this  as,

$$
\mathbb{P}_X(0 < X < 1/2) = \int_0^{ 1/2 } d\mathbb{P}_X(dx) = 1/2
$$

Now, let's consider a slightly more complicated and interesting example. As
before, suppose we have a uniform random variable, $X$ and let us introduce
another random variable defined,

$$
Y = 2 X
$$

  Now, what is the probability that $0 < Y < \frac{1}{2}$?
To express this in our framework, we write,

$$
Y \colon [0,1] \mapsto [0,2]
$$

 with corresponding,

$$
\mathbb{P}_Y \colon [0,2] \mapsto [0,1]
$$

 To answer the question, we need to measure the set $[0,1/2]$, with
the probability measure for $Y$, $\mathbb{P}_Y([0,1/2])$. How can we do this?
Because $Y$ is derived from the $X$ random variable, as with the fair-dice
throwing experiment, we have to create a set of equivalences in the target
space (i.e., $B$-type sets) that reflect back on the input space (i.e.,
$A$-type sets). That is, what is the interval $[0,1/2]$ equivalent to in terms
of the $X$ random variable? Because, functionally, $Y=2 X$, then the $B$-type
interval $[0,1/2]$ corresponds to the $A$-type interval $[0,1/4]$. From the
probability measure of $X$, we compute this with the integral,

$$
\mathbb{P}_Y([0,1/2]) =\mathbb{P}_X([0,1/4])=  \int_0^{1/4} dx  = 1/4
$$

Now, let's up the ante and consider the following random variable,

$$
Y = X^2
$$

 where now $X$ is still uniformly distributed, but now over the
interval $[-1/2,1/2]$.  We can express this in our framework as,

$$
Y \colon [-1/2,1/2] \mapsto [0,1/4]
$$

 with corresponding,

$$
\mathbb{P}_Y \colon [0,1/4] \mapsto [0,1]
$$

 What is the $\mathbb{P}_Y(Y < 1/8)$?  In other words, what is the
measure of the set $B_Y= [0,1/8]$? As before, because $X$ is derived from our
uniformly distributed random variable, we have to reflect the $B_Y$ set onto
sets of the $A$-type.  The thing to recognize is that because $X^2$
is symmetric about zero, all $B_Y$ sets reflect back into two sets.
This means that for any set $B_Y$, we have the correspondence $B_Y = A_X^+ \cup
A_X^{-}$.  So, we have,

$$
B_Y=\Big\lbrace 0<Y<\frac{1}{8}\Big\rbrace=\Big\lbrace 0<X<\frac{1}{\sqrt{8}} \Big\rbrace \bigcup \Big\lbrace -\frac{1}{\sqrt {8}}<X<0 \Big\rbrace
$$

  From this perspective, we have the following solution,

$$
\mathbb{P}_Y(B_Y)=\mathbb{P}(A_X^+)/2 + \mathbb{P}(A_X^{-})/2
$$

  where the $\frac{1}{2}$ comes from normalizing the $\mathbb{P}_Y$ to
one. Also,

$$
A_X^+    = \Big\lbrace 0<  X<\frac{1}{\sqrt{8}}   \Big\rbrace
$$

$$
\
A_X^{-}  = \Big\lbrace -\frac{1}{\sqrt {8}} <  X<0 \Big\rbrace
$$

  Therefore,

$$
\mathbb{P}_Y(B_Y) = \frac{1}{2\sqrt 8} + \frac{1}{2\sqrt 8}
$$

  because $\mathbb{P}(A_X^+) =\mathbb{P}(A_X^-) = 1/\sqrt 8$.  Let's
see if this comes out using the usual transformation of variables method from
calculus. Using this method, the density $f_Y(y) = f_X(\sqrt y)/(2 \sqrt y) =
\frac{1}{2 \sqrt y} $. Then, we obtain,

$$
\int_0^{\frac{1}{8}} \frac{1}{2 \sqrt y} dy = \frac{1}{\sqrt 8}
$$

  which is what we got using the sets method. Note that you would
favor the calculus method in practice, but it is important to
understand the deeper mechanics, because sometimes
the usual calculus method fails, as the next problem shows.

## Transformation of Variables Beyond Calculus

Suppose $X$ and $Y$ are uniformly distributed in the unit interval and we
define $Z$ as

$$
Z = \frac{X}{Y-X}
$$

  What is the $f_Z(z)$? If you try this using the usual calculus
method, you will fail (try it!). The problem is one
of the technical prerequisites for the calculus method is not in force.

The key observation is that $Z \notin [-1,0]$. If this were
possible, the $X$ and $Y$ would have different signs, which cannot happen,
given that $X$ and $Y$ are uniformly distributed over $[0,1]$. Now, let's
consider when $Z>0$. In this case, $Y>X$ because $Z$ cannot be positive
otherwise. For the density function, we are interested in the set
$\lbrace  0 < Z < z \rbrace $. We want to compute

$$
\mathbb{P}(Z<z) = \int \int B_1 dX dY
$$

 with,

$$
B_1 = \lbrace 0 < Z < z \rbrace
$$

  Now, we have to translate that interval into an interval
relevant to $X$ and $Y$. For $0 < Z$, we have $ Y > X$. For $Z < z $,
we have $Y > X(1/z+1)$. Putting this together gives

$$
A_1 = \lbrace \max (X,X(1/z+1)) < Y < 1 \rbrace
$$

  Integrating this over $Y$ as follows,

$$
\int_0^1\lbrace\max(X,X(1/z+1))<Y<1 \rbrace dY=\frac{z-X-Xz}{z}\mbox{ where } z > \frac{X}{1-X}
$$

 and integrating this one more time over $X$ gives

$$
\int_0^{\frac{z}{1+z}} \frac{-X+z-Xz}{z} dX = \frac{z}{2(z+1)} \mbox{ where }  z > 0
$$

 Note that this is the computation for the *probability*
itself, not the probability density function. To get that, all we have
to do is differentiate the last expression to obtain

$$
f_Z(z) = \frac{1}{(z+1)^2} \mbox{ where }  z > 0
$$

 Now we need to compute this density using the same process
for when $z < -1$.  We want  the interval $ Z < z $ for when $z < -1$.
For a fixed $z$, this is equivalent to $ X(1+1/z) < Y$. Because $z$
is negative, this also means that $Y < X$. Under these terms, we
have the following integral,

$$
\int_0^1 \lbrace X(1/z+1) <Y< X\rbrace dY = -\frac{X}{z} \mbox{ where }  z < -1
$$

 and integrating this one more time over $X$ gives the following

$$
-\frac{1}{2 z} \mbox{ where }  z < -1
$$

 To get the density for $z<-1$, we differentiate this with
respect to $z$ to obtain the following,

$$
f_Z(z) = \frac{1}{2 z^2} \mbox{ where }  z < -1
$$

 Putting this all together, we obtain,

$$
f_Z(z) =
\begin{cases}
      \frac{1}{(z+1)^2}     & \mbox{if }  z > 0 \\\
      \frac{1}{2 z^2}       & \mbox{if }  z < -1 \\\
      0 & \mbox{otherwise }
\end{cases}
$$

  We will leave it as an exercise to show that this
integrates out to one.

## Independent Random Variables

Independence is a standard assumption. Mathematically, the
necessary and sufficient condition for independence between two
random variables $X$  and $Y$ is the following:

$$
\mathbb{P}(X,Y) = \mathbb{P}(X)\mathbb{P}(Y)
$$

 Two random variables $X$  and $Y$
are *uncorrelated* if,

$$
\mathbb{E}(X-\overline{X})\mathbb{E}(Y-\overline{Y})=0
$$

 where $\overline{X}=\mathbb{E}(X)$ Note that uncorrelated random
variables are sometimes called *orthogonal* random variables.  Uncorrelatedness
is a weaker property than independence, however. For example, consider the
discrete random variables $X$ and $Y$ uniformly distributed over the set
$\lbrace 1,2,3 \rbrace$ where

$$
X =
\begin{cases}
1 & \mbox{if } \omega =1 \\\
0 & \mbox{if } \omega =2  \\\
-1 & \mbox{if } \omega =3
\end{cases}
$$

 and also,

$$
Y =
\begin{cases}
0 & \mbox{if } \omega =1 \\\
1 & \mbox{if } \omega =2 \\\
0 & \mbox{if } \omega =3
\end{cases}
$$

 Thus, $\mathbb{E}(X)=0$ and $\mathbb{E}(X Y)=0$, so
$X$ and $Y$ are uncorrelated. However, we have

$$
\mathbb{P}(X=1,Y=1)=0\neq \mathbb{P}(X=1)\mathbb{P}(Y=1)=\frac{1}{9}
$$

 So, these two random variables are *not* independent.
Thus, uncorrelatedness does not imply independence, generally, but
there is the important case of Gaussian random variables for which
it does. To see this, consider the probability density function
for two zero-mean, unit-variance Gaussian random variables $X$ and
$Y$,

$$
f_{X,Y}(x,y) = \frac{e^{\frac{x^2-2 \rho x
   y+y^2}{2 \left(\rho^2-1\right)}}}{2 \pi
   \sqrt{1-\rho^2}}
$$

 where $\rho:=\mathbb{E}(X Y)$  is the correlation coefficient.  In
the uncorrelated case where $\rho=0$, the probability density function factors
into the following,

$$
f_{X,Y}(x,y)=\frac{e^{-\frac{1}{2}\left(x^2+y^2\right)}}{2\pi}=\frac{e^{-\frac{x^2}{2}}}{\sqrt{2\pi}}\frac{e^{-\frac{y^2}{2}}}{\sqrt{2\pi}}    =f_X(x)f_Y(y)
$$

 which means that $X$ and $Y$ are independent.

Independence and conditional independence are closely related, as in the following:

$$
\mathbb{P}(X,Y\vert Z) =\mathbb{P}(X\vert Z) \mathbb{P}(Y\vert Z)
$$

 which says that $X$ and $Y$ and independent conditioned
on $Z$.  Conditioning independent random variables can break
their independence. For example, consider two independent
Bernoulli-distributed random variables,  $X_1, X_2\in\lbrace 0,1
\rbrace$.  We define $Z=X_1+X_2$. Note that $Z\in \lbrace
0,1,2 \rbrace$. In the case where $Z=1$, we have,

$$
\mathbb{P}(X_1\vert Z=1) >0
$$

$$
\
\mathbb{P}(X_2\vert Z=1) >0
$$

 Even though $X_1,X_2$ are independent,
after conditioning on $Z$, we have the following,

$$
\mathbb{P}(X_1=1,X_2=1\vert Z=1)=0\neq \mathbb{P}(X_1=1\vert Z=1)\mathbb{P}(X_2=1\vert Z=1)
$$

 Thus, conditioning on $Z$ breaks the independence of
$X_1,X_2$. This also works in the opposite direction ---
conditioning can make dependent random variables independent.
Define $Z_n=\sum_i^n X_i$ with $X_i$ independent, integer-valued
random variables.  The $Z_n$ variables are
dependent because they stack the same telescoping set of
$X_i$ variables. Consider the following,

$$
\begin{equation}
\mathbb{P}(Z_1=i,Z_3=j\vert Z_2=k) = \frac{\mathbb{P}(Z_1=i,Z_2=k,Z_3=j)}{\mathbb{P}(Z_2 =k)}
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation} \
=\frac{\mathbb{P}(X_1 =i)\mathbb{P}(X_2 =k-i)\mathbb{P}(X_3 =j-k) }{\mathbb{P}(Z_2 =k)}
\end{equation}
\label{eq:condIndep} \tag{2}
$$

 where the factorization comes from the independence of
the $X_i$ variables. Using the definition of conditional
probability,

$$
\mathbb{P}(Z_1=i\vert Z_2)=\frac{\mathbb{P}(Z_1=i,Z_2=k)}{\mathbb{P}(Z_2=k)}
$$

 We can continue to expand Equation ref{eq:condIndep},

$$
\mathbb{P}(Z_1=i,Z_3=j\vert Z_2=k) =\mathbb{P}(Z_1 =i\vert Z_2) \frac{\mathbb{P}( X_3 =j-k)\mathbb{P}( Z_2 =k)}{\mathbb{P}( Z_2 =k)}
$$

$$
\
                                   =\mathbb{P}(Z_1 =i\vert Z_2)\mathbb{P}(Z_3 =j\vert Z_2)
$$

 where $\mathbb{P}(X_3=j-k)\mathbb{P}(Z_2=k)=
\mathbb{P}(Z_3=j,Z_2)$.  Thus, we see that dependence between
random variables can be broken by conditioning to create
conditionally independent random variables. As we have just
witnessed, understanding how conditioning influences independence
is important and is the main topic of
study in Probabilistic Graphical Models, a field
with many algorithms and concepts to extract these
notions of conditional independence from graph-based
representations of random variables.

## Classic Broken Rod Example

Let's do one last example to exercise fluency in our methods by
considering the following classic problem: given a rod of unit-length,
broken independently and randomly at two places, what is the
probability that you can assemble the three remaining pieces into a
triangle? The first task is to find a representation of a triangle as
an easy-to-apply constraint. What we want is something like the
following:

$$
\mathbb{P}(\mbox{ triangle exists }) = \int_0^1 \int_0^1 \lbrace \mbox{ triangle exists }  \rbrace dX dY
$$

 where $X$ and $Y$ are independent and uniformly distributed
in the unit-interval.  Heron's formula for the area of the triangle,

$$
\mbox{ area } = \sqrt{(s-a)(s-b)(s-c)s}
$$

 where $s = (a+b+c)/2$ is what we need. The idea is that this
yields a valid area only when each of the terms under the square root is
greater than or equal to zero. Thus, suppose that we have

$$
\begin{eqnarray*}
a  & = &  X  \\\
b  & = & Y-X \\\
c  & = & 1-Y
\end{eqnarray*}
$$

 assuming that $Y>X$.  Thus, the criterion for a valid triangle boils down
to

$$
\lbrace (s > a) \wedge (s > b) \wedge (s > c) \wedge (X<Y) \rbrace
$$

 After a bit of manipulation, this  consolidates into:

$$
\Big\lbrace \frac{1}{2} < Y < 1 \bigwedge \frac{1}{2}(2 Y-1) < X < \frac{1}{2} \Big\rbrace
$$

  which we integrate out by $dX$ first to obtain

$$
\mathbb{P}(\mbox{ triangle exists }) = \int_{0}^1 \int_{0}^1 \Big\lbrace \frac{1}{2} < Y < 1 \bigwedge \frac{1}{2}(2 Y-1) < X < \frac{1}{2} \Big\rbrace dX dY
$$

$$
\mathbb{P}(\mbox{ triangle exists }) = \int_{\frac{1}{2}}^1 (1-Y) dY
$$

 and then by $dY$ to obtain finally,

$$
\mathbb{P}(\mbox{ triangle exists }) = \frac{1}{8}
$$

 when $Y>X$. By symmetry, we get the same result for $X>Y$. Thus, the
final result is the following:

$$
\mathbb{P}(\mbox{ triangle exists }) = \frac{1}{8}+\frac{1}{8} = \frac{1}{4}
$$

We can quickly check using this result using Python for the case $Y>X$ using
the following code:

```python
>>> import numpy as np
>>> x,y = np.random.rand(2,1000) # uniform rv
>>> a,b,c = x,(y-x),1-y # 3 sides
>>> s = (a+b+c)/2
>>> np.mean((s>a) & (s>b)  & (s>c) & (y>x)) # approx 1/8=0.125
```

    0.13700000000000001

**Programming Tip.**

The chained logical `&` symbols above tell Numpy that the logical operation
should be considered element-wise.


```python
!conda info
```

    Current conda install:

                   platform : win-64
              conda version : 4.3.22
           conda is private : False
          conda-env version : 4.3.22
        conda-build version : not installed
             python version : 3.6.1.final.0
           requests version : 2.14.2
           root environment : C:\Users\USER\Anaconda3  (writable)
        default environment : C:\Users\USER\Anaconda3
           envs directories : C:\Users\USER\Anaconda3\envs
                              C:\Users\USER\AppData\Local\conda\conda\envs
                              C:\Users\USER\.conda\envs
              package cache : C:\Users\USER\Anaconda3\pkgs
                              C:\Users\USER\AppData\Local\conda\conda\pkgs
               channel URLs : https://repo.continuum.io/pkgs/free/win-64
                              https://repo.continuum.io/pkgs/free/noarch
                              https://repo.continuum.io/pkgs/r/win-64
                              https://repo.continuum.io/pkgs/r/noarch
                              https://repo.continuum.io/pkgs/pro/win-64
                              https://repo.continuum.io/pkgs/pro/noarch
                              https://repo.continuum.io/pkgs/msys2/win-64
                              https://repo.continuum.io/pkgs/msys2/noarch
                config file : C:\Users\USER\.condarc
                 netrc file : None
               offline mode : False
                 user-agent : conda/4.3.22 requests/2.14.2 CPython/3.6.1 Windows/10 Windows/10.0.14393
              administrator : False

# Variables

```python
x = 2
y = '3'
print(x+int(y))

z = [1, 2, 3] #List
w = (2, 3, 4) #Tuple

import numpy as np
q = np.array([1, 2, 3]) #numpy.ndarray
type(q)
```

    5

    numpy.ndarray

# Console input and output

```python
MyName = input('My name is: ')
print('Hello, '+MyName)
```

    My name is: david
    Hello, david

# File input and output

```python
fid = open('msg.txt','w')
fid.write('demo of writing.\n')
fid.write('Second line')
fid.close()

fid = open('msg.txt','r')
msg = fid.readline()
print(msg)
msg = fid.readline()
print(msg)

fid.close()
```

    demo of writing.

    Second line

```python
fid = open('msg.txt','r')
msg = fid.readlines()
print(msg)
```

    ['demo of writing.\n', 'Second line']

```python
fid = open('msg.txt','r')
msg = fid.read()
print(msg)
```

    demo of writing.
    Second line

```python
import numpy as np
x = np.linspace(0, 2*np.pi,4)
y = np.cos(x)

#Stack arrays in sequence vertically (row wise).
data = np.vstack((x,y))  #上下對隊齊好
dataT = data.T #Transpose

np.savetxt('data.txt', data, delimiter=',')
z = np.loadtxt('data.txt', delimiter=',')

print(x)
print(y)
print(data)
print(dataT)
print(z)
```

    [ 0.          2.0943951   4.1887902   6.28318531]
    [ 1.  -0.5 -0.5  1. ]
    [[ 0.          2.0943951   4.1887902   6.28318531]
     [ 1.         -0.5        -0.5         1.        ]]
    [[ 0.          1.        ]
     [ 2.0943951  -0.5       ]
     [ 4.1887902  -0.5       ]
     [ 6.28318531  1.        ]]
    [[ 0.          2.0943951   4.1887902   6.28318531]
     [ 1.         -0.5        -0.5         1.        ]]

```python
import numpy as np
x = np.linspace(0, 2*np.pi,20)
y = np.cos(x)
z = np.sin(x)

%matplotlib inline
import matplotlib.pyplot as plt

#使用 help(plt.plot) 可以看到所有畫圖玩法
plt.plot(x,y,'b')
plt.plot(x,y,'go', label = 'cos(x)')
plt.plot(x,z,'r')
plt.plot(x,z,'go', label = 'sin(x)')
plt.legend(loc='best') # 放到最好的位置
plt.xlim([0, 2*np.pi])
```

```python
import numpy as np
x = np.linspace(0, 2*np.pi,20)
y = np.cos(x)
z = np.sin(x)

%matplotlib inline
import matplotlib.pyplot as plt

#使用 help(plt.plot) 可以看到所有畫圖玩法
plt.subplot(2,1,1) #分成兩張圖 形式是(row, column, order)
plt.plot(x,y,'b')
plt.plot(x,y,'go', label = 'cos(x)')
plt.legend(loc='best') #放到最好的位置

plt.subplot(2,1,2) #分成兩張圖
plt.plot(x,z,'r')
plt.plot(x,z,'go', label = 'sin(x)')
plt.legend(loc='best') #放到最好的位置

plt.xlim([0, 2*np.pi])
```

# Functions, Conditions, Loop

```python
import numpy as np

def f(x):
    return x**2

x = np.linspace(0,5,10)
y = f(x)

print(y)
```

    [  0.           0.30864198   1.2345679    2.77777778   4.9382716
       7.71604938  11.11111111  15.12345679  19.75308642  25.        ]

```python
import numpy as np

def f(x): #這是個奇怪的練習用函數
    res = x
    if res < 3:
        res = np.nan #<3就傳 Not a Number
    elif res < 15:
        res = x**3
    else:
        res = x**4
    return res

x = np.linspace(0,10,20)
y = np.empty_like(x)
#Return a new array with the same shape and type as a given array.
#傳一個跟x一樣的array回來

i = 0
for xi in x:
    y[i] = f(xi)
    i = i + 1
print(y)

%matplotlib inline
import matplotlib.pyplot as plt

plt.plot(x,y,'bp')
plt.xlim([0,11])
```

# Matrices, linear equations

```python
A = np.array([[1,2],[3,2]])
B = np.array([1,0])

# x = A^-1 * b
sol1 = np.dot(np.linalg.inv(A),B)
print(sol1)
sol2 = np.linalg.solve(A,B)
print(sol2)

import sympy as sym
sym.init_printing()
#This will automatically enable the best printer available in your environment.

x,y = sym.symbols('x y')
z = sym.linsolve([3*x+2*y-1,x+2*y],(x,y))
z
#sym.pprint(z) The ASCII pretty printer
```

# Non-linear equation

```python
from scipy.optimize import fsolve

def f(z):  #用z參數來表示x和y，做函數運算
    x = z[0]
    y = z[1]
    return [x+2*y, x**2+y**2-1]

z0 = [0,1]
z = fsolve(f,z0)
print(z)
print(f(z))
```

    [-0.89442719  0.4472136 ]
    [0.0, -1.1102230246251565e-16]

# Integration

```python
from scipy.integrate import quad

def f(x):
    return x**2

quad(f,0,2) #計算積分值

import sympy as sym
sym.init_printing()
x = sym.Symbol('x')
f = sym.integrate(x**2,x)
f.subs(x,2) #將值帶入函數中
f
```

# Derivative

```python
from scipy.misc import derivative

def f(x):
    return x**2

print(derivative(f,2,dx=0.01)) #dx表示精確程度

import sympy as sym
sym.init_printing()
x = sym.Symbol('x')
f = sym.diff(x**3,x)
f.subs(x,2) #將值帶入函數中，得解
f
```

# Interpolation

```python
from scipy.interpolate import interp1d #中間的字是1不是L喔!!!

x = np.arange(0,6,1)
y = np.array([0.2,0.3,0.5,1.0,0.9,1.1])

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y,'bo')

xp = np.linspace(0,5,100) #為了顯示差別把點增加

y1 = interp1d(x,y,kind='linear') #一階
plt.plot(xp,y1(xp),'r-')

y2 = interp1d(x,y,kind='quadratic') #二階
plt.plot(xp,y2(xp),'k--')

y3 = interp1d(x,y,kind='cubic') #三階
plt.plot(xp,y3(xp),'g--')

```

# Linear regression

```python
import numpy as np
x = np.array([0,1,2,3,4,5])
y = np.array([0.1,0.2,0.3,0.5,0.8,2.0 ])

#多項式逼近法，選擇階層
p1 = np.polyfit(x,y,1)
print(p1)
p2 = np.polyfit(x,y,2)
print(p2)
p3 = np.polyfit(x,y,3)
print(p3)

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y,'ro')

# np.polyval表示多項式的值，把係數p_帶入多項式x求出來的值
xp = np.linspace(0,5,100)
plt.plot(xp, np.polyval(p1,xp), 'b-', label='linear') #這個字是polyvaL喔!!
plt.plot(xp, np.polyval(p2,xp), 'g--', label='quadratic')
plt.plot(xp, np.polyval(p3,xp), 'k:', label='cubic')
plt.legend(loc='best')
```

# Nonlinear regression

```python
import numpy as np
from scipy.optimize import curve_fit

x = np.array([0,1,2,3,4,5])
y = np.array([0.1,0.2,0.3,0.5,0.8,2.0 ])

#多項式逼近法，選擇階層
p1 = np.polyfit(x,y,1)
print(p1)
p2 = np.polyfit(x,y,2)
print(p2)
p3 = np.polyfit(x,y,3)
print(p3)

#使用指數對數
def f(x,a):
    return 0.1 * np.exp(a*x)
a = curve_fit(f,x,y)[0] #非線性回歸，Use non-linear least squares to fit a function，取第0項
print('a='+str(a))

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(x,y,'ro')

# np.polyval表示多項式的值，把係數p_帶入多項式x求出來的值
xp = np.linspace(0,5,100)
plt.plot(xp, np.polyval(p1,xp), 'b-', label='linear') #這個字是polyvaL喔!!
plt.plot(xp, np.polyval(p2,xp), 'g--', label='quadratic')
plt.plot(xp, np.polyval(p3,xp), 'k:', label='cubic')
plt.plot(xp, f(xp,a), 'c', label='nonlinear')
plt.legend(loc='best')
```

# Differential equation

```python
from scipy.integrate import odeint

def dydt(y,t,a):
    return -a * y

a = 0.5
t = np.linspace(0,20)
y0 = 5.0
y = odeint(dydt,y0,t,args=(a,))

%matplotlib inline
import matplotlib.pyplot as plt
plt.plot(t,y)
plt.xlabel('time')
plt.ylabel('y')
```

# Nonlinear optimization

```python
#概念:要有Objective、Constraint，然後初始猜想值
import numpy as np
from scipy.optimize import minimize
```

```python
def objective(x): #此函數求最小值
    x1 = x[0]
    x2 = x[1]
    x3 = x[2]
    x4 = x[3]
    return x1*x4*(x1+x2+x3)+x3

#用減法做比較
def constraint1(x):
    return x[0]*x[1]*x[2]*x[3] - 25.0

#用減法做比較
def constraint2(x):
    sum_sq = 40.0
    for i in range(0,4):
        sum_sq = sum_sq - x[i]**2
    return sum_sq

#初始猜想值
x0 = [1,5,5,1]
print(objective(x0))

#設定值域
b = (1.0,5.0) #x的值域
bnds = (b,b,b,b) #四個值域都一樣b
con1 = {'type':'ineq','fun': constraint1} #第一個是不等式
con2 = {'type':'eq','fun': constraint2} #第二個需要等式
cons = [con1,con2] #cons合成一個list

sol = minimize(objective,x0,method='SLSQP',\
              bounds = bnds, constraints = cons)

```

    16

```python
print(sol)
```

         fun: 17.01401724563517
         jac: array([ 14.57227015,   1.37940764,   2.37940764,   9.56415057])
     message: 'Optimization terminated successfully.'
        nfev: 30
         nit: 5
        njev: 5
      status: 0
     success: True
           x: array([ 1.        ,  4.7429961 ,  3.82115462,  1.37940765])

```python
print(sol.fun)
```

    17.01401724563517

```python
print(sol.x)
```

    [ 1.          4.7429961   3.82115462  1.37940765]

```python

```


```python
%%HTML
<style>
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}
</style>
```

# Metody Numeryczne

## Elementy analizy numerycznej

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Informacje ogólne
- Katedra Automatyki i Robotyki, C3, p. 214
- Konsultacje
    - Czwartki 11:00-12:00
    (o ile nie ma Kolegium Wydziałowego lub seminarium)
- jb@agh.edu.pl
- wykłady dostępne tutaj: https://github.com/KAIR-ISZ/public_lectures

# Reprezentacja liczb

## Kod binarny

- Zapis liczby z wykorzystaniem dwóch symboli **1** i **0**
- Podstawa współczesnego sposobu reprezentacji informacji

## Zamierzchła historia

- Pingala, Chandaḥśāstra i Prozodia
    - Ok. 4 wiek pne
    - Wykorzystanie zapisu w formie zer i jedynek do opisu metrum
- Chiny, hexagramy, Shao Yong, I-Ching
- Leibniz

## Algebra Boole'a

$$
\begin{align}
x \land y & = xy & \mathsf{Koniunkcja}\\
x \lor y & = x+y-xy & \mathsf{Alternatywa}\\
\neg x & =1-x & \mathsf{Negacja}\\
x \rightarrow y & = (\neg x\lor y) & \mathsf{Implikacja}\\
x \oplus y & = (x \lor y)\land\neg(x\land y) & \mathsf{EXOR}\\
x = y & = \neg(x\oplus y) & \mathsf{Równoważność}\\
\end{align}
$$

## Nieco mniej zamierzchła historia
- 1937 Shannon – przekaźnikowa realizacja operacji binarnych i algebry Boole’a
- 1937 Stibitz – Pierwszy komputer przekaźnikowy (dodawanie)

## Kod binarny
| **0**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =2^5+2^3+2^1+2^0=32+8+2+1=43$

## Liczby naturalne
- Ogólnie zakres od 0 do 2n-1
- 8 bit – zakres od 0 do 255
- 16 bit – zakres od 0 do 65,535 (short, int)
- 32 bit – zakres od 0 do 4,294,967,295 (long)

W Pythonie i matlabie za bardzo nie przejmujemy się typami, chyba że je wymusimy

## Operacje na liczbach binarnych

- Dodawanie
    - 0+0=0
    - 0+1=1
    - 1+0=1
    - 1+1=0, przenieś 1
- Jak w dodawaniu pisemnym

`` 1 1 1 1 1   ``(cyfry przenoszone)
``   0 1 1 0 1 ``(1310)
``+  1 0 1 1 1 ``(2310)
``------------ ``
``=1 0 0 1 0 0 `` (3610)

## Operacje na liczbach binarnych

- Odejmowanie
    - 0-0=0
    - 0-1=1, pożyczka 1
    - 1-0=1
    - 1-1=0,
- Analogicznie

``    *   * * *  ``(pożyczki)
``  1 1 0 1 1 1 0``(11010)
``-     1 0 1 1 1``(2310)
``--------------- ``
``= 1 0 1 0 1 1 1`` (8710)

## Co z liczbami ujemnymi?

Uzupełniamy zapis o tzw. bit znaku

| **1**   | **0**   | **1**   | **0**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| S       | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =(-1)^1(2^3+2^1+2^0)=-(8+2+1)=-11$

Zmieniają się zakresy:
- 8 bit (-128 do 127)
- 16 bit (−32,768 do 32,767)
- itd

## Problemy

- Niepraktyczny zapis
- Trzeba przekodowywać wyniki operacji
- Potencjalnie podatniejsze na błędy

## Kod uzupełnienia do 2 (U2)
| **1**   | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $-2^{7}$| $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |

Co daje $ =-2^7+2^6+2^5+2^4+2^3+2^1+2^0$

$=-128+64+32+16+8+2+1=-5$

## Bardzo łatwa konwersja
- Liczby dodatnie są takie same jak były
- Aby zamienić liczbę na jej przeciwną wystarczy zanegować wszystkie bity i do wyniku dodać 1 (*w obie strony*)

| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | oryginał  |
|----------|---------|---------|---------|---------|---------|---------|---------|-----------------|-----------|
| 1    | 1   | 1   | 1   | 1   | 0   | 1   | 0   |                 | negacja   |
| **1**    | **1**   | **1**   | **1**   | **1**   | **0**   | **1**   | **1**   | -510 | dodanie 1 |
| 0    | 0   | 0   | 0   | 0   | 1   | 0   | 0   |                 | negacja   |
| **0**    | **0**   | **0**   | **0**   | **0**   | **1**   | **0**   | **1**   | 510  | dodanie 1 |
| -$2^{7}$ | $2^{6}$ | $2^{5}$ | $2^{4}$ | $2^{3}$ | $2^{2}$ | $2^{1}$ | $2^{0}$ |                 |           |

## Jaka z tego korzyść?
- Odejmowanie staje się dodawaniem (prawie)
$$ A - B = A + \neg B + 1$$
- Przykład 13 – 7 (na 8 bitach)

``  1 1 1 1     1  ``(cyfry przenoszone)
``  0 0 0 0 1 1 0 1``(1310)
``  1 1 1 1 1 0 0 0``(zanegowane 710)
``+               1``(jedynka)
``-----------------``
``= 0 0 0 0 0 1 1 0`` (610)

## Operacje na liczbach binarnych
Mnożenie również przypomina mnożenie pisemne

``          1 0 1 1`` 1110
``        * 1 0 1 0`` 1010
``      -----------``
``          0 0 0 0``
``  +     1 0 1 1   ``
``  +   0 0 0 0``
``  + 1 0 1 1``
``  ---------------``
``  = 1 1 0 1 1 1 0`` 11010

# Metody Numeryczne

## Reprezentacja liczb wymiernych

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## A co z ułamkami?
Są dwa sposoby zapisu liczb niecałkowitych
- Stałoprzecinkowy (stałopozycyjny)
- Zmiennoprzecinkowy (zmiennopozycyjny)

## Zapis stałoprzecinkowy
| **1**   | **0**   | **1**   | **1**   | **1**   | **0**   | **0**   | **0**   |
|---------|---------|---------|---------|---------|---------|---------|---------|
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ |

$$
2^1+2^{-1}+2^{-2}+2^{-3}=2+\frac{1}{2}+\frac{1}{4}+\frac{1}{8}=2.875
$$

## Zalety zapisu stałoprzecinkowego
- Nie ma różnicy w kodowaniu
- Mamy stale określoną dokładność, którą możemy w miarę dokładnie kształtować
- Stosunkowa prostota
- Małe wymagania sprzętowe

## Wady zapisu stałoprzecinkowego
Problemy z dokładnością, np. nie da się dokładnie przedstawić liczby 0.1
- Na 3 bitach części ułamkowej różnica wynosi 0.025
- Na 7 bitach części ułamkowej różnica wynosi ok. 0.001

## Jak wykonujemy działania?
- Działania wykonujemy traktując zapis liczby stałoprzecinkowej jako normalną binarną
- Kod U2 dalej działa
- Należy pamiętać, że wtedy liczba jest pomnożona przez 2n gdzie n to ilość bitów części ułamkowej
- W liczbach poddanych działaniu liczba bitów części całkowitej i ułamkowej musi być równa

## Działania stałoprzecinkowe
- Dodawanie wykonujemy identycznie
- W przypadku mnożenia wynik musimy podzielić przez 2n
- Mnożenie liczb stałoprzecinkowych przez potęgę 2 polega tylko na przesuwaniu bitów (bardzo proste w realizacji)

| 1             | 0             | 1   | 1   | 1   | 0   | 0   | 0   |   |
|---------------|---------------|-----|-----|-----|-----|-----|-----|---|
| 0             | 0             | 1   | 0   | 1   | 1   | 1   | 0   | Podzielenie przez $2^{2}$  |
| $2^{1}$ | $2^{0}$ | $2^{-1}$ | $2^{-2}$ | $2^{-3}$ | $2^{-4}$ | $2^{-5}$ | $2^{-6}$ ||

## Format zmiennoprzecinkowy
- Bardziej zaawansowany sposób przedstawiania liczb
- Ustandaryzowany normą IEEE
- Dający pod pewnymi względami większą dokładność

## Format zmiennoprzecinkowy

Reprezentacja liczby

$$
x=S\cdot M\cdot B^E
$$

- S – znak (*sign*)
- M – mantysa (*mantissa*, także *fraction*)
- B – podstawa (*base*, zazwyczaj 2, rzadziej 10)
- E - wykładnik (*exponent*)

## Mantysa
- Liczba odpowiadająca za ułamkową część zapisu
- Format stałoprzecinkowy, zazwyczaj liczba z przedziału [1,2)

## Podstawa i wykładnik

- Pozwalają na określenie szerokiego zakresu
- Ze względu na kodowanie, zazwyczaj podstawa to 2
- Wykładnik może być ujemny lub dodatni.
- Wykładnik koduje się w U2, lub też wprowadza się przesunięcie

## Działania na liczbach zmiennoprzecinkowych
Dodawanie i odejmowanie

$$
x_1\pm x_2=\left(M_1\pm M_2\cdot B^{E_2-E_1}\right)\cdot B^{E_1}
$$

Mnożenie i dzielenie

$$
x_1\cdot x_2=(S_1\cdot S_2)\cdot (M_1\cdot M_2)\cdot B^{E_1+E_2}
$$

$$
x_1 / x_2=(S_1\cdot S_2)\cdot (M_1/ M_2)\cdot B^{E_1-E_2}
$$

## Dzielenie
- Mając możliwość zapisu liczby ulamkowej można sformułować operację dzielenia.
- Istnieje wiele algorytmów np.
    - *restoring division*
    - *non-restoring division*
    - SRT
    - algorytm Newtona-Raphsona
    - algorytm Goldschmidta
- Są one już zaimplementowane, jedno dzielenie zazwyczaj wymaga przeprowadzenia 3-4 mnożeń

## Ważne formaty – IEEE Single precision

- 8 bitów wykładnika, wykładnik przesunięty o 127 (zamiana z -126 do 127 na 1 do 244)
- 24 bity mantysy, ale zawsze koduje się tylko 23 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float32``

## Ważne formaty – IEEE Double precision

- 11 bitów wykładnika, wykładnik przesunięty o 1023 (zamiana z -1022 do 1023 na 1 do 2046)
- 53 bity mantysy, ale zawsz koduje się tylko 52 po kropce, przed kropką jest 1
- Specjalne zapisy nieskończoności i błędów
- w NumPy - ``float64``, ale w zasadzie każda liczba w Pythonie i Matlabie to double, chyba że wymusimy inaczej

## Wyświetlanie liczb
- Normalnie
- Notacja inżynierska
    - $3700=3.7\cdot10^3$, $0.12=120\cdot10^{-3}$
- Notacja naukowa
    - ``3700=3.7E3``, ``0.12=1.2E-1``

# Metody Numeryczne

## Błedy numeryczne

### dr hab. inż. Jerzy Baranowski, Prof. AGH.

## Podstawowe definicje
Wartość dokładna
$$y=\tilde{y}+\varepsilon$$
- $\tilde{y}$ - wartość przybliżona
- $\varepsilon$ - błąd

## Błąd bezwzględny
Wartość bezwzględna różnicy między rozwiązaniem dokładnym i przybliżonym
$$ \varepsilon=|y-\tilde{y}|$$

## Błąd względny
Stosunek błędu bezwzględnego do wartości bezwzględnej rozwiązania
$$\eta=\frac{|y-\tilde{y}|}{|y|}=\left|\frac{y-\tilde{y}}{y}\right|=\left|1-\frac{\tilde{y}}{y}\right|$$
Czasami błąd względny wyrażamy w procentach

## Przykłady
Pierwiastek kwadratowy ze 122

$$
\begin{align}
y{}&=\sqrt{122}\approx 11.04536\\
\tilde{y}{}&=11\\
\varepsilon{}&=|y-\tilde{y}|=0.04536\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00411
\end{align}
$$

## Przykłady
Liczba obywateli Polski (stan na ostatni spis powszechny z 2011)

$$
\begin{align}
y{}&=38\ 538\ 447\\
\tilde{y}{}&=38\ 500\ 000\\
\varepsilon{}&=|y-\tilde{y}|=38\ 447\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=9.97627\cdot10^{-4}\approx 0.001
\end{align}
$$

## Przykłady
Obliczanie stałej grawitacji
$$
\begin{align}
y{}&=6.673841\cdot10^{-11}\\
\tilde{y}{}&=6.7\cdot10^{-11}\\
\varepsilon{}&=|y-\tilde{y}|=2.6159\cdot10^{-13}\\
\eta{}&=\frac{|y-\tilde{y}|}{|y|}=0.00391
\end{align}
$$

## Źródła błędów
Błędy powstające przy formułowaniu zagadnienia
- Błędy pomiaru
- Błędy wynikające z przyjęcia określonych przybliżeń opisu zjawisk fizycznych

Błędy powstające przy obliczeniach
- Błędy grube (pomyłki)
- Błędy metody (obcięcia)
- Błędy zaokrągleń

## Błędy grube
- Błąd przy wpisywaniu wzoru do komputera
np. ``x=A/b`` zamiast ``x=A\b``
- Zła implementacja algorytmu
- Niewłaściwa kolejność wykonywania działań

## Błędy metody (obcięcia)
- Błędy obcięcia są nieodłącznym elementem obliczeń numerycznych.
- Błąd obcięcia jest to błąd wynikający z tego, że do uzyskania dokładnego rozwiązania potrzebujemy wykonać nieskończenie wiele obliczeń

## Przykłady błędów metody
Można wykazać, że
$$
\begin{align}
\sin x={}&x-\frac{x^3}{3!}+\frac{x^5}{5!}-\frac{x^7}{7!}+\ldots=\\
={}&\sum\limits_{n=0}^\infty(-1)^n\frac{x^{2n+1}}{(2n+1)!}
\end{align}
$$
Błędem odcięcia będzie
$$
\sin x\approx x-\frac{x^3}{3!}+\frac{x^5}{5!}
$$

## Przykłady błędów metody

Metoda bisekcji

```python
def bisection(f,a,b,N):
    a_n = a
    b_n = b
    for n in range(1,N+1):
        m_n = (a_n + b_n)/2
        f_m_n = f(m_n)
        if f(a_n)*f_m_n < 0:
            a_n = a_n
            b_n = m_n
        elif f(b_n)*f_m_n < 0:
            a_n = m_n
            b_n = b_n
    return (a_n + b_n)/2
```

Szukamy pierwiastka wielomianu $x^2-2$, w przedziale $[1,2]$. Rozwiązanie to $\sqrt{2}$.

```python
f = lambda x: x**2 - 2 # definicja funkcji
bisection(f,1,2,5) # 5 kroków
```

    1.421875

```python
bisection(f,1,2,10) # 10 kroków
```

    1.41455078125

```python
bisection(f,1,2,15) # 15 kroków
```

    1.4141998291015625

```python
import numpy as np
np.sqrt(2)
```

    1.4142135623730951

## Błąd metody - podsumowanie
- Praktycznie wszystkie metody numeryczne mają jakiś błąd metody
- Dobre algorytmy podają jednak jego oszacowanie, w ten sposób wiemy jak daleko jesteśmy od rozwiązania nawet jak przerwiemy obliczenia

Metody Numeryczne 2020/2. Reprezentacja liczb/Reprezentacja liczb.ipynb


# Run-level analyses
In this notebook, we will explain how to aggregate data from different fMRI runs using "run-level analyses". We will use both simulated and real data to explain this concept and show how to implement this in Python as well as FSL.

If you haven't done the other two notebooks (`linux_and_the_CMD.ipynb` and `first_level_analyses.ipynb` yet), please go through these first.

**What you'll learn**: after this lab, you'll be able to ...

* set up a run-level model in FSL FEAT

**Estimated time needed to complete**: 1-2 hours

## Run-level analyses
More often than not, researchers split their experiment across different fMRI *runs* (a period of continuous fMRI acquisition). These runs may be grouped within a particular *session* (a set of MRI scans within a particular period that the participant is in the scanner) or split across different sessions (e.g., run 1 and 2 are done on day 1, and run 3 and 4 are done on day 2).

ToDo (1 point): Name one reason why researchers often rather split their experiment across different runs than having a single (longer) run.

YOUR ANSWER HERE

As discussed before, splitting your experiment in different runs (across multiple sessions) generates a new "level" within data. If we analyzed each run/session in separate first-level models, we need to somehow aggregate (or average) the effects ($c\hat{\beta}$) and their variance ($\mathrm{var}[c\hat{\beta}]$) across these different runs. **If your experiment indeed contains multiple runs, then the next step in the "summary statistics" approach will be to combine their results**.

Before discussing how the summary statistics approach would be implemented for fMRI data with multiple runs, let's focus on how a "traditional" hierarchical/multilevel model for this type of data would look like.

### The traditional multilevel model
In traditional hierarchical/multilevel (frequentists) GLM models, the data ($y$) and design matrices ($\mathbf{X}$) across different levels are simply concatenated. For example, for single-subject fMRI data with multiple runs, the signals ($y$) and design matrices ($\mathbf{X}$) are concatenated in time.

We'll show you how this is done for some example data. For example, suppose we have an experimental paradigm with two conditions ("A" and "B"), which we spread over 4 runs of 200 seconds/volumes (assuming a TR of 1 for simplicity).

We'll generate such a signal below:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
from niedu.utils.nii import simulate_signal  # ignore the warning!

duration = 200
onsets = np.linspace(0, duration, 10, endpoint=False)
n_run = 4
Xs, ys = [], []
for run in range(n_run):  # simulate 4 signals / design matrices
    y, X = simulate_signal(
        onsets=onsets, conditions=np.tile(['A', 'B'], len(onsets) // 2),
        TR=1, duration=duration, params_canon=[2, 0.5], std_noise=2, plot=False, rnd_seed=run
    )
    ys.append(y)
    Xs.append(X[:, :3])  # remove tderivs

print("Number of signals: %i" % len(ys))
print("Size of signal for each run: %i\n" % ys[0].size)

print("Number of design matrices: %i" % len(Xs))
print("Shape of design matrix per run: %s" % (Xs[0].shape,))
```

Let's plot the signals from the separate runs:

```python
plt.figure(figsize=(7, 8))
for i in range(n_run):
    plt.plot(ys[i] + i*10, np.arange(ys[i].size))

plt.ylim(ys[i].size, 0)
plt.xticks(np.arange(n_run) * 10, np.arange(1, n_run+1))
plt.ylabel("Time (sec.)", fontsize=20)
plt.xlabel("Run", fontsize=20)
plt.grid()
plt.show()
```

To create a "traditional" multilevel model, we need to concatenate the signals ($y$) in time:

```python
y = np.concatenate(ys)
print("Size of signal: %i" % y.size)

colors = plt.rcParams['axes.prop_cycle'].by_key()['color']

# Let's plot the concatenated signal (but color-code the runs)
plt.figure(figsize=(3, 16))
for i, ytmp in enumerate(ys):
    t = np.arange(duration * i, duration * (i + 1))
    plt.plot(ytmp, t, c=colors[i])

plt.ylim(y.size, 0)
plt.ylabel("Time (sec.)", fontsize=20)
plt.xlabel("Signal amplitude (A.U.)", fontsize=20)
plt.grid()
plt.show()
```

Similarly, we need to stack the run-wise design matrices, but these are stacked both vertically (in time) and horizontally (as separate predictors). In other words, each predictor (including the intercept!) in each run gets its own column (i.e., predictor) in our multilevel design matrix:

```python
X = np.zeros((len(Xs) * duration, len(Xs) * 3))
for i in range(len(Xs)):
    t = np.arange(duration * i, duration * (i + 1))
    X[t, i*3:(i+1)*3] = Xs[i]

# number of columns = number of conditions * number of runs
print("Shape of concatenated design matrix: %s" % (X.shape,))
```

It's probably easier to understand it if it's plotted. Below, we'll plot the concatenated signal ($y_{all}$) and the concatenated design matrix ($X_{all}$) in a single figure:

```python
fig, axes = plt.subplots(ncols=len(Xs) + 1, figsize=(15, 15), sharey=True, sharex=False)

for i, ytmp in enumerate(ys):
    t = np.arange(duration * i, duration * (i + 1))
    axes[0].plot(ytmp, t, c=colors[i])
    axes[0].set_ylim(duration * n_run, 0)
    axes[0].spines['right'].set_visible(False)
    axes[0].spines['top'].set_visible(False)

    axes[i].grid()

    axes[i+1].set_title(r"$X_{run\ %i}$" % (i+1), fontsize=25)
    for ii in range(3):
        pred = X[:, (i*3)+ii]
        t = np.arange(duration * n_run)
        axes[i+1].plot(pred + (ii*2), t, c=colors[i], lw=2)

    axes[i+1].spines['right'].set_visible(False)
    axes[i+1].spines['top'].set_visible(False)
    axes[i+1].set_xticks([0, 2, 4])
    axes[i+1].set_xticklabels(
        [r'$icept_{%i}$' % (i+1), r'$A_{%i}$' % (i+1),r'$B_{%i}$' % (i+1)], fontsize=15
    )

plt.figtext(0.61, 0.92, r'$X_{all}$', fontsize=30)
axes[0].set_title(r'$y_{all}$', fontsize=30, y=1.045)

axes[0].set_ylabel("Time (volumes)", fontsize=20)
axes[i+1].grid()
fig.show()
```

    ToDo (2 points)

Given the concatenated signal (the variable <tt>y</tt>) and concatenated design matrices (the variable <tt>X</tt>), you can run a single GLM to get the parameters for the different conditions across the four runs (i.e., $icept_{1}$, $icept_{2}$, $A_{1}$, $A_{2}$, $B_{1}$, $B_{2}$, etc.). Do this for our data (using <tt>y</tt> and <tt>X</tt>) and store the estimated parameters (there should be 12) in a new variable named <tt>av_betas</tt>.

Then, you can in fact specify a specific contrast vector that gives you the *average* effect (i.e., $\hat{\beta}_{A}$ or $\hat{\beta}_{B}$). Try to think of which particular contrast (which should also be of size 12) would "compute" the average effect. Create a separate contrast vector for the average effect of A (store this in a variable named <tt>cvec_a</tt>) and the average effect of B (store this in a variable named <tt>cvec_b</tt>).

Hint: you can of course check whether your contrast-vectors in fact yield the mean effect ($\hat{\beta}$) by comparing it to the result when you manually compute the mean!

```python
from numpy.linalg import inv

# Implement the ToDo here!
# YOUR CODE HERE
raise NotImplementedError()
```

```python
from niedu.tests.nii.week_5 import test_ffx_glm
test_ffx_glm(X, y, av_betas, cvec_a, cvec_b, n_run)
```

Now, the previous ToDo only concerned the estimation of the run-average *effects* (and the contrast), but not the estimation of the *variance of the run-average effect*. The exact way this is computed actually depends on whether you want to use a fixed-effect, random-effect, or mixed-effects approach. We will discuss this later in this notebook. First, let's take a look at how fMRI studies usually deal with multilevel data: using the summary statistics approach.

### The summary statistics approach
Most fMRI studies dealing with multilevel data often do not use the traditional multilevel model, but use a slightly different approach: instead of running one single GLM for the concatenated data, it runs a GLM *per* unit within each level and subsequently "averages" the contrast estimates using the "summary statistics" approach. For example, for multi-run data, a first-level model for each separate run is evaluated (instead of a single, concatenated model). This is also the approach FSL takes.

After estimating a first-level model for each run separately, the summary statistics approach uses the results (i.e., $c\hat{\beta}$, representing the "summary statistics") of the previous level as the "data-to-be-explained" (i.e., $y$) within the current level. Using a "run-level" GLM with a new design matrix, the data from the previous level can be aggregated as desired, where the results (i.e., $c\hat{\beta}$) of the *current* level GLM represent the aggregated data. This is both the case for aggregating results across runs (in run-level analyses) and across subjects (in group-level analyses). The global idea of this summary statistics approach is visualized in the figure below.

To put it another way, to aggregate first-level results across runs, we are going to construct another GLM at the "run-level". To distinguish components of run-level GLMs from first-level GLMs, we will add a superscript asterisk ($^{*}$) to the run-level GLM components.

So, in our run-level GLM, the first-level results ($c\hat{\beta}$) become our new dependent variable ($y^{*}$):

\begin{align}
\mathbf{y}^{*} = c\hat{\beta}
\end{align}

which is, similar to first-level GLMs, modeled using a (run-level) GLM with a specific run-level design matrix, $\mathbf{X}^{*}$, and associated run-level parameters, $\hat{\beta}^{*}$:

\begin{align}
\mathbf{y}^{*} = \mathbf{X}^{*}\beta^{*} + \epsilon^{*}
\end{align}

These run-level parameters are usually estimated using OLS:

\begin{align}
\hat{\beta}^{*} = (\mathbf{X}^{*T}\mathbf{X}^{*})^{-1}\mathbf{X}^{*T}\mathbf{y}^{*}
\end{align}

Importantly, the way you specify the run-level design matrix, $\mathbf{X}^{*}$, dictates the way the data is aggregated, which we'll come back to later. For now, it is important to understand that the summary statistics approach entails using the results from the previous level ($c\hat{\beta}$) as the target/dependent variable ($\mathbf{y}^{*}$) in the current level.

In what follows, we'll explain the process using the previously simulated dataset (with two conditions, "A" and "B"). First of all, we need to run our "first-level" models for the four runs separately.

    ToDo (2 points): Within a loop, compute the contrast against baseline for both condition "A" ($\beta_{A} > 0$) and condition "B" ($\beta_{B} > 0$) per run and store these results in the pre-allocated <tt>runwise_cb</tt> array, where the first row should represent the $\beta_{A} > 0$ results and the second row should represent the $\beta_{B} > 0$ results.

Use the variables <tt>Xs</tt> and <tt>ys</tt>, which are both lists of length 4, corresponding to the four runs. Each element in <tt>Xs</tt> contains a $200$ (N) $\times\ 2$ (P) array. Each element in <tt>ys</tt> contains a 1D array of size $200$ (N).

```python
''' Implement your ToDo here. '''
# array has shape: num contrasts x N_runs
runwise_cb = np.zeros((2, len(Xs)))

# Implement your loop here, in which you fill the
# runwise_cb array with the different
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_5 import test_ss_ffx_glm
test_ss_ffx_glm(Xs, ys, runwise_cb)
```

In case you couldn't figure out the previous ToDo, we'll load in the (approximately) correct `runwise_cb` values below, so we can continue the explanation (note that this will overwrite your own answer, and will give an error if you run the test cell above, but during grading we will only use your own implementation).

```python
runwise_cb = np.load('runwise_cb_answer.npy')
```

Okay, so within our summary statistics approach, the $c\hat{\beta}$ values will become our new data ($y^{*}$). For now, let's only focus on the condition "A" against baseline contrast (the first row in `runwise_cb`). Let's redefine this as `y_rl` ("y, run-level"):

```python
y_rl = runwise_cb[0, :]
```

We have defined our dependent variable, but what should our run-level design matrix ($\mathbf{X}^{*}$) be? This, of course, depends on what you are interested in. Often, people simply want to aggregate the results across different runs into a single "average" estimate.

It turns out that using a vector of ones as your design matrix ($\mathbf{X}^{*}$) will yield run-level parameter estimates ($\hat{\beta}^{*}$) that correspond to the average of your first-level results. So, when you use a vector of ones as your run-level design matrix, $\mathbf{X}^{*}$ ...

\begin{align}
\mathbf{X}^{*} = \begin{bmatrix}
             1 \\
             1 \\
             \vdots \\
             1
         \end{bmatrix}
\end{align}

then the parameter of this run-level GLM will correspond to the average of the first-level results ($c\hat{\beta}$):

\begin{align}
\mathrm{average}(c\hat{\beta}) = \hat{\beta}^{*} = (\mathbf{X}^{*T}\mathbf{X}^{*})^{-1}\mathbf{X}^{*T}y^{*}
\end{align}

So, why is this design — in which the average is just a vector of ones for each run — modelling the average of the first-level results? Well, let's investigate this using our previously simulated data. Now, we know that our GLM (with a vector of ones) should give us the mean. Let's calculate the mean 'manually' so that we can check later whether the GLM gives us the same answer.

```python
manual_mean = np.mean(y_rl)
print(manual_mean)
```

ToDo (1 point): Create a design-matrix of shape $(4, 1)$, in which the single column contains all ones (you can use the <tt>np.ones</tt> function for this!). Then run linear regression using the variable <tt>y_rl</tt> as our target and the using the design-matrix you just created (representing the "$\mathbf{X}$"). Store the resulting parameter-value in a variable named <tt>glm_mean</tt>.

```python
''' Implement your ToDo here.'''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo.'''
if isinstance(glm_mean, np.ndarray):
    ans = glm_mean[0]
else:
    ans = glm_mean

np.testing.assert_almost_equal(ans, manual_mean)
print("Well done!")
```

If you've done the previous ToDo correctly, you've seen that, as expected, the parameter calculated by the GLM when using a predictor with all ones (a run-level intercept, basically) reflects the mean of the dependent-variable. In fact, in the context of the GLM — which aims to minimize the residuals between the predictor(s) and the target — this makes sense! We show you this in the plot below:

```python
rnge = np.max(y_rl) - np.min(y_rl)
fig, axes = plt.subplots(ncols=2, figsize=(12, 11), sharey=True, sharex=True)
for i, ax in enumerate(axes):
    ax.plot(y_rl, np.arange(1, y_rl.size+1), marker='o', ms=12, lw=2)
    ax.plot(np.ones(n_run), np.arange(1, n_run+1), ls='--', lw=2)
    ax.legend([r'$Y^{*}$', r'$X^{*}$'], fontsize=15, frameon=False)
    ax.set_yticks(np.arange(1, n_run+1)[::-1])
    ax.set_yticklabels(np.arange(1, n_run + 1), fontsize=20)
    ax.grid()
    ax.set_xlim(0, np.max(y_rl) + 0.3 * rnge)
    ax.set_xlabel(r"Contrast-values ($c\beta$)", fontsize=20)

    if i == 0:
        ax.set_ylabel(r"Runs", fontsize=20)
        ax.set_title("Run-level GLM model", fontsize=25)
    else:
        ax.set_title("Run-level *fitted* GLM model", fontsize=25)
        ax.plot(np.repeat(manual_mean, n_run), np.arange(1, n_run+1), c='tab:orange', lw=3)
        for i in range(n_run):
            ax.plot((y_rl[i], manual_mean), (i+1, i+1), c='tab:red', ls='--', lw=3)

        ax.annotate(text='', xy=(0, 2.5), xytext=(manual_mean, 2.5), arrowprops=dict(arrowstyle='<->', lw=4))
        ax.text(0.5, 2.3, r'$\hat{\beta}^{*}$', fontsize=30)
        ax.legend([r'$Y^{*}$', r'$X^{*}$', r'$\hat{Y}^{*} = \hat{\beta}^{*}$', r'$residuals^{*}$'],
                  fontsize=15, frameon=False, loc='lower left')

fig.tight_layout()
```

From the plot above, you can nicely see that the vector of ones nicely models the mean of the dependent variable (the first-level $\beta_{A} > 0$ contrasts). Just like in first-level models, we can specify particular **run-level** contrasts. Implicitly, we used a **run-level** contrast against baseline here (because for our single run-level predictor, when $c^{*} = [1]$, then $c\hat{\beta}^{*} = \hat{\beta}^{*}$), but other contrasts are definitely possible (we'll take a look at that later).

But what if we want to test more than one *first-level* contrast, eventually, at the group-level? For example, suppose we also want to test $\beta_{B} > 0$. Should we then just run a *separate* run-level analysis for each first-level contrast? You surely can, but often it's easier to do it in a single model in which we concatenate the contrast-values from the different contrasts. We do this below:

```python
# order: A1, A2, A3, A4, B1, B2, B3, B4
y_AB = np.concatenate([runwise_cb[0, :], runwise_cb[1, :]])

plt.figure(figsize=(8, 15))
plt.plot(runwise_cb[0, :], np.arange(n_run), marker='o', ms=12, lw=2)
plt.plot(runwise_cb[1, :], np.arange(n_run, n_run * 2), c='tab:green', marker='o', ms=12, lw=2)
plt.yticks(np.arange(n_run * 2), ['Run ' + str(i) for i in np.tile(range(1, n_run+1), 2)], fontsize=15)
plt.xlim(-1, 5)
plt.ylabel("Runs", fontsize=20)
plt.xlabel(r"Contrast-values ($c\hat{\beta}$)", fontsize=20)
plt.title("Concatenated contrast-values, A and B", fontsize=25)
plt.axvline(0, c='black', ls='--', lw=0.5)
plt.legend([r'$Y_{A}$', r'$Y_{B}$'], fontsize=20)
plt.gca().invert_yaxis()
plt.grid()
plt.show()
```

    ToDo (2 points): Now, given the concatenated data (the variable <tt>y_AB</tt>), can you come up with a design-matrix ($X$) that models both the mean of the contrast-values of A-against-baseline (first four values) and the contrast-values of B-against-baseline (last four values)? Name your design-matrix <tt>X_concat_data</tt> (hint: it needs two columns), which should be a 2D numpy array. Then, run linear regression on the concatenated data using your design-matrix (<tt>X_concat_data</tt>); you can check your answer by verifying that the two parameters from the run-level regression model are the same as the mean across the columns of the <tt>runwise_cb</tt> array.

```python
''' Implement your ToDo here. '''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the above ToDo. '''
from niedu.tests.nii.week_5 import test_concat_design
test_concat_design(n_run, X_concat_data)
```

    ToDo (1 point): Right now, we only have the first-level contrasts against baseline to our disposal ($\beta_{A} > 0$ and $\beta_{B} > 0$). But suppose that I'm actually interested to see which voxels significantly activate in response to both stimuli across runs, i.e., $(\beta_{A} > 0)\ \&\ (\beta_{B} > 0)$. We can actually specify a particular run-level contrast that tests this. Create an array (with size 2) that implements this contrast vector, and store it in a variable named cvec_A_and_B.

```python
''' Implement your ToDo here. '''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the ToDo above. '''
from niedu.tests.nii.week_5 import test_cvec_A_and_B
test_cvec_A_and_B(cvec_A_and_B)
```

    ToDo (optional, bonus point): Suppose that, for some reason, I'm interested in investigating whether there are voxels that respond more strongly to condition "A" in the first two sessions than the last two sessions. Given our first-level results (stored in <tt>y_AB</tt>), how should my run-level design matrix ($\mathbf{X}_{rl}$) and run-level contrast vector look like? For a bonus point, create this matrix and contrast vector below, run linear regression on the run-level data (<tt>y_AB</tt>) using your design matrix, and compute the run-level contrast using your contrast vector. Store the result of this run-level contrast (i.e., a single nubmer) in a variable named <tt>runlevel_cb_optional</tt>.

```python
''' Implement the (optional) ToDo here. '''
# YOUR CODE HERE
raise NotImplementedError()
```

```python
''' Tests the ToDo above. '''
from niedu.tests.nii.week_5 import test_runlevel_cb_optional
test_runlevel_cb_optional(runlevel_cb_optional, y_AB)
```

### Fixed vs. random vs. mixed effects
Remember that we told you that hierarchical data, such as multi-subject fMRI data, is often analyzed with mixed models which compute the variance (uncertainty) of the effects using the variance estimates at each level within your model. This specific type of analysis, in which the final variance estimate is computed using variance components across all levels, is often called a **mixed-effects analysis**.

In the run-level models so far, though, we only focused on the run-level *effects* ($c\hat{\beta}_{rl}$), not their variance! In fact, for run-level analyses, it is common to "ignore" the run-level variance and simply average the first-level variance estimates across runs. In other words, this is treating the effect as "fixed" across runs and assuming the variance per run is similar (so that these variance estimates can be average). This specific analysis, in which the effect is assumed to be fixed across observations within an analysis level (e.g., across runs), is often called a **fixed-effects analysis**. Just like the mixed-effects analysis, they are all (variations on) the GLM with specific ways in which they estimate the variance component of our effects.

In the context of analysis of hierarchical fMRI data, **fixed-effects analyses** at the run-level are completely reasonable (at least, that's what most people do in practice). However, this is not true for the next level in our summary statistics approach: the group-level. Here, you actually need to incorporate the variance component from the group-level in order to draw valid population inferences. When you incorporate this group-level variance component, but ignore the lower-level variance components (i.e., at the first-level or run-level), this type of analysis is not a full mixed-effects analysis, but is often called a **random-effects analysis**. This is the approach that SPM takes (which is fine, given some assumptions). FSL, on the other hand, offers both random-effects as well as a full mixed-effects option, in which the latter is estimated by incorporating both first-level (or run-level) and group-level variance estimates (which approximates the "proper" mixed model without the summary statistics procedure).

But let's not get distracted (group-level models is the topic of next week!) and continue with this notebook's last section on how to implement run-level (fixed-effects) analyses in FSL.

### Run-level analyses in FSL
In FSL, any analysis that is not at the first-level is called a "higher-level analysis". In this section, we are going to set up such a higher-level analysis with the goal to average our first-level `flocBLOCKED` results. In particular, we are going to do this specifically for the following contrasts:

* $4\times\beta_{\mathbf{face}} - \beta_{body} - \beta_{place} - \beta_{character} - \beta_{object} > 0$ contrast (this was contrast number 4 in our original first-level analysis)
* $4\times\beta_{\mathbf{place}} - \beta_{body} - \beta_{face} - \beta_{character} - \beta_{object} > 0$ contrast (this was contrast number 5 in our original first-level analysis).

ToDo (not graded): Open FEAT and change the type of analysis from "First-level analysis" to "Higher-level analysis".

You should see that the "Pre-stats" and "Registration" tabs become unavailable, as FEAT assumes these things have been done in the first-level analyses.

ToDo (not graded): First, go to the "Stats" tab and change the analysis-type from "Mixed effects: FLAME 1" to "Fixed effects".

Now, let's tell FEAT which data we want to analyze. For the upcoming ToDos, we're going to average the "face > other" and "place > other" contrasts. The two first-level analyses (one for each run) have been run already and are located here:

```python
import os
data_dir = os.path.join(os.path.expanduser("~"), 'NI-edu-data')
print("Starting download of FEAT directory (+- 287MB) ...\n")
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/fsl/sub-03/flocBLOCKED/ses-2.feat/*"
print("\nDone!")
```

```python
feat_r1 = os.path.join(data_dir, 'derivatives', 'fsl', 'sub-03', 'flocBLOCKED', 'ses-1.feat')
print("The run 1 results are here: %s" % feat_r1)
feat_r2 = os.path.join(data_dir, 'derivatives', 'fsl', 'sub-03', 'flocBLOCKED', 'ses-2.feat')
print("The run 2 results are here: %s" % feat_r2)
```

ToDo (not graded):

Change the setting "Inputs are lower-level FEAT directories" to "Inputs are 3D cope images from FEAT directories". Set the "Number of inputs" to 4. Then, press the "Select cope images" button and add the different first-level contrast estimate files. The order should be: "face > other" (session 1), "face > other" (session 2), "place > other" (session 1), and "place > other" (session 2):

\begin{align}
y^{*} =
\begin{bmatrix}
c\hat{\beta}_{\mathrm{face>other,\ session\ 1}} \\
c\hat{\beta}_{\mathrm{face>other,\ session\ 2}} \\
c\hat{\beta}_{\mathrm{place>other,\ session\ 1}} \\
c\hat{\beta}_{\mathrm{place>other,\ session\ 2}} \\
\end{bmatrix}
\end{align}

Check the ToDo in section 4.1.4. to see which contrast ("COPE") numbers these first-level contrasts refer to.
Set the "Output directory" to the "week 5" folder. And, under the "Post-stats" tab, set the thresholding to "Uncorrected" at a threshold at $p < 0.005$.

Note that, instead of assuming that "Inputs are 3D cope images from FEAT directions", you could use the other option "Inputs are lower-level FEAT directories". This option allows you to apply a particular higher-level design ($\mathbf{X}^{*}$) and higher-level contrasts ($\mathbf{c}^{*}$) for *all first-level contrasts* at the same time. Here, we don't use this option as it is less clear what is happening "under the hood".

Assignment (2 points)

In the "Stats" tab, click on the "Full model setup" button. Now, you can directly specify your design matrix ($\mathbf{X}^{*}$) here in the "EVs" tab. Here, create a design matrix that allow you to average the first-level "face > other" and "place > other" contrasts within a (single) run-level GLM (hint: you might need to change the "Number of main EVs"). Then, go to the "Contrasts and F-tests" tab and create the contrasts that you need in order to average the "face > other" first-level contrast across runs and the "place > other" first-level contrast across runs. Give the EVs and contrasts sensible names.

When you're happy with your setup, save your setup in your <tt>week_5</tt> directory; give it the name <tt>setup_feat_runlevel</tt>. Then, add the following line to the text-cell below:

``

After adding this line and running the cell, the run-level design should be visible. If it is not, you probably saved the design in the wrong directory.

YOUR ANSWER HERE

## Checking out results from run-level FEAT analyses
We actually already ran the run-level analysis, of which the results are| downloaded in the cell below:

```python
print("Starting download of runlevel FEAT directory (+- 40MB) ...\n")
!aws s3 sync --no-sign-request s3://openneuro.org/ds003965 {data_dir} --exclude "*" --include "derivatives/fsl/sub-03/flocBLOCKED/runlevel_Week5.gfeat/*"
print("\nDone!")

runlevel_dir = os.path.join(data_dir, 'derivatives', 'fsl', 'sub-03', 'flocBLOCKED', 'runlevel_Week5.gfeat')
print(runlevel_dir)
```

    ToDo: Open a terminal and navigate to the above <tt>runlevel_Week5.gfeat</tt> directory (using <tt>cd</tt>). Then, print its contents to the terminal window (using <tt>ls</tt>)!

The `gfeat` suffix stands for "group FEAT" and is appended to each higher-level analysis output directory (i.e., any analysis that is not a first-level analysis).

In this `gfeat` directory, you should see similar files as you've seen in first-level FEAT directories: HTML-files with analysis summaries, files with design information, etc.

    ToDo (ungraded): In your terminal, open the <tt>report.html</tt> file in the <tt>gfeat</tt> directory with Firefox, which should open a new Firefox window. Then, click on the "Registration summary" tab.

The registration tab conveniently shows us the "functional → standard space" registration results again, but now for all four runs at the same time. Here you can easily check for possible large registration differences between runs/sessions. Note that this particular transformation is only executed in higher-level analyses. While the (linear and non-linear) registration parameters are already *computed* in the first-level analysis, they are *applied* in the higher-level analysis.

    ToThink (ungraded): Can you think of a reason why spatial resampling to standard space is postponed to the higher-level stage?

If you scroll further down, you see a figure with the caption "Sum of all input masks after transformation to standard space". This shows you how well the functional brain masks (delineating brain from non-brain matter, i.e., the skullstripping result for the different functional files) align. It uses a red-yellow colormap from 1 (red) to 4 (yellow), showing where all masks overlap (yellow) and where this is not the case (relatively red voxels). There are some voxels not included in all masks in orbitofrontal cortex (visible in the first row of brain slices), which might be caused by different amounts of signal dropout across runs, but that isn't much of a problem.

The figure below ("Unique missing-mask voxels") is a similar quality check, where it plots voxels missing in precisely *one* mask, which allows you to easily spot whether a single registration did not work as expected. Here, everything looks alright.

    ToDo (ungraded): Click on the "Results" tab. Note that the image of the design matrix is missing, because we have deleted this (as it would give away the answer to the previous assignment). Now, click on the "Higher-level FEAT results" link, which will open the "post-stats" tab of the higher-level FEAT analysis. Here, you can see two figures, showing the (thresholded) results of our run-level contrasts ($c\hat{\beta}^{*}$).

Within the `gfeat1` directory, of particular interest is the `cope1.feat` directory. This subdirectory contains the actual *results* of the run-level analysis. The reason this directory is called `cope1.feat` is because the other input option ("Inputs are lower-level FEAT directories") allows you to apply the higher-level design to multiple lower-level contrasts at the same time, which would create separate `cope*.feat` subdirectories (`cope1.feat`, `cope2.feat`, `cope3.feat`, etc.). The "Inputs are 3D cope images from FEAT directories" option only creates a single directory which is by default called `cope1.feat`.

    ToDo (ungraded): In your terminal, navigate into the <tt>cope1.feat</tt> directory and print its contents to the terminal window.

Here, you see that this `cope1.feat` subdirectory has the exact same structure and files as the first-level `.feat` directories that we inspected earlier!

Let's view the results in FSLeyes.

    ToDo (ungraded): Close any active overlays (Overlay → Remove all). Then, click on File → Add standard → click on <tt>MNI152_T1_2mm_brain.nii.gz</tt>, which is the standard space image that FSL used as the group template. Then, add the <tt>thresh_zstat1.nii.gz</tt> image from the <tt>cope1.feat</tt> directory (i.e., <tt>runlevel.gfeat/cope1.feat/thresh_zstat1.nii.gz</tt>). This file corresponds to the (thresholded) fixed-effects "average" effect of our lower-level $\beta_{face} > 0$ contrasts.

Change the colormap to "Red-Yellow".

Viewing results from run-level analyses is not fundamentally different from first-level analyses, with one exception: the data is now in "standard space" (i.e., MNI152, 2mm space). This allows us to use *atlases* to connect the location of our effects to particular anatomical brain regions.

    ToDo (ungraded): In the top menu, click on Settings → Ortho View 1 → Atlas panel. This should open a new panel in between the Overlay list and the Location panel.

The Atlases panel by default has to to atlases loaded: the [Harvard-Oxford Cortical Structural atlas](https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/Atlases) and the Harvard-Oxford Subcortical Structural atlas, which are based on the individual segmentations of 37 T1-weighted scans.

Importantly, these atlases are *probabilistic*, meaning that, for each voxel, they give a *probability* (or, actually, percentage) of belonging to a particular (set of) region(s). The exact percentage is based on the number of people (out of 37) in which that voxel belonged to that region. So, if, for a particular voxel, the atlas shows you "82% left amygdala", that means that that voxel was part of the left amygdala for 81% of those 37 people.

Consequently, most voxels actually have multiple labels (e.g., 81% left amygdala, 19% left hippocampus).

    ToDo (ungraded): Go to voxel $[25, 39, 25]$ and check out the regions listed in the atlas panel. Do you expect to find this region for this particular contrast/analysis?

    Tip!
    Before handing in your notebooks, we recommend restarting your kernel (Kernel → Restart & Clear Ouput) and running all your cells again (manually, or by Cell → Run all). By running all your cells one by one (from "top" to "bottom" of the notebook), you may spot potential errors that are caused by accidentally overwriting your variables or running your cells out of order (e.g., defining the variable 'x' in cell 28 which you then use in cell 15).