Table of Contents

1  Response to Thomas Icard's question about counterfactual implementation in probability trees.

1.1  Probability tree for the counterfactual example

1.2  Reference tree

1.3  Indicative tree

1.4  Subjunctive tree

1.5  Composed tree

1.6  Counterfactual tree

2  Can probability trees calculate the Natural Direct Effect?

3  Context-specific conditional independence with Probability trees

3.1  What is the probability tree associated with $E[Y]$?

3.2  What is the probability tree associated with $E[Z(x=1)]$?

3.3  What is the probability tree associated with $E[Y(x=0)]$?

3.4  What is the probability tree associated with $E[Y(x=0,Z(x=1)=1]$?

4  Zenna's question about the intuition of intervention on probability trees

5  Alex's question about intervening on $X=1$ or  $X=2$

6  Probabilistic truth versus logical truth

6.1  Alex Lew's first model

6.1.1  Probability tree of model

6.1.2  Intervention at $L=1$

6.2  Alex's second model

6.2.1  Probability tree of second model

6.2.2  Intervention of $L=1$ on second model

7  Independent choice semantics vs program trace semantics

7.1  James Koppel's example 1

7.2  Trace semantics probability tree

7.3  Independent choice semantics probability tree

7.4  Trace Semantics Petri net

8  Example involving conditioning and counterfactual on a graph with dynamic dependencies and interesting conditioning dependencies

8.1  Factual world Probability Tree mechanism for $Z=0$

8.2  Conditioned on $Z=0$

8.3  Intervene on $Z=1$

8.4  Counterfactual

8.5

```python

import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from probability_trees import MinCut, Critical, PTree
from IPython.display import Latex
```

# Response to Thomas Icard's question about counterfactual implementation in probability trees.

During the discussion, Thomas questioned whether "the operations defined for probability trees can represent genuine counterfactuals." In particular, he pointed out that simply composing `see()` and `do()` does not result in counterfactual inference.

For example, if you condition on $Z=1$ and then ask what would happen to $Z$ if $Y\leftarrow 1$, then $P(Z=0|Z=1,Y\leftarrow 1) = 0$.  However, the counterfactual method does not simply compose seeing and doing. It also splices together the referenced tree and the composed tree.  I will demonstrate the answer to this question using both figures and code.

## Probability tree for the counterfactual example

Note that

```python
def icard_tree( bvar ):
    if 'X' not in bvar:
        return [(0.5, 'X=0'),
                (0.5, 'X=1')]
    if bvar['X'] == '0':
        if 'Y' not in bvar:
            return [(0.25, 'Y=0'),
                    (0.75, 'Y=1')]
        if 'Z' not in bvar:
            if bvar['Y'] == '1':
                return  [(0.75, 'Z=0'),
                         (0.25, 'Z=1')]
    if 'Z' not in bvar:
        return [(0.25, 'Z=0'),
                (0.75, 'Z=1')]
    if 'Y' not in bvar:
        if bvar['Z'] == '0':
            return [(0.5, 'Y=0'),
                    (0.5, 'Y=1')]
        else:
            return [(0.75, 'Y=0'),
                    (0.25, 'Y=1')]
icard = PTree.fromFunc(icard_tree)
```

```python
icard = PTree()

# Set the root node and its sub-nodes.
icard.root('O = 1', [
    icard.child(0.5, 'X = 0', [
        icard.child(0.25, 'Y = 0'),
        icard.child(0.75, 'Y = 1',
                 [icard.child(0.75, 'Z = 0'),
                  icard.child(0.25, 'Z = 1')])
    ]),
    icard.child(0.5, 'X = 1', [
        icard.child(0.25, 'Z = 0',
                 [icard.child(0.5, 'Y = 0'),
                  icard.child(0.5, 'Y = 1')]),
        icard.child(0.75, 'Z = 1',
                 [icard.child(0.75, 'Y = 0'),
                  icard.child(0.25, 'Y = 1')])
    ])
])
```

    <probability_trees.Node at 0x7ff130e381d0>

## Reference tree
We want to know the probability that $Z=0$ before conditioning or intervention:

```python
Y1 = icard.prop('Y=1')
Z1 = icard.prop('Z=1')
Z0 = icard.prop('Z=0')
crit = icard.critical(Z0)
display(Latex(f'$$P(Z=0)={icard.prob(Z0)}$$'))
icard.show(show_prob=True,
           cut=Z0,
           crit=crit)
```

$$P(Z=0)=0.4375$$

## Indicative tree

We want to know the probability that $Z=0$ given that $Z=1$.

```python
see_Z1 = icard.see(Z1)
display(Latex(f'$$P(Z=0|Z=1)={see_Z1.prob(Z0)}$$'))
see_Z1.show(show_prob=True,
           cut=Z0,
           crit=crit)
```

$$P(Z=0|Z=1)=0.0$$

## Subjunctive tree

We want to know the probability that $Z=0$ given that $Y\leftarrow 1$.

```python
do_Y1 = icard.do(Y1)
display(Latex(f'$$P(Z=0|Y\leftarrow 1)={do_Y1.prob(Z0)}$$'))
do_Y1.show(show_prob=True,
           cut=Z0,
           crit=crit)
```

$$P(Z=0|Y\leftarrow 1)=0.5$$

## Composed tree

We want to know the probability that $Z=0$ given that $Z= 1$ and $Y\leftarrow 1$.

```python
see_Z1_do_Y1 = icard.see(Z1).do(Y1)
display(Latex(f'$$P(Z=0|Z=1,Y\leftarrow 1)={see_Z1_do_Y1.prob(Z0)}$$'))
see_Z1_do_Y1.show(show_prob=True,
           cut=Z0,
           crit=crit)
```

$$P(Z=0|Z=1,Y\leftarrow 1)=0.0$$

## Counterfactual tree

We want to know the probability that $Z_{Y\leftarrow 1}=0$ given that $Z=1$ and $Y\leftarrow 1$.

```python
see_Z1= icard.see(Z1)
counterfactual = icard.cf(see_Z1, Y1)
display(Latex('$$P(Z_{Y\leftarrow 1}=0|Z=1,Y\leftarrow 1)='
       f'{counterfactual.prob(Z0)}$$'))
counterfactual.show(show_prob=True,
           cut=Z0,
           crit=crit)
```

$$P(Z_{Y\leftarrow 1}=0|Z=1,Y\leftarrow 1)=0.25$$

This figure shows how the counterfactual tree was obtained:

The example shows a counterfactual probability tree generated by imposing $Y
\leftarrow 1$, given the factual premise $Z = 1$. Starting from a **reference
probability tree**, we first derive two additional trees: a **indicative tree**,
capturing the observations made in the factual world; and a **subjunctive tree**,
represented as an intervention on the **reference tree**.

To form the counterfactual we proceed as follows:
-   We slice both derived trees along the critical set
of the counterfactual premise.
-   Then, we compose the counterfactual tree by
taking the transition probabilities **upstream of the slice**
from the factual premise, and those **downstream of the slice**
from the counterfactual premise.

The events downstream then span a new scope containing copies
of the original random variables (marked with "∗"), ready to
adopt new values.

In particular note that $Z^\ast = 0$ can happen in our alternate
reality, even though we know that $Z = 1$.

# Can probability trees calculate the Natural Direct Effect?

$$DE_{x,x'}(Y) = E[Y(x',Z(x))] - E[Y(x)]$$

Here, $Z$ represents all parents of $Y$ excluding $X$, and the expression $Y(x′, Z(x))$ represents the value that $Y$ would attain under the operation of setting $X$ to $x′$ and, simultaneously, setting $Z$ to whatever value it would have obtained under the setting $X = x$. We see that $DE_{x,x′}(Y)$, the natural direct effect of the transition from $x$ to $x′$, involves probabilities of nested counterfactuals and cannot be written in terms of the $do(x)$ operator. Therefore, the natural direct effect cannot in general be identified, even with the help of ideal, controlled experiments.

Pearl, Judea. Causality (p. 131). Cambridge University Press. Kindle Edition.

we test this with the following SCM:

\begin{align}
x  = & bernoulli(0.75) \\
z = & x \\
y  = & x + z \\
\end{align}

So we

```python
def natural_direct_effect(bvar):
    if 'X' not in bvar:
        return [(0.25, 'X=0'),
                (0.75, 'X=1')]
    if 'Z' not in bvar:
        return z_if_x( bvar['X'])
    if 'Y' not in bvar:
        return y_if_x_plus_z( bvar['X'], bvar['Z'])
    return None

def y_if_x_plus_z( x, z ):
    if (x == '0') and (z == '0'):
        return [(1, 'Y=0'),
                (0, 'Y=1'),
                (0, 'Y=2')]
    elif (x == '1') and (z == '1'):
        return [(0, 'Y=0'),
                (0, 'Y=1'),
                (1, 'Y=2')]
    else:
        return [(0, 'Y=0'),
                (1, 'Y=1'),
                (0, 'Y=2')]

def z_if_x( x ):
    if x == '1':
        return [(0, 'Z=0'),
                (1, 'Z=1')]
    else:
        return [(1, 'Z=0'),
                (0, 'Z=1')]

nde = PTree.fromFunc(natural_direct_effect)
Y1 = nde.prop('Y=1')
X1 = nde.prop('X=1')
Z1 = nde.prop('Z=1')
X0 = nde.prop('X=0')
Y0 = nde.prop('Y=0')
Y2 = nde.prop('Y=2')
Z0 = nde.prop('Z=0')
```

# Context-specific conditional independence with Probability trees

```python

```

## What is the probability tree associated with $E[Y]$?

```python
display(Latex(f'$$E[Y]={nde.expect("Y")}$$'))
display(nde.show(show_prob=True, cut=Y0 | Y2, crit=nde.critical(Y0 | Y2)))
```

$$E[Y]=1.5$$

## What is the probability tree associated with $E[Z(x=1)]$?

```python
do_X = nde.do(X1)

display(Latex(f'$$E[Z(x)]={do_X.expect("Z")}$$'))
display(do_X.show(show_prob=True, cut=do_X.prop('Z=1'), crit=do_X.critical(do_X.prop('Z=1'))))
```

$$E[Z(x)]=1.0$$

## What is the probability tree associated with $E[Y(x=0)]$?

```python
do_notX = nde.do(X0)
display(Latex(f"$$E[Y(x=0)]={do_notX.expect('Y')}$$"))
display(do_notX.show(show_prob=True, cut=Y1, crit=do_notX.critical(Y1)))
```

$$E[Y(x=0)]=0.0$$

## What is the probability tree associated with $E[Y(x=0,Z(x=1)=1]$?

```python

do_notX_and_Z = nde.do( X0 & Z1 )
display(Latex(f"$$E[Y(x=0,Z(x=1))]={do_notX_and_Z.expect('Y')}$$"))
display(do_notX_and_Z.show(show_prob=True, cut=Y1, crit=do_notX_and_Z.critical(Y1)))
```

$$E[Y(x=0,Z(x=1))]=1.0$$

```python
see_Z1_do_X0 = nde.cf(nde.see(X1), X0)
display(Latex(f"$$E[Y(x',Z(x))]={see_Z1_do_X0.expect('Y')}$$"))
display(see_Z1_do_X0.show(show_prob=True, cut=Y1, crit=see_Z1_do_X0.critical(Y1)))
```

$$E[Y(x',Z(x))]=0.0$$

# Zenna's question about the intuition of intervention on probability trees

Zenna noted that the `see()` operator affects both probabilities upstream of the critical node and probabilities downstream of the critical node, whereas `do()` only affects probabilities downstream of the critical node, and asked why this makes sense.

This is the intuition I have:  Conditioning is about information flow. It asks, if I tell you information about $A$, does that give me new information about $B$?  In this case, $A$ could be upstream or downstream of $B$ and you could still gain information.  Intervening is about causality.  It asks, if I alter the barometer, will the weather change?   If the answer is yes, then the barometer weather weather. If no, then the barometer does not cause weather.

# Alex's question about intervening on $X=1$ or  $X=2$

```python
pt = PTree()
pt.root(
    'O = 1',
    [pt.child(0.3, 'X = 1'),
     pt.child(0.5, 'X = 2'),
     pt.child(0.2, 'X = 3')])

# Let's pick the negative event for our minimal prob tree.
cut = pt.prop('X = 1') | pt.prop('X=2')
crit = pt.critical(cut)

# Intervene.
pt_do = pt.do(cut)

# Show results.
print('Before the intervention:')
display(pt.show(cut=cut, crit=crit))
print('After the invention on "X <- 1 or X <- 2":')
display(pt_do.show(cut=cut, crit=crit))
```

    Before the intervention:

    After the invention on "X <- 1 or X <- 2":

Pearl also doesn't want to deal with non-atomic interventions, but allows for higher-order `replace()` interventions:

      We will confine our attention to actions in the form of do(X = x). Conditional actions of the form “$do(X = x)$ if $Z = z$” can be formalized using the replacement of equations by functions of $Z$, rather than by constants. We will not consider disjunctive actions of the form "$do(X = x \lor Z = z),$" since these complicate the probabilistic treatment of counterfactuals.

      -- Pearl, Judea. Causality (p. 204). Cambridge University Press. Kindle Edition.

But any disjunction of actions can be converted to a negative conjunction of negative actions:

```python
equivalent_cut = ~(~pt.prop('X=1') & ~pt.prop('X=2'))
equivalent_crit = pt.critical(equivalent_cut)

print('After the invention on "not (not X <- 1) and (not X <- 2)":')
display(pt.do(equivalent_cut).show(cut=equivalent_cut, crit=equivalent_crit))
```

    After the invention on "not (not X <- 1) and (not X <- 2)":

# Probabilistic truth versus logical truth
In [section 2.2.3](https://nbviewer.jupyter.org/github/COVID-19-Causal-Reasoning/probability_trees/blob/main/Causal_Reasoning_in_Probability_Trees.ipynb#Special-case:-probabilistic-truth-versus-logical-truth) of the Probability tree tutorial, the distinction between logical truth and probabilistic truth is made:

Let's have a look at one special case. Our definitions make a distinction
between **logical** and **probabilistic truth**. This is best seen in the
example below.

In this example, we have a probability tree with three outcomes: $X = 1, 2$, and
$3$. - $X = 1$ occurs with probability one.
- Hence, probabilistically, the event $X=1$ is resolved at the level of the
root node.
- However, it isn't resolved at the logical level, since $X = 2$ or $X = 3$
can happen logically, although with probability zero.

Distinguishing between logical truth and probabilistic truth is important for
stating counterfactuals. This will become clearer later.

```python
# First we add all the nodes.
pt = PTree()
pt.root('O = 1',
        [pt.child(1, 'X = 1'),
         pt.child(0, 'X = 2'),
         pt.child(0, 'X = 3')])

# Show the cut for 'X = 0'
cut = pt.prop('X = 1')
print('While the root node "O=1" does resolve the event "X=1"\n' +
      'probabilistically, it does not resolve the event logically.')
display(pt.show(cut=cut))
```

    While the root node "O=1" does resolve the event "X=1"
    probabilistically, it does not resolve the event logically.

## Alex Lew's first model

Say  $x_i$ is $1$ if there is an item present at position $i$ in the list, and $0$ otherwise. Situations like $x_1=0, x_2=1$ have probability mass $0$.

Let $L$ be the length of the list. Each choice of the $x_i$ with nonzero probability  determines a setting of $L$. But it is possible to intervene on $L$ without intervening on the $x_i$.

-- Alex Lew

```python
def L_logical_truth(bvar):
    if 'x1' not in bvar:
        return [(0.5, 'x1=0'),
                (0.5, 'x1=1')]
    if 'x2' not in bvar:
        if bvar['x1'] == '1':
            return [(0.5, 'x2=0'),
                    (0.5, 'x2=1')]
    if 'L' not in bvar:
        if bvar['x1'] == '0':
            return [(1, 'L=0')]
        if bvar['x2'] == '0':
            return [(1,'L=1')]
        if bvar['x2'] == '1':
            return [(1, 'L=2')]
    return None
```

```python

```

### Probability tree of model

```python
only_L = PTree.fromFunc(L_logical_truth)
what_if_L_was_1 = only_L.prop('L=1')
crit_L = only_L.critical(what_if_L_was_1)
show_L = only_L.show(show_prob=True, cut=what_if_L_was_1, crit=crit_L)
display(show_L)
```

### Intervention at $L=1$

I don’t see how to intervene on $L = 1$ without forcing us into the path where $x_1 = 1$ and $x_2 = 0$ here. (It seems like "$x_2 = 1$” and “$x_1 = 0$" are both in the false min-cut for the “$L = 1$” event.)

```python
do_only_L = only_L.do(what_if_L_was_1)
show_do_only_L = do_only_L.show(show_prob=True,
                                cut=what_if_L_was_1,
                                crit=do_only_L.critical(what_if_L_was_1))
display(show_do_only_L)
```

## Alex's second model

But maybe the problem is that I need to draw “$L = 0$,” “$L = 1,$” and “$L = 2$” as siblings at every node where $L$ is being assigned? I don’t see anything in the probability tree definition that requires this, but if it’s the case, then I can see how their notion of “intervention” corresponds more closely to the usual one. Maybe the right way to think about probability trees as causal models is that the edges represent ‘possible mechanisms of intervention,’ and by failing to draw outgoing edges to the alternative “$L = *$” cases, I am encoding that intervention changing $L$ but not $x_i$ is impossible.

```python
def L_probabilistic_truth(bvar):
    if 'x1' not in bvar:
        return [(0.5, 'x1=0'),
                (0.5, 'x1=1')]
    if 'x2' not in bvar:
        if bvar['x1'] == '1':
            return [(0.5, 'x2=0'),
                    (0.5, 'x2=1')]
    if 'L' not in bvar:
        if bvar['x1'] == '0':
            return [(1, 'L=0'),
                    (0, 'L=1'),
                    (0, 'L=2')]
        if bvar['x2'] == '0':
            return [(0, 'L=0'),
                    (1,'L=1'),
                    (0, 'L=2' )]
        if bvar['x2'] == '1':
            return [(0, 'L=0'),
                    (0, 'L=1'),
                    (1, 'L=2')]
    return None

```

### Probability tree of second model

```python
with_L = PTree.fromFunc(L_probabilistic_truth)
what_if_L_was_1 = with_L.prop('L=1')
crit_L = with_L.critical(what_if_L_was_1)
show_L = with_L.show(show_prob=True, cut=what_if_L_was_1, crit=crit_L)
display(show_L)
```

### Intervention of $L=1$ on second model

```python
do_with_L = with_L.do(what_if_L_was_1)
show_do_with_L = do_with_L.show(show_prob=True,
                                cut=what_if_L_was_1,
                                crit=do_with_L.critical(what_if_L_was_1))
display(show_do_with_L)
```

# Independent choice semantics vs program trace semantics

David Poole distinguishes between Independent choice semantics, where every variable is defined for all values in all worlds, and Program trace semantics, where variables are only defined for the values they take for a particular trace.

This lecture by David Poole is short (11 minutes) but contains a concise summary of  how probabilistic programming languages, structural causal models, bayes nets, and logic programming are related:

1. ([0.59](https://youtu.be/L_D9Xne6ATc?t=59)) All probabilistic programming languages are basically generalizations of structural causal models:  deterministic relationships with probability distributions over exogenous variables.

2. ([1:31](https://youtu.be/L_D9Xne6ATc?t=91))  All PPLs have the following features:
    * All PPL's use probabilistic inputs,
    * condition on observations, and
    * query for distributions.
    * Learning probabilities from data

    PPL's employ inference which is much faster than, but equivalent to, rejection sampling for computing posterior distributions

    Note:  Adding the `do()` operator enables meaningful counterfactual queries.

3. ([2:29](https://youtu.be/L_D9Xne6ATc?t=149))  Any Bayesian network can be represented as a probabilistic program or equivalently as a structural causal model (with probabilistic inputs and bidirectional deterministic relationships)

4.  ([4:16](https://youtu.be/L_D9Xne6ATc?t=256)) PPLs face several choices about what it means to assign values to variables encountered in the execution of a program
Rejection sampling semantics samples from the joint distribution and rejects any sample that does not meet the condition. (Simplest, but most inefficient approach)
Independent choice semantics treats each choice of value assignments to the random inputs as a possible world (This is the logic programming approach)
Program trace semantics only creates a possible world for each choice of value assignments encountered in an execution path. (This is what PPLs typically do)
Abductive semantics only creates a possible world for each choice needed to infer observations and a value for a query. (This is the most parsimonious approach. May be what Omega_C already does)

5. ([6:43](https://youtu.be/L_D9Xne6ATc?t=403)) variables are only defined in one trace and not another, this creates problems for inference and learning (These problems are mostly solved nowadays with MCMC and variational inference).

6.  ([9:03](https://youtu.be/L_D9Xne6ATc?t=543)) (How do we align ontologies of causal knowledge with ontologies of observations about the world?  (This problem is partially resolved with rich knowledge representation languages like BEL that are grounded in observation ontologies).

The video is missing details that are present in [these slides](https://www.cs.ubc.ca/~poole/talks/IndependentChoicesTalk2014.pdf), and the slides are missing details that are contained in  [this  paper](https://sciwheel.com/work/#/items/9526992/detail?collection=320250).

## James Koppel's example 1

```julia
A = bernoulli(0.5)
B = bernoulli(0.5)
If (A) {
  Y = 1
  Z = 1
} else {
  Y = 0
  Z = 0
}

If (B && Y) {
  W = 1
} else {
  W = 0
}
```

## Trace semantics probability tree

```python
trace_semantics = PTree.fromFunc(trace_semantics_tree)
trace_semantics.show(show_prob=True,cut=trace_semantics.prop('W=1'),
                    crit=trace_semantics.critical(trace_semantics.prop('W=1')))
```

## Independent choice semantics probability tree

```python
def independent_choice_semantics_tree( bvar ):
    if 'A' not in bvar:
        return [(0.5, 'A=0'),
                (0.5, 'A=1')]
    if 'B' not in bvar:
        return [(0.5, 'B=0'),
                (0.5, 'B=1')]
    if ('Y' not in bvar) or ('Z' not in bvar):
        if bvar['A'] == '1':
            return [(0, 'Y=0,Z=0'),
                    (0, 'Y=0,Z=1'),
                    (0, 'Y=1,Z=0'),
                    (1, 'Y=1,Z=1')]
        else:
            return [(1, 'Y=0,Z=0'),
                    (0, 'Y=0,Z=1'),
                    (0, 'Y=1,Z=0'),
                    (0, 'Y=1,Z=1')]
    if 'W' not in bvar:
        if (bvar['B'] == '1') and (bvar['Y'] == '1'):
            return [(0, 'W=0'),
                    (1, 'W=1'),]
        else:
            return [(1, 'W=0'),
                    (0, 'W=1')]
    return None
```

```python
independent_choice_semantics = PTree.fromFunc(independent_choice_semantics_tree)
independent_choice_semantics.show(show_prob=True,cut=independent_choice_semantics.prop('W=1'),
                    crit=independent_choice_semantics.critical(independent_choice_semantics.prop('W=1')))
```

```python
def trace_semantics_tree( bvar ):
    if 'A' not in bvar:
#       A = bernoulli(0.5)
        return [(0.5, 'A=0'),
                (0.5, 'A=1')]
    if 'B' not in bvar:
#       B = bernoulli(0.5)
        return [(0.5, 'B=0'),
                (0.5, 'B=1')]
    if ('Y' not in bvar) or ('Z' not in bvar):
        if bvar['A'] == '1':
#           If (A) {
#              Y = 1
#              Z = 1
#           }
            return [(1, 'Y=1,Z=1')]
        else:
#           else {
#             Y = 0
#             Z = 0
#           }
            return [(1,'Y=0,Z=0')]
    if 'W' not in bvar:
        if (bvar['B'] == '1') and (bvar['Y'] == '1'):
#           If (B && Y) {
#             W = 1
#           }
            return [(1, 'W=1')]
        else:
#           else {
#             W = 0
#           }
            return [(1, 'W=0')]
```

## Trace Semantics Petri net
0. two kinds of nodes: state and decision nodes
1. at most one token per node
2. every transition from a decision node has a weight
3. the weights of transitions off a single node sum to 1
4. if two transitions share a parent, then they have the exact same parent set.
5. the graph is acyclic.
One can imagine future work relaxing (5).

We split nodes into “state nodes” (representing a variable assignment) and “decision nodes” (representing a decision involving that variable about to be made). State nodes will be copied into one or more decision nodes . I’ll just be drawing these as direct edges between states.

```python
import snakes.plugins
nets = snakes.plugins.load('gv', 'snakes.nets', 'nets')
from nets import Place, PetriNet, Transition, MultiSet, Expression, Marking, OneOf, Substitution, Test,Variable,Value
dir(PetriNet)
```

    ['__class__',
     '__contains__',
     '__delattr__',
     '__dict__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__le__',
     '__lt__',
     '__module__',
     '__ne__',
     '__new__',
     '__plugins__',
     '__pnmldump__',
     '__pnmlload__',
     '__pnmltag__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__setattr__',
     '__sizeof__',
     '__str__',
     '__subclasshook__',
     '__weakref__',
     '_copy',
     '_copy_edges',
     '_pnml_dump_arc',
     '_set_marking',
     'add_input',
     'add_marking',
     'add_output',
     'add_place',
     'add_transition',
     'bbox',
     'copy',
     'copy_place',
     'copy_transition',
     'declare',
     'draw',
     'get_marking',
     'has_node',
     'has_place',
     'has_transition',
     'layout',
     'merge_places',
     'merge_transitions',
     'node',
     'place',
     'post',
     'pre',
     'remove_input',
     'remove_marking',
     'remove_output',
     'remove_place',
     'remove_transition',
     'rename',
     'rename_node',
     'set_marking',
     'shift',
     'transition',
     'transpose']

```python
def trace_semantics_petri_net():
    n = PetriNet("N")
    # State Nodes
    n.add_place(Place("Root=1", [0,1,0,1]))
    n.add_place(Place("A=0"))
    n.add_place(Place('A=1'))
    n.add_place(Place("B=0"))
    n.add_place(Place('B=1' ))
    n.add_place(Place("Y=0"))
    n.add_place(Place('Y=1'))
    n.add_place(Place("Z=0"))
    n.add_place(Place('Z=1'))
    n.add_place(Place("W=0"))
    n.add_place(Place('W=1'))

    # Decision Nodes
    n.add_transition(Transition('A if Root=1'))
    n.add_transition(Transition('B if Root=1'))
    n.add_transition(Transition('Y=1 if A=1', Expression("A==1")))
    n.add_transition(Transition('Z=1 if A=1', Expression("A==1")))
    n.add_transition(Transition('Y=0 if A=0', Expression("A==0")))
    n.add_transition(Transition('Z=0 if A=0', Expression("A==0")))
    n.add_transition(Transition('W=1 if B=1 and Y=1', Expression("(B==1) and (Y==1)")))
    n.add_transition(Transition('W=0 if B=0 or Y=0', Expression("(B==0) and (Y==0)")))

    # Arcs from States to Decisions
    n.add_input("Root=1","A if Root=1", Test(Variable("Root")))
    n.add_input("Root=1", "B if Root=1", Test(Variable("Root")))

    n.add_input("A=0","Y=0 if A=0", Variable('Y'))
    n.add_input("A=0","Z=0 if A=0", Variable("Z"))
    n.add_input("A=1","Y=1 if A=1", Variable("Y"))
    n.add_input("A=1","Z=1 if A=1", Variable("Z"))

    n.add_input("B=0", 'W=0 if B=0 or Y=0',Variable('W') )
    n.add_input("Y=0", 'W=0 if B=0 or Y=0',Variable('W'))

    n.add_input("B=1", 'W=1 if B=1 and Y=1', Variable('W'))
    n.add_input("Y=1", 'W=1 if B=1 and Y=1', Variable('W'))

    # Arcs from Decisions to States have a weight that sums to one.
    n.add_output("A=0", "A if Root=1", Value(0.5))
    n.add_output("A=1", "A if Root=1",  Value(0.5))
    n.add_output("B=0", "B if Root=1",  Value(0.5))
    n.add_output("B=1", "B if Root=1",  Value(0.5))

    n.add_output("Y=0", "Y=0 if A=0", Value(1))
    n.add_output("Z=0", "Z=0 if A=0",  Value(1))
    n.add_output("Y=1", "Y=1 if A=1", Value(1))
    n.add_output("Z=1", "Z=1 if A=1", Value(1))

    n.add_output( "W=0",'W=0 if B=0 or Y=0',  Value(1))
    n.add_output( "W=1", 'W=1 if B=1 and Y=1', Value(1))

    return n
```

```python
n = trace_semantics_petri_net()
dir(n)
```

    ['__class__',
     '__contains__',
     '__delattr__',
     '__dict__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__le__',
     '__lt__',
     '__module__',
     '__ne__',
     '__new__',
     '__plugins__',
     '__pnmldump__',
     '__pnmlload__',
     '__pnmltag__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__setattr__',
     '__sizeof__',
     '__str__',
     '__subclasshook__',
     '__weakref__',
     '_copy',
     '_copy_edges',
     '_declare',
     '_node',
     '_place',
     '_pnml_dump_arc',
     '_set_marking',
     '_trans',
     'add_input',
     'add_marking',
     'add_output',
     'add_place',
     'add_transition',
     'bbox',
     'clusters',
     'copy',
     'copy_place',
     'copy_transition',
     'declare',
     'draw',
     'get_marking',
     'globals',
     'has_node',
     'has_place',
     'has_transition',
     'layout',
     'merge_places',
     'merge_transitions',
     'name',
     'node',
     'place',
     'post',
     'pre',
     'remove_input',
     'remove_marking',
     'remove_output',
     'remove_place',
     'remove_transition',
     'rename',
     'rename_node',
     'set_marking',
     'shift',
     'transition',
     'transpose']

```python
n.draw('trace_semantics_pnet.png')
```

    Graph(['node_3', 'node_2', 'node_0', 'node_14', 'node_10', 'node_9', 'node_15', 'node_17', 'node_4', 'node_11', 'node_1', 'node_16', 'node_12', 'node_7', 'node_6', 'node_18', 'node_13', 'node_5', 'node_8'], [])

```python
n.get_marking()
```

    Marking({'Root=1': MultiSet([0, 0, 1, 1])})

```python
n.transition('A if Root=1').modes()
```

    [Substitution(Root=0),
     Substitution(Root=0),
     Substitution(Root=1),
     Substitution(Root=1)]

```python
n.transition('A if Root=1').fire(Substitution(Root=1))
```

```python
n.get_marking()
```

    Marking({'Root=1': MultiSet([0, 0, 1, 1]), 'A=0': MultiSet([0.5]), 'A=1': MultiSet([0.5])})

```python
n.transition('Y=1 if A=1').modes()
```

```python
dir(PetriNet)
```

    ['__class__',
     '__contains__',
     '__delattr__',
     '__dict__',
     '__dir__',
     '__doc__',
     '__eq__',
     '__format__',
     '__ge__',
     '__getattribute__',
     '__gt__',
     '__hash__',
     '__init__',
     '__init_subclass__',
     '__le__',
     '__lt__',
     '__module__',
     '__ne__',
     '__new__',
     '__pnmldump__',
     '__pnmlload__',
     '__pnmltag__',
     '__reduce__',
     '__reduce_ex__',
     '__repr__',
     '__setattr__',
     '__sizeof__',
     '__str__',
     '__subclasshook__',
     '__weakref__',
     '_pnml_dump_arc',
     '_set_marking',
     'add_input',
     'add_marking',
     'add_output',
     'add_place',
     'add_transition',
     'copy',
     'copy_place',
     'copy_transition',
     'declare',
     'get_marking',
     'has_node',
     'has_place',
     'has_transition',
     'merge_places',
     'merge_transitions',
     'node',
     'place',
     'post',
     'pre',
     'remove_input',
     'remove_marking',
     'remove_output',
     'remove_place',
     'remove_transition',
     'rename',
     'rename_node',
     'set_marking',
     'transition']

```python
help(PetriNet.add_input)
```

    Help on function add_input in module snakes.nets:

    add_input(self, place, trans, label)
        Add an input arc between `place` and `trans` (nodes names).
        An input arc is directed from a place toward a transition.

        >>> n = PetriNet('N')
        >>> n.add_place(Place('p', range(3)))
        >>> n.add_transition(Transition('t', Expression('x!=1')))
        >>> try : n.add_input('p', 't', Expression('2*x'))
        ... except ConstraintError : print(sys.exc_info()[1])
        'Expression' not allowed on input arcs
        >>> n.add_input('p', 't', Variable('x'))
        >>> (n.post('p'), n.pre('t')) == (set(['t']), set(['p']))
        True
        >>> n.transition('t').modes()
        [Substitution(x=0), Substitution(x=2)]
        >>> n.place('p').tokens == MultiSet([0, 1, 2])
        True
        >>> n.transition('t').fire(Substitution(x=0))
        >>> n.place('p').tokens == MultiSet([1, 2])
        True
        >>> try : n.add_input('p', 't', Value(42))
        ... except ConstraintError: print(sys.exc_info()[1])
        already connected to 'p'

        @param place: the name of the place to connect
        @type place: `str`
        @param trans: the name of the transition to connect
        @type trans: `str`
        @param label: the annotation of the arc
        @type label: `ArcAnnotation`
        @raise ConstraintError: in case of anything not allowed

```python
dir(snakes.nets)
```

    ['ArcAnnotation',
     'Binder',
     'BlackToken',
     'Collection',
     'ConstraintError',
     'CrossProduct',
     'DomainError',
     'Evaluator',
     'Expression',
     'Flush',
     'Greater',
     'GreaterOrEqual',
     'Inhibitor',
     'Instance',
     'Less',
     'LessOrEqual',
     'List',
     'Mapping',
     'Marking',
     'ModeError',
     'MultiArc',
     'MultiSet',
     'Names',
     'NetElement',
     'Node',
     'NodeError',
     'OneOf',
     'PY3',
     'PetriNet',
     'Place',
     'PluginError',
     'Range',
     'Renamer',
     'Set',
     'SnakesError',
     'StateGraph',
     'Substitution',
     'Symbol',
     'Test',
     'Token',
     'Transition',
     'Tree',
     'Tuple',
     'Type',
     'TypeCheck',
     'UnificationError',
     'Unparser',
     'Value',
     'Variable',
     'WordSet',
     '__builtins__',
     '__cached__',
     '__doc__',
     '__file__',
     '__loader__',
     '__name__',
     '__package__',
     '__spec__',
     'ast',
     'bind',
     'builtins',
     'cross',
     'defaultencoding',
     'dot',
     'dumps',
     'getvars',
     'hashable',
     'hdict',
     'hlist',
     'hset',
     'imp',
     'inspect',
     'io',
     'iterate',
     'loads',
     'operator',
     'os',
     'pickle',
     'pkgutil',
     're',
     'reduce',
     'rename',
     'snakes',
     'sys',
     'tAll',
     'tBlackToken',
     'tBoolean',
     'tDict',
     'tFloat',
     'tInteger',
     'tList',
     'tNatural',
     'tNone',
     'tNothing',
     'tNumber',
     'tPair',
     'tPositive',
     'tString',
     'tTuple',
     'unhash',
     'unicode',
     'unparse',
     'version',
     'xml',
     'xor',
     'xrange']

```python
help(PetriNet.add_input)
```

    Help on function add_input in module snakes.nets:

    add_input(self, place, trans, label)
        Add an input arc between `place` and `trans` (nodes names).
        An input arc is directed from a place toward a transition.

        >>> n = PetriNet('N')
        >>> n.add_place(Place('p', range(3)))
        >>> n.add_transition(Transition('t', Expression('x!=1')))
        >>> try : n.add_input('p', 't', Expression('2*x'))
        ... except ConstraintError : print(sys.exc_info()[1])
        'Expression' not allowed on input arcs
        >>> n.add_input('p', 't', Variable('x'))
        >>> (n.post('p'), n.pre('t')) == (set(['t']), set(['p']))
        True
        >>> n.transition('t').modes()
        [Substitution(x=0), Substitution(x=2)]
        >>> n.place('p').tokens == MultiSet([0, 1, 2])
        True
        >>> n.transition('t').fire(Substitution(x=0))
        >>> n.place('p').tokens == MultiSet([1, 2])
        True
        >>> try : n.add_input('p', 't', Value(42))
        ... except ConstraintError: print(sys.exc_info()[1])
        already connected to 'p'

        @param place: the name of the place to connect
        @type place: `str`
        @param trans: the name of the transition to connect
        @type trans: `str`
        @param label: the annotation of the arc
        @type label: `ArcAnnotation`
        @raise ConstraintError: in case of anything not allowed

```python
help(PetriNet)
```

    Help on class PetriNet in module snakes.nets:

    class PetriNet(builtins.object)
     |  PetriNet(name)
     |
     |  A Petri net. As soon as nodes are added to a `PetriNet`, they
     |  should be handled by name instead of by the `Place` or
     |  `Transition` instance. For instance:
     |
     |  >>> n = PetriNet('N')
     |  >>> t = Transition('t')
     |  >>> n.add_transition(t)
     |  >>> n.has_transition('t') # use 't' and not t
     |  True
     |  >>> n.transition('t') is t
     |  True
     |
     |  Methods defined here:
     |
     |  __contains__(self, name)
     |      `name in net` is a shortcut for `net.has_node(name)`
     |
     |      >>> n = PetriNet('N')
     |      >>> 't' in n
     |      False
     |      >>> 'p' in n
     |      False
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_place(Place('p'))
     |      >>> 't' in n
     |      True
     |      >>> 'p' in n
     |      True
     |
     |      @param name: the name of the searched node
     |      @type name: `str`
     |      @return: `True` if a node called `name` is present in the net,
     |          `False` otherwise
     |      @rtype: `bool`
     |
     |  __hash__(self)
     |      Return hash(self).
     |
     |  __init__(self, name)
     |      Initialise with a name that may be an arbitrary string.
     |
     |      >>> PetriNet('N')
     |      PetriNet('N')
     |
     |      @param name: the name of the net
     |      @type name: `str`
     |
     |  __pnmldump__(self)
     |      Dump a `PetriNet` as a PNML tree
     |
     |      >>> n = PetriNet('N')
     |      >>> n.declare('x = "foo" + "bar"')
     |      >>> n.globals['y'] = 'egg'
     |      >>> for i, l in enumerate((Value(dot),
     |      ...                        MultiArc([Value(dot), Value(dot)]),
     |      ...                        Value(1))) :
     |      ...    n.add_place(Place('p%u' % i))
     |      ...    n.add_transition(Transition('t%u' % i))
     |      ...    n.add_input('p%u' % i, 't%u' % i, l)
     |      >>> n.__pnmldump__()
     |      <?xml version="1.0" encoding="utf-8"?>
     |      <pnml>
     |       <net id="N">
     |        <declare>
     |         x = "foo" + "bar"
     |        </declare>
     |        <global name="y">
     |         <object type="str">
     |          egg
     |         </object>
     |        </global>
     |        <place id="...">
     |        ...
     |        </place>
     |        <place id="...">
     |        ...
     |        </place>
     |        <place id="...">
     |        ...
     |        </place>
     |        <transition id="..."/>
     |        <transition id="..."/>
     |        <transition id="..."/>
     |        <arc id="..." source="..." target="...">
     |        ...
     |        </arc>
     |        <arc id="..." source="..." target="...">
     |        ...
     |        </arc>
     |        <arc id="..." source="..." target="...">
     |        ...
     |        </arc>
     |       </net>
     |      </pnml>
     |
     |      @return: PNML tree
     |      @rtype: `pnml.Tree`
     |
     |  __repr__(self)
     |      Return a string suitable for `eval` to represent the net.
     |
     |      >>> repr(PetriNet('N'))
     |      "PetriNet('N')"
     |
     |      @return: the textual representation of the net
     |      @rtype: `str`
     |
     |  __str__(self)
     |      Return the name of the net.
     |
     |      >>> str(PetriNet('N'))
     |      'N'
     |
     |      @return: the name if the net
     |      @rtype: `str`
     |
     |  add_input(self, place, trans, label)
     |      Add an input arc between `place` and `trans` (nodes names).
     |      An input arc is directed from a place toward a transition.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p', range(3)))
     |      >>> n.add_transition(Transition('t', Expression('x!=1')))
     |      >>> try : n.add_input('p', 't', Expression('2*x'))
     |      ... except ConstraintError : print(sys.exc_info()[1])
     |      'Expression' not allowed on input arcs
     |      >>> n.add_input('p', 't', Variable('x'))
     |      >>> (n.post('p'), n.pre('t')) == (set(['t']), set(['p']))
     |      True
     |      >>> n.transition('t').modes()
     |      [Substitution(x=0), Substitution(x=2)]
     |      >>> n.place('p').tokens == MultiSet([0, 1, 2])
     |      True
     |      >>> n.transition('t').fire(Substitution(x=0))
     |      >>> n.place('p').tokens == MultiSet([1, 2])
     |      True
     |      >>> try : n.add_input('p', 't', Value(42))
     |      ... except ConstraintError: print(sys.exc_info()[1])
     |      already connected to 'p'
     |
     |      @param place: the name of the place to connect
     |      @type place: `str`
     |      @param trans: the name of the transition to connect
     |      @type trans: `str`
     |      @param label: the annotation of the arc
     |      @type label: `ArcAnnotation`
     |      @raise ConstraintError: in case of anything not allowed
     |
     |  add_marking(self, marking)
     |      Add a marking to the current one. If an error occurs during
     |      the process, the marking is left unchanged. Places in the
     |      marking that do not belong to the net are ignored.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2', range(3)))
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 1, 2])})
     |      True
     |      >>> n.add_marking(Marking(p1=MultiSet(range(2)), p2=MultiSet([1])))
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 1, 1, 2]), 'p1': MultiSet([0, 1])})
     |      True
     |
     |      @param marking: the new marking
     |      @type marking: `Marking`
     |
     |  add_output(self, place, trans, label)
     |      Add an output arc between `place` and `trans` (nodes names).
     |
     |      An output arc is directed from a transition toward a place.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p'))
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_output('p', 't', Value(42))
     |      >>> (n.post('t'), n.pre('p')) == (set(['p']), set(['t']))
     |      True
     |      >>> n.place('p').tokens == MultiSet([])
     |      True
     |      >>> n.transition('t').fire(Substitution())
     |      >>> n.place('p').tokens == MultiSet([42])
     |      True
     |      >>> try : n.add_output('p', 't', Value(42))
     |      ... except ConstraintError : print(sys.exc_info()[1])
     |      already connected to 'p'
     |
     |      @param place: the name of the place to connect
     |      @type place: `str`
     |      @param trans: the name of the transition to connect
     |      @type trans: `str`
     |      @param label: the annotation of the arc
     |      @type label: `ArcAnnotation`
     |      @raise ConstraintError: in case of anything not allowed
     |
     |  add_place(self, place)
     |      Add a place to the net. Each node in a net must have a name
     |      unique to this net, which is checked when it is added.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.place('p')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: place 'p' not found
     |      >>> n.add_place(Place('p', range(3)))
     |      >>> n.place('p')
     |      Place('p', MultiSet([...]), tAll)
     |      >>> n.place('p').tokens == MultiSet([0, 1, 2])
     |      True
     |      >>> n.add_place(Place('p'))
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: place 'p' exists
     |
     |      @param place: the place to add
     |      @type place: `Place`
     |      @raise ConstraintError: when a place with the same name exists
     |          already in the net
     |
     |  add_transition(self, trans)
     |      Add a transition to the net. Each node in a net must have a
     |      name unique to this net, which is checked when it is added.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.transition('t')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: transition 't' not found
     |      >>> n.add_transition(Transition('t', Expression('x==1')))
     |      >>> n.transition('t')
     |      Transition('t', Expression('x==1'))
     |      >>> n.add_transition(Transition('t'))
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: transition 't' exists
     |
     |      @param trans: the transition to add
     |      @type trans: `Transition`
     |      @raise ConstraintError: when a transition with the same name
     |          exists already in the net
     |
     |  copy(self, name=None)
     |      Return a complete copy of the net, including places,
     |      transitions, arcs and declarations.
     |
     |      >>> PetriNet('N').copy()
     |      PetriNet('N')
     |      >>> PetriNet('N').copy('x')
     |      PetriNet('x')
     |
     |      @param name: if not `None`, the name of the copy
     |      @type name: `str`
     |      @return: a copy of the net
     |      @rtype: `PetriNet`
     |
     |  copy_place(self, source, targets)
     |      Make copies of the `source` place (use place names).
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p', range(3)))
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_input('p', 't', Value(0))
     |      >>> n.copy_place('p', ['bis', 'ter'])
     |      >>> n.copy_place('bis', 'more')
     |      >>> list(sorted(n.place(), key=str))
     |      [Place('bis', MultiSet([...]), tAll),
     |       Place('more', MultiSet([...]), tAll),
     |       Place('p', MultiSet([...]), tAll),
     |       Place('ter', MultiSet([...]), tAll)]
     |      >>> list(sorted(n.pre('t'), key=str))
     |      ['bis', 'more', 'p', 'ter']
     |
     |      @param source: the name of the place to copy
     |      @type source: `str`
     |      @param targets: a name or a list of names for the copie(s)
     |      @type targets: `str`
     |
     |  copy_transition(self, source, targets)
     |      Make copies of the `source` transition (use transition names).
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_transition(Transition('t', Expression('x==1')))
     |      >>> n.add_place(Place('p'))
     |      >>> n.add_input('p', 't', Value(0))
     |      >>> n.copy_transition('t', ['bis', 'ter'])
     |      >>> n.copy_transition('bis', 'more')
     |      >>> list(sorted(n.transition(), key=str))
     |      [Transition('bis', Expression('x==1')),
     |       Transition('more', Expression('x==1')),
     |       Transition('t', Expression('x==1')),
     |       Transition('ter', Expression('x==1'))]
     |      >>> list(sorted(n.post('p')))
     |      ['bis', 'more', 't', 'ter']
     |
     |      @param source: the name of the transition to copy
     |      @type source: `str`
     |      @param targets: a name or a list of names for the copie(s)
     |      @type targets: `str`
     |
     |  declare(self, statements, locals=None)
     |      Execute `statements` in the global dictionnary of the net.
     |      This has also on the dictionnarie of the instances of
     |      `Expression` in the net (guards of the transitions and labels
     |      on the arcs) so the declarations have an influence over the
     |      elements embedded in the net. If `locals` is given, most of
     |      the declared objects will be placed in it instead of the
     |      global dictionnary, see the documentation of Python for more
     |      details about local and global environments.
     |
     |      >>> n = PetriNet('N')
     |      >>> t = Transition('t', Expression('x==0'))
     |      >>> n.add_transition(t)
     |      >>> t.guard(Substitution())
     |      Traceback (most recent call last):
     |        ...
     |      NameError: name 'x' is not defined
     |      >>> n.declare('x=0')
     |      >>> t.guard(Substitution())
     |      True
     |      >>> n.add_place(Place('p'))
     |      >>> n.add_output('p', 't', Expression('math.pi'))
     |      >>> t.fire(Substitution())
     |      Traceback (most recent call last):
     |        ...
     |      NameError: name 'math' is not defined
     |      >>> n.declare('import math')
     |      >>> t.fire(Substitution())
     |      >>> n.place('p')
     |      Place('p', MultiSet([3.14...]), tAll)
     |
     |      @param statements: a Python instruction suitable to `exec`
     |      @type statements: `str`
     |      @param locals: a `dict` used as locals when `statements` is
     |          executed, or `None`
     |      @type locals: `dict`
     |
     |  get_marking(self)
     |      Return the current marking of the net, omitting empty places.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p0', range(0)))
     |      >>> n.add_place(Place('p1', range(1)))
     |      >>> n.add_place(Place('p2', range(2)))
     |      >>> n.add_place(Place('p3', range(3)))
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 1]), 'p3': MultiSet([0, 1, 2]), 'p1': MultiSet([0])})
     |      True
     |
     |      @return: the current marking
     |      @rtype: `Marking`
     |
     |  has_node(self, name)
     |      Check if there is a transition called `name` in the net.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.has_node('t')
     |      False
     |      >>> n.has_node('p')
     |      False
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_place(Place('p'))
     |      >>> n.has_node('t')
     |      True
     |      >>> n.has_node('p')
     |      True
     |
     |      @param name: the name of the searched node
     |      @type name: `str`
     |      @return: `True` if a node called `name` is present in the net,
     |          `False` otherwise
     |      @rtype: `bool`
     |
     |  has_place(self, name)
     |      Check if there is a place called `name` in the net.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.has_place('p')
     |      False
     |      >>> n.add_place(Place('p'))
     |      >>> n.has_place('p')
     |      True
     |
     |      @param name: the name of the searched place
     |      @type name: `str`
     |      @return: `True` if a place called `name` is present in the
     |          net, `False` otherwise
     |      @rtype: `bool`
     |
     |  has_transition(self, name)
     |      Check if there is a transition called `name` in the net.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.has_transition('t')
     |      False
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.has_transition('t')
     |      True
     |
     |      @param name: the name of the searched transition
     |      @type name: `str`
     |      @return: `True` if a transition called `name` is present in
     |          the net, `False` otherwise
     |      @rtype: `bool`
     |
     |  merge_places(self, target, sources)
     |      Create a new place by merging those in `sources`. Markings
     |      are added, place types are 'or'ed and arcs labels are joinded
     |      into multi-arcs, the sources places are not removed. Use
     |      places names.
     |
     |      >>> n = PetriNet('n')
     |      >>> n.add_place(Place('p1', [1], tInteger))
     |      >>> n.add_place(Place('p2', [2.0], tFloat))
     |      >>> n.add_transition(Transition('t1'))
     |      >>> n.add_transition(Transition('t2'))
     |      >>> n.add_output('p1', 't1', Value(1))
     |      >>> n.add_output('p2', 't2', Value(2.0))
     |      >>> n.add_output('p2', 't1', Value(2.0))
     |      >>> n.merge_places('p', ['p1', 'p2'])
     |      >>> (n.pre('p'), n.post('t1')) == (set(['t2', 't1']), set(['p2', 'p', 'p1']))
     |      True
     |      >>> list(sorted(n.node('p').pre.items()))
     |      [('t1', MultiArc((Value(1), Value(2.0)))),
     |       ('t2', Value(2.0))]
     |      >>> n.node('p').tokens == MultiSet([1, 2.0])
     |      True
     |      >>> n.node('p').checker()
     |      (Instance(int) | Instance(float))
     |
     |      @param target: the name of the created place
     |      @type target: `str`
     |      @param sources: the list of places names to be merged (or a
     |          single place name)
     |      @type sources: `list`
     |
     |  merge_transitions(self, target, sources)
     |      Create a new transition by merging those in `sources`.
     |      Guards are 'and'ed and arcs labels are joinded into multi-
     |      arcs, the sources transitions are not removed. Use transitions
     |      names.
     |
     |      >>> n = PetriNet('n')
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2'))
     |      >>> n.add_transition(Transition('t1', Expression('x==1')))
     |      >>> n.add_transition(Transition('t2', Expression('y==2')))
     |      >>> n.add_output('p1', 't1', Value(1))
     |      >>> n.add_output('p2', 't2', Value(2.0))
     |      >>> n.add_output('p2', 't1', Value(2.0))
     |      >>> n.merge_transitions('t', ['t1', 't2'])
     |      >>> list(sorted(n.post('t'), key=str))
     |      ['p1', 'p2']
     |      >>> list(sorted(n.pre('p2'), key=str))
     |      ['t', 't1', 't2']
     |      >>> n.transition('t')
     |      Transition('t', Expression('(x==1) and (y==2)'))
     |      >>> n.node('t').post
     |      {'p2': MultiArc((Value(2.0), Value(2.0))), 'p1': Value(1)}
     |
     |      @param target: the name of the created transition
     |      @type target: `str`
     |      @param sources: the list of transitions names to be merged (or
     |          a single transition name)
     |      @type sources: `list`
     |
     |  node(self, name=None)
     |      Return one (if `name` is not `None`) or all the nodes.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_place(Place('p'))
     |      >>> n.node('t')
     |      Transition('t', Expression('True'))
     |      >>> n.node('x')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: node 'x' not found
     |      >>> list(sorted(n.node(), key=str))
     |      [Place('p', MultiSet([]), tAll), Transition('t', Expression('True'))]
     |
     |      @param name: the name of the node to retrieve or `None` to get
     |          the list of all the nodes in the net
     |      @type name: `str`
     |      @return: a node whose name is `name` or a list of nodes
     |      @rtype: `object`
     |      @raise ConstraintError: when a node requested does not exist
     |
     |  place(self, name=None)
     |      Return one (if `name` is not `None`) or all the places.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2'))
     |      >>> n.place('p1')
     |      Place('p1', MultiSet([]), tAll)
     |      >>> n.place('p')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: place 'p' not found
     |      >>> n.place()
     |      [Place('p2', MultiSet([]), tAll), Place('p1', MultiSet([]), tAll)]
     |
     |      @param name: the name of the place to retrieve or `None` to
     |          get the list of all the places in the net
     |      @type name: `str`
     |      @return: a place whose name is `name` or a list of places
     |      @rtype: `object`
     |      @raise ConstraintError: when a place requested does not exist
     |
     |  post(self, nodes)
     |      Return the set of nodes names succeeding `nodes`. `nodes`
     |      can be a single node name ot a list of nodes names.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2'))
     |      >>> n.add_transition(Transition('t1'))
     |      >>> n.add_transition(Transition('t2'))
     |      >>> n.add_output('p1', 't1', Value(1))
     |      >>> n.add_output('p2', 't2', Value(2))
     |      >>> n.post('t1') == set(['p1'])
     |      True
     |      >>> n.post(['t1', 't2']) == set(['p2', 'p1'])
     |      True
     |
     |      @param nodes: a single node name or a list of node names
     |      @type nodes: `list`
     |      @return: a set of node names
     |      @rtype: `set`
     |
     |  pre(self, nodes)
     |      Return the set of nodes names preceeding `nodes`. `nodes`
     |      can be a single node name ot a list of nodes names.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2'))
     |      >>> n.add_transition(Transition('t1'))
     |      >>> n.add_transition(Transition('t2'))
     |      >>> n.add_output('p1', 't1', Value(1))
     |      >>> n.add_output('p2', 't2', Value(2))
     |      >>> n.pre('p1') == set(['t1'])
     |      True
     |      >>> n.pre(['p1', 'p2']) == set(['t2', 't1'])
     |      True
     |
     |      @param nodes: a single node name or a list of node names
     |      @type nodes: `list`
     |      @return: a set of node names
     |      @rtype: `set`
     |
     |  remove_input(self, place, trans)
     |      Remove an input arc between `place` and `trans` (nodes names).
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p', range(3)))
     |      >>> n.add_transition(Transition('t', Expression('x!=1')))
     |      >>> n.add_input('p', 't', Variable('x'))
     |      >>> (n.post('p'), n.pre('t')) == (set(['t']), set(['p']))
     |      True
     |      >>> n.remove_input('p', 't')
     |      >>> (n.post('p'), n.pre('t')) == (set([]), set([]))
     |      True
     |      >>> try : n.remove_input('p', 't')
     |      ... except ConstraintError : print(sys.exc_info()[1])
     |      not connected to 'p'
     |
     |      @param place: the name of the place to disconnect
     |      @type place: `str`
     |      @param trans: the name of the transition to disconnect
     |      @type trans: `str`
     |      @raise ConstraintError: when this arc does not exist
     |
     |  remove_marking(self, marking)
     |      Substract a marking from the current one. If an error
     |      occurs during the process, the marking is left unchanged.
     |      Places in the marking that do not belong to the net are
     |      ignored.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2', range(3)))
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 1, 2])})
     |      True
     |      >>> try : n.remove_marking(Marking(p1=MultiSet(range(2)), p2=MultiSet([1])))
     |      ... except ValueError : print(sys.exc_info()[1])
     |      not enough occurrences
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 1, 2])})
     |      True
     |      >>> n.remove_marking(Marking(p2=MultiSet([1])))
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 2])})
     |      True
     |
     |      @param marking: the new marking
     |      @type marking: `Marking`
     |
     |  remove_output(self, place, trans)
     |      Remove an output arc between `place` and `trans` (nodes
     |      names).
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p'))
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_output('p', 't', Value(42))
     |      >>> (n.post('t'), n.pre('p')) == (set(['p']), set(['t']))
     |      True
     |      >>> n.remove_output('p', 't')
     |      >>> (n.post('t'), n.pre('p')) == (set([]), set([]))
     |      True
     |      >>> try : n.remove_output('p', 't')
     |      ... except ConstraintError : print(sys.exc_info()[1])
     |      not connected to 'p'
     |
     |      @param place: the name of the place to disconnect
     |      @type place: `str`
     |      @param trans: the name of the transition to disconnect
     |      @type trans: `str`
     |      @raise ConstraintError: when this arc does not exist
     |
     |  remove_place(self, name)
     |      Remove a place (given by its name) from the net.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.remove_place('p')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: place 'p' not found
     |      >>> n.add_place(Place('p', range(3)))
     |      >>> n.place('p')
     |      Place('p', MultiSet([...]), tAll)
     |      >>> n.remove_place('p')
     |      >>> n.place('p')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: place 'p' not found
     |
     |      @param name: the name of the place to remove
     |      @type name: `str`
     |      @raise ConstraintError: when no place with this name exists
     |          in the net
     |
     |  remove_transition(self, name)
     |      Remove a transition (given by its name) from the net.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.remove_transition('t')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: transition 't' not found
     |      >>> n.add_transition(Transition('t', Expression('x==1')))
     |      >>> n.transition('t')
     |      Transition('t', Expression('x==1'))
     |      >>> n.remove_transition('t')
     |      >>> n.transition('t')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: transition 't' not found
     |
     |      @param name: the name of the transition to remove
     |      @type name: `str`
     |      @raise ConstraintError: when no transition with this name
     |          exists in the net
     |
     |  rename(self, name)
     |      Change the name of the net.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.rename('Long name!')
     |      >>> n
     |      PetriNet('Long name!')
     |
     |      @param name: the new name
     |      @type name: `str`
     |
     |  rename_node(self, old, new)
     |      Change the name of a node.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p'))
     |      >>> n.add_transition(Transition('t'))
     |      >>> n.add_output('p', 't', Value(0))
     |      >>> list(sorted(n.node(), key=str))
     |      [Place('p', MultiSet([]), tAll), Transition('t', Expression('True'))]
     |      >>> n.post('t') == set(['p'])
     |      True
     |      >>> n.rename_node('p', 'new_p')
     |      >>> list(sorted(n.node(), key=str))
     |      [Place('new_p', MultiSet([]), tAll), Transition('t', Expression('True'))]
     |      >>> n.post('t') == set(['new_p'])
     |      True
     |      >>> try : n.rename_node('new_p', 't')
     |      ... except ConstraintError : print(sys.exc_info()[1])
     |      node 't' exists
     |      >>> try : n.rename_node('old_t', 'new_t')
     |      ... except ConstraintError : print(sys.exc_info()[1])
     |      node 'old_t' not found
     |
     |      @param old: the current name of the node
     |      @type old: `str`
     |      @param new: the new name for the node
     |      @type new: `str`
     |
     |  set_marking(self, marking)
     |      Assign a marking to the net. Places not listed in the
     |      marking are considered empty, the corresponding place in the
     |      net is thus emptied. If the marking has places that do not
     |      belong to the net, these are ignored (as in the last
     |      instruction below). If an error occurs during the assignment,
     |      the marking is left unchanged.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_place(Place('p0', range(5), tInteger))
     |      >>> n.add_place(Place('p1'))
     |      >>> n.add_place(Place('p2'))
     |      >>> n.add_place(Place('p3'))
     |      >>> n.get_marking() == Marking({'p0': MultiSet([0, 1, 2, 3, 4])})
     |      True
     |      >>> n.set_marking(Marking(p1=MultiSet([0]), p2=MultiSet([0, 1]), p3=MultiSet([0, 1, 2])))
     |      >>> n.get_marking() == Marking({'p2': MultiSet([0, 1]), 'p3': MultiSet([0, 1, 2]), 'p1': MultiSet([0])})
     |      True
     |      >>> n.set_marking(Marking(p=MultiSet([0])))
     |      >>> n.get_marking()
     |      Marking({})
     |      >>> try : n.set_marking(Marking(p2=MultiSet([1]), p0=MultiSet([3.14])))
     |      ... except ValueError : print(sys.exc_info()[1])
     |      forbidden token '3.14'
     |      >>> n.get_marking() # unchanged
     |      Marking({})
     |
     |      @param marking: the new marking
     |      @type marking: `Marking`
     |
     |  transition(self, name=None)
     |      Return one (if `name` is not `None`) or all the transitions.
     |
     |      >>> n = PetriNet('N')
     |      >>> n.add_transition(Transition('t1'))
     |      >>> n.add_transition(Transition('t2'))
     |      >>> n.transition('t1')
     |      Transition('t1', Expression('True'))
     |      >>> n.transition('t')
     |      Traceback (most recent call last):
     |        ...
     |      ConstraintError: transition 't' not found
     |      >>> n.transition()
     |      [Transition('t2', Expression('True')),
     |       Transition('t1', Expression('True'))]
     |
     |      @param name: the name of the transition to retrieve or `None`
     |          to get the list of all the transitions in the net
     |      @type name: `str`
     |      @return: a transition whose name is `name` or a list of
     |          transitions
     |      @rtype: `object`
     |      @raise ConstraintError: when a place requested does not exist
     |
     |  ----------------------------------------------------------------------
     |  Class methods defined here:
     |
     |  __pnmlload__(tree) from builtins.type
     |      Create a `PetriNet` from a PNML tree
     |
     |      @param tree: the tree to convert
     |      @type tree: `pnml.Tree`
     |      @return: the net built
     |      @rtype: `PetriNet`
     |
     |      >>> old = PetriNet('N')
     |      >>> old.declare('x = "foo" + "bar"')
     |      >>> old.globals['y'] = 'egg'
     |      >>> for i, l in enumerate((Value(dot),
     |      ...                        MultiArc([Value(dot), Value(dot)]),
     |      ...                        Value(1))) :
     |      ...    old.add_place(Place('p%u' % i))
     |      ...    old.add_transition(Transition('t%u' % i))
     |      ...    old.add_input('p%u' % i, 't%u' % i, l)
     |      >>> new = PetriNet.__pnmlload__(old.__pnmldump__())
     |      >>> new.globals == old.globals
     |      True
     |      >>> sorted(new.place(), key=lambda x: x.name)
     |      [Place('p0', MultiSet([]), tAll),
     |       Place('p1', MultiSet([]), tAll),
     |       Place('p2', MultiSet([]), tAll)]
     |      >>> sorted(new.transition(), key=lambda x: x.name)
     |      [Transition('t0', Expression('True')),
     |       Transition('t1', Expression('True')),
     |       Transition('t2', Expression('True'))]
     |      >>> for t in sorted(new.transition(), key=lambda x: x.name) :
     |      ...     print('%s %s' % (t.name, t.input()))
     |      t0 [(Place('p0', MultiSet([]), tAll), Value(dot))]
     |      t1 [(Place('p1', MultiSet([]), tAll), MultiArc((Value(dot), Value(dot))))]
     |      t2 [(Place('p2', MultiSet([]), tAll), Value(1))]
     |
     |  ----------------------------------------------------------------------
     |  Data descriptors defined here:
     |
     |  __dict__
     |      dictionary for instance variables (if defined)
     |
     |  __weakref__
     |      list of weak references to the object (if defined)
     |
     |  ----------------------------------------------------------------------
     |  Data and other attributes defined here:
     |
     |  __pnmltag__ = 'net'

# Example involving conditioning and counterfactual on a graph with dynamic dependencies and interesting conditioning dependencies

```julia
X = bernoulli(0.8)

If (X) {
  Y = bernoulli(0.6)
  Z = Y XOR bernoulli(0.2)
} else {
  Z= bernoulli(0.8)
  Y =Z XOR bernoulli(0.4)
}

W = Y && Z && bernoulli(0.6)
```

```python
def dynamic_dependencies( bvar ):
    if 'X' not in bvar:
        # X = bernoulli(0.8)
        return [(0.2, 'X=0'),
                (0.8, 'X=1')]
    if 'Y' not in bvar:
        if bvar['X'] == '1':
            # Y = bernoulli(0.6)
            return [(0.4, 'Y=0'),
                    (0.6, 'Y=1')]
        if 'Z' not in bvar:
            # Z = bernoulli(0.8) when X==0
            return [(0.2, 'Z=0'),
                    (0.8, 'Z=1')]
        if bvar['Z'] == '0':
            # Y = Z XOR bernoulli(0.4) when Z == 0
            return [(0.6,'Y=0'),
                    (0.4,'Y=1')]
        else:
            # Y = Z XOR bernoulli(0.4) when Z == 1
            return [(0.4, 'Y=0'),
                    (0.6, 'Y=1')]
    if 'Z' not in bvar:
        if bvar['X'] == '1':
            if bvar['Y'] == '0':
                # Z = Y XOR bernoulli(0.2) when Y==0
                return [(0.8, 'Z=0'),
                        (0.2, 'Z=1')]
            else:
                # Z = Y XOR bernoulli(0.2) when Y==1
                return [(0.2, 'Z=0'),
                        (0.8, 'Z=1')]
        else:
            # Z = bernoulli(0.8)
            return [(0.2, 'Z=0'),
                    (0.8, 'Z=1')]
    if 'W' not in bvar:
        if (bvar['Y'] == '1') and (bvar['Z'] == '1'):
            # W = Y && Z && bernoulli(0.6) when Y==1 and Z==1
            return [(0.4, 'W=0'),
                    (0.6, 'W=1')]
        else:
            # W = Y && Z && bernoulli(0.6) == 0 when Y==0 or Z==0
            return [(1, 'W=0'),
                    (0, 'W=1')]
    return None

```

## Factual world Probability Tree mechanism for $Z=0$

```python
factual = PTree.fromFunc(dynamic_dependencies)
Z_is_0 = factual.prop('Z=0')
show_factual_Z_is_0 = factual.show(show_prob=True,
                                              cut=Z_is_0,
                                              crit=factual.critical(Z_is_0) )
#show_example3_factual_Z_is_0
```

```python
W = factual.prop('W=1')
show_factual_W = factual.show(show_prob=True,
                                              cut=W,
                                              crit=factual.critical(W) )
display(Latex(f'$$P(W)={factual.prob(W)}$$'))
show_factual_W
```

$$P(W)=0.288$$

## Conditioned on $Z=0$

```python
see_Z_is_0 = factual.see(Z_is_0)
show_see_z_is_0 = see_Z_is_0.show(show_prob=True, cut=Z_is_0, crit=see_Z_is_0.critical(Z_is_0))
#show_see_z_is_0
```

```python
show_W_given_z_is_0 = see_Z_is_0.show(show_prob=True, cut=W, crit=see_Z_is_0.critical(W))
display(Latex('$$P(W|Z=0)='
             f'{see_Z_is_0.prob(W)}$$'))
show_W_given_z_is_0
```

$$P(W|Z=0)=0.0$$

## Intervene on $Z=1$

```python
Z_is_1 = see_Z_is_0.prop('Z=1')
do_Z_is_1 = see_Z_is_0.do(Z_is_1)
show_do_Z_is_1 = do_Z_is_1.show(show_prob=True,
                                cut=Z_is_1,
                                crit=do_Z_is_1.critical(Z_is_1))
#show_do_Z_is_1
```

```python
show_W_given_do_Z_is_1 = do_Z_is_1.show(show_prob=True,
                                               cut=W,
                                               crit=do_Z_is_1.critical(W))
display(Latex('$$P(W|Z=0,Z\leftarrow 1)='
             f'{do_Z_is_1.prob(W)}$$'))
show_W_given_do_Z_is_1
```

$$P(W|Z=0,Z\leftarrow 1)=0.18367346938775506$$

## Counterfactual

```python
counterfactual = factual.cf(tree_prem=see_Z_is_0,
                                     cut_subj =Z_is_1)
show_counterfactual = counterfactual.show(show_prob=True,
                                          cut=Z_is_1,
                                          crit=counterfactual.critical(Z_is_1))
#show_counterfactual
```

Note that the counterfactual value for $W_{Z\leftarrow 1}$ is the same as the factual value for $W$ when intervening on $Z\leftarrow 1$. This is not always necessarily the case as demonstrated in [section 2.7.1](https://nbviewer.jupyter.org/github/COVID-19-Causal-Reasoning/probability_trees/blob/main/Causal_Reasoning_in_Probability_Trees.ipynb?flush_cache=true#Computing-a-counterfactual) of the Causal Reasoning in Probability Trees tutorial.

```python
given_Z_is_0_show_W_in_a_world_where_Z_is_1 = counterfactual.show(show_prob=True,
                                                                  cut=W,
                                                                  crit=counterfactual.critical(W))
display(Latex('$$P(W_{Z\leftarrow 1}|Z=0)='
              f'{counterfactual.prob(W)}$$'))
given_Z_is_0_show_W_in_a_world_where_Z_is_1
```

$$P(W_{Z\leftarrow 1}|Z=0)=0.18367346938775503$$

##

notebooks/PPL and Causality Probability Trees.ipynb


## Elements of Machine Learning

We consider ML problems involving data points with real-valued labels $y$, which represent some quantity of interest. We often refer to such ML problems as **regression problems**. In this notebook, we will apply some basic ML methods to solve a simple regression problem. These methods aim at finding or learning a good predictor function $h(\mathbf{x})$, which predicts the label $y$ based on the features $\mathbf{x}$ of a data point. The different methods are differnt combinations of particular choices for the type of predictor functions (hypothesis space) and loss function (the quality measure used to rank predictors). These different combinations offer different tradeoffs between **computational complexity, robustness (against outliers) and accuracy**.

## Learning goals

- know how to formulate a simple regression problem by identifying data points, their features, and labels
- know how to represent features and labels of data points using matrices and vectors
- apply ready-made regression methods to learn a good predictor function from labeled data
- assess and compare the computational complexity and accuracy of different regression methods

## Relevant Sections in [Course Book](https://arxiv.org/abs/1805.05052)

Chapter 2; Section 3.1-3.2; Chapter 4 and 5

## Background Material

[Video Lecture](https://www.youtube.com/watch?v=kHwlB_j7Hkc) on regression by [Prof. Andrew Ng](https://en.wikipedia.org/wiki/Andrew_Ng)

Additional information on the Python libraries used in this exercise can be found here:

- [NumPy](http://cs231n.github.io/python-numpy-tutorial/)
- [matplotlib](https://matplotlib.org/tutorials/index.html#introductory)
- [Pandas](https://pandas.pydata.org/pandas-docs/stable/getting_started/10min.html#min)
- [Slicing numpy arrays](https://www.pythoninformer.com/python-libraries/numpy/index-and-slice/)

## The Problem

Assume you have secured an internship at a real estate broker in downtown Helsinki. Your firm has gathered a dataset obtained from previously sold houses in different areas of the city. Your task is to find out the relation between different characteristics (features) $\mathbf{x}$ and the price (label) $y$ of a sold house.

In order to solve this task you can use ML methods to find a good predictor map $h(\mathbf{x})$ such that $h(\mathbf{x}) \approx y$. The basic principle behind these methods is simple: Out of a given set of predictor functions (e.g. the set of linear functions) pick one that fits well the data points $(\mathbf{x}^{(i)},y^{(i)})$, $i=1,\ldots,m$, for which we know the true label values $y^{(i)}$. For the house price prediction problem, this labeled data is provided by the recordings of previous house sales.

## The Data

Our goal is to learn a good predictor $h(\mathbf{x})$ for the price $y$ of a house. The prediction $h(\mathbf{x})$ is based on several features (characteristics) $\mathbf{x} = \big(x_{1},\ldots,x_{n}\big)^{T}$ such as the average number of rooms per dwelling $x_{1}$ or the nitric oxides concentration $x_{2}$ near the house.

To learn a good predictor $h(\mathbf{x})$, we use historic recordings of house sales. These recordings consist of $m$ data points. Each data pints is characterized by the house features $\mathbf{x}^{(i)} \in \mathbb{R}^{n}$ and the selling price $y^{(i)}$. ML methods find or learn a good predictor $h(\mathbf{x})$ by minimizing the average error $h(\mathbf{x}^{(i)}) - y^{(i)}$ for $i=1,\ldots,m$.

Demo. Loading the Data.

The following code snippet defines a function `X,y= GetFeaturesLabels(m,n)` which reads in data of previous house sales. The input parameters are the number `m` of data points and the number `n` of features to be used for each data point. The function returns a matrix $\mathbf{X}$ and vector $\mathbf{y}$.

The features $\mathbf{x}^{(i)}$ of the sold houses are stored in the rows of the numpy array `X` (of shape (m,n)) and the corresponding selling prices $y^{(i)}$ in the numpy array `y` (shape (m,1)). The two arrays represent the feature matrix $\mathbf{X} = \begin{pmatrix} \mathbf{x}^{(1)} & \ldots & \mathbf{x}^{(m)} \end{pmatrix}^{T}$ and the label vector $\mathbf{y} = \big( y^{(1)}, \ldots, y^{(m)} \big)^{T}$.

```python
# import "Pandas" library/package (and use shorthand "pd" for the package)
# Pandas provides functions for loading (storing) data from (to) files
import pandas as pd
from matplotlib import pyplot as plt
from IPython.display import display, HTML
import numpy as np
from sklearn.datasets import load_boston
import random

def GetFeaturesLabels(m=10, n=10):

    house_dataset = load_boston()
    house = pd.DataFrame(house_dataset.data, columns=house_dataset.feature_names)
    x1 = house['RM'].values.reshape(-1,1)   # vector whose entries are the average room numbers for each sold houses
    x2 = house['NOX'].values.reshape(-1,1)  # vector whose entries are the nitric oxides concentration for sold houses

    x1 = x1[0:m]
    x2 = x2[0:m]
    np.random.seed(30)
    X = np.hstack((x1,x2,np.random.randn(n,m).T))

    X = X[:,0:n]
    X = X - X.mean(axis = 0)

    y = house_dataset.target.reshape(-1,1)  # creates a vector whose entries are the labels for each sold house
    y = y[0:m]
    y = y - y.mean(axis = 0)

    return X, y

```

## Visualize Data

Scatter plots visualize data points by representing them as "dots" in the two-dimensional plane. Scatter plots can help to develop an intuition for the relation between features and labels of data points.

Demo. Scatter Plots.

The code snippet below creates one scatterplot showing the first feature $x^{(i)}_{1}$ and the label $y^{(i)}$ (house price) for each data point. It also creates a second scatterplot with the second feature $x^{(i)}_{2}$ and the label $y^{(i)}$ (house price) for each data point.

```python
from matplotlib import pyplot as plt

X,y = GetFeaturesLabels(10,10)
fig, axs = plt.subplots(1, 2,figsize=(15,5))
axs[0].scatter(X[:,0], y)
axs[0].set_title('average number of rooms per dwelling vs. price')
axs[0].set_xlabel(r'feature $x_{1}$')
axs[0].set_ylabel('house price $y$')

axs[1].scatter(X[:,1], y)
axs[1].set_xlabel(r'feature $x_{2}$')
axs[1].set_title('nitric oxide level vs. price')
axs[1].set_ylabel('house price $y$')

plt.show()
```

## Linear Regression

Our goal is to predict the price $y$ of a house based on $n$ properties or features $\mathbf{x}=(x_{1},\ldots,x_{n})^{T} \in \mathbb{R}^{n}$ of that house. To this end, we try to find (or learn) a predictor function $h(\mathbf{x})$ such that $y \approx h(\mathbf{x})$.

Within linear regression, we restrict the predictor function to be a linear function. The resulting hypothesis space of all such linear predictor functions is
\begin{equation*}
 \mathcal{H} = \{h^{(\mathbf{w})}(\mathbf{x}) = \mathbf{w}^{T} \mathbf{x} \mbox{ for some } \mathbf{w} \in \mathbb{R}^{n}\}.
\label{eq1}
\tag{1}
\end{equation*}
Note that,for two vectors $\mathbf{w}=\big(w_{1},\ldots,w_{n}\big)^{T} \in \mathbb{R}^{n}$ and $\mathbf{x}=\big(x_{1},\ldots,x_{n}\big)^{T}\in \mathbb{R}^{n}$, we denote $\mathbf{w}^{T} \mathbf{x} = \sum_{r=1}^{n} w_{r} x_{r}$.

To measure the quality of a particular predictor $h^{(\mathbf{w})}(\mathbf{x}) = \mathbf{w}^{T} \mathbf{x}$, obtained for some particular choice for the weight vector $\mathbf{w} \in \mathbb{R}^{n}$, we try it on the historic recordings of house sales. For the $i$th house sale record we know the house price $y^{(i)}$ that has been achieved and can compare it to the predicted price $h(\mathbf{x}^{(i)})$.

The prediction $h^{(\mathbf{w})}(\mathbf{x}^{(i)})$ will typically incur a non-zero __prediction error__ $y^{(i)} - h^{(\mathbf{w})}(\mathbf{x}^{(i)})$. To measure the error or **loss** incurred by predicting the true label value $y$ of a data point using the predicted label $\hat{y}=h(\mathbf{x})$ (which is the result of applying the predictor function $h(\cdot)$ to the features $\mathbf{x}$), we need to define a loss function $\mathcal{L}(y,\hat{y})$. In principle, the loss function is a design parameter which can be chosen freely by the ML scientist or engineer (based on the application at hand).

For regression problems with numeric labels $y$, a popular choice for the loss is the [squared error loss](https://scikit-learn.org/stable/modules/model_evaluation.html#mean-squared-error) $\mathcal{L}(y,\hat{y}) = (y-\hat{y})^{2}$. We highlight that to evaluate the loss we need to know the true label $y$ of a data point. However, this is the case for the data points $(\mathbf{x}^{(i)},y^{(i)})$ representing previous house sales. We can then average the predictor loss over all $m$ data points $(\mathbf{x}^{(i)},y^{(i)})$ of previously sold houses:
\begin{equation*}
 \mathcal{E} (\mathbf{w}) = (1/m) \sum^{m}_{i=1}(y^{(i)} - \mathbf{w}^{T} \mathbf{x}^{(i)})^2.
\label{eq2}
\tag{2}
\end{equation*}

The optimal weight vector $\mathbf{w}_{\rm opt}$ is any weight vector which achives the minimum value of $ \mathcal{E} (\mathbf{w})$, i.e., $$\mathcal{E} (\mathbf{w}_{\rm opt})= \min_{\mathbf{w} \in \mathbb{R}^{n}}\mathcal{E} (\mathbf{w}).$$
An optimal predictor is then obtained as $h(\mathbf{x}) = \mathbf{w}_{\rm opt}^{T} \mathbf{x}$. The average loss

$$\mathcal{E} (\mathbf{w}_{\rm opt}) = (1/m) \sum^{m}_{i=1}(y^{(i)} - \mathbf{w}_{\rm opt}^{T} \mathbf{x}^{(i)})^2$$

incurred by the optimal predictor $h(\mathbf{x}) = \mathbf{w}_{\rm opt}^{T} \mathbf{x}$ is also known as the **training error**, since it is the error obtained when fitting or training a linear model to some labeled data points, which serve as the **training data**.

Demo. Learn a Linear Predictor.

The code snippet below shows how to use the `LinearRegression` class from the Python library `scikit-learn` to learn an optimal predictor from the set of linear predictors $h(\mathbf{x}) = \mathbf{w}^{T} \mathbf{x}$ using the squared error loss.

The optimal weight vector $\mathbf{w}_{\rm opt}$ can be found by the function `LinearRegression.fit()`. The resulting optimal weight vector is stored in the variable `LinearRegression.coef_`. Using the optimal weight vector, we compute the training error
\begin{equation}
(1/m) \sum_{i=1}^{m} \big( y^{(i)} - \mathbf{w}_{\rm opt}^{T} \mathbf{x}^{(i)} \big)^{2}.
\end{equation}
and store it in the variable `training_error`.

You can find the documentation of the `LinearRegression` class under [this link](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html).

```python
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from IPython.display import display, Math

reg = LinearRegression(fit_intercept=False)
reg = reg.fit(X, y)
training_error = mean_squared_error(y, reg.predict(X))

display(Math(r'$\mathbf{w}_{\rm opt} ='))
optimal_weight = reg.coef_
optimal_weight = optimal_weight.reshape(-1,1)
print(optimal_weight)
print("\nThe resuling training error is ",training_error)
```

$\displaystyle \mathbf{w}_{\rm opt} =$

    [[ 32.5939049 ]
     [-19.90275233]
     [  2.6463634 ]
     [ 16.20766644]
     [ 10.69852006]
     [  0.38989091]
     [-19.20441205]
     [ -8.90217068]
     [ 14.3588369 ]
     [ 13.58536119]]

    The resuling training error is  5.198593365406468e-28

Student Task Varying Number of Features.

In principle, we can freely choose how many of the available features $x_{1}, x_{2},\ldots,x_{n}$
of a house you want to use, to predict the house price $y$. You are now to explore the effect of using a varying number $r \leq n$ of features on the resulting training error and computational complexity (runtime) of linear regression. For each $r=1,2,\ldots,10$, fit a linear model (using `LinearRegression(fit_intercept=False)` to the house sales dataset (using $m=10$ data points) by using only the first $r$ features $x_{1},...,x_{r}$ of a house.

- You can get the first $r$ features and labels for the previously sold houses using `GetFeaturesLabels(m,r)`.

- For each value of $r$, determine the computation time of the fitting method `LinearRegression.fit()` and the resulting training error (using the Python function `mean_squared_error()`) of the fitted linear model.

- The results should be stored in two vectors `linreg_time` (multiply by 1000 to get time in milliseconds) and `linreg_error`.

- You can measure the computation time with Python function `time.time()` as demonstrated in Round 1.

```python
import time

m = 10                            # we use 10 data points of the house sales database
max_r = 10                        # maximum number of features used

X,y = GetFeaturesLabels(m,max_r)  # read in m data points using max_r features

linreg_time = np.zeros(max_r)     # vector for storing the exec. times of LinearRegresion.fit() for each r
linreg_error = np.zeros(max_r)    # vector for storing the training error of LinearRegresion.fit() for each r

# linreg_time = ...
# linreg_error = ...
# Hint: loop "r" times.

### BEGIN SOLUTION
for r in range(max_r):
    reg = LinearRegression(fit_intercept=False)
    start_time = time.time()
    reg = reg.fit(X[:,:(r+1)], y)
    end_time = (time.time() - start_time)*1000
    linreg_time[r] = end_time
    pred = reg.predict(X[:,:(r+1)])
    linreg_error[r] = mean_squared_error(y, pred)
### END SOLUTION

plot_x = np.linspace(1, max_r, max_r, endpoint=True)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))
axes[0].plot(plot_x, linreg_error, label='MSE', color='red')
axes[1].plot(plot_x, linreg_time, label='time', color='green')
axes[0].set_xlabel('features')
axes[0].set_ylabel('empirical error')
axes[1].set_xlabel('features')
axes[1].set_ylabel('time (ms)')
axes[0].set_title('training error vs number of features')
axes[1].set_title('computation time vs number of features')
axes[0].legend()
axes[1].legend()
plt.tight_layout()
plt.show()
```

Demo. Varying Number of Data Points.

Beside the number of features which are used to describe a data point, another important parameter of a data set is the number $m$ of data points it contains. Intuitively, having more labeled data points to find or learn a good predictor should be beneficial. Let us now explore the effect of using an increasing number $m$ of data points on the resulting training error.

- For each choice $m=1,2,\ldots,10$ read in the first $m$ data points of previously sold houses using $n=2$ features. This can be done using `GetFeaturesLabels(m,2)`.
- For each $m$, fit a linear model (using `LinearRegression(fit_intercept=False)` to the first $m$ data points.
- Store the resulting training errors in the numpy array `train_error` of shape (10,1). The $m$th entry of this array should be the training error obtained when using $m$ data points. Note that the first entry of a numpy array has index $0$.

```python
import time

max_m = 10                            # maximum number of data points
X, y = GetFeaturesLabels(max_m, 2)      # read in max_m data points using n=2 features
train_error = np.zeros(max_m)         # vector for storing the training error of LinearRegresion.fit() for each r

for r in range(max_m):
    reg = LinearRegression(fit_intercept=False)
    reg = reg.fit(X[:(r+1),:], y[:(r+1)])
    y_pred = reg.predict(X[:(r+1),:])
    train_error[r] = mean_squared_error(y[:(r+1)], y_pred)

print(train_error[2])
plot_x = np.linspace(1, max_r, max_r, endpoint=True)
fig, axes = plt.subplots(nrows=1, ncols=1, figsize=(8, 4))
axes.plot(plot_x, train_error, label='MSE', color='red')
axes.set_xlabel('number of data points (sample size)')
axes.set_ylabel('training error')
axes.set_title('training error vs. number of data points')
axes.legend()
plt.tight_layout()
plt.show()
```

Demo. Robustness Against Outliers.

We now consider an important aspect of ML methods, i.e., the robustness to outliers. An **outlier** refers to a data point which is intrinsically different from all other data points, e.g., due to measurement errors. Imagine that for some reason, the first data point in our house sales database is such an erroneous outlier. We would then prefer ML methods not to be affected too much by an error in a single data point.

The code snippet below considers fitting a linear model for house prices $y$ based on a single feature $x_{1}$. The resulting linear predictor minimizes the average squared error loss on the training data. We then intentionally perturb the first data point by setting $y_{1}^{(1)}$ to an unreasonable value. Using this corrupted data set, we then fit again a linear model and compare the so-obtained linear predictor to the linear predictor obtained from the "clean" data set.

```python
from sklearn import linear_model

X,y = GetFeaturesLabels(10,1)   # read in 10 data points with single feature x_1 and label y

### fit a linear model to the clean data
reg = linear_model.LinearRegression(fit_intercept=False)
reg = reg.fit(X, y)
y_pred = reg.predict(X)

# now we intentionally perturb the label of the first data point

y_perturbed = np.copy(y)
y_perturbed[8] = 15;

### fit a linear model to the perturbed data

reg1 = linear_model.LinearRegression(fit_intercept=False)
reg1 = reg1.fit(X, y_perturbed)
y_pred_perturbed = reg1.predict(X)

fig, axes = plt.subplots(1, 2, figsize=(15, 4))
axes[0].scatter(X, y, label='data')
axes[0].plot(X, y_pred, color='green', label='Fitted model')

#np.savetxt("foo.csv", np.hstack((X,y)), delimiter=",",header="x,y")
df2 = pd.DataFrame(np.hstack((X,y,y_pred)),columns=['x', 'y','yhat'])
df2.to_csv("cleandata.csv",index=False)

df3 = pd.DataFrame(np.hstack((X,y_perturbed,y_pred_perturbed)),columns=['x', 'y','yhat'])
df3.to_csv("corrupteddata.csv",index=False)

# now add individual line for each error point
axes[0].plot((X[0], X[0]), (y[0], y_pred[0]), color='red', label='errors') # add label to legend
for i in range(len(X)-1):
    lineXdata = (X[i+1], X[i+1]) # same X
    lineYdata = (y[i+1], y_pred[i+1]) # different Y
    axes[0].plot(lineXdata, lineYdata, color='red')

axes[0].set_title('fitted model using clean data')
axes[0].set_xlabel('feature x_1')
axes[0].set_ylabel('house price y')
axes[0].legend()

axes[1].scatter(X, y_perturbed, label='data')
axes[1].plot(X, y_pred_perturbed, color='green', label='Fitted model')

# now add individual line for each error point
axes[1].plot((X[0], X[0]), (y_perturbed[0], y_pred_perturbed[0]), color='red', label='errors') # add label to legend
for i in range(len(X)-1):
    lineXdata = (X[i+1], X[i+1]) # same X
    lineYdata = (y_perturbed[i+1], y_pred_perturbed[i+1]) # different Y
    axes[1].plot(lineXdata, lineYdata, color='red')

axes[1].set_title('fitted model using perturbed training data')
axes[1].set_xlabel('feature x_1')
axes[1].set_ylabel('house price y')
axes[1].legend()

plt.show()
plt.close('all') # clean up after using pyplot

print("optimal weight w_opt by fitting to (training on) clean training data : ", reg.coef_)
print("optimal weight w_opt by fitting to (training on) perturbed training data : ", reg1.coef_)

print(y)

```

## Using Different Loss Function

We observe from the demo above that the resulting linear predictor is heavily affected by corrupting only one single data point. The reason for this sensitivity is rooted in the properties of the squared error loss function used by the class `LinearRegression()`. Indeed by using the loss $(\hat{y} - y)^{2}$ we force the predictor $\hat{y}$ to not be too far away from any data point with very large value $y$ of the true label.

It turns out that using a different loss function to learn a linear predictor can make the learning robust against few outliers. One such robust loss function is known as "Huber loss" $\mathcal{L}(\hat{y},y)$. Given a data point with label $y$ and a predicted label $\hat{y}=h(\mathbf{x})$ the Huber loss is defined as
$$\mathcal{L}(y,\hat{y}) = \begin{cases} (1/2) (y-\hat{y})^{2} & \mbox{ for } |y-\hat{y}| \leq  c \\
c (|y-\hat{y}| - c/2) & \mbox{ else. }\end{cases}$$

Note that the Huber loss contains a parameter $c$ which has to be adapted to the application at hand. To learn a linear predictor $h(\mathbf{x}) = \mathbf{w}^{T} \mathbf{x}$ which minimizes the average Huber loss over the labeled data points in the training set we can use the [`HuberRegressor()`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.HuberRegressor.html) class.

The Huber loss contains two important special cases. The first special case is obtained when $c$ is chosen very large (the precise value depending on the value range of the features and labels) such that the condition $|y-\hat{y}| \leq  c$ is always satisfied. In this case, the Huber loss becomes the squared error loss $(y-\hat{y})^{2}$ (with an additional factor 1/2). The second special case is obtained for choosing $c$ very small (close to $0$) such that the condition $|y-\hat{y}| \leq  c$ is never satisfied. In this case, the Huber loss becomes the absolute loss $|y - \hat{y}|$ scaled by a factor $c$.

Demo. Squared Error and Huber Loss.

The code below plots the squared error loss and the Huber loss for different choices of the parameter $c$. Note that the Huber loss reduces to the squared error loss for a sufficiently large value of the parameter $c$.

```python
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
import numpy as np
from matplotlib import pyplot as plt

#------------------------------------------------------------
# Define the Huber loss
def Phi(t, c):
    t = abs(t)
    flag = (t > c)
    return (~flag) * (0.5 * t ** 2) - (flag) * c * (0.5 * c - t)

#------------------------------------------------------------
# Plot for several values of c
fig = plt.figure(figsize=(10, 3.75))
ax = fig.add_subplot(111)

x = np.linspace(-5, 5, 100)

for c in (1,2,10):
    y = Phi(x, c)
    ax.plot(x, y, '-k')

    if c > 10:
        s = r'\infty'
    else:
        s = str(c)

    ax.text(x[6], y[6], '$c=%s$' % s,
            ha='center', va='center',
            bbox=dict(boxstyle='round', ec='k', fc='w'))

ax.plot(x,np.square(x),label="squared loss")

ax.set_xlabel(r'$y - \hat{y}$')
ax.set_ylabel(r'loss $\mathcal{L}(y,\hat{y})$')
ax.legend()
plt.show()
```

Demo. Robustness Against Outliers II.

The code snippet below fits a linear model for house prices $y$ based on a single feature $x_{1}$ using the Huber loss. We also intentionally perturb the first data point by setting $y_{1}^{(1)}$ to an unreasonable value. Using this corrupted data set, we fit a linear model (under Huber loss) again and compare the so-obtained linear predictor to the linear predictor obtained from the "clean" data set.

```python
from sklearn import linear_model
from sklearn.linear_model import HuberRegressor

X,y = GetFeaturesLabels(10,1)   # read in 10 data points with single feature x_1 and label y

### fit a linear model (using Huber loss) to the clean data
reg = HuberRegressor(fit_intercept=False)
reg = reg.fit(X, y)
y_pred = reg.predict(X).reshape(-1,1)

# now we intentionaly perturb the label of the first data point

y_perturbed = np.copy(y)
y_perturbed[8] = 0;

### fit a linear model (using Huber loss) to the perturbed data

#reg1 = linear_model.LinearRegression(fit_intercept=False)
reg1 = HuberRegressor (fit_intercept=False) ;

reg1 = reg1.fit(X, y_perturbed)
y_pred_perturbed = reg1.predict(X).reshape(-1,1)

fig, axes = plt.subplots(1, 2, figsize=(15, 4))
axes[0].scatter(X, y, label='data')

#np.savetxt("foo.csv", np.hstack((X,y)), delimiter=",",header="x,y")
df2 = pd.DataFrame(np.hstack((X,y,y_pred)),columns=['x', 'y','yhat'])
df2.to_csv("cleandata_huber.csv",index=False)

df3 = pd.DataFrame(np.hstack((X,y_perturbed,y_pred_perturbed)),columns=['x', 'y','yhat'])
df3.to_csv("corrupteddata_huber.csv",index=False)

axes[0].plot(X, y_pred, color='green', label='Fitted model')

# now add individual line for each error point
axes[0].plot((X[0], X[0]), (y[0], y_pred[0]), color='red', label='errors') # add label to legend
for i in range(len(X)-1):
    lineXdata = (X[i+1], X[i+1]) # same X
    lineYdata = (y[i+1], y_pred[i+1]) # different Y
    axes[0].plot(lineXdata, lineYdata, color='red')

axes[0].set_title('fitted model using clean data')
axes[0].set_xlabel('feature x_1')
axes[0].set_ylabel('house price y')
axes[0].legend()

axes[1].scatter(X, y_perturbed, label='data')
axes[1].plot(X, y_pred_perturbed, color='green', label='Fitted model')

# now add individual line for each error point
axes[1].plot((X[0], X[0]), (y_perturbed[0], y_pred_perturbed[0]), color='red', label='errors') # add label to legend
for i in range(len(X)-1):
    lineXdata = (X[i+1], X[i+1]) # same X
    lineYdata = (y_perturbed[i+1], y_pred_perturbed[i+1]) # different Y
    axes[1].plot(lineXdata, lineYdata, color='red')

axes[1].set_title('fitted model using perturbed data')
axes[1].set_xlabel('feature x_1')
axes[1].set_ylabel('house price y')
axes[1].legend()

plt.show()
plt.close('all') # clean up after using pyplot

print("optimal weight w_opt by fitting on clean data : ", reg.coef_)
print("optimal weight w_opt by fitting on perturbed data : ", reg1.coef_)
```

Demo Varying Number of Features with Huber Loss.

In principle you can choose how many of the available features $x_{1}, x_{2},\ldots,$
of a house you want to use to in order to predict the house price $y$. Let us now explore
the effect of using a varying number $r$ of features on the resulting error and
computational complexity (runtime).

In particular, for each $r=1,2,\ldots,10$, the code snippet below fits a linear model under
Huber loss to the house sales dataset (using $m=10$ data points) by using only the
first $r$ features $x_{1},...,x_{r}$ of a house.

- The first $r$ features and labels for the previously sold houses can be obtained using `GetFeaturesLabels(m,r)`.

- For each value of $r$, the computation time of the fitting method `HuberRegressor.fit()` and the resulting training error (using the Python function `mean_squared_error()`) of the fitted linear model.

- The results are stored in two vectors `linreg_time` (multiply by 1000 to get time in milliseconds) and `linreg_error`.

```python
import time

m = 10                            # we use 100 data points of the house sales database
max_r = 10                        # maximum number of features used

X,y = GetFeaturesLabels(m,max_r)  # read in 100 data points using 10 features

linreg_time = np.zeros(max_r)     # vector for storing the exec. times of LinearRegresion.fit() for each r
linreg_error = np.zeros(max_r)    # vector for storing the training error of LinearRegresion.fit() for each r

for r in range(max_r):
    reg_hub = HuberRegressor(fit_intercept=False)
    start_time = time.time()
    reg_hub = reg_hub.fit(X[:,:(r+1)], y)
    end_time = (time.time() - start_time)*1000
    linreg_time[r] = end_time
    pred = reg_hub.predict(X[:,:(r+1)])
    linreg_error[r] = mean_squared_error(y, pred)

plot_x = np.linspace(1, max_r, max_r, endpoint=True)
fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(8, 4))
axes[0].plot(plot_x, linreg_error, label='MSE', color='red')
axes[1].plot(plot_x, linreg_time, label='time', color='green')
axes[0].set_xlabel('features')
axes[0].set_ylabel('empirical error')
axes[1].set_xlabel('features')
axes[1].set_ylabel('Time (ms)')
axes[0].set_title('training error vs number of features')
axes[1].set_title('computation time vs number of features')
axes[0].legend()
axes[1].legend()
plt.tight_layout()
plt.show()
```

```python

```

```python

```

```python

```