# Recommendation engines explained

We start by importing the relevant [Apache Spark](https://spark.apache.org/) modules and create a Spark session:

```python
%matplotlib inline

from pyspark import SparkContext, SparkConf
from pyspark.sql import Row, SparkSession
import matplotlib.pyplot as plt

spark = SparkSession.builder \
        .master("local") \
        .appName("recommender") \
        .config("spark.sql.crossJoin.enabled", "true") \
        .getOrCreate()
```

## The MovieLens dataset

The MovieLens dataset (already available with this notebook, originally [here](https://grouplens.org/datasets/movielens/)) is a popular dataset used in recommendation engines research.
In this workshop we will use the "small" variant, consisting of $100836$ ratings for $9742$ movies by $610$ users.
This dataset consist of several CSV files, namely
 * A ratings data file (`ratings.csv`)
 * A tags data file (`tags.csv`)
 * A movies data file (`movies.csv`)
 * A links data file (`links.csv`)
 * A tag genome (`genome-scores.csv` and `genome-tags.csv`)

 For the scope of this workshop, we are mainly interest in the rating and movies data files, which have, respectively, the following structure:
 The ratings data:
 * `userId`, a unique numerical user id
 * `movieId`, a unique numerical movie id
 * `rating`, a numerical rating, as an integer from $0$ to $5$
 * `timestamp`, a numerical timestamp (not used in this exercise)
 The movies data:
 * `movieId`, as above
 * `title`, the movie title
 * `genres` a string label with the movie genre (as listed [here](http://files.grouplens.org/datasets/movielens/ml-20m-README.html)).

 ### Loading the data

 We start by using Apache Spark's built in functionality to load text files into the Spark cluster, `.textFile()` (documented [here](http://spark.apache.org/docs/2.1.0/api/python/pyspark.html#pyspark.SparkContext.textFile)), which will return an `RDD` of strings.
 Since `.textFile()` will read each row in the CSV file as an RDD entry, we further process it by then splitting each row into a list of CSV entries and keep only the relevant fields (i.e. `userId`, `movieId` and `rating` for the ratings, and `movieId` and `title` for the movies file).

 We then create a Spark [dataframe](https://spark.apache.org/docs/latest/sql-programming-guide.html) containing the entries.

```python
def load_data(path, header, token_fun):
    """Loads the CSV file, strips the header and keeps only the specified fields"""
    rows = spark.sparkContext.textFile(path).filter(lambda x: x!=header).map(lambda x: x.split(",")).map(token_fun)
    # create a Spark dataframe from the RDDs
    return spark.createDataFrame(rows).cache()

# Load the movie data (movie id, title)
movies = load_data('ml-latest-small/movies.csv',
                   'movieId,title,genres',
                   lambda tokens: Row(item=int(tokens[0]),title=tokens[1]))

# Load the ratings (user id, movie id, rating)
ratings = load_data('ml-latest-small/ratings.csv',
                    'userId,movieId,rating,timestamp',
                    lambda tokens: Row(user=int(tokens[0]), item=int(tokens[1]), rating=float(tokens[2])))
```

```python
print("Number of movies: {}".format(movies.count()))
print("Number of ratings: {}".format(ratings.count()))
print("Number of users: {}".format(ratings.select('user').distinct().count()))
movies.take(5)
```

    Number of movies: 9742
    Number of ratings: 100836
    Number of users: 610

    [Row(item=1, title='Toy Story (1995)'),
     Row(item=2, title='Jumanji (1995)'),
     Row(item=3, title='Grumpier Old Men (1995)'),
     Row(item=4, title='Waiting to Exhale (1995)'),
     Row(item=5, title='Father of the Bride Part II (1995)')]

We can now start by exploring the data.
Let's look at the ratings distribution:

```python
import seaborn as sns

r = ratings.select('rating').collect()
sns.distplot(r, kde=False)
```

We can see that most of the ratings are $4$ starts, followed by $3$ and $3.5$ stars. Users generally do not give very low ratings (e.g. $0.5$, $1$).

Let's try to see now the distribution of the number of ratings per user. That is, so user give many ratings on average?

```python
ratings_per_user = ratings.groupBy("user").count()
sns.distplot(ratings_per_user.select("count").collect(), hist_kws={'log':True}, kde_kws={'clip': (1.0, 3000.0)})
```

We can see that we have "super users", i.e. users that give more than 2000 ratings!
From this histogram its not easy to see how many they are, so lets find out.

```python
from pyspark.sql.functions import col

ratings_per_user.filter(col('count') > 1500).groupBy('user').count().agg({"count":"sum"}).show()
```

    +----------+
    |sum(count)|
    +----------+
    |         4|
    +----------+

So we only have $4$ "super" users. Let's now see how many users we have with less that 100 ratings:

```python
ratings_per_user.filter(col('count') < 200).groupBy('user').count().agg({"count":"sum"}).show()
```

    +----------+
    |sum(count)|
    +----------+
    |       476|
    +----------+

We can see that the vast majority of users gave less that $200$ ratings, which is a sensible number.

What about the movies? Do movies tend to have the same number of reviews or not? We can check by running:

```python
ratings_per_movie = ratings.groupBy("item").count()
sns.distplot(ratings_per_movie.select("count").collect(), hist_kws={'log':True}, kde_kws={'clip': (0, 3000.0)})
```

```python
ratings_per_movie.filter(col('count') < 2).groupBy('item').count().agg({"count":"sum"}).show()
```

    +----------+
    |sum(count)|
    +----------+
    |      3446|
    +----------+

# Alternating Least Squares (ALS)

Alternating Least Squares (ALS) is a standard algorithm for collaborative filtering. Its main rationale is that (conditioned on some item) users that agreed before are more likely to agree in the future.
In ALS, we aim at building a correlation matrix $M_{u,i}$ such that, given a rating $r$ given by a user $u$ to an item $i$ we have:

$$M_{u,i}=\begin{cases}
    r,& \text{if user}\ u\ \text{rated item}\ i\\
    0,& \text{if user}\ u\ \text{didn't rate item}\ i
\end{cases}$$

Provided we have a factor matrix $A$ for the users and a factor matrix $B$ for the movies, as the name implies, we use alternating least squares to first estimate $B\left(A\right)$ and then $A\left(B\right)$ until we reach convergence.

An important characteristics is that we want to penalise items without ratings, and as such we use

$$ w_{u,i}=\begin{cases}
    0,& \text{if}\ M_{u,i}=0\\
    1,& \text{otherwise}
\end{cases}$$

The cost functions to minimise are:
$$
\begin{align}
J\left(a_u\right) &= \left(m_u - a_u B\right)W_u\left(m_u-a_u B\right)^T + \lambda a_u a_u^T \\
J\left(b_i\right) &= \left(m_i - A b_i B\right)W_i\left(m_i-A b_i \right)^T + \lambda b_i b_i^T
\end{align}
$$

This is minimised using

$$
\begin{align}
a_u = \left(B W_u B^T + \lambda I\right)^{-1}B W_u m_u \\
b_u = \left(A W_i A^T + \lambda I\right)^{-1}A W_i m_i
\end{align}
$$

## Training the model

To train Spark's ALS model, we need to perform a few steps.
The first step is to define a metric to quantify "how good" our model is. This is done by chosing an error measure, which typically is the _mean squared error_ (MSE). The mean square error is formally defined by

$$
\text{MSE} = \frac{1}{N}\sum_{i=1}^{N}\epsilon_i^2, \qquad \epsilon_i = r_i - \hat{r}_i.
$$

That is, we first calculate the difference, $\epsilon_i$, between each pair of true rating ($r_i$) and it's corresponding _predicted_ rating ($\hat{r}_i$) and then calculated the average of the square of these errors. As example, consider a list of ratings `r1` and corresponding predictiong `r2`:

```python
r1 = [4.5, 3.0, 1.0, 5.0, 4.0]
r2 = [4.56, 2.9, 1.1, 4.9, 3.2]
```

```python
def calc_mse(r, p):
    return sum([(pair[0] - pair[1])**2 for pair in zip(r, p)]) / len(r)

print("MSE = {}".format(calc_mse(r1, r2)))
```

    MSE = 0.13471999999999992

As we've seen above, the prediction for user $i$ and item $j$, $\hat{r}_{i,j,}$ is simply the product between the corresponding user and item vectors, that is

$$\hat{r}_{i,j} = U^T_{i}P_{j}$$

which, in practice, leads to a calculation of the Squared Error (SE) as below, which we will then average across the dataset used for validation:

```python
def calc_se(rating, user_factor, item_factor):
    """Squared Error (SE) for a single rating and prediction"""
    prediction = user_factor.T.dot(item_factor)
    return (rating - prediction) ** 2
```

Since we've established our error measure, we can proceed to the next step.
We now need to split our data into a _training_, _test_ and _validation_ subsets.
We will use respectively the proportions 63%, 18% and 18% (you can try other ratios).

```python
def split_sets(ratings, proportions):
    split = ratings.randomSplit(proportions, seed=42)
    return {'training': split[0], 'validation': split[1], 'test': split[2]}

sets = split_sets(ratings, [0.63212056, 0.1839397, 0.1839397])
print("Training dataset size = {}".format(sets['training'].count()))
print("Validation dataset size = {}".format(sets['validation'].count()))
print("Test dataset size = {}".format(sets['test'].count()))
```

    Training dataset size = 63952
    Validation dataset size = 18464
    Test dataset size = 18420

According to Spark's documentation, the [parameters available](https://spark.apache.org/docs/latest/api/python/pyspark.ml.html?highlight=als#module-pyspark.ml.recommendation) for the Spark ALS class are:

```
ALS(rank=10, maxIter=10, regParam=0.1, numUserBlocks=10,
    numItemBlocks=10, implicitPrefs=False, alpha=1.0,
    userCol='user', itemCol='item', seed=None, ratingCol='rating',
    nonnegative=False, checkpointInterval=10,
    intermediateStorageLevel='MEMORY_AND_DISK',
    finalStorageLevel='MEMORY_AND_DISK', coldStartStrategy='nan')
```
which are described as follow:

* `rank`, Rank of the feature matrices computed (number of features, default $10$)
* `maxIter`, Maximum number of iterations of ALS. (default: 10)
* `regParam`, Regularization parameter. (default: 0.1)
* `numUserBlocks`, Number of blocks used to parallelize the computation. A value of -1 will use an auto-configured number of blocks. (default: 10)
* `nonnegative`, A value of True will solve least-squares with nonnegativity constraints. (default: False)
* `seed`, Random seed for initial matrix factorization model. A value of None will use system time as the seed. (default: None)
* `implicitPrefs`, whether we are using implicit data or not (more on that later)
* `userCol`, name of the Dataframe column representing user ids
* `itemCol`, name of the Dataframe column representing movie ids
* `coldStrategy`, Strategy for dealing with unknown or new users/items at prediction time. This may be useful in cross-validation or production scenarios, for handling user/item ids the model has not seen in the training data. (either `nan` (default) or `drop`)

Let's try an ALS model with some arbitrary parameters, namely `rank=3`, `maxIter=5`, `regParam=0.1` and `coldStrategy='drop'`.

```python
from pyspark.ml.recommendation import ALS

simple_als = ALS(maxIter=5, regParam=0.01, rank=3, coldStartStrategy='drop')
```

After the `ALS` class instantiation, we specify the data to be used for training using the `.fit()` method:

```python
%time simple_model = simple_als.fit(sets['training'])
```

    CPU times: user 24.1 ms, sys: 8.38 ms, total: 32.5 ms
    Wall time: 3.21 s

As we can see from the above, the training was quite quick ($2$ seconds in my case, yours may be different).

We can now calculate the MSE for this model, with the test datasets. To do so, we use the [Spark's](http://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#pyspark.ml.evaluation.RegressionEvaluator) `RegressionEvaluator` class. In this class we can specify the error measure (`mse`, `rmse`, etc.) and the names of the labels (in our case the ratings) and the predictions columns.

```python
from pyspark.ml.evaluation import RegressionEvaluator

evaluator = RegressionEvaluator(
    metricName="mse",
    labelCol="rating",
    predictionCol="prediction")

prediction = simple_model.transform(sets['test'])
mse = evaluator.evaluate(prediction)
print("MSE = {}".format(mse))
```

    MSE = 1.2082838419149144

A common sanity test for a model is also to check if there the error measure is similar to using the _mean rating_ as the prediction.
We start by calculating the mean rating from the dataframe:

```python
mean_rating = sets['test'].groupBy().mean('rating').collect()[0]['avg(rating)']
print("Mean rating = {}".format(mean_rating))
```

    Mean rating = 3.4911780673181325

We create a dataframe to store all the movie ids, but we'll replace the actual prediction from the previous model, with the mean rating:

```python
from pyspark.sql.functions import lit

mean_prediction = prediction.withColumn('prediction', lit(mean_rating))
mean_prediction.show()
```

    +----+------+----+------------------+
    |item|rating|user|        prediction|
    +----+------+----+------------------+
    | 471|   2.5| 599|3.4911780673181325|
    | 471|   4.0| 603|3.4911780673181325|
    | 471|   4.5| 182|3.4911780673181325|
    | 471|   1.0| 500|3.4911780673181325|
    | 471|   5.0| 520|3.4911780673181325|
    | 471|   4.0| 411|3.4911780673181325|
    | 471|   3.0| 541|3.4911780673181325|
    | 471|   4.5| 260|3.4911780673181325|
    |1088|   3.0| 606|3.4911780673181325|
    |1088|   2.5| 599|3.4911780673181325|
    |1088|   4.0| 563|3.4911780673181325|
    |1088|   3.5| 381|3.4911780673181325|
    |1088|   4.0| 200|3.4911780673181325|
    |1088|   3.5| 600|3.4911780673181325|
    |1088|   3.0|  42|3.4911780673181325|
    |1088|   1.0| 517|3.4911780673181325|
    |1238|   4.0| 325|3.4911780673181325|
    |1342|   1.0| 223|3.4911780673181325|
    |1342|   2.0|  19|3.4911780673181325|
    |1342|   3.5| 425|3.4911780673181325|
    +----+------+----+------------------+
    only showing top 20 rows

Finally we calculate the MSE for this new set of "predictions":

```python
mse = evaluator.evaluate(mean_prediction)
print("MSE (using mean) = {}".format(mse))
```

    MSE (using mean) = 1.0893233836723415

We can see that although the model we've just trained is better than the mean baseline, it could be much better perhaps?
Next we will se how to improve our model, namely how to select the best parameters instead of using arbitrary ones.

### Parameter estimation

The question now is, how do we know which parameters are the best? We first start by the definition of "best".
In this case, the "best" parameters will be the ones that would generate a model which when used with a dataset (other than the training), would produce the most accurate predictions. We've [seen above](#Training-the-model) that a way of determining this accuracy would be to use an error measure such the MSE.

To determine the parameters, we then use a "brute force" method: we train the model with a range of parameters, calculate the MSE, and choose the parameter combination which provides the smallest MSE.

We could of course do this manually, but Spark provives an utility method to automate the task for us.
We use the [`RegressionEvaluator` class](http://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#pyspark.ml.evaluation.RegressionEvaluator) to calculate the MSE for each parameter combination and a [`TrainValidationSplit` class](http://spark.apache.org/docs/2.2.0/api/python/pyspark.ml.html#pyspark.ml.tuning.TrainValidationSplit) to perform the actual training.

(Below, we add the `coldStartStrategy` to the ALS class so that we automatically drop any `NaN` predictions.)

```python
from pyspark.ml.recommendation import ALS
from pyspark.ml.tuning import TrainValidationSplit, ParamGridBuilder
from pyspark.ml.evaluation import RegressionEvaluator

als = ALS(coldStartStrategy="drop")

param_grid = ParamGridBuilder() \
    .addGrid(als.rank, [6, 8]) \
    .addGrid(als.maxIter,[10, 12]) \
    .build()

evaluator = RegressionEvaluator(
    metricName="mse",
    labelCol="rating",
    predictionCol="prediction")

tvs = TrainValidationSplit(
    estimator=als,
    estimatorParamMaps=param_grid,
    evaluator=evaluator,
)

%time model = tvs.fit(sets['training'])
```

    CPU times: user 388 ms, sys: 107 ms, total: 495 ms
    Wall time: 59.9 s

The `TrainValidationSplit` instance (`model`) stores the model with the smallest MSE in `model.bestModel`.
We can access the best parameters from our proposal in `model.bestModel`.

```python
print("Best rank = {}".format(model.bestModel.rank))
print("Best maxIter = {}".format(model.bestModel._java_obj.parent().getMaxIter()))
```

    Best rank = 6
    Best maxIter = 12

If you want, you can search for the best parameter giving more options. Keep in mind, however, that the parameter grid search explores all _combinations_ of parameters, therefore the number trained models will grow exponentially, and of course, so will the time taken to perform the computations.

The `model` instance has a method (`.transform()`) which allows us to train the model. It will automatically assume the best model.
Lets perform the training and display the predictions. We will also join the prediction dataframe (which contains the movie ids) with the `movies` dataframe so we can see the actual titles of the movies.

```python
prediction = model.transform(sets['test'])

prediction.alias('p').join(movies.alias('m'), col('p.item') == col('m.item')) \
    .select([col('p.user'), col('m.title'), col('p.prediction'), col('p.rating')]).show(100, truncate=False)
```

    +----+--------------------------------+----------+------+
    |user|title                           |prediction|rating|
    +----+--------------------------------+----------+------+
    |599 |"Hudsucker Proxy                |2.8028913 |2.5   |
    |603 |"Hudsucker Proxy                |2.7718585 |4.0   |
    |182 |"Hudsucker Proxy                |3.5782845 |4.5   |
    |500 |"Hudsucker Proxy                |2.202407  |1.0   |
    |520 |"Hudsucker Proxy                |3.2265217 |5.0   |
    |411 |"Hudsucker Proxy                |2.8057125 |4.0   |
    |541 |"Hudsucker Proxy                |3.6663718 |3.0   |
    |260 |"Hudsucker Proxy                |3.4036603 |4.5   |
    |606 |Dirty Dancing (1987)            |3.7067513 |3.0   |
    |599 |Dirty Dancing (1987)            |2.7773557 |2.5   |
    |563 |Dirty Dancing (1987)            |3.5890899 |4.0   |
    |381 |Dirty Dancing (1987)            |3.6757362 |3.5   |
    |200 |Dirty Dancing (1987)            |3.9650793 |4.0   |
    |600 |Dirty Dancing (1987)            |2.9308026 |3.5   |
    |42  |Dirty Dancing (1987)            |3.9532409 |3.0   |
    |517 |Dirty Dancing (1987)            |2.9788518 |1.0   |
    |325 |Local Hero (1983)               |4.4329467 |4.0   |
    |223 |Candyman (1992)                 |2.363716  |1.0   |
    |19  |Candyman (1992)                 |2.4370744 |2.0   |
    |425 |Candyman (1992)                 |2.884019  |3.5   |
    |580 |Men in Black (a.k.a. MIB) (1997)|3.517845  |4.0   |
    |368 |Men in Black (a.k.a. MIB) (1997)|2.9470463 |3.0   |
    |93  |Men in Black (a.k.a. MIB) (1997)|4.4043994 |5.0   |
    |355 |Men in Black (a.k.a. MIB) (1997)|3.7632709 |4.0   |
    |274 |Men in Black (a.k.a. MIB) (1997)|3.371884  |3.0   |
    |280 |Men in Black (a.k.a. MIB) (1997)|3.6479337 |3.5   |
    |474 |Men in Black (a.k.a. MIB) (1997)|3.3700995 |4.5   |
    |500 |Men in Black (a.k.a. MIB) (1997)|2.8322043 |4.0   |
    |570 |Men in Black (a.k.a. MIB) (1997)|3.2649727 |4.0   |
    |489 |Men in Black (a.k.a. MIB) (1997)|2.9875088 |3.0   |
    |202 |Men in Black (a.k.a. MIB) (1997)|3.4634233 |4.0   |
    |263 |Men in Black (a.k.a. MIB) (1997)|3.5830278 |3.0   |
    |495 |Men in Black (a.k.a. MIB) (1997)|3.890459  |3.5   |
    |555 |Men in Black (a.k.a. MIB) (1997)|3.682169  |2.0   |
    |391 |Men in Black (a.k.a. MIB) (1997)|2.7820096 |2.0   |
    |361 |Men in Black (a.k.a. MIB) (1997)|3.1165917 |3.5   |
    |249 |Men in Black (a.k.a. MIB) (1997)|3.8090856 |4.5   |
    |219 |Men in Black (a.k.a. MIB) (1997)|3.1851234 |3.0   |
    |352 |Men in Black (a.k.a. MIB) (1997)|3.637846  |2.5   |
    |272 |Men in Black (a.k.a. MIB) (1997)|3.1388712 |4.0   |
    |534 |Men in Black (a.k.a. MIB) (1997)|3.8058543 |4.0   |
    |522 |Men in Black (a.k.a. MIB) (1997)|3.5455909 |4.0   |
    |357 |Men in Black (a.k.a. MIB) (1997)|3.6635323 |3.5   |
    |153 |Men in Black (a.k.a. MIB) (1997)|1.5741858 |0.5   |
    |18  |Men in Black (a.k.a. MIB) (1997)|3.4596581 |3.5   |
    |187 |Men in Black (a.k.a. MIB) (1997)|2.745304  |2.0   |
    |19  |Spawn (1997)                    |1.7458444 |2.0   |
    |217 |Spawn (1997)                    |2.3070567 |2.0   |
    |288 |Spawn (1997)                    |2.5276318 |3.0   |
    |464 |Spawn (1997)                    |3.074071  |1.0   |
    |298 |Spawn (1997)                    |2.6382265 |1.5   |
    |305 |The Devil's Advocate (1997)     |4.164713  |2.5   |
    |603 |The Devil's Advocate (1997)     |2.743326  |3.0   |
    |182 |The Devil's Advocate (1997)     |3.0768993 |4.5   |
    |542 |The Devil's Advocate (1997)     |3.0558462 |5.0   |
    |590 |The Devil's Advocate (1997)     |3.3096359 |3.0   |
    |586 |The Devil's Advocate (1997)     |4.030625  |3.5   |
    |69  |The Devil's Advocate (1997)     |4.1731668 |4.0   |
    |160 |The Devil's Advocate (1997)     |2.9772968 |3.0   |
    |480 |The Devil's Advocate (1997)     |3.3234901 |3.0   |
    |608 |The Devil's Advocate (1997)     |3.915698  |4.5   |
    |606 |Chinese Box (1997)              |1.9332248 |3.5   |
    |577 |Out of Africa (1985)            |3.3838975 |4.0   |
    |202 |Out of Africa (1985)            |3.8321576 |4.0   |
    |368 |Children of the Corn (1984)     |1.9362082 |2.0   |
    |260 |Children of the Corn (1984)     |1.5695186 |3.5   |
    |608 |"American Tail: Fievel Goes West|2.5575922 |2.0   |
    |294 |"American Tail: Fievel Goes West|2.7496076 |1.0   |
    |332 |King Kong (1933)                |3.8258653 |3.5   |
    |508 |King Kong (1933)                |2.2000468 |1.5   |
    |527 |"Buddy Holly Story              |3.0359251 |4.0   |
    |155 |Galaxy Quest (1999)             |3.6359649 |4.0   |
    |91  |Galaxy Quest (1999)             |3.7321358 |3.5   |
    |328 |Galaxy Quest (1999)             |3.2317429 |1.5   |
    |391 |Galaxy Quest (1999)             |2.935434  |2.0   |
    |249 |Galaxy Quest (1999)             |3.728284  |3.0   |
    |167 |Galaxy Quest (1999)             |3.3815227 |3.5   |
    |156 |Galaxy Quest (1999)             |3.4646103 |4.5   |
    |141 |Galaxy Quest (1999)             |3.2673461 |4.0   |
    |294 |Galaxy Quest (1999)             |2.9182374 |3.0   |
    |313 |Galaxy Quest (1999)             |2.9484942 |4.0   |
    |89  |Galaxy Quest (1999)             |2.2904923 |2.5   |
    |603 |Hellbound: Hellraiser II (1988) |3.8221264 |3.0   |
    |274 |Hellbound: Hellraiser II (1988) |2.933885  |3.5   |
    |610 |Dungeons & Dragons (2000)       |2.2855659 |1.0   |
    |599 |"Land Before Time               |2.38566   |2.5   |
    |249 |I Spy (2002)                    |2.9534469 |3.0   |
    |606 |Mississippi Masala (1991)       |3.835687  |4.0   |
    |356 |American Splendor (2003)        |3.8250668 |2.0   |
    |477 |American Splendor (2003)        |4.0505104 |5.0   |
    |287 |American Splendor (2003)        |3.873679  |5.0   |
    |187 |American Splendor (2003)        |4.3718905 |5.0   |
    |271 |10 (1979)                       |2.6210253 |2.0   |
    |448 |10 (1979)                       |2.8787282 |3.5   |
    |345 |10 (1979)                       |2.5806093 |4.5   |
    |599 |"Tale of Two Sisters            |3.1128097 |3.5   |
    |474 |Before Sunset (2004)            |3.4546523 |4.0   |
    |560 |Before Sunset (2004)            |2.724901  |3.0   |
    |221 |Before Sunset (2004)            |4.0141273 |4.5   |
    |105 |Before Sunset (2004)            |2.5512233 |4.0   |
    +----+--------------------------------+----------+------+
    only showing top 100 rows

By glancing at this dataframe, we can see that many predictions are close to the true value, but some are way off.
First, lets visualise the error distribution.

```python
errors = prediction.withColumn('error', col('prediction')-col('rating')).select('error').collect()
ax = sns.distplot(errors, rug=True, hist=True)
ax.set_xlim(0, 5)
```

We can see that the majority of errors is close to zero, which is always good.
But we want to quantify how good the model is, and for that we will use the MSE again.

The `evaluator` instance that can calculate this for us (recall from above that we specified `mse` as our error measure when creating the instance). We pass the `prediction` dataframe as the evaluator is expecting a column named `prediction`.

```python
mse = evaluator.evaluate(prediction)
print("MSE = {}".format(mse))
```

    MSE = 0.8370703145656093

So we can see that with these new parameters, the error is lower that with our arbirtary ones, which was the goal of this exercise.

## Prediction

Now that we have a trained model, we want to make predictions. That is, we want to know given a certain user and a movie, which will be the predicted rating. Also, given the nature of the ALS algorithm (where users and items latent factors are stored as matrices), we could look a prediction for all the movies for a certain user and vice-versa.

Spark has several methods to perform the prediction given an ALS model.

For instance, to predict the top-$k$ movies for all users, we can use:

```python
# top 10 movies for all users
k = 10
model.bestModel.recommendForAllUsers(k).show(10)
```

    +----+--------------------+
    |user|     recommendations|
    +----+--------------------+
    | 471|[[170355, 5.04989...|
    | 463|[[5075, 5.347768]...|
    | 496|[[26171, 5.779956...|
    | 148|[[5013, 5.0821075...|
    | 540|[[5075, 5.6096983...|
    | 392|[[40491, 5.878360...|
    | 243|[[5075, 6.150671]...|
    |  31|[[87234, 6.304914...|
    | 516|[[3200, 5.2486925...|
    | 580|[[60766, 5.37418]...|
    +----+--------------------+
    only showing top 10 rows

And to predict the top-$k$ users (users that would give the highest rating) for each movie, we could use:

```python
# top 10 users for all movies
model.bestModel.recommendForAllItems(k).show(10)
```

    +------+--------------------+
    |  item|     recommendations|
    +------+--------------------+
    |  1580|[[53, 4.964338], ...|
    |  4900|[[53, 5.145113], ...|
    |  5300|[[206, 3.9970074]...|
    |  6620|[[236, 5.163564],...|
    |  7340|[[543, 4.217016],...|
    | 32460|[[53, 6.1421156],...|
    | 54190|[[43, 5.8928227],...|
    |   471|[[360, 4.7487717]...|
    |  1591|[[258, 4.420768],...|
    |140541|[[461, 4.829395],...|
    +------+--------------------+
    only showing top 10 rows

If we wanted to see the prediction for a certain pair of `(user, movie)` we could create a dataframe containing the information needed:

```python
pairs = [(233, 901), (451, 300), (344, 934)] # (user, item)
rows = [Row(user=pair[0], item=pair[1]) for pair in pairs]
df = spark.createDataFrame(rows)
```

And then use, as before, the `.transform()` method from our trained `model`:

```python
prediction = model.transform(df)
prediction.show()
```

    +----+----+----------+
    |item|user|prediction|
    +----+----+----------+
    | 300| 451| 3.8386116|
    | 901| 233| 2.0794456|
    | 934| 344|  2.312513|
    +----+----+----------+

### Your personal recommendations

We can now used the model as our personal movie recommender!

You will create a new user profile for yourself and start adding your own ratings.
Based on these ratings, you will then be able to predict ratings for movies you haven't seen it (at least rated).

First, you'll determine your `user` id:

```python
new_user_id = ratings.groupBy().max('user').first()['max(user)'] + 1
print("New user id = {}".format(new_user_id))
```

    New user id = 611

Next, we create a list of tuples (pairs) consisting of a `item` id and our own `rating`.
The following list is just a example, if you want to create your own list, evaluate the following method with a partial name of movie for which you want to find the id and replace it on the list below.

```python
from pyspark.sql.functions import lower

def find_movie_id(partial_title):
    movies.where(lower(col('title')).like("%{}%".format(partial_title.lower())))\
    .show(truncate=False)

find_movie_id('heart')
```

    +----+----------------------------+
    |item|title                       |
    +----+----------------------------+
    |110 |Braveheart (1995)           |
    |653 |Dragonheart (1996)          |
    |988 |Grace of My Heart (1996)    |
    |1423|Hearts and Minds (1996)     |
    |2350|Heart Condition (1990)      |
    |2417|Heartburn (1986)            |
    |2443|Playing by Heart (1998)     |
    |2476|Heartbreak Ridge (1986)     |
    |2738|Crimes of the Heart (1986)  |
    |2906|Random Hearts (1999)        |
    |2996|Music of the Heart (1999)   |
    |3111|Places in the Heart (1984)  |
    |3466|Heart and Souls (1993)      |
    |3565|Where the Heart Is (2000)   |
    |3706|Angel Heart (1987)          |
    |3714|Clara's Heart (1988)        |
    |3914|"Broken Hearts Club         |
    |4067|Untamed Heart (1993)        |
    |4228|Heartbreakers (2001)        |
    |4639|America's Sweethearts (2001)|
    +----+----------------------------+
    only showing top 20 rows

After finding your favourite movies, replace them (along with your rating) on the following list:

```python
my_ratings = [(484, 1), # Sorry, I didn't really liked Lassie
(485, 4), # Last Action Hero, it's a guilty pleasure
(492, 4), # Manhattan Murder Mystery, a great movie
(519, 1), # RoboCop 3, what a way to spoil a great franchise
(541, 5), # Blade Runner, enough said
(2968, 5), # Time Bandits, a classic
(587, 1), # eeech, Ghost
(608, 5), # Fargo, another classic
(680, 4), # Alphaville, great, but not easy
(741, 4), # Ghost in the Shell, always fun
(750, 5), # Dr. Strangelove, again, enough said
(784, 1), # Cable Guy. not. funny.
(1288, 5), # This Is Spinal Tap. essential.
(2067, 2), # Doctor Zhivago, classic but not my thing.
(3081, 3), # Sleepy Hollow enjoyable
(4848, 5), # Mulholland Drive. What? Yes!
(4878, 3), # Donnie Darko, twas ok
(4973, 4), # Amelie
(5147, 5), # Wild Strawberries
(6383, 1)] # 2 Fast 2 Furious

my_ratings_df = spark.createDataFrame([Row(user=new_user_id, item=r[0], rating=r[1]) for r in my_ratings])
my_ratings_df.show()
```

    +----+------+----+
    |item|rating|user|
    +----+------+----+
    | 484|     1| 611|
    | 485|     4| 611|
    | 492|     4| 611|
    | 519|     1| 611|
    | 541|     5| 611|
    |2968|     5| 611|
    | 587|     1| 611|
    | 608|     5| 611|
    | 680|     4| 611|
    | 741|     4| 611|
    | 750|     5| 611|
    | 784|     1| 611|
    |1288|     5| 611|
    |2067|     2| 611|
    |3081|     3| 611|
    |4848|     5| 611|
    |4878|     3| 611|
    |4973|     4| 611|
    |5147|     5| 611|
    |6383|     1| 611|
    +----+------+----+

We will now join the two dataframes. The original ratings from the MovieLens dataset and our newly created ones:

```python
new_ratings = ratings.union(my_ratings_df)
```

We will retrain the model so it can now take into account our personal ratings.
In theory, we should also estimate the parameters again, since there's no guarantee the previous parameters will still be the best.

However, since we've only added a few ratings to an already existing dataset of hundreds of thousands, we can say that, in practice, this won't make a big difference. However, the best practice is always to retrain the model after a significant number of ratings is added. How you define "significant" can be assessed by comparing the old parameters with a new search. Due to the limited amount of time, we won't do that now.

```python
best_rank = model.bestModel.rank
best_iterations = model.bestModel._java_obj.parent().getMaxIter()
%time new_model = ALS(rank=best_rank, maxIter=best_iterations, coldStartStrategy="drop").fit(new_ratings)
```

    CPU times: user 33.2 ms, sys: 9.62 ms, total: 42.8 ms
    Wall time: 4.37 s

Now let's predict the ratings for the movies you _haven't_ seen.
We start by creating a dataframe consisting of all the movie ids _except_ the ones you have rated previously.

```python
unseen_movies = movies.alias('m').join(my_ratings_df.alias('r'), col('m.item') == col('r.item'), how='left_anti').select('item')
unseen_movies.show(10)
print("Number of unseen movies = {}".format(unseen_movies.count()))
```

    +----+
    |item|
    +----+
    |  26|
    |  29|
    | 474|
    |1677|
    |1806|
    |1950|
    |2040|
    |2453|
    |2529|
    |2927|
    +----+
    only showing top 10 rows

    Number of unseen movies = 9722

We now create a dataframe consisting of your `user` id and the `item` id of each movie you haven't seen.

```python
from pyspark.sql.functions import lit

unseen_movies_user = unseen_movies.withColumn("user", lit(new_user_id))
unseen_movies_user.show()
```

    +----+----+
    |item|user|
    +----+----+
    |  26| 611|
    |  29| 611|
    | 474| 611|
    |1677| 611|
    |1806| 611|
    |1950| 611|
    |2040| 611|
    |2453| 611|
    |2529| 611|
    |2927| 611|
    |3091| 611|
    |3506| 611|
    |3764| 611|
    |4823| 611|
    |5385| 611|
    |5409| 611|
    |5556| 611|
    |6721| 611|
    |7225| 611|
    |8484| 611|
    +----+----+
    only showing top 20 rows

As previously, we use the model's `.transform()` method to create a prediction dataframe:

```python
unseen_ratings = new_model.transform(unseen_movies_user)
unseen_ratings.show(10)
```

    +----+----+----------+
    |item|user|prediction|
    +----+----+----------+
    | 148| 611|  4.253159|
    | 471| 611|  3.117791|
    | 496| 611|  4.253159|
    | 833| 611|  1.199823|
    |1088| 611| 1.9459069|
    |1238| 611|  4.428846|
    |1342| 611| 2.6041217|
    |1580| 611| 2.6076708|
    |1591| 611| 2.1376724|
    |1645| 611| 2.4451044|
    +----+----+----------+
    only showing top 10 rows

To help visualise the result we will, as previously, join the prediction dataframe with the movie titles.

```python
unseen_ratings_titles = unseen_ratings.alias('r')\
                        .join(movies.alias('m'), col('r.item') == col('m.item'))\
                        .select(['user', 'title', 'prediction'])
unseen_ratings_titles.show(10, truncate=False)
```

    +----+--------------------------------+----------+
    |user|title                           |prediction|
    +----+--------------------------------+----------+
    |611 |"Awfully Big Adventure          |4.253159  |
    |611 |"Hudsucker Proxy                |3.117791  |
    |611 |What Happened Was... (1994)     |4.253159  |
    |611 |High School High (1996)         |1.199823  |
    |611 |Dirty Dancing (1987)            |1.9459069 |
    |611 |Local Hero (1983)               |4.428846  |
    |611 |Candyman (1992)                 |2.6041217 |
    |611 |Men in Black (a.k.a. MIB) (1997)|2.6076708 |
    |611 |Spawn (1997)                    |2.1376724 |
    |611 |The Devil's Advocate (1997)     |2.4451044 |
    +----+--------------------------------+----------+
    only showing top 10 rows

If we want to see the predictions ordered by their rating, in descending order, we simply apply the `.orderBy()` transformation to the dataframe:

```python
unseen_ratings_titles.orderBy(col('prediction').desc()).show(20)
```

    +----+--------------------+----------+
    |user|               title|prediction|
    +----+--------------------+----------+
    | 611|Dragon Ball Z: Th...|  6.140652|
    | 611|Mulholland Dr. (1...|  6.120075|
    | 611|"Tale of Two Sisters| 6.0905924|
    | 611|"Million Dollar H...| 5.9996037|
    | 611|      Yojimbo (1961)|  5.918203|
    | 611|     "Skin I Live In| 5.8376017|
    | 611|        Opera (1987)| 5.8291016|
    | 611|        Ikiru (1952)|  5.776032|
    | 611|           Moonlight|  5.573234|
    | 611|     "Act of Killing|  5.573234|
    | 611|  Indignation (2016)|  5.573234|
    | 611|Who Killed Chea V...|  5.573234|
    | 611|   Umberto D. (1952)|  5.573234|
    | 611|I Am Not Your Neg...|  5.573234|
    | 611|Won't You Be My N...|  5.573234|
    | 611|Wait Until Dark (...| 5.5370984|
    | 611|Kwaidan (Kaidan) ...|  5.497493|
    | 611| "Match Factory Girl|  5.473209|
    | 611|    Marwencol (2010)|  5.453308|
    | 611|My Architect: A S...|  5.433885|
    +----+--------------------+----------+
    only showing top 20 rows

### Post-processing

Lets do some post-processing to the results we just got.
We can start to see what's the difference if we remove predictions made for movies with more than 100 ratings.
We start by getting the movie ids from movies with less that $100$ ratings:

```python
ratings_per_movie = ratings.groupBy('item').count()
enough_ratings = ratings_per_movie.filter(col('count') < 100)
enough_ratings.show()
```

    +------+-----+
    |  item|count|
    +------+-----+
    |  2529|   56|
    | 60756|   28|
    |   474|   70|
    |    26|   13|
    | 72011|   32|
    |  1806|    8|
    |  2040|    7|
    |  2453|    3|
    |102993|    4|
    |    29|   38|
    |  6721|    7|
    | 96829|   10|
    |  3764|    5|
    |106100|   17|
    |106002|   16|
    |  4823|   25|
    | 45726|    6|
    | 51709|    7|
    | 69720|    1|
    | 91261|    1|
    +------+-----+
    only showing top 20 rows

We can see that the results (sorted in rating descending order), are now constrained to be lower than $5$.
Why do you think that is?

```python
training_100 = unseen_ratings.alias('r')\
    .join(enough_ratings.alias('e'), col('r.item') == col('e.item'), how='left_anti')\
    .select(['item', 'user', 'prediction']).orderBy(col('prediction').desc())
training_100.show()
```

    +----+----+----------+
    |item|user|prediction|
    +----+----+----------+
    | 858| 611| 4.6375756|
    |1208| 611|  4.588269|
    |1136| 611| 4.5728636|
    | 296| 611|  4.564739|
    |1732| 611|  4.469977|
    |1221| 611| 4.4622774|
    |6874| 611|  4.424946|
    |1206| 611| 4.3651676|
    |7438| 611| 4.3647456|
    |3996| 611|   4.31933|
    | 111| 611| 4.3121367|
    |2858| 611|  4.259027|
    |4226| 611| 4.2484784|
    |1089| 611| 4.2392826|
    |  50| 611|   4.17518|
    |2959| 611|  4.155445|
    |1193| 611|  4.152857|
    | 260| 611|  4.109965|
    |4993| 611|   4.10192|
    |1196| 611| 3.9933815|
    +----+----+----------+
    only showing top 20 rows

Just out of curiosity, we will show the actual titles of the movies which are highest rated for you (if you changed the list above):

```python
training_100.alias('t').join(movies.alias('m'), col('t.item') == col('m.item')) \
    .select(['user', 'title', 'prediction']).show(100, truncate=False)
```

    +----+------------------------------------------------------------------------------+----------+
    |user|title                                                                         |prediction|
    +----+------------------------------------------------------------------------------+----------+
    |611 |"Godfather                                                                    |4.6375756 |
    |611 |Apocalypse Now (1979)                                                         |4.588269  |
    |611 |Monty Python and the Holy Grail (1975)                                        |4.5728636 |
    |611 |Pulp Fiction (1994)                                                           |4.564739  |
    |611 |"Big Lebowski                                                                 |4.469977  |
    |611 |"Godfather: Part II                                                           |4.4622774 |
    |611 |Kill Bill: Vol. 1 (2003)                                                      |4.424946  |
    |611 |"Clockwork Orange                                                             |4.3651676 |
    |611 |Kill Bill: Vol. 2 (2004)                                                      |4.3647456 |
    |611 |"Crouching Tiger                                                              |4.31933   |
    |611 |Taxi Driver (1976)                                                            |4.3121367 |
    |611 |American Beauty (1999)                                                        |4.259027  |
    |611 |Memento (2000)                                                                |4.2484784 |
    |611 |Reservoir Dogs (1992)                                                         |4.2392826 |
    |611 |"Usual Suspects                                                               |4.17518   |
    |611 |Fight Club (1999)                                                             |4.155445  |
    |611 |One Flew Over the Cuckoo's Nest (1975)                                        |4.152857  |
    |611 |Star Wars: Episode IV - A New Hope (1977)                                     |4.109965  |
    |611 |"Lord of the Rings: The Fellowship of the Ring                                |4.10192   |
    |611 |Star Wars: Episode V - The Empire Strikes Back (1980)                         |3.9933815 |
    |611 |2001: A Space Odyssey (1968)                                                  |3.983112  |
    |611 |Goodfellas (1990)                                                             |3.959895  |
    |611 |Alien (1979)                                                                  |3.9365551 |
    |611 |"Silence of the Lambs                                                         |3.8818905 |
    |611 |"Lord of the Rings: The Return of the King                                    |3.8742137 |
    |611 |"Princess Bride                                                               |3.8692129 |
    |611 |Groundhog Day (1993)                                                          |3.8636773 |
    |611 |Interview with the Vampire: The Vampire Chronicles (1994)                     |3.8623075 |
    |611 |Léon: The Professional (a.k.a. The Professional) (Léon) (1994)                |3.8442123 |
    |611 |Raiders of the Lost Ark (Indiana Jones and the Raiders of the Lost Ark) (1981)|3.836926  |
    |611 |Casablanca (1942)                                                             |3.8211513 |
    |611 |"Dark Knight                                                                  |3.8026252 |
    |611 |"Shining                                                                      |3.7893515 |
    |611 |"Lord of the Rings: The Two Towers                                            |3.7717173 |
    |611 |Trainspotting (1996)                                                          |3.752823  |
    |611 |Aliens (1986)                                                                 |3.7519138 |
    |611 |Eternal Sunshine of the Spotless Mind (2004)                                  |3.7401786 |
    |611 |"Shawshank Redemption                                                         |3.7084436 |
    |611 |Heat (1995)                                                                   |3.696199  |
    |611 |"Departed                                                                     |3.6751428 |
    |611 |Twelve Monkeys (a.k.a. 12 Monkeys) (1995)                                     |3.6721525 |
    |611 |Full Metal Jacket (1987)                                                      |3.665713  |
    |611 |Ghostbusters (a.k.a. Ghost Busters) (1984)                                    |3.6641202 |
    |611 |Seven (a.k.a. Se7en) (1995)                                                   |3.6636682 |
    |611 |Clerks (1994)                                                                 |3.657603  |
    |611 |Indiana Jones and the Last Crusade (1989)                                     |3.651164  |
    |611 |Schindler's List (1993)                                                       |3.634584  |
    |611 |Austin Powers: The Spy Who Shagged Me (1999)                                  |3.5721326 |
    |611 |"Matrix                                                                       |3.551618  |
    |611 |"Sixth Sense                                                                  |3.507343  |
    |611 |Austin Powers: International Man of Mystery (1997)                            |3.5027006 |
    |611 |"Fifth Element                                                                |3.49884   |
    |611 |"Terminator                                                                   |3.4847937 |
    |611 |American History X (1998)                                                     |3.4782953 |
    |611 |"Incredibles                                                                  |3.4494095 |
    |611 |Star Wars: Episode VI - Return of the Jedi (1983)                             |3.4271898 |
    |611 |Inception (2010)                                                              |3.4235704 |
    |611 |Willy Wonka & the Chocolate Factory (1971)                                    |3.355763  |
    |611 |Ferris Bueller's Day Off (1986)                                               |3.3478699 |
    |611 |Saving Private Ryan (1998)                                                    |3.34226   |
    |611 |Die Hard (1988)                                                               |3.3300853 |
    |611 |"Truman Show                                                                  |3.3015227 |
    |611 |WALL·E (2008)                                                                 |3.298211  |
    |611 |Terminator 2: Judgment Day (1991)                                             |3.2807825 |
    |611 |Toy Story (1995)                                                              |3.1642158 |
    |611 |Back to the Future (1985)                                                     |3.1641514 |
    |611 |"Breakfast Club                                                               |3.0954108 |
    |611 |There's Something About Mary (1998)                                           |3.0950463 |
    |611 |Finding Nemo (2003)                                                           |3.0707853 |
    |611 |V for Vendetta (2006)                                                         |3.0681136 |
    |611 |Gladiator (2000)                                                              |3.0567915 |
    |611 |Pirates of the Caribbean: The Curse of the Black Pearl (2003)                 |3.0524235 |
    |611 |Babe (1995)                                                                   |3.0494535 |
    |611 |Spider-Man (2002)                                                             |3.0392647 |
    |611 |Aladdin (1992)                                                                |3.0377197 |
    |611 |"Fugitive                                                                     |3.0239449 |
    |611 |"Green Mile                                                                   |3.0103395 |
    |611 |Batman (1989)                                                                 |3.0076027 |
    |611 |"Bourne Identity                                                              |2.9942975 |
    |611 |E.T. the Extra-Terrestrial (1982)                                             |2.991442  |
    |611 |Batman Begins (2005)                                                          |2.9821024 |
    |611 |"Monsters                                                                     |2.9748406 |
    |611 |Good Will Hunting (1997)                                                      |2.9743466 |
    |611 |Ace Ventura: Pet Detective (1994)                                             |2.9061546 |
    |611 |Ocean's Eleven (2001)                                                         |2.881152  |
    |611 |Shrek (2001)                                                                  |2.8635664 |
    |611 |Braveheart (1995)                                                             |2.8631084 |
    |611 |Forrest Gump (1994)                                                           |2.853273  |
    |611 |"Beautiful Mind                                                               |2.8320637 |
    |611 |Four Weddings and a Funeral (1994)                                            |2.8174558 |
    |611 |Catch Me If You Can (2002)                                                    |2.77349   |
    |611 |Jurassic Park (1993)                                                          |2.7555625 |
    |611 |Indiana Jones and the Temple of Doom (1984)                                   |2.7359176 |
    |611 |Minority Report (2002)                                                        |2.715933  |
    |611 |Apollo 13 (1995)                                                              |2.6692479 |
    |611 |"Lion King                                                                    |2.6667078 |
    |611 |Up (2009)                                                                     |2.6458108 |
    |611 |X-Men (2000)                                                                  |2.6214192 |
    |611 |Dumb & Dumber (Dumb and Dumber) (1994)                                        |2.6080747 |
    |611 |Men in Black (a.k.a. MIB) (1997)                                              |2.6076708 |
    +----+------------------------------------------------------------------------------+----------+
    only showing top 100 rows

## Implicit data

We usually consider using ALS on a set of `(user, product)` ratings. But what if the data isn't so self explanatory?

### A day trip to the library

Consider, for example, the data collected by a local library. The library records which users took out each books and how long they kept the books before returning them.

As such, we have no explicit indication that a user liked or disliked the books they took out - Just because you borrowed a book does not mean that you enjoyed it, or even read it.
Furthermore, the missing data is of interest - the fact that a user has not taken out a specific book could indicate that they dislike that genre, or that they haven't been to that section of the library.

Furthermore the same user action could have many different causes. Suppose you withdraw a book three times. That might indicate that you loved the book, but it may also indicate that the book doesn't appeal to you as strongly as some other books you withdrew so you never got round to reading it the first two times.

To make the situation even worse, implicit data is often dirty. For example, a user may withdraw a library book for their child using their account, or they may accidentally pick up a book that was sitting on the counter.

### The solution
Based on the standard ALS implementation, [Hu et al. (2008)](https://www.google.ca/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=0ahUKEwi899eAu6baAhUurlkKHaVvB6UQFggsMAA&url=http%3A%2F%2Fyifanhu.net%2FPUB%2Fcf.pdf&usg=AOvVaw3WIcPGTpxR8m7C32F8whFx) presented a methodolgy for carrying out ALS when dealing with implicit data.

The general idea is that we have some recorded observations $r_{u,i}$ denoting the level of interaction user $u$ had with product $i$. For example, if a user $1$ borrowed book $4$ once we may set $r_{1,4}=1$. Alternatively we may wish to allow $r_{u,i}$ to hold information about how many days the book was borrowed for. (There is a lot of freedom in this set up, so we need to make some data specific decisions regarding how we will select $r_{u,i}$).

Given the set of observations $r_{u,i}$, a binary indicator $p_{u,i}$ is introduced where:

$$p_{u,i} = \begin{cases} 1 & \mbox{if } r_{i,j}>0 \\
0 & \mbox{otherwise.} \end{cases}
$$

A confidence parameter $\alpha$ lets the user determine how much importance they wish to place on the recorded $r_{u,i}$. This leads to the introduction of $c_{u,i}$ which we take to be the confidence we have in the strength of user $u$'s reaction to product $i$:
$c_{u,i} = 1 + \alpha r_{u,i}$.

Let $N_u$ denote the number of users, and $N_p$ denote the number of products. Let $k\in \mathbb{R}^+$ be a user defined number of factors.
Now, in implicit ALS the goal is to find matrices $X\in \mathbb{R}^{N_u \times k}$ and $Y\in \mathbb{R}^{N_p \times k}$ such that the following cost function is minimised:

$$\sum_{u,i} c_{u,i}(p_{u,i}-X_u^T Y_i)^2 + \lambda (\sum_u \| X_u\|^2 + \sum_{i} \| y_u\|^2), $$

where $X_u$ is the $u$th row of X, $Y_i$ is the $i$th row of Y,
\lambda is a user defined parameter which prevents overfitting.

With this minimisation at hand, we are able to recover estimates of $c_{u,i}$, and thus of $r_{u,i}$ for interactions which have not yet occured.

### Let's get going

We are going to run implicit ALS using the implementation given in the `pyspark.mllib.recommendation` module.

The data used in the [Book Crossings](http://www2.informatik.uni-freiburg.de/~cziegler/BX/) dataset. We load the data into Spark from the CSV file, as previously and perform some cleaning task on the data.

```python
implicit_ratings_raw = spark.sparkContext.textFile('BX-Book-Ratings.csv')\
                   .filter(lambda x: x!='"User-ID";"ISBN";"Book-Rating"')\
                   .map(lambda x: x.replace('"',"")) \
                   .map(lambda x:x.split(";"))\
                   .map(lambda x:(int(x[0]), str(x[1]), int(x[2])))
```

Let's look at how it looks.

```python
implicit_ratings_raw.take(10)
```

    [(276725, '034545104X', 0),
     (276726, '0155061224', 5),
     (276727, '0446520802', 0),
     (276729, '052165615X', 3),
     (276729, '0521795028', 6),
     (276733, '2080674722', 0),
     (276736, '3257224281', 8),
     (276737, '0600570967', 6),
     (276744, '038550120X', 7),
     (276745, '342310538', 10)]

We are interested in the files `BX-Books-Ratings.csv` and `BX-Books.csv`. The first three columns of `BX-Book-Ratings` are:
* the user id
* an isbn which identifies the book
* and a rating.

A `0` in the rating column is used to denote that an implicit interaction occured between the user an the book. It is this data that we are interested in, and we extract such rows using the following command:

```python
implicit_ratings = implicit_ratings_raw.filter(lambda x: x[2] == 0)
```

Let's look at the result.

```python
implicit_ratings.take(10)
```

    [(276725, '034545104X', 0),
     (276727, '0446520802', 0),
     (276733, '2080674722', 0),
     (276746, '0425115801', 0),
     (276746, '0449006522', 0),
     (276746, '0553561618', 0),
     (276746, '055356451X', 0),
     (276746, '0786013990', 0),
     (276746, '0786014512', 0),
     (276747, '0451192001', 0)]

The implicit ALS function we are going to use requires that product ids are integers.
At the moment we have unique ISBNs, which contain a mixture of numbers and letters, so we must convert to integers.
This can be done using the `.zipWithIndex()` function which takes an RDD and joins unique ids to each entry:

```python
# Extract unique isbns.
isbns = implicit_ratings.map(lambda x: x[1]).distinct()

# Associates an integer with each unique isbn.
isbns_with_indices = isbns.zipWithIndex()

# Sets isbn as the key
reordered_implicit_ratings = implicit_ratings.map(lambda x: (x[1], (x[0], x[2])))
joined = reordered_implicit_ratings.join(isbns_with_indices) # joins with indexes
joined.take(10)
```

    [('034545104X', ((276725, 0), 0)),
     ('034545104X', ((6543, 0), 0)),
     ('034545104X', ((23768, 0), 0)),
     ('034545104X', ((28266, 0), 0)),
     ('034545104X', ((28523, 0), 0)),
     ('034545104X', ((39002, 0), 0)),
     ('034545104X', ((56157, 0), 0)),
     ('034545104X', ((59102, 0), 0)),
     ('034545104X', ((59287, 0), 0)),
     ('034545104X', ((77940, 0), 0))]

The data is now in the form `(isbn, ((user-id, rating), isbn-id-integer))`.
We use the `.map()` function to get it the form `(user-id, isbn-id-integer, rating)` which is the form we will use to create a dataframe with the schema expected by Spark's ALS function.

```python
ratings_int_nice = joined.map(lambda x: (x[1][0][0], x[1][1], x[1][0][1]))
ratings_int_nice.take(10)
```

    [(276725, 0, 0),
     (6543, 0, 0),
     (23768, 0, 0),
     (28266, 0, 0),
     (28523, 0, 0),
     (39002, 0, 0),
     (56157, 0, 0),
     (59102, 0, 0),
     (59287, 0, 0),
     (77940, 0, 0)]

However, we weed `1`s not `0`s, since the matrix is singular if we use `0`s as the ratings.
This means we will use `1` to indicate response, not `0`.

```python
ratings_ones = ratings_int_nice.map(lambda x: Row(user=x[0], item=x[1], rating=1))
implicit_ratings = spark.createDataFrame(ratings_ones).cache()
implicit_ratings.show()
```

    +----+------+------+
    |item|rating|  user|
    +----+------+------+
    |   0|     1|276725|
    |   0|     1|  6543|
    |   0|     1| 23768|
    |   0|     1| 28266|
    |   0|     1| 28523|
    |   0|     1| 39002|
    |   0|     1| 56157|
    |   0|     1| 59102|
    |   0|     1| 59287|
    |   0|     1| 77940|
    |   0|     1| 81977|
    |   0|     1|123981|
    |   0|     1|132081|
    |   0|     1|133235|
    |   0|     1|135045|
    |   0|     1|138543|
    |   0|     1|144315|
    |   0|     1|144486|
    |   0|     1|145451|
    |   0|     1|147799|
    +----+------+------+
    only showing top 20 rows

We can now split the dataset and train the model.
Notice that training is very similar to the non-implicit data. We will still use the `TrainValidationSplit` and `RegressionEvaluator` classes.
In fact the only change is that now we add the `implicitPrefs=True` to the ALS instantiation.
The `nonnegative` argument specifies whether or not to use non-negative constraints for least squares (defaults to `false`).
Essentially, it is informaing Spark on which solver to use: `CholeskySolve`r or `NNLS` (conjugated gradient).

```python
implicit_sets = split_sets(implicit_ratings, [0.63212056, 0.1839397, 0.1839397])
print("Training dataset size (implicit) = {}".format(implicit_sets['training'].count()))
print("Validation dataset size (implicit) = {}".format(implicit_sets['validation'].count()))
print("Test dataset size (implicit) = {}".format(implicit_sets['test'].count()))
```

    Training dataset size (implicit) = 452587
    Validation dataset size (implicit) = 132345
    Test dataset size (implicit) = 131177

```python
ials = ALS(coldStartStrategy="drop", implicitPrefs=True)

iparam_grid = ParamGridBuilder() \
    .addGrid(ials.rank, [6, 8]) \
    .addGrid(ials.maxIter,[10, 12]) \
    .build()

ievaluator = RegressionEvaluator(
    metricName="mse",
    labelCol="rating",
    predictionCol="prediction")

itvs = TrainValidationSplit(
    estimator=ials,
    estimatorParamMaps=iparam_grid,
    evaluator=ievaluator,
)

%time imodel = itvs.fit(implicit_sets['training'])
```

    CPU times: user 1.15 s, sys: 329 ms, total: 1.48 s
    Wall time: 5min 9s

Also, as previously, we use the `.transform()` method to calculate predictions on a datase:

```python
iprediction = imodel.transform(implicit_sets['test'])
iprediction.show()
```

    +----+------+------+---------------+
    |item|rating|  user|     prediction|
    +----+------+------+---------------+
    | 148|     1|274549|   2.1587999E-4|
    | 463|     1|164926|    0.016497733|
    | 463|     1|127429|     0.07327433|
    | 463|     1|236283|     0.17530283|
    | 463|     1|162529|-1.06537574E-23|
    | 463|     1|223431|    0.009114619|
    | 463|     1|265595|   0.0049092765|
    | 463|     1|106845|   0.0041168053|
    | 463|     1|145619|    0.069575585|
    | 463|     1| 54547|   2.3011443E-4|
    | 463|     1| 62891|    0.018543275|
    | 463|     1|113611|   2.4358055E-4|
    | 463|     1|252071|     0.08285412|
    | 463|     1| 63644|   0.0013208847|
    | 463|     1|230522|     0.23360755|
    | 463|     1|167349|     0.07951713|
    | 463|     1|167556|    7.484213E-4|
    | 463|     1| 76942|     0.07771825|
    | 463|     1|161299|    0.014341331|
    | 463|     1| 28139|    0.004866551|
    +----+------+------+---------------+
    only showing top 20 rows

Let's have a look at user $8$. We wish to make predictions on what books the user will like, based on their interactions.

```python
iprediction.filter(col('user') == 8).show()
```

    +----+------+----+-----------+
    |item|rating|user| prediction|
    +----+------+----+-----------+
    |6925|     1|   8|1.767715E-5|
    +----+------+----+-----------+

Now we want to exclude books with which user 8 already interacted with:

```python
seen = implicit_ratings.filter(col('user')==8).select('item')
unseen_books = implicit_ratings.alias('r')\
    .join(seen.alias('s'), col('r.item') == col('s.item'), how='left_anti')\
    .select('item').distinct()\
    .withColumn('user', lit(8))\
    .cache()
unseen_books.show(10)
print("Number of unseen books = {}".format(unseen_books.count()))
```

    +----+----+
    |item|user|
    +----+----+
    |  26|   8|
    |  29|   8|
    | 474|   8|
    |1950|   8|
    |2214|   8|
    |2250|   8|
    |2453|   8|
    |2927|   8|
    |3506|   8|
    |4590|   8|
    +----+----+
    only showing top 10 rows

    Number of unseen books = 246713

We will create a predictions dataframe using the `.transform()` method and then we will sort it in descending order of predicted value:

```python
imodel.transform(unseen_books).orderBy(col('prediction').desc()).show(20)
```

    +-----+----+------------+
    | item|user|  prediction|
    +-----+----+------------+
    | 8107|   8|0.0014625812|
    | 9945|   8|0.0014404728|
    | 2487|   8| 0.001429715|
    | 1595|   8|0.0014291506|
    | 1445|   8|0.0014289932|
    | 9391|   8|0.0013860163|
    |  238|   8|0.0013758251|
    | 1698|   8|0.0013576851|
    | 6819|   8|0.0013438873|
    |  783|   8|0.0013374606|
    | 4369|   8|0.0013187684|
    | 1031|   8|0.0013123228|
    | 1694|   8|0.0013054439|
    | 6773|   8|0.0013006659|
    |  215|   8| 0.001297561|
    |10561|   8| 0.001286359|
    | 7917|   8|0.0012797969|
    | 8126|   8|0.0012593766|
    | 1655|   8|0.0012584074|
    |  689|   8|0.0012445278|
    +-----+----+------------+
    only showing top 20 rows

Remember that to check the models MSE for a predictions dataset, we used the `.evaluate()` method from the evaluator.

```python
print("MSE (implicit) = {}".format(ievaluator.evaluate(iprediction)))
```

    MSE (implicit) = 0.9504550166369008

## Streaming (Stochastic Gradient Descent) ALS

```python
import utils

ratings, pairs = utils.load_data('ml-latest-small/ratings.csv')
print("Ratings shape = {}".format(ratings.shape))
sns.heatmap(ratings[200:, :200].todense())
```

As previously, we will split the ratings data. This time we will use simple a _training_ and _test_ subsets and will use the proportions $80/20\%$:

We are using `numpy` now, so we'll do this by suffling the `(row, col)` pairs and then selecting the first $80\%$ of the indices as training and the remaining as testing.

```python
from scipy import sparse
import numpy as np

np.random.seed(42)

np.random.shuffle(pairs)
rows, cols = pairs.shape
training_indices, test_indices = pairs[:int(rows*0.8),:], pairs[int(rows*0.8):,:]
```

Since we have a high proportion of missing ratings, we will create a [sparse matrix](https://en.wikipedia.org/wiki/Sparse_matrix) to contain both sets.

Since we need to keep the unique user and movie identifiers, we will create two matrices with the same size of the original ratings:

```python
training = sparse.lil_matrix(ratings.shape)
for indices in training_indices:
    training[indices[0]-1, indices[1]-1] = ratings[indices[0]-1, indices[1]-1]

testing = sparse.lil_matrix(ratings.shape)
for indices in test_indices:
    testing[indices[0]-1, indices[1]-1] = ratings[indices[0]-1, indices[1]-1]

sns.heatmap(testing[200:, :200].todense())
```

As expected, most values of the matrix are empty (in black).

The next step is to create the _user factors_ and _item factors_.
These will be matrices taking a number of rows corresponding to the number of features we are using (_rank_) and the number of columns corresponding the total number of users or items.

These matrices will be initialised as random, as we will use a uniform distribution to fill them (by calling `numpy.random.rand`).
We will chose and arbitrary rank of $k=20$.

```python
rank = 20
n_item, n_user = ratings.shape

user_factors = np.random.rand(rank, n_user)
item_factors = np.random.rand(rank, n_item)

print(user_factors)
print(item_factors)

sns.heatmap(user_factors)
```

We continue by specify some more arbitrary parameters. In this case the $\gamma$, $\lambda_{u}$, $\lambda_{i}$ and the number of iterations.
We recall that the gradient update is calculated using

$$
\nabla U_i = \gamma + \left(\epsilon_{i,j} P_j - \lambda_u U_i\right) \\
\nabla P_j = \gamma + \left(\epsilon_{i,j} U_i - \lambda_p P_j\right)
$$

Where $\gamma$ is the learning rate, $\epsilon_{i,j}$ is the error between the true rating $r_{i,j}$ and the predicted rating for $(i,j)$, $\hat{r}_{i,j}$.
$\lambda_u$ and $\lambda_p$ are respectively the user and item regression parameters.
The calculation of the new latent factor will be iterated for a specified number of iteratios, which we will, as before, `maxIter`:

```python
learning_rate = 0.01 # "gamma"
lambda_user = 0.1
lambda_item = 0.7
maxIter = 20
rate = 1.3
```

We are now ready to start the ALS-SGD training.

```python
np.random.seed(42)
errors = []
for it in range(maxIter):

    np.random.shuffle(training_indices)

    learning_rate /= rate

    for pair in training_indices:
        row = pair[0] - 1
        col = pair[1] - 1
        item_factor = item_factors[:, row]
        user_factor = user_factors[:, col]
        error = training[row, col] - user_factor.T.dot(item_factor)

        item_factors[:, row] += learning_rate * (error * user_factor - lambda_item * item_factor)
        user_factors[:, col] += learning_rate * (error * item_factor - lambda_user * user_factor)

    # calculate the intermediate MSE
    mse = 0
    for pair in training_indices:
        row = pair[0] - 1
        col = pair[1] - 1
        mse += calc_se(training[row, col], user_factors[:, col], item_factors[:, row])
    mse /= len(training_indices)
    print("step = {}, MSE (training) = {}.".format(it + 1, mse))

    errors.append(mse)
```

    step = 1, MSE (training) = 1.016509243649517.
    step = 2, MSE (training) = 0.9310447943381437.
    step = 3, MSE (training) = 0.9108730348585418.
    step = 4, MSE (training) = 0.8866558283896934.
    step = 5, MSE (training) = 0.8723890955634594.
    step = 6, MSE (training) = 0.86460188742356.
    step = 7, MSE (training) = 0.8602621441351882.
    step = 8, MSE (training) = 0.8556057302722733.
    step = 9, MSE (training) = 0.8527915821073694.
    step = 10, MSE (training) = 0.8521074741223279.
    step = 11, MSE (training) = 0.8489033563178858.
    step = 12, MSE (training) = 0.8478134507159334.
    step = 13, MSE (training) = 0.8481661460132137.
    step = 14, MSE (training) = 0.84669913391902.
    step = 15, MSE (training) = 0.8463972990589317.
    step = 16, MSE (training) = 0.8459203812806537.
    step = 17, MSE (training) = 0.8455972295632478.
    step = 18, MSE (training) = 0.8452319874308323.
    step = 19, MSE (training) = 0.8452398374140437.
    step = 20, MSE (training) = 0.8451412668660666.

We can now visualise the MSE evolution with each iteration:

```python
sns.lineplot(range(len(errors)), errors)
```

And finally we can calculate the MSE for the training set:

```python
mse = 0
for i in test_indices:
    row = i[0] - 1
    col = i[1] - 1
    mse += calc_se(testing[row, col], user_factors[:, col], item_factors[:, row])
mse /= len(test_indices)
print("MSE (test) = {}".format(mse))
```

    MSE (test) = 0.9429504496618373

```python

```


# Python 101

**ENGSCI233: Computational Techniques and Computer Systems**

*Department of Engineering Science, University of Auckland*

The purpose of this notebook is to give you the **very basics** of Python (and computer code in general). You will not be an expert by the end. But you will be on [The Path](https://zen-of-python.info/).

When writing computer code, there are ***rules*** and there are ***conventions***.

- If you break a ***rule***, the code will not work.
- If you break a ***convention***, someone, somewhere puts a mark against your name in a book. At the end, you will be called to account.

Some Python rules:

- ***Syntax*** - we have *very* precise expectations about how you write computer code. If you type an opening bracket, `(`, you must close it again later, `)`. Some control structures are terminated by a colon, `:` - if you omit this, the code will not work. ***Yes, learning syntax for a new code is very pedantic and a total pain.*** But you have to do it anyway, and at least Python returns readable error messages to help you understand your missteps...
- ***Indentation*** - Python uses this to determine when control structures begin and end. More on what a **control structure** is later.

Some Python conventions:

- ***Commenting*** - it is helpful for the poor soul who has to read your poorly written Python (sometimes that is you, weeks or months later) if you have included little **'sign-posts'** in the code, articulating what you are doing. These are called comments. They begin with a `#` symbol, after which you can write whatever you like and it will not be executed as a command.
- ***Sensible variable names*** - relating to the thing the variable represents but not too long. For example, if a variable contains the mean temperature, then `Tmean` is a sensible variable name, where as `the_mean_temperature_of_the_profile` or `a1` are not sensible variable names.

Hang-on, what's a **variable**?

**Execute the cell below by clicking inside it and hitting Ctrl+Enter**

```python
# this is a comment, nothing happens when Python reads this line
hello_string = 'Hello, world'
print(hello_string)
```

The snippet of Python code above does a number of things.

1. The first **line** is a comment - Python sees the `#` symbol and ignores everything that follows **on that line**.
2. The second line creates a **variable** called `hello_string` and **assigns to it** the ***value*** `'Hello, world'`.
3. The third line uses a **function** called `print` to display the value of `hello_string` to the screen. `hello_string` was **passed** to `print` as an **input** or **argument**.

Ooooooh. LOTS of terminology there. Let's list and define the terms:

- ***line*** - just as you read a book one sentence at a time, Python executes a computer program one line at a time. If Python is executing the 4th line of a program, it *will* have knowledge of the three lines preceding it, *but no* knowledge of the lines that follow.
- ***variable*** - think of this as a 'container' inside the computer code. A variable has a **name** (in this case, `hello_string`) and a **value**.
- ***value*** - the thing sitting inside the 'container', in this case it is `'Hello, world'`, which is a particular **type** of variable called a string.
- ***type*** - a classification of the variable, e.g., `'hi'` and `'Hello, world'` are both *strings*, `3.2`, `5.0` and `-0.12e5` are *floats*, and `2`, `-3412` and `0` are *integers*. There are other types, we will get to some of them later.
- ***assign to*** - the act of giving a *value* to a *variable*, usually accomplished by an 'equals sign', e.g., `variable = value`.
- ***function*** - a sequence of Python commands, written down somewhere else, to achieve some small (or large) task. Some functions are given to you as part of Python and its modules. Others you will have to define yourself. Functions inputs are given inside of **round brackets**.
- ***input/argument*** - a variable that is used by a function as it executes its tasks.

## Learning by doing

Obviously, the best way to master any new skill is to practice it. Work your way through the cells and exercises below to grow your Python foundation.

## 1 We can use Python as a glorified calculator

In the example below, we create a few variables, assign them some simple numbers, and confirm that Python can manage basic arithmetic.

```python
a = 2           # a variable named a, assigned a value of 2, which is of type 'integer'
b = 3
c = a+b         # adding two variables together to create a third one
print(c)

# Can you make a change? Define the variable d as the sum of a and b and c. Print it out.
```

See also

```python
print(a-b)      # Python does subtraction
print(a*b)      # and multiplication
print(a/b)      # and division
print(a**b)     # and exponentiation

# Can you make a change? Print the value of b minus a, instead of a minus b .
```

There are heaps of other mathematical operations we can perform with Python, many of which are made available through the NumPy module.

```python
import numpy as np
print(np.sin(a))
```

In the cell above we **imported** `numpy` and told Python to make the module available as a variable called `np`. Then, if I want to use one of its special functions, I use the **syntax** `module.function`.

Some more examples below

```python
print(np.cos(b))
print(np.log(a))
print(np.log10(b))
print(np.sinh(a))
print(np.arctan2(a,b))
```

Not sure what a particular function does? Write it down and replace the round brackets with a question mark `?`

```python
np.arctan2?

# what does np.exp do?
```

#### <homework>

Calculate viscosity of magma according to the forumula given by [Costa et al. [2007]](http://onlinelibrary.wiley.com/doi/10.1029/2008GC002138/abstract)

\begin{equation}
\eta(\phi) = \frac{1+\varphi^\delta}{\left[1-F(\varphi, \varepsilon, \gamma)^{B\cdot\phi_*}\right]},\quad\text{where}\quad F=(1-\xi)\cdot\text{erf}\left[\frac{\sqrt{\pi}}{2\cdot(1-\xi)}\varphi\cdot(1+\varphi^\gamma)\right]\quad\text{with}\quad\varphi=\frac{\phi}{\phi_*}.
\end{equation}

```python
# parameters
phi = 0.5
phi_star = 0.66
pi = 3.14159
xi = 0.0009
gamma = 9.8
B = 2.5          # Einstein coefficient
delta = 1.3

# note, the error function is given by erf(x)
from scipy.special import erf           # call this as a function, e.g., erf(2)

# **complete the code below**
# uncomment the partially written code below
# calculate viscosity in three steps
# - calculate new variable phi_scaled
# - calculate F
# - calculate viscosity

#phi_scaled = phi/phi_star

#F = ____

#print(visc)

```

***How does increasing each of the parameters, $\phi$, $\phi_*$, $\xi$ (`xi`), $\gamma$, $B$, $\delta$, change the viscosity, $\eta$?***

#### </homework>

## 2 Whoops!

I have replicated the very first example in this notebook, **except** that the variable names have been changed AND I have **reversed the order** of the commands.

***Run the cell below***

```python
# this is a comment, nothing happens when Python reads this line
print(night_string)
night_string = 'Good night, world'
```

What you see above is a Python error message. ***Reading these messages and using them to debug your code is an invaluable skill.***

There are two key pieces of information:

1. **WHERE** is the error. In this case, the problem occurs on Line 2. We know this, because there is an arrow pointing to it, i.e., `----> 2 print(night_string)`
2. **WHAT** is the error. In this case, it is that we are attempting to use a variable called `night_string` before it has been created. We know this from the very last piece of information in the error message, i.e., `name 'night_string' is not defined`.

Deciphering these errors **gets easier with practice**, because there are a handful of easy-to-make mistakes that will crop up frequently.

Another tip, sometimes the error is not on the line that `---->` points to, but instead the command immediately above or below it.

***Fix the error above so that the cell runs correctly.***

To do this, swap the order of the two commands. The key here is that `night_string` has to be defined (`night_string = 'Good night, world'`) before it can be used in a function (`print(night_string)`).

***See if you can fix the errors in the code below.***

```python
# calculate the harmonic average of the numbers a and b
hamonic_avg = 2/(1/a+1/b
a = 2
b = 3
print(hamonic_avg)
```

## 3 Lists and arrays

There are these things called lists. They are as you would expect, literally a list of ordered items. An example is given below.

```python
my_list = [1,2,3]                                            # this list has three items
empty_list = []                                              # this list has no items
mixed_list = [-0.2, 300, 'a string', 5.3, True, my_list]     # this list has four items of different types, one is another list

print(mixed_list)
```

Note, lists use square brackets [] whereas functions use round brackets () - ***syntax!***

We can **access** the items of a list by passing an **index** to the variable name. The indices begin at 0 (the first item in a list) and increment by 1. For example

```python
print(my_list[0])
```

```python
print(my_list[1]+my_list[2])
```

We can also use indices to 'count backward' from the end of the list, for example

```python
print(my_list[2], my_list[-1])                 # these access the same element of the list
print(my_list[1], my_list[-2])                 # so do these
```

We can pull out a smaller list from the list using **index slicing**. This uses the colon, `:`, which essentially says 'and everything in between'. For example,

```python
print(mixed_list)
print(mixed_list[1:3])                         # all items between index 1 and 3 NOT including 3
print(mixed_list[:3])                          # all items from the START of the list up to index 3 NOT including 3
print(mixed_list[3:])                          # all items from index 3 up to the END of the list
```

Or we can even reverse a list.

```python
print(mixed_list[::-1])
```

An **array** is a list of numbers. It has special properties

```python
T = np.array([1,2,3])                          # array of 1st order temperature measurements
dT = np.array([0.1, 0.1, -0.1])                # array of corresponding 2nd order temperature deviation
print(T+dT)                                   # total temperature change
print(T*dT)                                   # product of 1st and 2nd order effects
```

Make sure your arrays are the same length though! ***Execute the cell below and interpret the error message***

```python
b1 = np.array([4,5,6])
b2 = np.array([0.1,-0.1])
print(b1+b2)
```

## 4 Doing things over and over

Scripting is useful to automate tasks that have to be performed over and over. For instance, consider calculating the value of $e$ using the exponential series

$$ e = 1+\frac{1}{1!}+\frac{1}{2!}+\frac{1}{3!}+\frac{1}{4!}+\cdots $$

We could do it the long way...

```python
from math import factorial
e = 1 + 1/factorial(1) + 1/factorial(2) + 1/factorial(3) + 1/factorial(4) + 1/factorial(5)    # I got tired and gave up here
print(e)
```

Or we could write a **for loop** to do it for us.

```python
e = 1        # the initial value
for i in range(1,21):        # create a variable called i, initially assign it the value of 1, then 2, then 3, ... then 20
    e = e + 1/factorial(i)   # each time the loop 'goes around', execute the commands 'in the loop'
print(e)
```

What's happening above? The loop we have written is identical to the sequence of commands below

```python
e = 1
i = 1
e = e + 1/factorial(i)
i = 2
e = e + 1/factorial(i)
i = 3
e = e + 1/factorial(i)
i = 4
e = e + 1/factorial(i)
i = 5
e = e + 1/factorial(i)
i = 6
e = e + 1/factorial(i)
i = 7
e = e + 1/factorial(i)
i = 8
e = e + 1/factorial(i)
i = 9
e = e + 1/factorial(i)
i = 10
e = e + 1/factorial(i)
i = 11
e = e + 1/factorial(i)
i = 12
e = e + 1/factorial(i)
i = 13
e = e + 1/factorial(i)
i = 14
e = e + 1/factorial(i)
i = 15
e = e + 1/factorial(i)
i = 16
e = e + 1/factorial(i)
i = 17
e = e + 1/factorial(i)
i = 18
e = e + 1/factorial(i)
i = 19
e = e + 1/factorial(i)
i = 20
e = e + 1/factorial(i)
print(e)
```

but with A LOT less repetition. Can you see which command is 'inside the loop' and gets executed over and over? Can you see which variable has its value changed with each iteration of the loop?

***In the cell below, write a for loop to calculate the "sum of squares" of the list of pressure observations.***

```python
a = [7,3,-2,0,4.5,9.0,-1,-1,15,0.1]              # pressure deviations (in kPa), from p0 = 101 kPa
sum_squares = 0
# **your code here**
```

## 5 Making the computer program a little bit smart

Humans are allegedly an intelligent species. One aspect of this intelligence is the ability to **see how things are and act accordingly**. What?

As an example, to cross a busy street, you first check for cars. ***If*** there is a car coming, you do not cross, ***else*** you do.

We can write this in Python.

```python
street = 'busy'
if street == 'busy':
    print('dont cross the street')
else:
    print('cross the street')
```

The `if` statement above evaluates a **condition** (essentially a question asked of Python, is the 'value' of `street` equal to `'busy'`?).

If the condition evaluates to `True`, then the command `print('dont cross the street')` is executed. If it evaluates to `'False'`, then the command `print('cross the street')` is executed instead.

The key here though is that it is **one or the other** and the outcome depends on stuff that happened earlier in the code (in this case, on the first line when I assigned a value to the variable `street`).

Have another play with the example below.

```python
street = 'busy'
attempt_to_cross = True     # this is a type of variable called a 'boolean' - it is either True or False, no other options

if street == 'busy' and attempt_to_cross is False:      # the first condition to check
    print('dont cross, good decision')
elif street != 'busy' and attempt_to_cross is True:     # if the first condition is False, then check this one
    print('safe to cross, well done')
elif street == 'busy' and attempt_to_cross is True:     # if the first AND second conditions are False, check this one
    print('you dead')
```

In the example above, we see that the **special statement**, `and`, allows us to evaluate two conditions at once and require that they BOTH be true. Alternatively, we could have used `or`, which allows either one, or both to be true.

***Make changes to the cell above to generate a "successful crossing" outcome.***

The example below demonstrates different conditions and combinations of conditions:

```python
print('1',True and True)
print('2',True and False)
print('3',True or False)
print('4',False or False)
print('5',not False)
print('6', False or not False)
print('6b', False or not (True or not True))
print('7', 3>4)
print('8', 3<4)
print('9', 4<4)
print('10', 4<=4)
print('11', 4==4)
density = 2650.
print('12', density>2400. and density<2800.)
print('13', 2400.<density<2800.)
```

#### <neat> List Comprehension

This concept is subtle, and you should ideally be already quite comfortable with for loops before experimenting with [list comprehensions](http://treyhunner.com/2015/12/python-list-comprehensions-now-in-color/).

Compare the following snippets of codes and verify that they generate the same output.

```python
a1 = []                 # an empty list
for i in range(10):     # for all integers up to (but not including) ten
    a1.append(i**2)           # append to the end of the list, the integer**2
print(a1)

a2 = [i**2 for i in range(10)]   # the same list, now constructed using list comprehension (one line!)
print(a2)
```

"*Can you make it fancier?*"

Oh, yes. Consider the following, equivalent, list transformations.

```python
b1 = []
for ai in a1:
    if (ai%2 == 1):        # if the value is odd, append it to the new list
        b1.append(ai)      #  (% is the 'remainder' operator, i.e., what is the remainder when ai is divided by 2)
    else:
        b1.append(-ai)     # otherwise, make the value negative and append it
print(b1)

b2 = [ai if ai%2 == 1 else -ai for ai in a1]    # (one line!)
print(b2)

# or if there's not 'else' case to worry about, the if condition comes at the end
# e.g., just keep the odd numbers
# b3 = [ai for ai in a1 if ai%2 == 1]
# print(b3)
```

#### </neat>

### 5.1 Making a loop a little bit smart

We have [seen](python101.ipynb#0.4-Doing-things-over-and-over) how a for loop can be used to repeat a computation over and over a **fixed number of times**. But what if we don't know ahead of time how many times a loop should go around?

Taking the example from before, say we want to calculate the value of $e$ to a fixed level of accuracy, say 4 decimal places. That is, as soon as the next term in the series - $1/i!$ - is smaller than $10^{-5}$, we should stop the loop.

To do this, we can use a **while loop**, which makes use of a condition to determine when the loop should stop.

```python
from math import factorial
e = 1        # the initial value
i = 1        # unlike the for loop, we have to initialise and increment the counter manually
de = 1/factorial(i)   # calculate the update to e

# initialise the while loop with a condition, each time the loop goes around the value of de changes,
# and the condition is rechecked. The loop stops when the condition evaluates to False.

while de > 1.e-5:                 # check the condition
    e = e + de                    # update the estimate of e
    i = i + 1                     # increment the counter
    de = 1/factorial(i)           # calculate a new value for de
    print('i =',i,', e =', e)     # print progress so far

# MAKE A CHANGE
# - modify the loop so that it exits when e has been calculated to 6 decimal places
# - modify the loop so that it exits when e has been calculated to 8 decimal places, or when 12 terms
# have been calculated, whichever comes first
```

## 6 Classes, objects, attributes and methods

Python is an [**object-oriented**](https://en.wikipedia.org/wiki/Object-oriented_programming) programming language. Most people initially become familiar with **procedural, structured programming**, computer code organised into a logical procession of statements, loops, control blocks and functions. We can build on that understanding and introduce the idea of **objects, with attributes and methods**.

The best introduction is perhaps a direct demonstration.

**Execute the cell below to define a new *Class* for a geothermal well.**

```python
class Well(object):          # defining a class is similar to defining a function in that there is precise syntax
    ''' An object to represent an arbitrary Well.
    '''
    def __init__(self):
        ''' Define what properties the object should have when it is brought into existence
        '''
        self.location = []                  # these are called attributes, we have defined 3: location, name and depth
        self.name = 'unnamed'               # they are like variables, but they *belong* to the object
        self.depth = None                   # we can access and change them using the notation OBJECT.ATTRIBUTE
```

Think of the **Class** as a new "kind" or a "type" of object (along with floats, integers, strings, and arrays). Much like a function, once it is defined, we can begin to use it.

```python
# Create an *instance* of the Well object. A duplicate, to be modified independently of other instances.
well1 = Well()               # note the use of brackets in creating the object
well2 = Well()               # now we have two 'instances' of the Well object

print(well1.name, well2.name)
```

We can modify their **attributes** in the usual way a variable is modified.

```python
# let's make the first object personal (change for yourself)
well1.location = [0.5, 8.2]
well1.name = 'tvz25'
well1.depth = 2305.

# let's make the second object a beloved pet (change for yourself)
well2.location = [6.3, -2.4]
well2.name = 'tvz32'
well2.depth = 1680.

print(well1.name, well2.name)               # verifying we have changed the attributes
print(well1.depth > well2.depth)            # verifying attributes are subject to the usual computer arithmetic
```

#### <neat> `__repr__`

Try **printing** an object directly.

```python
print(well2)
```

The standard output is not very **informative**...

We can modify this by including a **specialised method** called '__repr__' in the class definition

```python
class Well(object):
    ''' An object to represent an arbitrary Well.
    '''
    def __init__(self):
        ''' Define what properties the object should have when it is brought into existence
        '''
        self.location = []
        self.name = 'unnamed'
        self.depth = None
    def __repr__(self):
        ''' What information to print to the screen when the object is printed.
        '''
        return '{:s}'.format(self.name)

# create and print the new object
well2 = Well()
well2.location = [6.3, -2.4]
well2.name = 'tvz32'
well2.depth = 1680.
print(well2)
```

#### </neat>

An object's attributes are **specific** to it. For example,

```python
print(well1.name)                   # the 'name' *attribute* has been defined for the well1 object
print(well2.name)                   # the 'name' *attribute* has been defined for the well2 object
print(name)                         # 'name' on its own is a *variable* that has yet to be defined
```

In much the same way that attributes are just variables **specific to an object**, we can define **methods**, which are functions **specific to an object**.

***Execute the cell below to update the Well class with a method `bhp` that computes bottomhole pressure.***

```python
class Well(object):
    ''' An object to represent an arbitrary Well.
    '''
    def __init__(self):
        ''' Define what properties the object should have when it is brought into existence
        '''
        self.location = []
        self.name = 'unnamed'
        self.depth = None
    def __repr__(self):
        ''' What information to print to the screen when the object is printed.
        '''
        return '{:s}'.format(self.name)

    def bhp(self, density=1000., whp=0.1):
        ''' Computes the bottomhole pressure (in MPa) for given fluid density and wellhead pressure.
        '''
        # set gravity
        g = 9.81
        # compute pressure due to water column, in Pa
        dP = density*self.depth*g
        # convert to MPa
        dP = dP/1.e6
        # add wellhead pressure
        bhp = whp + dP
        # return value
        return bhp
```

***Execute the cell below to call the `bhp` method for a range of inputs.***

```python
# define a well
well2 = Well()
well2.location = [6.3, -2.4]
well2.name = 'tvz32'
well2.depth = 1680.

# compute downhole pressure assuming density of 1000. and whp of 12 bar (1.2 MPa)
bottomholepressure = well2.bhp(density=1000., whp=1.2)
print('at 1000 kg/m^3 and 12 bar WHP, the BHP is', bottomholepressure)

# compute downhole pressure with density of 980 and atmospheric pressure (default 0.1)
bottomholepressure = well2.bhp(density=980.)
print('at 980 kg/m^3 and open, the BHP is', bottomholepressure)

# compute downhole pressure using all defaults
bottomholepressure = well2.bhp()
print('at 1000 kg/m^3 and open, the BHP is', bottomholepressure)
```

# What now?

There you go. That's your crash course in computer coding with Python. Obviously, we're only scraping the surface, and you shouldn't expect to really feel like you "*know what you're doing*" until you've been writing Python for a month or so.

The rewards though. So great.


Table of Contents

1  Question

1.1  Answer

2  Question

2.1  Answer

3  Question

3.1  Answer

4  Question

4.1  Answer

# Question
**Question 1.1** Write a Python Program(with class concepts) to find the area of the triangle using the below formula.

`area = (s*(s-a)*(s-b)*(s-c)) ** 0.5`

Function to take the length of the sides of triangle from user should be defined in the parent class and function to calculate the area should be defined in subclass.

**BreakDown points**

`Class Parent : Should Read inputs from User fr the sides of triangle`

`Class Child  : Should Calculate the Area Of Triangle with Variables from parent`

**Heron's Formula**
\begin{align}
s = \frac{a + b + c}{2}
\end{align}

\begin{align}
Area Of Triangle = \sqrt{(s*(s-a)*(s-b)*(s-c))}
\end{align}

## Answer

```python
# Parent Class : Will get the input from User and display
class LengthOfTheSides():
    def __init__(self):
        print('Please provide length of the sides of triangle : \n')
        self.a = float(input('side1 : '))
        self.b = float(input('side2 : '))
        self.c = float(input('side3 : '))

    def __repr__(self):
        return 'Provided Values are ...\nside1 = {}, side2 = {}, side3 = {}\n\n'.format(str(self.a),str(self.b), str(self.c))

# Child Class : Will Calculate the Area Of Triangle
class Area(LengthOfTheSides):
    def __init__(self):
        LengthOfTheSides.__init__(self)

    def AreaOfTriangle(self):
        s, a, b, c = (self.a + self.b + self.c)/2, self.a, self.b, self.c
        return (s*(s-a)*(s-b)*(s-c)) ** 0.5

AreaOfTriangle = Area()
print('\nArea Of Triangle : {}'.format(AreaOfTriangle.AreaOfTriangle()))
```

    Please provide length of the sides of triangle :

    side1 : 5
    side2 : 4
    side3 : 3

    Area Of Triangle : 6.0

# Question

**Question 1.2** Write a function filter_long_words() that takes a list of words and an integer n and returns the list of words that are longer than n.

## Answer

```python
def filter_long_words(words_to_check, max_length):
    # Check Provided Arg[0] is List
    if not isinstance(words_to_check, list):
        raise Exception('Please enter arg[0] as list of words')
    # Check Provided Arg[1] is int
    if not isinstance(max_length, int):
        raise Exception('Please enter arg[1] as int')

    return [i for i in words_to_check if len(i)>max_length]
```

```python
# Correct Arguments Passed

filter_long_words(['Vignesh', 'Vicky'], 5)
```

    ['Vignesh']

```python
# Arg[0] is String

filter_long_words('Vignesh', 5)
```

```python
# Arg[1] is String

filter_long_words(['Vignesh', 'Vicky'], "5")
```

# Question

**Question 2.1** Write a Python program using function concept that maps list of words into a list of integers representing the lengths of the corresponding words.

**Hint:** If a list [ ab,cde,erty] is passed on to the python function output should come as [2,3,4]. Here 2,3 and 4 are the lengths of the words in the list.

## Answer

```python
def words_to_int(words_to_check):
    # Check Provided Arg[0] is List
    if not isinstance(words_to_check, list):
        raise Exception('Please enter arg[0] as list of words')

    return [len(i) for i in words_to_check]
```

```python
words_to_int(['Vignesh', 'Vicky'])
```

    [7, 5]

# Question

**Question 2.2** Write a Python function which takes a character (i.e. a string of length 1) and returns True if it is a vowel, False otherwise.

## Answer

```python
def check_vowels(char_to_check):
    vowels = ('a', 'e', 'i', 'o', 'u')

    # Check Provided Arg[0] is a Single Charecter String
    if isinstance(char_to_check, str) and len(char_to_check) == 1:
        if char_to_check.lower() in vowels:
            return True
        else:
            return False
    else:
        raise Exception('Please enter arg[0] as a Single Charecter String')
```

```python
check_vowels('V')
```

    False

```python
check_vowels('I')
```

    True