---
layout: global
title: 分类和回归
displayTitle: 分类和回归
---

$$
\newcommand{\R}{\mathbb{R}}
\newcommand{\E}{\mathbb{E}}
\newcommand{\x}{\mathbf{x}}
\newcommand{\y}{\mathbf{y}}
\newcommand{\wv}{\mathbf{w}}
\newcommand{\av}{\mathbf{\alpha}}
\newcommand{\bv}{\mathbf{b}}
\newcommand{\N}{\mathbb{N}}
\newcommand{\id}{\mathbf{I}}
\newcommand{\ind}{\mathbf{1}}
\newcommand{\0}{\mathbf{0}}
\newcommand{\unit}{\mathbf{e}}
\newcommand{\one}{\mathbf{1}}
\newcommand{\zero}{\mathbf{0}}
$$

本章节涵盖分类和回归算法，讨论了线性方法、树和集成方法等特定类型的算法。

# 分类

## Logistic回归

Logistic回归是一个流行的分类问题预测方法。它是[广义线性模型](https://en.wikipedia.org/wiki/Generalized_linear_model)的一个特例，预测类别概率。
在`spark.ml`的Logistic回归中二项Logistic回归可用于预测二值输出，而多项Logistic回归可用于预测多值输出。 用`family`参数来在这两算法中选择，也可以不设置它，Spark会推断正确的选择。

  > 把`family`参数设置成`"multinomial"`，多项Logistic回归可用于二分类。它将产生两组系数和截距。

  > 当在有非零常数列的数据集上不带截距拟合`LogisticRegressionModel`时，Spark MLlib对非零常数列输出零系数。这种行为与R glmnet相同但是不同于LIBSVM。

### 二项Logistic回归

关于二项Logistic回归实现的更多背景和细节，请参考[`spark.mllib`中的Logistic回归](mllib-linear-methods.md#logistic-regression)文档。

**样例**

下面的例子展示了如何在二分类问题上训练带elastic net正则化的二项和多项Logistic回归模型。`elasticNetParam`对应于$\alpha$，`regParam`对应于$\lambda$。

关于参数的更多细节可以在[Python API文档](api/python/pyspark.ml.html#pyspark.ml.classification.LogisticRegression)中找到。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/logistic_regression_with_elastic_net.py"找到

```python
from pyspark.ml.classification import LogisticRegression

# 载入训练数据
training = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)

# 拟合模型
lrModel = lr.fit(training)

# 打印Logistic回归的系数和截距
print("Coefficients: " + str(lrModel.coefficients))
print("Intercept: " + str(lrModel.intercept))

# 我们也可以在二分类问题上用多项分布族
mlr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8, family="multinomial")

# 拟合模型
mlrModel = mlr.fit(training)

# 打印多项分布族Logistic回归的系数和截距
print("Multinomial coefficients: " + str(mlrModel.coefficientMatrix))
print("Multinomial intercepts: " + str(mlrModel.interceptVector))
```

`spark.ml`中Logistic回归的实现也支持提取一个训练集上一个模型的摘要。请注意，作为`DataFrame`存储在`LogisticRegressionSummary`中的预测和指标被标注为`@transient`，因此仅对驱动程序可用。

[`LogisticRegressionTrainingSummary`](api/python/pyspark.ml.md#pyspark.ml.classification.LogisticRegressionSummary)
提供[`LogisticRegressionModel`](api/python/pyspark.ml.md#pyspark.ml.classification.LogisticRegressionModel)的一个摘要。
对于二分类的情况, 还有些额外指标，比如ROC曲线。请参考[`BinaryLogisticRegressionTrainingSummary`](api/python/pyspark.ml.md#pyspark.ml.classification.BinaryLogisticRegressionTrainingSummary)。

接着之前的例子:

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/logistic_regression_summary_example.py"找到

```python
from pyspark.ml.classification import LogisticRegression

# 从之前的例子中训练返回的LogisticRegressionModel实例中提取摘要
trainingSummary = lrModel.summary

# Obtain the objective per iteration
objectiveHistory = trainingSummary.objectiveHistory
print("objectiveHistory:")
for objective in objectiveHistory:
    print(objective)

# Obtain the receiver-operating characteristic as a dataframe and areaUnderROC。
trainingSummary.roc.show()
print("areaUnderROC: " + str(trainingSummary.areaUnderROC))

# Set the model threshold to maximize F-Measure
fMeasure = trainingSummary.fMeasureByThreshold
maxFMeasure = fMeasure.groupBy().max('F-Measure').select('max(F-Measure)').head()
bestThreshold = fMeasure.where(fMeasure['F-Measure'] == maxFMeasure['max(F-Measure)']) \
    .select('threshold').head()['threshold']
lr.setThreshold(bestThreshold)
```

### 多项Logistic回归

通过多项logistic(softmax)回归支持多分类问题。在多项Logistic回归中，
算法产生$K$组系数，或一个$K \times J$维矩阵，其中$K$是结果类别数，$J$是特征数。如果算法带一个截距项拟合，则还得到一个长为$K$的截距向量。

  > `coefficientMatrix`为多项分布系数，`interceptVector`为截距。

  > 一个以多项分布族训练的Logistic回归模型不支持`coefficients`和`intercept`方法。可用`coefficientMatrix`和`interceptVector`替代。

结果类别的条件概率$k \in \{1, 2, ..., K\}$由softmax函数刻画。

$$
   P(Y=k|\mathbf{X}, \boldsymbol{\beta}_k, \beta_{0k}) =  \frac{e^{\boldsymbol{\beta}_k \cdot \mathbf{X}  + \beta_{0k}}}{\sum_{k'=0}^{K-1} e^{\boldsymbol{\beta}_{k'} \cdot \mathbf{X}  + \beta_{0k'}}}
$$

我们使用一个多项分布响应模型最小化负对数似然，配合elastic-net惩罚以控制过拟合。

$$
\min_{\beta, \beta_0} -\left[\sum_{i=1}^L w_i \cdot \log P(Y = y_i|\mathbf{x}_i)\right] + \lambda \left[\frac{1}{2}\left(1 - \alpha\right)||\boldsymbol{\beta}||_2^2 + \alpha ||\boldsymbol{\beta}||_1\right]
$$

详细推导请参考[这里](https://en.wikipedia.org/wiki/Multinomial_logistic_regression#As_a_log-linear_model)。

**样例**

下面的例子展示了如何训练一个带elastic net正则化的多分类Logistic回归模型，并且提取多分类训练的摘要来评估模型。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/multiclass_logistic_regression_with_elastic_net.py"找到

```python
from pyspark.ml.classification import LogisticRegression

# 载入训练数据
training = spark \
    .read \
    .format("libsvm") \
    .load("data/mllib/sample_multiclass_classification_data.txt")

lr = LogisticRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)

# 拟合模型
lrModel = lr.fit(training)

# 打印多项Logistic回归的参数和截距
print("Coefficients: \n" + str(lrModel.coefficientMatrix))
print("Intercept: " + str(lrModel.interceptVector))

trainingSummary = lrModel.summary

# Obtain the objective per iteration
objectiveHistory = trainingSummary.objectiveHistory
print("objectiveHistory:")
for objective in objectiveHistory:
    print(objective)

# for multiclass, 我们可以检查每个标签上的指标
print("False positive rate by label:")
for i, rate in enumerate(trainingSummary.falsePositiveRateByLabel):
    print("label %d: %s" % (i, rate))

print("True positive rate by label:")
for i, rate in enumerate(trainingSummary.truePositiveRateByLabel):
    print("label %d: %s" % (i, rate))

print("Precision by label:")
for i, prec in enumerate(trainingSummary.precisionByLabel):
    print("label %d: %s" % (i, prec))

print("Recall by label:")
for i, rec in enumerate(trainingSummary.recallByLabel):
    print("label %d: %s" % (i, rec))

print("F-measure by label:")
for i, f in enumerate(trainingSummary.fMeasureByLabel()):
    print("label %d: %s" % (i, f))

accuracy = trainingSummary.accuracy
falsePositiveRate = trainingSummary.weightedFalsePositiveRate
truePositiveRate = trainingSummary.weightedTruePositiveRate
fMeasure = trainingSummary.weightedFMeasure()
precision = trainingSummary.weightedPrecision
recall = trainingSummary.weightedRecall
print("Accuracy: %s\nFPR: %s\nTPR: %s\nF-measure: %s\nPrecision: %s\nRecall: %s"
      % (accuracy, falsePositiveRate, truePositiveRate, fMeasure, precision, recall))
```

## 决策树分类器

决策树以及其集成算法是机器学习分类和回归问题中非常流行的算法。
关于`spark.ml`实现的更多信息可以在[决策树](#决策树)一节中找到。

**样例**

下面的例子导入LibSVM格式数据，并将之划分为训练数据和测试数据。使用第一部分数据进行训练，剩下数据来测试。训练之前我们使用了两种数据预处理方法来对特征进行转换，并且向`DataFrame`添加了元数据来让树结构算法能够识别。

关于参数的更多细节可以在[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.DecisionTreeClassifier)中找到。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/decision_tree_classification_example.py"找到

```python
from pyspark.ml import Pipeline
from pyspark.ml.classification import DecisionTreeClassifier
from pyspark.ml.feature import StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Load the data stored in LIBSVM format as a DataFrame.
data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

# Index labels, adding metadata to the label column.
# Fit on whole dataset to include all labels in index.
labelIndexer = StringIndexer(inputCol="label", outputCol="indexedLabel").fit(data)
# Automatically identify categorical features, and index them.
# We specify maxCategories so features with > 4 distinct values are treated as continuous.
featureIndexer =\
    VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a DecisionTree model.
dt = DecisionTreeClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures")

# Chain indexers and tree in a Pipeline
pipeline = Pipeline(stages=[labelIndexer, featureIndexer, dt])

# Train model.  This also runs the indexers.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

# Select example rows to display.
predictions.select("prediction", "indexedLabel", "features").show(5)

# Select (prediction, true label) and compute test error
evaluator = MulticlassClassificationEvaluator(
    labelCol="indexedLabel", predictionCol="prediction", metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("Test Error = %g " % (1.0 - accuracy))

treeModel = model.stages[2]
# summary only
print(treeModel)
```

## 随机森林分类器

随机森林是常用的分类和回归算法。
关于`spark.ml`实现的更多信息可以在[随机森林](#随机森林)一节中找到。

**样例**

下面的例子导入LibSVM格式数据，并将之划分为训练数据和测试数据。使用第一部分数据进行训练，剩下数据来测试。训练之前我们使用了两种数据预处理方法来对特征进行转换，并且向`DataFrame`添加了元数据来让树结构算法能够识别。

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.RandomForestClassifier)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/random_forest_classifier_example.py"找到

```python
from pyspark.ml import Pipeline
from pyspark.ml.classification import RandomForestClassifier
from pyspark.ml.feature import IndexToString, StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Load and parse the data file, converting it to a DataFrame.
data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

# Index labels, adding metadata to the label column.
# Fit on whole dataset to include all labels in index.
labelIndexer = StringIndexer(inputCol="label", outputCol="indexedLabel").fit(data)

# Automatically identify categorical features, and index them.
# Set maxCategories so features with > 4 distinct values are treated as continuous.
featureIndexer =\
    VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a RandomForest model.
rf = RandomForestClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures", numTrees=10)

# Convert indexed labels back to original labels.
labelConverter = IndexToString(inputCol="prediction", outputCol="predictedLabel",
                               labels=labelIndexer.labels)

# Chain indexers and forest in a Pipeline
pipeline = Pipeline(stages=[labelIndexer, featureIndexer, rf, labelConverter])

# Train model.  This also runs the indexers.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

# Select example rows to display.
predictions.select("predictedLabel", "label", "features").show(5)

# Select (prediction, true label) and compute test error
evaluator = MulticlassClassificationEvaluator(
    labelCol="indexedLabel", predictionCol="prediction", metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("Test Error = %g" % (1.0 - accuracy))

rfModel = model.stages[2]
print(rfModel)  # summary only
```

## 梯度提升树分类器

梯度提升树基于决策树的集成，是一种常用的分类和回归算法。
关于`spark.ml`实现的更多信息可以在[梯度提升树](#梯度提升树)一节中找到。

**样例**

下面的例子导入LibSVM格式数据，并将之划分为训练数据和测试数据。使用第一部分数据进行训练，剩下数据来测试。训练之前我们使用了两种数据预处理方法来对特征进行转换，并且向`DataFrame`添加了元数据来让树结构算法能够识别。

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.GBTClassifier)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/gradient_boosted_tree_classifier_example.py"找到

```python
from pyspark.ml import Pipeline
from pyspark.ml.classification import GBTClassifier
from pyspark.ml.feature import StringIndexer, VectorIndexer
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# Load and parse the data file, converting it to a DataFrame.
data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

# Index labels, adding metadata to the label column.
# Fit on whole dataset to include all labels in index.
labelIndexer = StringIndexer(inputCol="label", outputCol="indexedLabel").fit(data)
# Automatically identify categorical features, and index them.
# Set maxCategories so features with > 4 distinct values are treated as continuous.
featureIndexer =\
    VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a GBT model.
gbt = GBTClassifier(labelCol="indexedLabel", featuresCol="indexedFeatures", maxIter=10)

# Chain indexers and GBT in a Pipeline
pipeline = Pipeline(stages=[labelIndexer, featureIndexer, gbt])

# Train model.  This also runs the indexers.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

# Select example rows to display.
predictions.select("prediction", "indexedLabel", "features").show(5)

# Select (prediction, true label) and compute test error
evaluator = MulticlassClassificationEvaluator(
    labelCol="indexedLabel", predictionCol="prediction", metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("Test Error = %g" % (1.0 - accuracy))

gbtModel = model.stages[2]
print(gbtModel)  # summary only
```

## 多层感知机

多层感知机是基于前向人工神经网络[feedforward artificial neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network)的一种分类器。
多层感知机含有多层节点，每层节点与网络的下一层节点完全连接。输入层的节点代表输入数据，其他层的节点通过将输入数据与层上节点的权重$\wv$以及偏差$\bv$线性组合且应用一个激活函数，得到该层输出。
$K+1$层多层感知机分类器可以写成如下矩阵形式：

$$
\mathrm{y}(\x) = \mathrm{f_K}(...\mathrm{f_2}(\wv_2^T\mathrm{f_1}(\wv_1^T \x+b_1)+b_2)...+b_K)
$$

中间层节点使用sigmoid(logistic)函数：

$$
\mathrm{f}(z_i) = \frac{1}{1 + e^{-z_i}}
$$

输出层使用softmax函数:

$$
\mathrm{f}(z_i) = \frac{e^{z_i}}{\sum_{k=1}^N e^{z_k}}
$$

输出层中$N$代表类别数目。

多层感知机通过反向传播来学习模型，其中我们使用logistic损失函数以及L-BFGS优化算法。

**样例**

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.MultilayerPerceptronClassifier)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/multilayer_perceptron_classification.py"找到

```python
from pyspark.ml.classification import MultilayerPerceptronClassifier
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# 载入训练数据
data = spark.read.format("libsvm")\
    .load("data/mllib/sample_multiclass_classification_data.txt")

# Split the data into train and test
splits = data.randomSplit([0.6, 0.4], 1234)
train = splits[0]
test = splits[1]

# specify layers for the neural network:
# input layer of size 4 (features), two intermediate of size 5 and 4
# and output of size 3 (classes)
layers = [4, 5, 4, 3]

# create the trainer and set its parameters
trainer = MultilayerPerceptronClassifier(maxIter=100, layers=layers, blockSize=128, seed=1234)

# 训练模型
model = trainer.fit(train)

# compute accuracy on the test set
result = model.transform(test)
predictionAndLabels = result.select("prediction", "label")
evaluator = MulticlassClassificationEvaluator(metricName="accuracy")
print("Test set accuracy = " + str(evaluator.evaluate(predictionAndLabels)))
```

## 线性支持向量机

[支持向量机](https://en.wikipedia.org/wiki/Support_vector_machine)构造一个高维或无限维空间中的一个超平面或超平面的集合，可用于分类、回归或其他任务。
直观来看，一个好的超平面形成的分割需要离任何类别最近的训练数据点距离最大（所谓functional margin），因为一般来说类间距越大分类器泛化误差越低。
Spark ML中的`LinearSVC`用线性SVM支持二分类问题。内部使用OWLQN算法优化[Hinge损失](https://en.wikipedia.org/wiki/Hinge_loss)。

**样例**

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.LinearSVC)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/linearsvc.py"找到

```python
from pyspark.ml.classification import LinearSVC

# 载入训练数据
training = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

lsvc = LinearSVC(maxIter=10, regParam=0.1)

# 拟合模型
lsvcModel = lsvc.fit(training)

# 打印线性支持向量机的系数和截距
print("Coefficients: " + str(lsvcModel.coefficients))
print("Intercept: " + str(lsvcModel.intercept))
```

## 一对多分类器

[OneVsRest](http://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest)将一个给定的二分类算法有效地扩展到多分类问题中，也叫做“One-vs-All.”。

`OneVsRest`作为一个`Estimator`来实现。它采用一个基础的`Classifier`然后对于k个类别分别创建二分类问题。类别i的二分类分类器用来预测类别为i还是不为i，即将i类和其他类别区分开来。

最后，通过依次对k个二分类分类器进行评估，取置信最高的分类器的标签作为i类别的标签。

**样例**

下面的例子导入[鸢尾花数据集](http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/multiclass/iris.scale)，解析为`DataFrame`，用`OneVsRest`分类。计算测试集错误来衡量算法准确率。

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.OneVsRest)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/one_vs_rest_example.py"找到

```python
from pyspark.ml.classification import LogisticRegression, OneVsRest
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# load data file.
inputData = spark.read.format("libsvm") \
    .load("data/mllib/sample_multiclass_classification_data.txt")

# generate the train/test split.
(train, test) = inputData.randomSplit([0.8, 0.2])

# instantiate the base classifier.
lr = LogisticRegression(maxIter=10, tol=1E-6, fitIntercept=True)

# instantiate the One Vs Rest Classifier.
ovr = OneVsRest(classifier=lr)

# train the multiclass model.
ovrModel = ovr.fit(train)

# score the model on test data.
predictions = ovrModel.transform(test)

# obtain evaluator.
evaluator = MulticlassClassificationEvaluator(metricName="accuracy")

# compute the classification error on test data.
accuracy = evaluator.evaluate(predictions)
print("Test Error = %g" % (1.0 - accuracy))
```

## 朴素Bayes

[朴素Bayes分类器](http://en.wikipedia.org/wiki/Naive_Bayes_classifier)是基于贝叶斯定理与（朴素的）特征条件独立假设的分类方法。

朴素Bayes可以非常高效地训练。通过在训练数据上一次遍历，对于给出的待分类项，求解在此项出现的条件下各个类别出现的概率。预测时在没有其它可用信息下，我们会选择条件概率最大的类别作为此待分类项应属的类别。

MLlib支持[多项朴素Bayes](http://en.wikipedia.org/wiki/Naive_Bayes_classifier#Multinomial_naive_Bayes)
和[Bernoulli朴素Bayes](http://nlp.stanford.edu/IR-book/html/htmledition/the-bernoulli-model-1.html)。

*输入数据*:
这类模型常用于[文档分类](http://nlp.stanford.edu/IR-book/html/htmledition/naive-bayes-text-classification-1.html)。
此处每个观测是一个文档，每个词项表示为一个特征。
一个特征值为该词项的频率（多项朴素Bayes）或0/1指示该词项是否出现在文档中（Bernoulli朴素Bayes）。
特征值必须*非负*。模型类型由一个可选参数`"multinomial"`或`"bernoulli"`决定，默认值是`"multinomial"`。
对于文档分类，输入特征向量经常是稀疏向量。
由于训练数据仅使用一次，没必要缓存它。

[附加平滑](http://en.wikipedia.org/wiki/Lidstone_smoothing)可通过参数$\lambda$设定（默认值$1.0$）。

**样例**

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.classification.NaiveBayes)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/naive_bayes_example.py"找到

```python
from pyspark.ml.分类import NaiveBayes
from pyspark.ml.evaluation import MulticlassClassificationEvaluator

# 载入训练数据
data = spark.read.format("libsvm") \
    .load("data/mllib/sample_libsvm_data.txt")

# Split the data into train and test
splits = data.randomSplit([0.6, 0.4], 1234)
train = splits[0]
test = splits[1]

# create the trainer and set its parameters
nb = NaiveBayes(smoothing=1.0, modelType="multinomial")

# 训练模型
model = nb.fit(train)

# select example rows to display。
predictions = model.transform(test)
predictions.show()

# compute accuracy on the test set
evaluator = MulticlassClassificationEvaluator(labelCol="label", predictionCol="prediction",
                                              metricName="accuracy")
accuracy = evaluator.evaluate(predictions)
print("Test set accuracy = " + str(accuracy))
```

# 回归

## 线性回归

线性回归模型与摘要的接口类似于Logistic回归。

  > 当在有非零常数列的数据集上不带截距拟合`LogisticRegressionModel`时，Spark MLlib对非零常数列输出零系数。这种行为与R glmnet相同但是不同于LIBSVM。

**样例**

下面的例子展示了训练一个elastic net正则化的线性回归模型并提取模型摘要统计。
<!--- TODO: Add python model summaries once implemented -->

关于参数的更多细节可以在[Python API文档](api/python/pyspark.ml.md#pyspark.ml.regression.LinearRegression)中找到。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/linear_regression_with_elastic_net.py"找到

```python
from pyspark.ml.regression import LinearRegression

# 载入训练数据
training = spark.read.format("libsvm")\
    .load("data/mllib/sample_linear_regression_data.txt")

lr = LinearRegression(maxIter=10, regParam=0.3, elasticNetParam=0.8)

# 拟合模型
lrModel = lr.fit(training)

# 打印线性回归系数和截距
print("Coefficients: %s" % str(lrModel.coefficients))
print("Intercept: %s" % str(lrModel.intercept))

# 训练集上的模型摘要并打印一些指标
trainingSummary = lrModel.summary
print("numIterations: %d" % trainingSummary.totalIterations)
print("objectiveHistory: %s" % str(trainingSummary.objectiveHistory))
trainingSummary.residuals.show()
print("RMSE: %f" % trainingSummary.rootMeanSquaredError)
print("r2: %f" % trainingSummary.r2)
```

## 广义线性模型

与线性回归假设输出服从高斯分布不同，[广义线性模型](https://en.wikipedia.org/wiki/Generalized_linear_model)（GLMs）指定线性模型的因变量$Y_i$服从[指数族分布](https://en.wikipedia.org/wiki/Exponential_family)。
Spark的`GeneralizedLinearRegression`接口允许指定GLMs包括线性回归、泊松回归、逻辑回归等来处理多种预测问题。
目前`spark.ml`仅支持指数族分布中的一部分类型，[如下](#available-families)：

**注意**：目前Spark的`GeneralizedLinearRegression`仅支持最多4096个特征，如果特征超过4096个将会抛出异常。请看[advanced section](ml-advanced.md)了解更多细节。
对于线性回归和Logistic回归，如果模型特征数量太多，则可通过`LinearRegression`和`LogisticRegression`来训练。

GLMs要求的指数型分布可以为正则或者自然形式[自然形式指数族分布](https://en.wikipedia.org/wiki/Natural_exponential_family)。自然指数族分布为如下形式：

$$
f_Y(y|\theta, \tau) = h(y, \tau)\exp{\left( \frac{\theta \cdot y - A(\theta)}{d(\tau)} \right)}
$$

其中$\theta$是强度参数，$\tau$是分散度参数。在GLM中因变量$Y_i$服从自然指数族分布：

$$
Y_i \sim f\left(\cdot|\theta_i, \tau \right)
$$

其中强度参数$\theta_i$与因变量$\mu_i$的期望值联系如下：

$$
\mu_i = A'(\theta_i)
$$

其中$A'(\theta_i)$由所选择的分布形式所决定。GLMs同样允许指定连接函数，连接函数决定了因变量期望值与 _线性预测器_ $\eta_i$之间的关系：

$$
g(\mu_i) = \eta_i = \vec{x_i}^T \cdot \vec{\beta}
$$

通常，连接函数的选择如$A' = g^{-1}$，在强度参数$\theta$与线性预测器$\eta$之间产生一个简单的关系。这种情况下，连接函数$g(\mu)$也称为正则连接函数：

$$
\theta_i = A'^{-1}(\mu_i) = g(g^{-1}(\eta_i)) = \eta_i
$$

GLM通过最大化似然函数来求得回归系数$\vec{\beta}$：

$$
\max_{\vec{\beta}} \mathcal{L}(\vec{\theta}|\vec{y},X) =
\prod_{i=1}^{N} h(y_i, \tau) \exp{\left(\frac{y_i\theta_i - A(\theta_i)}{d(\tau)}\right)}
$$

其中强度参数$\theta_i$和回归系数$\vec{\beta}$的联系如下：

$$
\theta_i = A'^{-1}(g^{-1}(\vec{x_i} \cdot \vec{\beta}))
$$

Spark的广义线性模型接口也提供摘要统计来诊断GLM模型的拟合程度，包括残差、p值、偏度、Akaike信息量等等。

可参考更全面的广义线性模型和应用[复习](http://data.princeton.edu/wws509/notes/)。

###  可用的分布族

<table class="table">
  <thead>
    <tr>
      <th>分布族</th>
      <th>因变量类型</th>
      <th>支持的连接类型</th></tr>
  </thead>
  <tbody>
    <tr>
      <td>高斯</td>
      <td>连续型</td>
      <td>Identity*, Log, Inverse</td>
    </tr>
    <tr>
      <td>二项</td>
      <td>二值型</td>
      <td>Logit*, Probit, CLogLog</td>
    </tr>
    <tr>
      <td>泊松</td>
      <td>计数型</td>
      <td>Log*, Identity, Sqrt</td>
    </tr>
    <tr>
      <td>伽马</td>
      <td>连续型</td>
      <td>Inverse*, Idenity, Log</td>
    </tr>
    <tr>
      <td><a href="https://en.wikipedia.org/wiki/Tweedie_distribution">Tweedie</a></td>
      <td>零膨胀连续型</td>
      <td>幂连接函数</td>
    </tr>
  </tbody>
  <tfoot><tr><td colspan="4">*正则连接函数</td></tr></tfoot>
</table>

**样例**

下面的例子展示了训练一个高斯分布响应与恒等连接函数的GLM并提取模型摘要统计。

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.regression.GeneralizedLinearRegression)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/generalized_linear_regression_example.py"找到

```python
from pyspark.ml.regression import GeneralizedLinearRegression

# 载入训练数据
dataset = spark.read.format("libsvm")\
    .load("data/mllib/sample_linear_regression_data.txt")

glr = GeneralizedLinearRegression(family="gaussian", link="identity", maxIter=10, regParam=0.3)

# 拟合模型
model = glr.fit(dataset)

# 打印广义线性模型的系数和截距
print("Coefficients: " + str(model.coefficients))
print("Intercept: " + str(model.intercept))

# 训练集上的模型摘要并打印一些指标
summary = model.summary
print("Coefficient Standard Errors: " + str(summary.coefficientStandardErrors))
print("T Values: " + str(summary.tValues))
print("P Values: " + str(summary.pValues))
print("Dispersion: " + str(summary.dispersion))
print("Null Deviance: " + str(summary.nullDeviance))
print("Residual Degree Of Freedom Null: " + str(summary.residualDegreeOfFreedomNull))
print("Deviance: " + str(summary.deviance))
print("Residual Degree Of Freedom: " + str(summary.residualDegreeOfFreedom))
print("AIC: " + str(summary.aic))
print("Deviance Residuals: ")
summary.residuals().show()
```

## 回归树

决策树以及其集成算法是机器学习分类和回归问题中非常流行的算法。
关于`spark.ml`实现的更多信息可以在[决策树](#决策树)一节中找到。

**样例**

下面的例子导入LibSVM格式数据，并将之划分为训练数据和测试数据。使用第一部分数据进行训练，剩下数据来测试。训练之前我们使用了一种数据预处理方法来对特征进行转换，并且向`DataFrame`添加了元数据来让树结构算法能够识别。

关于参数的更多细节可以在[Python API文档](api/python/pyspark.ml.md#pyspark.ml.regression.DecisionTreeRegressor)中找到。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/decision_tree_regression_example.py"找到

```python
from pyspark.ml import Pipeline
from pyspark.ml.regression import DecisionTreeRegressor
from pyspark.ml.feature import VectorIndexer
from pyspark.ml.evaluation import RegressionEvaluator

# Load the data stored in LIBSVM format as a DataFrame.
data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

# Automatically identify categorical features, and index them.
# We specify maxCategories so features with > 4 distinct values are treated as continuous.
featureIndexer =\
    VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a DecisionTree model.
dt = DecisionTreeRegressor(featuresCol="indexedFeatures")

# Chain indexer and tree in a Pipeline
pipeline = Pipeline(stages=[featureIndexer, dt])

# Train model.  This also runs the indexer.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

# Select example rows to display.
predictions.select("prediction", "label", "features").show(5)

# Select (prediction, true label) and compute test error
evaluator = RegressionEvaluator(
    labelCol="label", predictionCol="prediction", metricName="rmse")
rmse = evaluator.evaluate(predictions)
print("Root Mean Squared Error (RMSE) on test data = %g" % rmse)

treeModel = model.stages[1]
# summary only
print(treeModel)
```

## 随机森林回归

随机森林是机器学习分类和回归问题中非常流行的算法。
关于`spark.ml`实现的更多信息可以在[随机森林](#随机森林)一节中找到。

**样例**

下面的例子导入LibSVM格式数据，并将之划分为训练数据和测试数据。使用第一部分数据进行训练，剩下数据来测试。训练之前我们使用了一种数据预处理方法来对特征进行转换，并且向`DataFrame`添加了元数据来让树结构算法能够识别。

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.regression.RandomForestRegressor)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/random_forest_regressor_example.py"找到

```python
from pyspark.ml import Pipeline
from pyspark.ml.regression import RandomForestRegressor
from pyspark.ml.feature import VectorIndexer
from pyspark.ml.evaluation import RegressionEvaluator

# Load and parse the data file, converting it to a DataFrame.
data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

# Automatically identify categorical features, and index them.
# Set maxCategories so features with > 4 distinct values are treated as continuous.
featureIndexer =\
    VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a RandomForest model.
rf = RandomForestRegressor(featuresCol="indexedFeatures")

# Chain indexer and forest in a Pipeline
pipeline = Pipeline(stages=[featureIndexer, rf])

# Train model.  This also runs the indexer.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

# Select example rows to display.
predictions.select("prediction", "label", "features").show(5)

# Select (prediction, true label) and compute test error
evaluator = RegressionEvaluator(
    labelCol="label", predictionCol="prediction", metricName="rmse")
rmse = evaluator.evaluate(predictions)
print("Root Mean Squared Error (RMSE) on test data = %g" % rmse)

rfModel = model.stages[1]
print(rfModel)  # summary only
```

## 梯度提升树回归

梯度提升树回归是机器学习分类和回归问题中非常流行的算法。
关于`spark.ml`实现的更多信息可以在[梯度提升树](#梯度提升树)一节中找到。

**样例**

**注意**：下面的例子中，`GBTRegressor`仅迭代了一次，在实际操作中是不现实的。

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.regression.GBTRegressor)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/gradient_boosted_tree_regressor_example.py"找到

```python
from pyspark.ml import Pipeline
from pyspark.ml.regression import GBTRegressor
from pyspark.ml.feature import VectorIndexer
from pyspark.ml.evaluation import RegressionEvaluator

# Load and parse the data file, converting it to a DataFrame.
data = spark.read.format("libsvm").load("data/mllib/sample_libsvm_data.txt")

# Automatically identify categorical features, and index them.
# Set maxCategories so features with > 4 distinct values are treated as continuous.
featureIndexer =\
    VectorIndexer(inputCol="features", outputCol="indexedFeatures", maxCategories=4).fit(data)

# Split the data into training and test sets (30% held out for testing)
(trainingData, testData) = data.randomSplit([0.7, 0.3])

# Train a GBT model.
gbt = GBTRegressor(featuresCol="indexedFeatures", maxIter=10)

# Chain indexer and GBT in a Pipeline
pipeline = Pipeline(stages=[featureIndexer, gbt])

# Train model.  This also runs the indexer.
model = pipeline.fit(trainingData)

# Make predictions.
predictions = model.transform(testData)

# Select example rows to display.
predictions.select("prediction", "label", "features").show(5)

# Select (prediction, true label) and compute test error
evaluator = RegressionEvaluator(
    labelCol="label", predictionCol="prediction", metricName="rmse")
rmse = evaluator.evaluate(predictions)
print("Root Mean Squared Error (RMSE) on test data = %g" % rmse)

gbtModel = model.stages[1]
print(gbtModel)  # summary only
```

## 生存回归

在`spark.ml`中，我们实现[加速失效时间模型（AFT）](https://en.wikipedia.org/wiki/Accelerated_failure_time_model)，对于截尾数据它是一个参数化生存回归的模型。
它描述了一个有对数生存时间的模型，所以它也常被称为生存分析的对数线性模型。
不同于应对同样问题的[比例危险模型](https://en.wikipedia.org/wiki/Proportional_hazards_model)，AFT模型中每个实例对目标函数的贡献是独立的，所以其更容易并行化。

给定协变量$x^{'}$的值，对于$i = 1, ..., n$，可能右截尾的随机生存时间$t_{i}$，AFT模型下的似然函数如下：

$$
L(\beta,\sigma)=\prod_{i=1}^n[\frac{1}{\sigma}f_{0}(\frac{\log{t_{i}}-x^{'}\beta}{\sigma})]^{\delta_{i}}S_{0}(\frac{\log{t_{i}}-x^{'}\beta}{\sigma})^{1-\delta_{i}}
$$

其中$\delta_{i}$是指示事件i发生了，即有无检测到。
令$\epsilon_{i}=\frac{\log{t_{i}}-x^{'}\beta}{\sigma}$，则对数似然函数为以下形式：

$$
\iota(\beta,\sigma)=\sum_{i=1}^{n}[-\delta_{i}\log\sigma+\delta_{i}\log{f_{0}}(\epsilon_{i})+(1-\delta_{i})\log{S_{0}(\epsilon_{i})}]
$$

其中$S_{0}(\epsilon_{i})$是基线生存函数，$f_{0}(\epsilon_{i})$是对应的密度函数。

最常用的AFT模型基于Weibull分布的生存时间。
Weibull分布的生存时间对应于生存时间对数的极值分布，$S_{0}(\epsilon)$函数如下：

$$
S_{0}(\epsilon_{i})=\exp(-e^{\epsilon_{i}})
$$

$f_{0}(\epsilon_{i})$函数如下：

$$
f_{0}(\epsilon_{i})=e^{\epsilon_{i}}\exp(-e^{\epsilon_{i}})
$$

Weibull分布的生存时间AFT模型对数似然函数如下：

$$
\iota(\beta,\sigma)= -\sum_{i=1}^n[\delta_{i}\log\sigma-\delta_{i}\epsilon_{i}+e^{\epsilon_{i}}]
$$

由于最小化负对数似然函数等于最大化后验概率，所以我们要优化的损失函数为$-\iota(\beta,\sigma)$。
分别对$\beta$以及$\log\sigma$求导：

$$
\frac{\partial (-\iota)}{\partial \beta}=\sum_{1=1}^{n}[\delta_{i}-e^{\epsilon_{i}}]\frac{x_{i}}{\sigma}
$$
$$
\frac{\partial (-\iota)}{\partial (\log\sigma)}=\sum_{i=1}^{n}[\delta_{i}+(\delta_{i}-e^{\epsilon_{i}})\epsilon_{i}]
$$

可以证明AFT模型是一个凸优化问题，即是说找到凸函数$-\iota(\beta,\sigma)$的最小值取决于系数向量$\beta$以及尺度参数的对数$\log\sigma$。
在`spark.ml`中实现的优化算法为L-BFGS。
这个实现与R中的生存函数相匹配
[survreg](https://stat.ethz.ch/R-manual/R-devel/library/survival/html/survreg.html)

  > 当在有非零常数列的数据集上不带截距拟合`AFTSurvivalRegressionModel`时，Spark MLlib对非零常数列输出零系数。这种行为不同于R的`survival::survreg`。

**样例**

请参考[Python API文档](api/python/pyspark.ml.md#pyspark.ml.regression.AFTSurvivalRegression)了解更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/aft_survival_regression.py"找到

```python
from pyspark.ml.regression import AFTSurvivalRegression
from pyspark.ml.linalg import Vectors

training = spark.createDataFrame([
    (1.218, 1.0, Vectors.dense(1.560, -0.605)),
    (2.949, 0.0, Vectors.dense(0.346, 2.158)),
    (3.627, 0.0, Vectors.dense(1.380, 0.231)),
    (0.273, 1.0, Vectors.dense(0.520, 1.151)),
    (4.199, 0.0, Vectors.dense(0.795, -0.226))], ["label", "censor", "features"])
quantileProbabilities = [0.3, 0.6]
aft = AFTSurvivalRegression(quantileProbabilities=quantileProbabilities,
                            quantilesCol="quantiles")

model = aft.fit(training)

# Print the coefficients, intercept and scale parameter for AFT survival regression
print("Coefficients: " + str(model.coefficients))
print("Intercept: " + str(model.intercept))
print("Scale: " + str(model.scale))
model.transform(training).show(truncate=False)
```

## 保序回归
[保序回归](http://en.wikipedia.org/wiki/Isotonic_regression)是一种回归算法。保序回归的形式化问题是给定一个实数的有限集合$Y = {y_1, y_2, ..., y_n}$表示观测到的因变量，$X = {x_1, x_2, ..., x_n}$表示未知因变量，拟合模型最小化函数

$$
\begin{equation}
  f(x) = \sum_{i=1}^n w_i (y_i - x_i)^2
\end{equation}
$$

使得满足全序$x_1\le x_2\le ...\le x_n$，其中$w_i$是正权重。
其结果函数称为保序回归，而且其解是唯一的。
它可以被视为有顺序约束下的最小二乘法问题。
实际上最好拟合原始数据点的保序回归是一个[单调函数](http://en.wikipedia.org/wiki/Monotonic_function)。

我们实现了[pool adjacent violators算法](http://doi.org/10.1198/TECH.2010.10111)，它采用一种[并行保序回归](http://doi.org/10.1007/978-3-642-99789-1_10)。
训练数据是一个`DataFrame`，包含标签、特征值以及权重三列。
另外保序算法还有一个参数名为`isotonic`默认为`true`，它指定保序回归为保序（单调递增）或者反序（单调递减）。

训练返回一个保序回归模型，可以被用于来预测已知或者未知特征值的标签。保序回归的结果是分段线性函数，预测规则如下：

* 如果预测输入与训练中的特征值完全匹配，则返回相应标签。如果一个特征值对应多个预测标签值，则返回其中一个，具体是哪一个未指定（类似`java.util.Arrays.binarySearch`）。

* 如果预测输入比训练中的所有特征值都高（或者都低），则相应返回最高特征值或者最低特征值对应标签。如果一个特征值对应多个预测标签值，则相应返回其中最高值或者最低值。

* 如果预测输入落入两个特征值之间，则预测将会是一个分段线性函数，其值由两个最近的特征值的预测值插值计算得到。如果一个特征值对应多个预测标签值，则使用上述两种情况中的处理方式解决。

**样例**

请参考[`IsotonicRegression`的Python文档](api/python/pyspark.ml.md#pyspark.ml.regression.IsotonicRegression)了解API的更多细节。

完整样例代码可以在[Spark仓库](https://github.com/apache/spark)中的"examples/src/main/python/ml/isotonic_regression_example.py"找到

```python
from pyspark.ml.regression import IsotonicRegression

# Loads data.
dataset = spark.read.format("libsvm")\
    .load("data/mllib/sample_isotonic_regression_libsvm_data.txt")

# Trains an isotonic regression model.
model = IsotonicRegression().fit(dataset)
print("Boundaries in increasing order: %s\n" % str(model.boundaries))
print("Predictions associated with the boundaries: %s\n" % str(model.predictions))

# Makes predictions.
model.transform(dataset).show()
```

# 线性方法

我们实现了常用的线性方法比如logistic回归与带$L_1$或$L_2$正则化的线性最小方。
请参考[RDD接口的线性方法指南](mllib-linear-methods.md)了解实现与调优的更多细节，这些信息仍然有效。

我们也包含一个[Elastic net](http://en.wikipedia.org/wiki/Elastic_net_regularization)的`DataFrame`，一个[Zou et al, Regularization and variable selection via the elastic net](http://users.stat.umn.edu/~zouxx019/Papers/elasticnet.pdf)中提出的混合$L_1$与$L_2$正则化。
在数学上定义为一个$L_1$与$L_2$正则化项的凸组合：

$$
\alpha \left( \lambda \|\wv\|_1 \right) + (1-\alpha) \left( \frac{\lambda}{2}\|\wv\|_2^2 \right) , \alpha \in [0, 1], \lambda \geq 0
$$

通过恰当设置$\alpha$，$L_1$与$L_2$正则化是elastic net的特例。举例来说，如果一个[线性回归](https://en.wikipedia.org/wiki/Linear_regression)模型以elastic net参数$\alpha$设为$1$来训练，那么它等价于[Lasso](http://en.wikipedia.org/wiki/Least_squares#Lasso_method)模型。
反之若$\alpha$设为$0$，训练出的模型成为一个[岭回归](http://en.wikipedia.org/wiki/Tikhonov_regularization)模型。
我们为带elastic net正则化的线性回归和logistic回归两者实现了管道API。

# 决策树

[决策树](http://en.wikipedia.org/wiki/Decision_tree_learning)以及其集成算法是机器学习分类和回归问题中非常流行的算法。

决策树因其易解释性、可处理类别特征、易扩展到多分类问题、不需特征缩放以及能够捕捉非线性和特征相互作用等性质被广泛使用。树集成算法如随机森林以及boosting算法几乎是解决分类和回归问题中表现最优的算法。

决策树是一个贪心算法递归地将特征空间划分为两个部分，在同一个叶子节点的数据最后会拥有同样的标签。每次划分通过贪心的以获得最大信息增益为目的，从可选择的分裂方式中选择最佳的分裂节点。节点不纯度有节点所含类别的同质性来衡量。工具提供为分类提供两种不纯度衡量（基尼不纯度和熵），为回归提供一种不纯度衡量（方差）。

`spark.ml`支持二分类、多分类以及回归的决策树算法，适用于连续特征以及类别特征。这里的实现以行划分数据，容许百万至十亿级实例的分布式训练。

用户可以在[MLlib决策树指南](mllib-decision-tree.md)中找到决策树算法的更多信息。
这个API与[之前的MLlib决策树API](mllib-decision-tree.md)之间的主要差别如下:

* 支持机器学习管道
* 区分了用于分类或回归的决策树
* 使用`DataFrame`元数据来区分连续和类别特征

决策树的管道API比之前的API提供了稍微多一点的功能。对于分类问题，工具可以返回属于每种类别的概率（类别条件概率），对于回归问题工具可以返回预测在偏置样本上的方差。

树集成算法（随机森林和梯度提升树）在[树集成](#树集成)一节中说明。

## 输入和输出

我们在这里列出了输入和输出（预测）列的类型。
所有输出列都是可选的；若需要排除一个列，将它的对应参数设为一个空字符串。

### 输入列

<table class="table">
  <thead>
    <tr>
      <th align="left">参数名</th>
      <th align="left">类型</th>
      <th align="left">默认值</th>
      <th align="left">描述</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>labelCol</td>
      <td>Double</td>
      <td>"label"</td>
      <td>预测标签</td>
    </tr>
    <tr>
      <td>featuresCol</td>
      <td>Vector</td>
      <td>"features"</td>
      <td>特征向量</td>
    </tr>
  </tbody>
</table>

### 输出列

<table class="table">
  <thead>
    <tr>
      <th align="left">参数名</th>
      <th align="left">类型</th>
      <th align="left">默认值</th>
      <th align="left">描述</th>
      <th align="left">备注</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>predictionCol</td>
      <td>Double</td>
      <td>"prediction"</td>
      <td>预测标签</td>
      <td></td>
    </tr>
    <tr>
      <td>rawPredictionCol</td>
      <td>Vector</td>
      <td>"rawPrediction"</td>
      <td>类别数长度的向量，表示在做出预测的树节点上的训练实例计数</td>
      <td>仅限分类</td>
    </tr>
    <tr>
      <td>probabilityCol</td>
      <td>Vector</td>
      <td>"probability"</td>
      <td>类别数长度的向量，等于一个多项分布归一化的`rawPrediction`</td>
      <td>仅限分类</td>
    </tr>
    <tr>
      <td>varianceCol</td>
      <td>Double</td>
      <td></td>
      <td>有偏差的预测样本方差</td>
      <td>仅限回归</td>
    </tr>
  </tbody>
</table>

# 树集成

`DataFrame`API支持两种主要的树集成算法：[随机森林](http://en.wikipedia.org/wiki/Random_forest)和[梯度提升树](http://en.wikipedia.org/wiki/Gradient_boosting)。
两者都使用[`spark.ml`决策树](#决策树)作为基础模型。

用户可以在[MLlib集成指南](mllib-ensembles.md)中找到集成算法的更多信息。
在这一节中，我们演示集成的`DataFrame`API。

这个API与[之前的MLlib集成API](mllib-ensembles.md)之间的主要差别如下:

* 支持`DataFrame`和机器学习管道
* 区分了分类或回归
* 使用`DataFrame`元数据来区分连续和类别特征
* 随机森林功能更多：估计特征重要性，还有对于分类问题，可以返回属于每种类别的概率（类别条件概率）。

## 随机森林

[随机森林](http://en.wikipedia.org/wiki/Random_forest)是[决策树](#决策树)的集成算法。

随机森林包含多个决策树来降低过拟合的风险。随机森林同样具有易解释性、可处理类别特征、易扩展到多分类问题、不需特征缩放等性质。

随机森林分别训练一系列的决策树，所以训练过程是并行的。因算法中加入随机过程，所以每个决策树又有少量区别。通过合并每个树的预测结果来减少预测的方差，提高在测试集上的性能表现。

随机性体现：

* 每次迭代时，对原始数据进行二次抽样来获得不同的训练数据。

* 对于每个树节点，考虑不同的随机特征子集来进行分裂。

除此之外，决策时的训练过程和单独决策树训练过程相同。

对新实例进行预测时，随机森林需要整合其各个决策树的预测结果。回归和分类问题的整合的方式略有不同。分类问题采取投票制，每个决策树投票给一个类别，获得最多投票的类别为最终结果。回归问题每个树得到的预测结果为实数，最终的预测结果为各个树预测结果的平均值。

`spark.ml`支持二分类、多分类以及回归的随机森林算法，同时适用于连续特征和类别特征。

请参考[`spark.mllib`的随机森林文档](mllib-ensembles.md#random-forests)了解算法的更多信息。

## 输入和输出

我们在这里列出了输入和输出（预测）列的类型。
所有输出列都是可选的；若需要排除一个列，将它的对应参数设为一个空字符串。

#### 输入列

<table class="table">
  <thead>
    <tr>
      <th align="left">参数名</th>
      <th align="left">类型</th>
      <th align="left">默认值</th>
      <th align="left">描述</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>labelCol</td>
      <td>Double</td>
      <td>"label"</td>
      <td>预测标签</td>
    </tr>
    <tr>
      <td>featuresCol</td>
      <td>Vector</td>
      <td>"features"</td>
      <td>特征向量</td>
    </tr>
  </tbody>
</table>

### 输出（预测）列

<table class="table">
  <thead>
    <tr>
      <th align="left">参数名</th>
      <th align="left">类型</th>
      <th align="left">默认值</th>
      <th align="left">描述</th>
      <th align="left">备注</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>predictionCol</td>
      <td>Double</td>
      <td>"prediction"</td>
      <td>预测标签</td>
      <td></td>
    </tr>
    <tr>
      <td>rawPredictionCol</td>
      <td>Vector</td>
      <td>"rawPrediction"</td>
      <td>类别数长度的向量，表示在做出预测的树节点上的训练实例计数</td>
      <td>仅限分类</td>
    </tr>
    <tr>
      <td>probabilityCol</td>
      <td>Vector</td>
      <td>"probability"</td>
      <td>类别数长度的向量，等于一个多项分布归一化的`rawPrediction`</td>
      <td>仅限分类</td>
    </tr>
  </tbody>
</table>

## 梯度提升树

[梯度提升树（GBT）](http://en.wikipedia.org/wiki/Gradient_boosting)是一种决策树的集成算法。它通过反复迭代训练决策树来最小化损失函数。决策树类似，梯度提升树具有可处理类别特征、易扩展到多分类问题、不需特征缩放等性质。`Spark.ml`通过使用现有决策树工具来实现。

梯度提升树依次迭代训练一系列的决策树。在一次迭代中，算法使用现有的集成来对每个训练实例的类别进行预测，然后将预测结果与真实的标签值进行比较。通过重新标记，来赋予预测结果不好的实例更高的权重。所以，在下次迭代中，决策树会对先前的错误进行修正。

对实例标签进行重新标记的机制由损失函数来指定。每次迭代过程中，梯度迭代树在训练数据上进一步减少损失函数的值。`Spark.ml`为分类问题提供一种损失函数（Log Loss），为回归问题提供两种损失函数（平方误差与绝对误差）。

`Spark.ml`支持二分类的梯度提升树算法，适用于连续特征以及类别特征。

请参考[`spark.mllib`梯度提升树文档](mllib-ensembles.md#gradient-boosted-trees-gbts)了解算法的更多信息。

## 输入和输出

我们在这里列出了输入和输出（预测）列的类型。
所有输出列都是可选的；若需要排除一个列，将它的对应参数设为一个空字符串。

#### 输入列

<table class="table">
  <thead>
    <tr>
      <th align="left">参数名</th>
      <th align="left">类型</th>
      <th align="left">默认值</th>
      <th align="left">描述</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>labelCol</td>
      <td>Double</td>
      <td>"label"</td>
      <td>预测标签</td>
    </tr>
    <tr>
      <td>featuresCol</td>
      <td>Vector</td>
      <td>"features"</td>
      <td>特征向量</td>
    </tr>
  </tbody>
</table>

**注意**：`GBTClassifier`目前仅支持二值标签。

### 输出（预测）列

<table class="table">
  <thead>
    <tr>
      <th align="left">参数名</th>
      <th align="left">类型</th>
      <th align="left">默认值</th>
      <th align="left">描述</th>
      <th align="left">备注</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>predictionCol</td>
      <td>Double</td>
      <td>"prediction"</td>
      <td>预测标签</td>
      <td></td>
    </tr>
  </tbody>
</table>

将来`GBTClassifier`也会像`RandomForestClassifier`一样输出`rawPrediction`和`probability`列。


# Supply Network Design 2

## Objective and Prerequisites

Take your supply chain network design skills to the next level in this example. We’ll show you how – given a set of factories, depots, and customers – you can use mathematical optimization to determine which depots to open or close in order to minimize overall costs.

This model is example 20 from the fifth edition of Model Building in Mathematical Programming, by H. Paul Williams on pages 275-276 and 332-333.

This example is of beginning difficulty; we assume that you know Python and have some knowledge of the Gurobi Python API and building mathematical optimization models.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

---
## Problem Description

In this problem, we have six end customers, each with a known demand for a product.  Customer demand can be satisfied from a set of six depots, or directly from a set of two factories.  Each depot can support a maximum volume of product moving through it, and each factory can produce a maximum amount of product.  There are known costs associated with transporting the product, from a factory to a depot, from a depot to a customer, or from a factory directly to a customer. This extension provides the opportunity to choose which four of the six possible depots to open.  It also provides an option of expanding capacity at one specific depot.

Our supply network has two factories, in Liverpool and Brighton, that produce a product.  Each has a maximum production capacity:

| Factory | Supply (tons) |
| --- | --- |
| Liverpool | 150,000 |
| Brighton |  200,000 |

The product can be shipped from a factory to a set of six depots.  Each depot has a maximum throughput.  Depots don't produce or consume the product; they simply pass the product through to customers.

| Depot | Throughput (tons) |
| --- | --- |
| Newcastle | 70,000 |
| Birmingham | 50,000 |
| London | 100,000 |
| Exeter | 40,000 |
| Bristol | 30,000 |
| Northampton | 25,000 |

We can actually only choose four of the six depots to open.  Opening a depot has a cost:

| Depot | Cost to open |
| --- | --- |
| Newcastle | 10,000 |
| Exeter | 5,000 |
| Bristol | 12,000 |
| Northampton | 4,000 |

(Note that the description in the book talks about the cost of opening Bristol or Northampton, and the savings from closing Newcastle or Exeter, but these are simply different ways of phrasing the same choice).

We also have the option of expanding the capacity at Birmingham by 20,000 tons, for a cost of \$3000.

Our network has six customers, each with a given demand.

| Customer | Demand (tons) |
| --- | --- |
| C1 | 50,000 |
| C2 | 10,000 |
| C3 | 40,000 |
| C4 | 35,000 |
| C5 | 60,000 |
| C6 | 20,000 |

Shipping costs are given in the following table (in dollars per ton).  Columns are source cities and rows are destination cities.  Thus, for example, it costs $1 per ton to ship the product from Liverpool to London.  A '-' in the table indicates that that combination is not possible, so for example it is not possible to ship from the factory in Brighton to the depot in Newcastle.

| To | Liverpool | Brighton | Newcastle | Birmingham | London | Exeter | Briston | Northhampton
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Depots |
| Newcastle   | 0.5 |   - |
| Birmingham  | 0.5 | 0.3 |
| London      | 1.0 | 0.5 |
| Exeter      | 0.2 | 0.2 |
| Bristol     | 0.6 | 0.4 |
| Northampton | 0.4 | 0.3 |
| Customers |
| C1 | 1.0 | 2.0 |   - | 1.0 |   - |   - | 1.2 |   - |
| C2 |   - |   - | 1.5 | 0.5 | 1.5 |   - | 0.6 | 0.4 |
| C3 | 1.5 |   - | 0.5 | 0.5 | 2.0 | 0.2 | 0.5 |   - |
| C4 | 2.0 |   - | 1.5 | 1.0 |   - | 1.5 |   - | 0.5 |
| C5 |   - |   - |   - | 0.5 | 0.5 | 0.5 | 0.3 | 0.6 |
| C6 | 1.0 |   - | 1.0 |   - | 1.5 | 1.5 | 0.8 | 0.9 |

The questions to be answered: (i) Which four depots should be opened? (ii) Should Birmingham be expanded? (iii) Which depots should be used to satisfy customer demand?

---
## Model Formulation

### Sets and Indices

$f \in \text{Factories}=\{\text{Liverpool}, \text{Brighton}\}$

$d \in \text{Depots}=\{\text{Newcastle}, \text{Birmingham}, \text{London}, \text{Exeter}, \text{Bristol}, \text{Northampton}\}$

$c \in \text{Customers}=\{\text{C1}, \text{C2}, \text{C3}, \text{C4}, \text{C5}, \text{C6}\}$

$\text{Cities} = \text{Factories} \cup \text{Depots} \cup \text{Customers}$

### Parameters

$\text{cost}_{s,t} \in \mathbb{R}^+$: Cost of shipping one ton from source $s$ to destination $t$.

$\text{supply}_f \in \mathbb{R}^+$: Maximum possible supply from factory $f$ (in tons).

$\text{through}_d \in \mathbb{R}^+$: Maximum possible flow through depot $d$ (in tons).

$\text{demand}_c \in \mathbb{R}^+$: Demand for goods at customer $c$ (in tons).

$\text{opencost}_d \in \mathbb{R}^+$: Cost of opening depot $d$ (in dollars).

### Decision Variables

$\text{flow}_{s,t} \in \mathbb{N}^+$: Quantity of goods (in tons) that is shipped from source $s$ to destionation $t$.

$\text{open}_{d} \in [0,1]$: Is depot $d$ open?

$\text{expand} \in [0,1]$: Should Birmingham be expanded?

### Objective Function

- **Cost**: Minimize total shipping costs plus costs of opening depots.

\begin{equation}
\text{Minimize} \quad Z = \sum_{(s,t) \in \text{Cities} \times \text{Cities}}{\text{cost}_{s,t}*\text{flow}_{s,t}} +
                          \sum_{{d} \in \text{Depots}}{\text{opencost}_d*\text{open}_d} +
                          3000 * \text{expand}
\end{equation}

### Constraints

- **Factory output**: Flow of goods from a factory must respect maximum capacity.

\begin{equation}
\sum_{t \in \text{Cities}}{\text{flow}_{f,t}} \leq \text{supply}_{f} \quad \forall f \in \text{Factories}
\end{equation}

- **Customer demand**: Flow of goods must meet customer demand.

\begin{equation}
\sum_{s \in \text{Cities}}{\text{flow}_{s,c}} = \text{demand}_{c} \quad \forall c \in \text{Customers}
\end{equation}

- **Depot flow**: Flow into a depot equals flow out of the depot.

\begin{equation}
\sum_{s \in \text{Cities}}{\text{flow}_{s,d}} =
\sum_{t \in \text{Cities}}{\text{flow}_{d,t}}
\quad \forall d \in \text{Depots}
\end{equation}

- **Depot capacity (all but Birmingham)**: Flow into a depot must respect depot capacity, and is only allowed if the depot is open.

\begin{equation}
\sum_{s \in \text{Cities}}{\text{flow}_{s,d}} \leq \text{through}_{d} * \text{open}_{d}
\quad \forall d \in \text{Depots} - \text{Birmingham}
\end{equation}

- **Depot capacity (Birmingham)**: Flow into Birmingham must respect depot capacity, which may have been expanded.

\begin{equation}
\sum_{s \in \text{Cities}} \text{flow}_{s,\text{Birmingham}} \leq \text{through}_{\text{Birmingham}} + 20000 * \text{expand}
\end{equation}

- **Open depots**: At most 4 open depots (no choice for Birmingham or London).

\begin{equation}
\sum_{d \in \text{Depots}}{\text{open}_{d}} \leq 4
\end{equation}

\begin{equation}
\text{open}_{\text{Birmingham}} = \text{open}_{\text{London}} = 1
\end{equation}

---
## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
%pip install gurobipy
```

```python
import numpy as np
import pandas as pd

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input Data
We define all the input data for the model.

```python
# Create dictionaries to capture factory supply limits, depot throughput limits, cost of opening depots, and customer demand.

supply = dict({'Liverpool': 150000,
               'Brighton': 200000})

through = dict({'Newcastle': 70000,
                'Birmingham': 50000,
                'London': 100000,
                'Exeter': 40000,
                'Bristol': 30000,
                'Northampton': 25000})

opencost = dict({'Newcastle': 10000,
                 'Birmingham': 0,
                 'London': 0,
                 'Exeter': 5000,
                 'Bristol': 12000,
                 'Northampton': 4000})

demand = dict({'C1': 50000,
               'C2': 10000,
               'C3': 40000,
               'C4': 35000,
               'C5': 60000,
               'C6': 20000})

# Create a dictionary to capture shipping costs.

arcs, cost = gp.multidict({
    ('Liverpool', 'Newcastle'): 0.5,
    ('Liverpool', 'Birmingham'): 0.5,
    ('Liverpool', 'London'): 1.0,
    ('Liverpool', 'Exeter'): 0.2,
    ('Liverpool', 'Bristol'): 0.6,
    ('Liverpool', 'Northampton'): 0.4,
    ('Liverpool', 'C1'): 1.0,
    ('Liverpool', 'C3'): 1.5,
    ('Liverpool', 'C4'): 2.0,
    ('Liverpool', 'C6'): 1.0,
    ('Brighton', 'Birmingham'): 0.3,
    ('Brighton', 'London'): 0.5,
    ('Brighton', 'Exeter'): 0.2,
    ('Brighton', 'Bristol'): 0.4,
    ('Brighton', 'Northampton'): 0.3,
    ('Brighton', 'C1'): 2.0,
    ('Newcastle', 'C2'): 1.5,
    ('Newcastle', 'C3'): 0.5,
    ('Newcastle', 'C5'): 1.5,
    ('Newcastle', 'C6'): 1.0,
    ('Birmingham', 'C1'): 1.0,
    ('Birmingham', 'C2'): 0.5,
    ('Birmingham', 'C3'): 0.5,
    ('Birmingham', 'C4'): 1.0,
    ('Birmingham', 'C5'): 0.5,
    ('London', 'C2'): 1.5,
    ('London', 'C3'): 2.0,
    ('London', 'C5'): 0.5,
    ('London', 'C6'): 1.5,
    ('Exeter', 'C3'): 0.2,
    ('Exeter', 'C4'): 1.5,
    ('Exeter', 'C5'): 0.5,
    ('Exeter', 'C6'): 1.5,
    ('Bristol', 'C1'): 1.2,
    ('Bristol', 'C2'): 0.6,
    ('Bristol', 'C3'): 0.5,
    ('Bristol', 'C5'): 0.3,
    ('Bristol', 'C6'): 0.8,
    ('Northampton', 'C2'): 0.4,
    ('Northampton', 'C4'): 0.5,
    ('Northampton', 'C5'): 0.6,
    ('Northampton', 'C6'): 0.9
})
```

## Model Deployment

We create a model and the variables. The 'flow' variables simply capture the amount of product that flows along each allowed path between a source and destination.  The 'open' variable capture decisions about which depots to open.  The 'expand' variable captures the choice of whether to expand Birmingham.  Objective coefficients are provided here, so we don't need to provide an optimization objective later.

```python
model = gp.Model('SupplyNetworkDesign2')

depots = through.keys()
flow = model.addVars(arcs, obj=cost, name="flow")
open = model.addVars(depots, obj=opencost, vtype=GRB.BINARY, name="open")
expand = model.addVar(obj=3000, vtype=GRB.BINARY, name="expand")

open['Birmingham'].lb = 1
open['London'].lb = 1
model.objcon = -(opencost['Newcastle'] + opencost['Exeter']) # Phrased as 'savings from closing'
```

    Using license file c:\gurobi\gurobi.lic

Our first constraints require the total flow along arcs leaving a factory to be at most as large as the supply capacity of that factory.

```python
# Production capacity limits

factories = supply.keys()
factory_flow = model.addConstrs((gp.quicksum(flow.select(factory, '*')) <= supply[factory]
                                 for factory in factories), name="factory")
```

Our next constraints require the total flow along arcs entering a customer to be equal to the demand from that customer.

```python
# Customer demand

customers = demand.keys()
customer_flow = model.addConstrs((gp.quicksum(flow.select('*', customer)) == demand[customer]
                                  for customer in customers), name="customer")
```

Our final constraints relate to depots.  The first constraints require that the total amount of product entering the depot must equal the total amount leaving.

```python
# Depot flow conservation

depot_flow = model.addConstrs((gp.quicksum(flow.select(depot, '*')) == gp.quicksum(flow.select('*', depot))
                               for depot in depots), name="depot")
```

The second set limits the product passing through the depot to be at most equal the throughput of that deport, or 0 if the depot isn't open.

```python
# Depot throughput

all_but_birmingham = list(set(depots) - set(['Birmingham']))

depot_capacity = model.addConstrs((gp.quicksum(flow.select(depot, '*')) <= through[depot]*open[depot]
                                   for depot in all_but_birmingham), name="depot_capacity")

```

The capacity constraint for Birmingham is different.  The depot is always open, but we have the option of expanding its capacity.

```python
birmingham_capacity = model.addConstr(gp.quicksum(flow.select('*', 'Birmingham')) <= through['Birmingham'] +
                                      20000*expand, name="birmingham_capacity")
```

Finally, there's a limit of at most 4 open depots

```python
# Depot count

depot_count = model.addConstr(open.sum() <= 4)
```

We now optimize the model

```python
model.optimize()
```

    Gurobi Optimizer version 9.1.0 build v9.1.0rc0 (win64)
    Thread count: 4 physical cores, 8 logical processors, using up to 8 threads
    Optimize a model with 21 rows, 49 columns and 119 nonzeros
    Model fingerprint: 0x140cc3a9
    Variable types: 42 continuous, 7 integer (7 binary)
    Coefficient statistics:
      Matrix range     [1e+00, 1e+05]
      Objective range  [2e-01, 1e+04]
      Bounds range     [1e+00, 1e+00]
      RHS range        [4e+00, 2e+05]
    Presolve removed 0 rows and 2 columns
    Presolve time: 0.00s
    Presolved: 21 rows, 47 columns, 113 nonzeros
    Variable types: 42 continuous, 5 integer (5 binary)

    Root relaxation: objective 1.740000e+05, 17 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

    *    0     0               0    174000.00000 174000.000  0.00%     -    0s

    Explored 0 nodes (17 simplex iterations) in 0.02 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 1: 174000

    Optimal solution found (tolerance 1.00e-04)
    Best objective 1.740000000000e+05, best bound 1.740000000000e+05, gap 0.0000%

---
## Analysis

The product demand from all of our customers can be satisfied for a total cost of $\$174,000$ by opening a depot in Northampton, closing the depot in Newcastle, and expanding the depot in Birmingham:

```python
print('List of open depots:', [d for d in depots if open[d].x > 0.5])
if expand.x > 0.5:
    print('Expand Birmingham')
```

    List of open depots: ['Birmingham', 'London', 'Exeter', 'Northampton']
    Expand Birmingham

```python
product_flow = pd.DataFrame(columns=["From", "To", "Flow"])
for arc in arcs:
    if flow[arc].x > 1e-6:
        product_flow = product_flow.append({"From": arc[0], "To": arc[1], "Flow": flow[arc].x}, ignore_index=True)
product_flow.index=[''] * len(product_flow)
product_flow
```

      From
      To
      Flow

      Liverpool
      Exeter
      40000.0

      Liverpool
      C1
      50000.0

      Liverpool
      C6
      20000.0

      Brighton
      Birmingham
      70000.0

      Brighton
      London
      10000.0

      Brighton
      Northampton
      25000.0

      Birmingham
      C2
      10000.0

      Birmingham
      C4
      10000.0

      Birmingham
      C5
      50000.0

      London
      C5
      10000.0

      Exeter
      C3
      40000.0

      Northampton
      C4
      25000.0

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```


```python
import numpy as np
from numpy import poly1d as p, polyint
from scipy.linalg import eigh
np.set_printoptions(suppress=True)

%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use(['fivethirtyeight', './00_mplrc'])
from matplotlib import rcParams
l_colors = rcParams['axes.color_cycle']

from IPython.display import HTML, Latex, display
HTML(open("00_custom.css","r").read())
```

<style>
@font-face {
    font-family: 'Charis SIL';
    src: url('fonts/CharisSILEur-R.woff') format('woff');
}
@font-face {
    font-family: 'Architects Daughter';
    font-style: normal;
    src: url(https://fonts.gstatic.com/s/architectsdaughter/v6/RXTgOOQ9AAtaVOHxx0IUBM3t7GjCYufj5TXV5VnA2p8.woff2) format('woff2');
    unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2212, U+2215, U+E0FF, U+EFFD, U+F000;
}
.prompt{display: None;}
div.cell{margin: auto;width:900px;}

div #notebook { /* centre the content */
    background: #fffaf0;
    margin: auto;
    padding: 0em;
    padding-top: 1em;
}
div #notebook_container {width: 960px!important;}
#notebook li { /* More space between bullet points */
    margin-top:0.2em;
}

div.text_cell_render{
    font-family: "Charis SIL", Cambria,  serif;
    line-height: 155%;
    font-size: 130%;
}
.CodeMirror{
    font-family: Consolas, monospace;
    font-size: 140%;
    background-color:#fcffff!important;
}

.output_area {
    font-family: Consolas,monospace;font-size: 120%;
    background-color:#fcffff!important;
    margin-top:0.8em;
}

div.output_latex{overflow:hidden}

h1 {
    font-family: 'Architects Daughter', serif;
    font-size:   48pt!important;
    text-align:  center;
    text-shadow: 4px 4px 4px #aaa;
    padding-bottom: 48pt;
}

.warning{color: rgb( 240, 20, 20 )}
</style>

# 3 DOF System

In the figure above

<ol type='a'>
 <li> the system under investigation, with the two supported masses and
      the dynamical degrees of freedom that describe the system deformation
      (top left);
 <li> the three diagrams of bending moment (in red positive bending moments,
      in blue negative ones) that derive from application of external unit
      forces, corresponding to each of the three degrees of freedom.
</ol>

The same bending moments are represented in the following data structure in terms of polynomials of first degree `p((linear_coefficient, constant_coefficient))`, each row corresponding to a load condition while the terms in each row are corresponding, the first 4 to the segments on length `L` on the horizontal part, from left to right (1,2,3) and from rigth to left (4), the fifth is corresponding to the vertical part, from top to bottom.

```python
bm = [[p(( 1, 0)), p(( 1, 1)), p(( 1, 2)), p(( 3, 0)), p(( 0, 0))],
      [p(( 0, 0)), p(( 0, 0)), p(( 1, 0)), p(( 1, 0)), p(( 0, 0))],
      [p(( 0, 0)), p(( 0,-1)), p(( 0,-1)), p((-1, 0)), p((-1, 0))]]
```

To compute the flexibilities we sum the integrals of the products of bending moments  on each of the five spans of unit length that we are using and place the results in a 2D data structure that is eventually converted to a matrix by `np.mat`.

```python
F = np.mat([[sum(polyint(bm0[i]*bm1[i])(1) for i in range(5))
              for bm1 in bm] for bm0 in bm])

print('F = 1/6 * L^3/EJ *')
print(F*6)
```

    F = 1/6 * L^3/EJ *
    [[ 72.  14. -30.]
     [ 14.   4.  -5.]
     [-30.  -5.  16.]]

we invert the flexibility matrix to obtain the stiffness matrix

```python
K = F.I
print('K = 3/136 * EJ/L^3 *')
print(K*136/3)
```

    K = 3/136 * EJ/L^3 *
    [[  39.  -74.   50.]
     [ -74.  252.  -60.]
     [  50.  -60.   92.]]

and eventually we define the mass matrix

```python
M = np.mat(np.eye(3)) ; M[2,2]=2
print('M = m *')
print (M)
```

    M = m *
    [[ 1.  0.  0.]
     [ 0.  1.  0.]
     [ 0.  0.  2.]]

```python
evals, evecs = eigh(K,M)
print("Eigenvalues, w_0^2 *", evals)
for i in range(3):
    if evecs[0,i]<0: evecs[:,i]*=-1
print("Matrix of mass normalized eigenvectors,")
print(evecs)
```

    Eigenvalues, w_0^2 * [ 0.05916788  1.06327173  6.31138392]
    Matrix of mass normalized eigenvectors,
    [[ 0.83520734  0.45567642  0.30787609]
     [ 0.15827504  0.3369755  -0.9281145 ]
     [-0.37240952  0.58258326  0.14801316]]

## The Load

The load is $F_0\,\boldsymbol{r}\,f(t)$ with $F_0 = \delta EJ/L^3$,  $\boldsymbol{r}=\begin{Bmatrix}1&0&0\end{Bmatrix}^T$ and
$f(t) = 2\sin^2(\omega_0t/2)=1-\cos(\omega_0t)$ for $0\le \omega_0 t\le 2\pi$ while $f(t)=0$ otherwise.

```python
pi = np.pi
t1 = np.linspace(0,2*pi,601)

plt.plot(t1,1-np.cos(t1))

plt.xlabel(r'$\omega_0t$', size=20)
plt.ylabel(r'$p(t)\,\frac{L^3}{\delta\,EJ}$', size=20)

plt.xlim((0,2*pi))
plt.ylim((-0.05,2.05))

plt.xticks((0,pi/2,pi,pi*1.5,2*pi),
           (r'$0$', r'$\pi/2$', r'$\pi$', r'$3\pi/2$', r'$2\pi$'), fontsize=20)

plt.title('The normalized load')
plt.show()
```

## The Particular Integrals

For our load, each modal equation of motion can be written as

\begin{align}
  m \ddot q_i + m \Lambda_i^2\omega_0^2 q_i &=
    \delta\frac{EJ}{L^3}\boldsymbol\psi_i^T\boldsymbol{r}\,
    (1-\cos(\omega_0t))\Rightarrow\\
    \ddot q_i + \Lambda_i^2\omega_0^2 q_i &= G_i \delta\omega_0^2 \,
    (1-\cos(\omega_0t))
\end{align}

with $G_i = \boldsymbol\psi_i^T\boldsymbol{r}.$

With $\xi_i = C_i + D_i \cos(\omega_0 t)$, substituting in the equation of motion and considering separately the constant terms and the cosine terms, with appropriate simplifications we have

\begin{align}
    \Lambda_i^2\,C_i &= +G_i \, \delta\\
    (\Lambda_i^2-1) \, D_i &= -G_i\,\delta
\end{align}

and consequently

$$ C_i = +\delta\,\frac{\boldsymbol\psi_i^T\boldsymbol{r}}{\Lambda^2_i},\qquad
   D_i = -\delta\,\frac{\boldsymbol\psi_i^T\boldsymbol{r}}{\Lambda^2_i-1}.$$

```python
r = np.array((1,0,0))
w = np.sqrt(evals)
C = np.dot(evecs.T,r)/evals
D = np.dot(evecs.T,r)/(1-evals)
display(Latex(r'\begin{align}' +
              r'\\'.join(r"""
              \frac{\xi_%d(t)}\delta &= %+g %+g \cos(\omega_0 t),
              && \text{for } 0 \le \omega_0 t \le 2\pi.
              """ % (i+1,C[i],D[i]) for i in range(3)) +
              r'\end{align}'))
```

\begin{align}
              \frac{\xi_1(t)}\delta &= +14.1159 +0.887733 \cos(\omega_0 t),
              && \text{for } 0 \le \omega_0 t \le 2\pi.
              \\
              \frac{\xi_2(t)}\delta &= +0.428561 -7.2019 \cos(\omega_0 t),
              && \text{for } 0 \le \omega_0 t \le 2\pi.
              \\
              \frac{\xi_3(t)}\delta &= +0.0487811 -0.0579653 \cos(\omega_0 t),
              && \text{for } 0 \le \omega_0 t \le 2\pi.
              \end{align}

```python
for i in 0, 1, 2:
    plt.plot(t1, C[i]+D[i]*np.cos(t1), label=r'$\xi_%d(t)$'%(i+1))

plt.xlabel(r'$\omega_0t$', size=20)
plt.ylabel(r'$\xi/\delta$', size=20)
plt.legend(loc=0, ncol=3)
plt.xlim((0,2*pi))
plt.xticks((0,pi/2,pi,pi*1.5,2*pi),
           (r'$0$', r'$\pi/2$', r'$\pi$', r'$3\pi/2$', r'$2\pi$'))
plt.title('The particular integrals, mode by mode')
plt.show()
```

## Modal Responses

With respect to the forced phase, the modal responses have the generic expression

\begin{align}
   q_i(t) & = A_i\cos(\Lambda_i\omega_0t)
            + B_i\sin(\Lambda_i\omega_0t) + C_i + D_i\cos(\omega_0t),\\
   \dot q_i(t) & = \Lambda_i\omega_0 \left(
       B_i\cos(\Lambda_i\omega_0t) - A_i\sin(\Lambda_i\omega_0t) \right) -
       \omega_0 D_i \sin(\omega_0t),
\end{align}

and we can write, for the specified initial rest conditions, that

$$ A_i + C_i + D_i = 0, \qquad B_i = 0$$

hence

\begin{align}
   q_i(t) & = (1-\cos(\Lambda_i\omega_0t)) C_i
            + (\cos(\omega_0t) - \cos(\Lambda_i\omega_0t)) D_i,\\
   {\dot q}_i(t) & = \Lambda_i\omega_0 (C_i+D_i) \sin(\Lambda_i\omega_0t) -
                   \omega_0 D_i \sin(\omega_0t).
\end{align}

```python
A = -C - D
L = np.sqrt(evals)

t1 = np.linspace(0,2*pi,601)
q1 = [A[i]*np.cos(L[i]*t1) + C[i] + D[i]*np.cos(t1) for i in (0,1,2)]

display(Latex(r'\begin{align}' +
              r'\\'.join(r"""
\frac{q_%d(t)}\delta &= %+g %+g \cos(\omega_0 t) %+g \cos(%g\omega_0t), &&
\text{for } 0 \le \omega_0 t \le 2\pi.
                          """ % (i+1,C[i],D[i],A[i],L[i]) for i in range(3)) +
              r'\end{align}'))
```

\begin{align}
\frac{q_1(t)}\delta &= +14.1159 +0.887733 \cos(\omega_0 t) -15.0036 \cos(0.243244\omega_0t), &&
\text{for } 0 \le \omega_0 t \le 2\pi.
                          \\
\frac{q_2(t)}\delta &= +0.428561 -7.2019 \cos(\omega_0 t) +6.77334 \cos(1.03115\omega_0t), &&
\text{for } 0 \le \omega_0 t \le 2\pi.
                          \\
\frac{q_3(t)}\delta &= +0.0487811 -0.0579653 \cos(\omega_0 t) +0.00918425 \cos(2.51225\omega_0t), &&
\text{for } 0 \le \omega_0 t \le 2\pi.
                          \end{align}

With respect to the free response phase, $2\pi \le \omega_0t$, writing

$$
  q^*_i(t) = A^*_i \cos(\Lambda_i\omega_0t) + B^*_i \sin(\Lambda_i\omega_0t)
$$

imposing the continuity of modal displacements and modal velocities we have

\begin{align}
    q_i(t_1) &= A^*_i \cos(\Lambda_i\omega_0t_1) + B^*_i \sin(\Lambda_i\omega_0t_1)\\
    \dot q_i(t_1) &= \big(
      B^*_i \cos(\Lambda_i\omega_0t_1) - A^*_i \sin(\Lambda_i\omega_0t_1)
    \big) \Lambda_i\omega_0
\end{align}

that gives

\begin{align}
    A^*_i &= \frac{q_i(t_1)\Lambda_i\omega_0\cos(\Lambda_i\omega_0t_1) - \dot q_i(t_1)\sin(\Lambda_i\omega_0t_1)}{\Lambda_i\omega_0} \\
    B^*_i &= \frac{q_i(t_1)\Lambda_i\omega_0\sin(\Lambda_i\omega_0t_1) + \dot q_i(t_1)\cos(\Lambda_i\omega_0t_1)}{\Lambda_i\omega_0} \\
\end{align}

```python
ct1 = np.cos(L*2*pi)
st1 = np.sin(L*2*pi)

q0t1 = C + D*np.cos(2*pi) + A*ct1
q1t1 =   - D*np.sin(2*pi) - A*st1*L

print(q0t1, q1t1)
As = (q0t1*L*ct1 - q1t1*st1)/L
Bs = (q0t1*L*st1 + q1t1*ct1)/L

print(As*ct1+Bs*st1, L*(Bs*ct1-As*st1))
t2 = np.linspace(2*pi, 4*pi, 601)
q2 = [As[i]*np.cos(L[i]*t2) + Bs[i]*np.sin(L[i]*t2) for i in (0,1,2)]

display(Latex(r'\begin{align}' +
              r'\\'.join(r"""
\frac{q^*_%d(t)}\delta &= %+g \cos(%g\omega_0 t) %+g \sin(%g\omega_0t), &&
\text{for } 2\pi \le \omega_0 t.
                          """ % (i+1, As[i], L[i], Bs[i], L[i]) for i in range(3)) +
              r'\end{align}'))
```

    [ 14.36696893  -0.12932443  -0.01834132] [ 3.64626164 -1.35830012  0.0017737 ]
    [ 14.36696893  -0.12932443  -0.01834132] [ 3.64626164 -1.35830012  0.0017737 ]

\begin{align}
\frac{q^*_1(t)}\delta &= -14.367 \cos(0.243244\omega_0 t) +14.9901 \sin(0.243244\omega_0t), &&
\text{for } 2\pi \le \omega_0 t.
                          \\
\frac{q^*_2(t)}\delta &= +0.129324 \cos(1.03115\omega_0 t) -1.31727 \sin(1.03115\omega_0t), &&
\text{for } 2\pi \le \omega_0 t.
                          \\
\frac{q^*_3(t)}\delta &= +0.0183413 \cos(2.51225\omega_0 t) +0.00070602 \sin(2.51225\omega_0t), &&
\text{for } 2\pi \le \omega_0 t.
                          \end{align}

### Plotting the modal responses

Let's plot the modal responses, first one by one, to appreciate the details of the single modal response

```python
for i in (0,1,2):
    plt.plot(t1/pi,q1[i], color=l_colors[i],
             label='$q_{%d}(t)$'%(i+1))
    plt.plot(t2/pi,q2[i], color=l_colors[i])
    plt.xlabel(r'$\omega_0t/\pi$', fontsize=18)
    plt.ylabel(r'$q/\delta$', fontsize=18)
    plt.legend(loc=0, fontsize=18)
    plt.show()
```

then all of them in a single plot, to appreciate the relative magnutudes of the different modal responses

```python
for i in (0,1,2):
    plt.plot(t1/pi,q1[i], color=l_colors[i],
             label='$q_{%d}(t)$'%(i+1))
    plt.plot(t2/pi,q2[i], color=l_colors[i])

plt.xlabel(r'$\omega_0t/\pi$', fontsize=18)
plt.ylabel(r'$q/\delta$', fontsize=18)
plt.legend(loc=0, fontsize=18)
plt.show()
```

## System Response in Natural Coordinates

We stack together the times and the modal responses for the forced and the free phases in two single vectors, then we compute the nodal response by premultiplying the modal response by the eigenvectors matrix

```python
t = np.hstack((t1, t2))
q = np.hstack((q1, q2))
x = np.dot(evecs, q)
```

### Plotting of the natural coordinate responses

All of them in a single plot, as they have the same order of magnitude

```python
for i in (0,1,2): plt.plot(t/pi,x[i],
                           label='$x_{%d}(t)$'%(i+1))

plt.xlabel(r'$\omega_0t/\pi$', fontsize=18)
plt.ylabel(r'$x/\delta$', fontsize=18)
plt.legend(loc=0, fontsize=18)
plt.show()
```

## Final Displacements and Final Velocities

Say that $t_2=4\pi/\omega_0$, we compute the vectors of sines and cosines with different frequencies at $t_2$, then we compute the modal displacements and velocities (note that the dimensional velocities are these adimensional velocities multiplied by $\omega_0\,\delta$) and eventually we compute the nodal quantities by premultiplication by the eigenvectors matrix.

```python
ct2 = np.cos(L*4*pi)
st2 = np.sin(L*4*pi)

q0t2 = As*ct2+Bs*st2 ; q1t2 = L*(Bs*ct2-As*st2)

display(Latex(r"$\boldsymbol x(t_2) = \{"+
              ",".join("%10.6f"%x for x in np.dot(evecs,q0t2))+
              "\}\,\delta$"))
display(Latex(r"$\boldsymbol v(t_2) = \{"+
              ",".join("%10.6f"%x for x in np.dot(evecs,q1t2))+
              "\}\,\omega_0\,\delta$"))
```

$\boldsymbol x(t_2) = \{ 12.848823,  2.320919, -6.024918\}\,\delta$

$\boldsymbol v(t_2) = \{ -3.383877, -0.963441,  0.480770\}\,\omega_0\,\delta$

```python

```


## Burgers Equation - Forward Euler/0-estimation
#### Parameter estimation for Burgers' Equation using Gaussian processes (Forward Euler scheme)

#### Problem Setup

$u_t + u u_{x} = \nu u_{x}$

$u(x,t) = \frac{x}{1+t}$ => We'd expect $\nu = 0$

$u_0(x) := u(x,0) = x$

$x \in [0, 1], t \in \{0, \tau \}$

Using the forward Euler scheme, the equation can be re-written as:

$\frac{u_n - u_{n-1}}{\tau} + u_{n-1} \frac{d}{dx}u_{n-1}  = \nu \frac{d^2}{dx}u_{n-1}$

and setting the factor $u_{n-1}(x) = u_0(x) = x$ (no mean used! Should give a better result) to deal with the non-linearity:

$\tau \nu \frac{d^2}{dx}u_{n-1} - \tau x \frac{d}{dx}u_{n-1} + u_{n-1} = u_{n}$

Consider $u_{n-1}$ to be a Gaussian process.

$u_{n-1} \sim \mathcal{GP}(0, k_{uu}(x_i, x_j, \theta))$

And the linear operator:

$\mathcal{L}_x^\nu = \cdot + \tau \nu \frac{d}{dx}\cdot - \tau x \frac{d}{dx} \cdot$

so that

$\mathcal{L}_x^\nu u_{n-1} = u_n$

Problem at hand: estimate $\nu$ (should be $\nu = 0$ in the end).

For the sake of simplicity, take $u := u_{n-1}$ and $f := u_n$.

#### step 1: Simulate data

Take data points at $t = 0$ for $(u_{n-1})$ and $t = \tau$ for $(u_n)$, where $\tau$ is the time step.

```python
import numpy as np
import sympy as sp
from scipy.optimize import minimize
import matplotlib.pyplot as plt
import warnings
import time
```

```python
tau = 0.001
def get_simulated_data(tau, n=20):
    x = np.random.rand(n)
    y_u = x
    y_f = x/(1+tau)
    return (x, y_u, y_f)

(x, y_u, y_f) = get_simulated_data(tau)
```

```python
plt.plot(x, y_u, 'ro')
plt.show()
```

```python
plt.plot(x, y_f, 'bo')
plt.show()
```

#### Step 2:Evaluate kernels

$k_{nn}(x_i, x_j; \theta) = \theta exp(-\frac{1}{2l}(x_i-x_j)^2)$

```python
x_i, x_j, theta, l, nu = sp.symbols('x_i x_j theta l nu')
kuu_sym = theta*sp.exp(-1/(2*l)*((x_i - x_j)**2))
kuu_fn = sp.lambdify((x_i, x_j, theta, l), kuu_sym, "numpy")
def kuu(x, theta, l):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kuu_fn(x[i], x[j], theta, l)
    return k
```

$k_{ff}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\nu \mathcal{L}_{x_j}^\nu k_{uu}(x_i, x_j; \theta) \\
= k_{uu} + \tau \nu \frac{d}{dx_i}k_{uu} - \tau x_i \frac{d}{dx_i}k_{uu} + \tau \nu \frac{d}{dx_j}k_{uu} + \tau^2 \nu^2 \frac{d}{dx_i} \frac{d}{dx_j}k_{uu} - \tau^2 \nu x_i\frac{d^2}{dx_i dx_j} k_{uu} - \tau x_j \frac{d}{dx_j}k_{uu} - \tau^2 \nu x_j \frac{d^2}{dx_i dx_j} k_{uu} + \tau^2 x_i x_j \frac{d^2}{dx_i dx_j}k_{uu}$

```python
kff_sym = kuu_sym \
        + tau*nu*sp.diff(kuu_sym, x_i) \
        - tau*x_i*sp.diff(kuu_sym, x_i) \
        + tau*nu*sp.diff(kuu_sym, x_j) \
        + tau**2*nu**2*sp.diff(kuu_sym, x_j, x_i) \
        - tau**2*nu*x_i*sp.diff(kuu_sym, x_j, x_i) \
        - tau*x_j*sp.diff(kuu_sym, x_j) \
        - tau**2*nu*x_j*sp.diff(kuu_sym, x_j, x_i) \
        + tau**2*x_i*x_j*sp.diff(kuu_sym, x_j, x_i)
kff_fn = sp.lambdify((x_i, x_j, theta, l, nu), kff_sym, "numpy")
def kff(x, theta, l, nu):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kff_fn(x[i], x[j], theta, l, nu)
    return k
```

$k_{fu}(x_i,x_j;\theta,\phi) \\
= \mathcal{L}_{x_i}^\nu k_{uu}(x_i, x_j; \theta) \\
= k_{uu} + \tau \nu \frac{d}{dx_i}k_{uu} - \tau x_i\frac{d}{dx_i}k_{uu}$

```python
kfu_sym = kuu_sym + tau*nu*sp.diff(kuu_sym, x_i) - tau*x_i*sp.diff(kuu_sym, x_i)
kfu_fn = sp.lambdify((x_i, x_j, theta, l, nu), kfu_sym, "numpy")
def kfu(x, theta, l, nu):
    k = np.zeros((x.size, x.size))
    for i in range(x.size):
        for j in range(x.size):
            k[i,j] = kfu_fn(x[i], x[j], theta, l, nu)
    return k
```

```python
def kuf(x, theta, l, nu):
    return kfu(x,theta, l, nu).T
```

#### Step 3: Compute NLML

```python
def nlml(params, x, y1, y2, s):
    theta_exp = np.exp(params[0])
    l_exp = np.exp(params[1])
    K = np.block([
        [kuu(x, theta_exp, l_exp) + s*np.identity(x.size), kuf(x, theta_exp, l_exp, params[2])],
        [kfu(x, theta_exp, l_exp, params[2]), kff(x, theta_exp, l_exp, params[2]) + s*np.identity(x.size)]
    ])
    y = np.concatenate((y1, y2))
    val = 0.5*(np.log(abs(np.linalg.det(K))) + np.mat(y) * np.linalg.inv(K) * np.mat(y).T)
    return val.item(0)
```

#### Step 4: Optimise hyperparameters

```python
m = minimize(nlml, np.random.rand(3), args=(x, y_u, y_f, 1e-7), method="Nelder-Mead", options = {'maxiter' : 1000})
```

```python
m.x[2]
```

    0.0003466983650267611

```python
m
```

     final_simplex: (array([[6.87006135e+00, 7.96630233e+00, 3.46698365e-04],
           [6.87003829e+00, 7.96626970e+00, 3.47929472e-04],
           [6.87004386e+00, 7.96627739e+00, 3.48762203e-04],
           [6.86996662e+00, 7.96621025e+00, 3.45925101e-04]]), array([-296.99997953, -296.99997838, -296.99997738, -296.99997381]))
               fun: -296.99997952966345
           message: 'Optimization terminated successfully.'
              nfev: 232
               nit: 126
            status: 0
           success: True
                 x: array([6.87006135e+00, 7.96630233e+00, 3.46698365e-04])

#### Step 5: Analysis w.r.t. the number of data points (up to 25):

In this section we want to analyze the error of our algorithm using two different ways and its time complexity.

```python
res = np.zeros((5,25))
timing = np.zeros((5,25))
warnings.filterwarnings("ignore")
for k in range(5):
    for n in range(25):
        start_time = time.time()
        (x, y_u, y_f) = get_simulated_data(tau, n)
        m = minimize(nlml, np.random.rand(3), args=(x, y_u, y_f, 1e-7), method="Nelder-Mead")
        res[k][n] = m.x[2]
        timing[k][n] = time.time() - start_time
```

###### Plotting the error in our estimate for $\nu$ (Error = $| \nu_{estimate} - \nu_{true} |$):

```python
lin  =  np.linspace(1, res.shape[1], res.shape[1])
est = np.repeat(0.01, len(lin))

f, (ax1, ax2) = plt.subplots(ncols=2, nrows=2, figsize=(13,7))
ax1[0].plot(lin, np.abs(res[0,:]), color = 'green')
ax1[0].plot(lin, est, color='blue', linestyle='dashed')
ax1[0].set(xlabel= r"Number of data points", ylabel= "Error")
ax1[1].plot(lin, np.abs(res[1,:]), color = 'green')
ax1[1].plot(lin, est, color='blue', linestyle='dashed')
ax1[1].set(xlabel= r"Number of data points", ylabel= "Error")
ax2[0].plot(lin, np.abs(res[2,:]), color = 'green')
ax2[0].plot(lin, est, color='blue', linestyle='dashed')
ax2[0].set(xlabel= r"Number of data points", ylabel= "Error")
ax2[1].plot(lin, np.abs(res[3,:]), color = 'green')
ax2[1].plot(lin, est, color='blue', linestyle='dashed')
ax2[1].set(xlabel= r"Number of data points", ylabel= "Error");
```

```python
lin = np.linspace(1, res.shape[1], res.shape[1])

for i in range(res.shape[0]):
    plt.plot(lin, np.abs(res[i,:]))
    plt.ylabel('Error')
    plt.xlabel('Number of data points')
    plt.show()
```

All in one plot:

```python
lin = np.linspace(1, res.shape[1], res.shape[1])

for i in range(res.shape[0]):
    plt.plot(lin, np.abs(res[i,:]))
    plt.ylabel('Error')
    plt.xlabel('Number of data points')

est = np.repeat(0.01, len(lin))
plt.plot(lin, est, color='blue', linestyle='dashed')
plt.show()
```

We see that for n sufficiently large (in this case $n \geq 3$), we can assume the error to be bounded by 0.01.

###### Plotting the error between the solution and the approximative solution:

###### Plotting the execution time:

```python
lin = np.linspace(1, timing.shape[1], timing.shape[1])

for i in range(timing.shape[0]):
    plt.plot(lin, timing[i,:])
    plt.ylabel('Execution time in seconds')
    plt.xlabel('Number of data points')
    plt.show()
```

```python
lin = np.linspace(1, timing.shape[1], timing.shape[1])

for i in range(timing.shape[0]):
    plt.plot(lin, timing[i,:])
    plt.ylabel('Execution time in seconds')
    plt.xlabel('Number of data points')

est = lin**(1.25)
plt.plot(lin, est, color='blue', linestyle='dashed')
plt.show()
```

#Curiously, the time complexity seems to be around $\mathcal{O}(n^{5/4})$ (blue-dashed line).

#Assuming an equal amount of function evaluations in the Nelder-Mead algorithm for different values of n,
#we would expect a time complexity of $\mathcal{O}(n^3)$, due to the computation of the inverse of an $n\times n$-matrix in every evaluation of $\textit{nlml}$.


# 第三讲：线性回归的概率解释、局部加权回归、逻辑回归

我们先来回顾一下上一讲中的内容。

* 我们用$\left(x^{(i)},\ y^{(i)}\right)$表示训练集中的第$i$个样本；
* 用向量$\vec\theta=(\theta_0,\ \cdots,\ \theta_n)$表示特征值的系数向量；
* 接着用$h_\theta\left(x^{(i)}\right)$表示用于拟合训练集中样本的假设函数（假设函数基于参数$\vec\theta$），并定义$h_\theta\left(x^{(i)}\right)=\displaystyle\sum_{j=0}^n\theta_jx_j^{(i)}=\theta^Tx$，$j$表示训练集中各样本的特征值$(x_1,\ \cdots,\ x_n)$的序号，共有$n$个特征值，而$x_0=1$，此为常数项$\theta_0$（即截距项）；
* 最后定义了成本函数$J(\theta)=\frac{1}{2}\left(\displaystyle\sum_{i=1}^mh_\theta\left(x^{(i)}\right)-y^{(i)}\right)^2$，其中$m$表示训练集中共有$m$个训练样本，其形式化描述为$\theta=\left(X^TX\right)^{-1}X^Ty$。

下面我们将接着上一讲的内容，从概率解释的角度介绍最小二乘的合理性。

## 3. 线性模型的概率解释

当我们遇到一个回归问题时，可能会有疑问，为什么使用线性回归？更具体的可能会问，为什么使用最小二乘成本函数$J$？这样合理吗？有那么多别的指标，比如预测值与实际值之差的绝对值、四次方等，为什么偏偏选择差的平方作为优化指标？在这一节我们将从一系列基于概率的假设中推出最小二乘回归的合理性。

仍旧使用“公寓租金”例子，我们假设价格是一些特征的线性函数加上$\epsilon^{(i)}$，于是有下面的式子将目标值与输入向量联系起来：

$$y^{(i)}=\theta^Tx^{(i)}+\epsilon^{(i)}$$

式中的$\epsilon^{(i)}$是误差项，它捕获了可能的未建模影响因素（比如我们的回归算法并没有纳入某个影响公寓租金的关键因素），也可能包含了存在于样本中的随机噪音。我们做出进一步假设，$\epsilon^{(i)}$是独立同分布（IDD: independently and identically distributed）的，服从期望为$0$方差为$\sigma^2$的高斯分布（或自然分布，Gaussian distribution/Normal distribution），即$\epsilon^{(i)}\sim N\left(0,\sigma^2\right)$，即$\epsilon^{(i)}$的概率密度函数为：

$$p\left(\epsilon^{(i)}\right)=\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\left(\epsilon^{(i)}\right)^2}{2\sigma^2}\right)$$

则可以得到：

$$p\left(y^{(i)}\mid x^{(i)};\theta\right)=\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\left(y^{(i)}-\theta^Tx^{(i)}\right)^2}{2\sigma^2}\right)$$

（这里教授简要说明了为什么使用高斯分布，一个原因是因为高斯分布便于计算；另一个原因是在处理实际问题的过程中，如果我们使用线性回归让后尝试测量误差的分布，大多情况下误差都服从高斯分布。高斯模型对于这类对于这类回归问题中的误差来说是一个很好的假设。另外，许多随机变量之和趋于服从高斯分布，如果误差是有许多效应共同导致的，例如卖家的情绪、买家的情绪、房子是否有花园、房间的壁炉数量等等，所有这些效应都是独立的，那么根据中心极限定理，这些效应的总和会近似的服从高斯分布。）

$p\left(y^{(i)}\mid x^{(i)};\theta\right)$表示以$\theta$为参数的对于给定$x^{(i)}$下$Y$的条件概率。我们不能将$\theta$也纳入条件（即不能记为$p\left(y^{(i)}\mid x^{(i)},\theta\right)$，这表示$y$是以$\theta,\ X$的为条件的概率，属于贝叶斯学派的观点），因为$\theta$并不是一个随机变量（频率学派的观点，我们不把$\theta$当做随机变量，我们认为$\theta$确有其值，但是我们现在不知道是多少）。我们也可以将$y^{(i)}$的分别表示为$y^{(i)}\mid x^{(i)};\theta\sim N\left(\theta^Tx^{(i)},\sigma^2\right)$。（即在给定$x^{(i)}$下假设函数的期望是$\theta^Tx^{(i)}$。）

现在，已知$X$（包含所有训练样本$x^{(i)}$的设计矩阵）和$\theta$，求$y^{(i)}$的分布。概率由$p\left(\vec y\mid X;\theta\right)$给出，这个量通常被视为一个关于$\vec y$（也可能关于$X$）的函数，有着固定参数$\theta$。现在我们明确的将这个量视为一个关于$\theta$的函数，我们仍叫它**似然函数（likelihood function）**（$\theta$的似然性是关于给定$X$下数据$Y$以$\theta$作为参数的概率，虽然$\theta$的似然性与数据$Y$的概率是一样的，而且尽管似然性和概率几乎是一样的意义，然而我们在这里使用似然性这个词是想强调，我们想要把$p\left(\vec y\mid X;\theta\right)$看做是在$\vec y,\ X$确定时关于$\theta$的函数，以后我们也会继续使用“参数的似然性”和“数据的概率”这种称谓，而不是“数据的似然性”和“参数的概率”这种说法。）：

$$L(\theta)=L\left(\theta;X,\vec y\right)=p\left(\vec y\mid X;\theta\right)$$

利用$\epsilon^{(i)}$独立的假设（也因为$y^{(i)}$是对于给定$x^{(i)}$的），上式可以写为：

$$\begin{align}L(\theta)&=\prod_{i=1}^mp\left(y^{(i)}\mid x^{(i)};\theta\right)\\&=\prod_{i=1}^m\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\left(y^{(i)}-\theta^Tx^{(i)}\right)^2}{2\sigma^2}\right)\end{align}$$

我们现在推出了将$x^{(i)}$与$y^{(i)}$联系在一起的概率模型，接下来需要考虑如何合理的对参数$\theta$做出猜测？最大似然估计告诉我们，应该选择使数据概率尽可能大的$\theta$，于是，我们应该选择能够使$L(\theta)$取最大值的$\theta$。

比起最大化$L(\theta)$，我们也可以选择最大化任意关于$L(\theta)$的严格增函数。因此，为了便于求导，我们可以最大化**对数似然估计（log likelihood）**$\mathscr{l}(\theta)$：

$$\begin{align}\mathscr{l}(\theta)&=\log L(\theta)\\&=\log \prod_{i=1}^m\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\left(y^{(i)}-\theta^Tx^{(i)}\right)^2}{2\sigma^2}\right)\\&=\sum_{i=1}^m\frac{1}{\sqrt{2\pi}\sigma}\exp\left(-\frac{\left(y^{(i)}-\theta^Tx^{(i)}\right)^2}{2\sigma^2}\right)\\&=m\log \frac{1}{\sqrt{2\pi}\sigma}-\frac{1}{\sigma^2}\cdot\frac{1}{2}\sum_{i=1}^m\left(y^{(i)}-\theta^Tx^{(i)}\right)^2\end{align}$$

于是，我们发现，最大化$\mathscr{l}(\theta)$的过程变成了最小化$\frac{1}{2}\sum_{i=1}^m\left(y^{(i)}-\theta^Tx^{(i)}\right)^2$，这个式子正是成本函数$J(\theta)$的定义式。

综上，可以看出，在关于训练数据的概率假设中，最小二乘回归与$\theta$的最大似然估计一致。也正是因为最小二乘回归其实是再做最大似然估计，所以我们才会强调最小二乘是一种“很自然”的方法。（不过，概率假设并不是证明“最小二乘是一种非常易用的、合理的求解过程”这一结论的*必要条件*，这只是众多假设中的一种，最小二乘在这种假设下合理，除此之外还有其他假设也可以证明这一结论。）

还需要注意的是，我们对$\theta$的求解并不依赖与$\sigma^2$，即使不知道$\sigma$我们依然完成了运算。我们将在后面关于指数族和一般线性模型的课程中使用这一结论。

## 欠拟合和过拟合

继续使用上一讲的例子，当我们考虑使用特征变量$x\in\mathbb{R}$来预测$y$，下图最左为使用直线$y=\theta_0+\theta_1x$拟合训练集，可以发现得到的直线并不能较为准确的描述训练数据的形态，我们说这不是一个良好的拟合。

如果我们再加入一个特征值$x^2$，然后对$y=\theta_0+\theta_1x+\theta_2x^2$做拟合，于是我们得到一个稍好的拟合，如中图所示。看上去似乎是特征值取的越多越好，其实不然，取的过多也会有问题。如图中最右所示，这是用五阶多项式$y=\displaystyle\sum_{j=0}^5\theta_jx_j$得到的结果，可以看出曲线恰好能够经过每一个训练样本，但这并不是我们期望的良好的拟合，比如在“公寓租金”的例子中，这个曲线并不能良好的反映面积$x$对租金$y$的影响。我们将在后面的课程中形式化定义这个现象，并告诉大家如何鉴定什么才是一个“良好的”假设，现在大家只需要了解，左图叫做**欠拟合（underfitting）**，表示数据的特征并没有被拟合曲线良好的表达出来；而右图叫做**过拟合（overfitting）**。

从上面的例子，我们可以清楚的认识到，对特征的选择会在很大程度上影响学习算法的表现。（当学到模型选择时，我们也会见到自动选择“好”特征值的算法。）

## 4. 局部加权回归（LWR: locally weighted linear regression）

如果我们有足够的训练数据，特征值的选择就没那么重要了。在前面学到的线性回归中，为了查询$x$（即得到$h(x)$），我们将执行下面的步骤：

1. 寻找使$\sum_i\left(y^{(i)}-\theta^Tx^{(i)}\right)^2$取最小的$\theta$；
2. 给出预测结果$\theta^Tx$。

相比上面的过程，局部加权回归执行以下步骤：

1. 针对给定的查询点$x$，寻找使$\sum_i\omega^{(i)}\left(y^{(i)}-\theta^Tx^{(i)}\right)^2$取最小的$\theta$；
2. 给出预测结果$\theta^Tx$。

这里的$\omega$称为权值，是一个非负数。直观上看，如果对于某个$i$，$\omega^{(i)}$取值较大，则在计算$\theta$取值时，我们将尽可能的减小$\left(y^{(i)}-\theta^Tx^{(i)}\right)^2$项的取值（精确拟合）；反之，如果$\omega^{(i)}$取值较小，则$\left(y^{(i)}-\theta^Tx^{(i)}\right)^2$所得到的的误差项将足够小而忽略不计。

一种较为合理的权值设置（这里我们使用指数衰减函数，exponential decay function，权值的设置还有很多其他选择）：

$$\omega^{(i)}=exp\left(-\frac{\left(x^{(i)}-x\right)^2}{2\tau^2}\right)$$

易看出，对于给定的查询点$x$，权值的大小与训练样本$x^{(i)}$相对于$x$的位置密切相关：

* 如果$x^{(i)}$距离$x$很近，则$\omega^{(i)}$将取靠近$1$的值；
* 如果$x^{(i)}$距离$x$很远，则$\omega^{(i)}$将取到$0$附近。

因此，通过这种方法得到的$\theta$将更注重查询点$x$附近的训练样本（权重较高）的精确拟合，而忽略那些距离较远的训练样本（权重较低）。（尽管权值函数的样子看起来很像高斯分布的概率密度函数，但是这个函数跟高斯分布并没有直接联系，另外$\omega^{(i)}$并不是随机变量，也并不服从高斯分布，它只是一个样子恰好类似钟形的曲线。）

函数中的$\tau$称作**带宽（bandwidth）**（或波长）参数，它控制了权值随距离下降的速率。如果$\tau$取值较小，则会得到一个较窄的钟形曲线，这意味着离给定查询点$x$较远的训练样本$x^{(i)}$的权值（对查询点$x$附近训练样本的拟合的影响）将下降的非常快；而$\tau$较大时，则会得到一个较为平缓的曲线，于是查询点附近的训练样本的权重随距离而下降的速度就会相对比较慢。

值得注意的是，这是一个非参数算法，我们在使用这个方法时，是针对给定的查询点$x$进行计算，也就是每当我们对于一个给定的$x$做出预测时，都需要根据整个训练集重新进行拟合运算，如果训练集很大而查询很频繁的话，这个算法的代价将非常高。关于提高这个算法的效率，可以参考Andrew Moore关于KD-Tree的工作。


# FEM: 1D Beispiel mit Python
Dr. Zoufiné Lauer-Baré

## Lineare Elastiziät: Gleichungen
Im Gebiet (den Bauteilen) bei isotropem Material
\begin{align}
-\nabla \cdot \mathbf{\Sigma}&=\mathbf{f} &\text{ Impulserhaltung}\\
\mathbf{\Sigma} & =2\mu \mathbf{E}+\lambda \text{tr}(\mathbf{E})\mathbf{I} &\text{ Hook-Gesetz: Spannung-Dehnung}\\
\mathbf{E} & = \frac{1}{2}\left(\nabla \mathbf{u}+(\nabla \mathbf{u})^T\right)&\text{ Dehnung-Verschiebung}
\end{align}
An jeder Stelle des Randes müssen Randbedingungen definiert sein. Hierbei ist $\mathbf{f}$ eine Volumenkraft, $$\mathbf{u}(x_1,x_2,x_3)= \begin{pmatrix}u_1(x_1,x_2,x_3)\\u_2(x_1,x_2,x_3)\\u_3(x_1,x_2,x_3) \end{pmatrix}$$ die Verschiebung.

Hierbei ist die Dehnung $\mathbf{E}=\left(e_{ij}\right)_{ij=1,2,3}$ im Detail gegeben durch ($i,j=1,2,3$) $$e_{ij}=\frac{1}{2}\left(\frac{\partial u_i}{\partial x_j}+\frac{\partial u_j}{\partial x_i}\right).$$

Die Komponenten der oberen Spannungsmatrix $\mathbf{\Sigma}=\left(\sigma_{ij}\right)_{ij=1,2,3}$ sind ($i\neq j$) $$\sigma_{ij}=2\mu e_{ij}.$$

Für $i=j$ $$\sigma_{ii}=2\mu e_{ii}+\lambda\nabla\cdot\mathbf{u}$$

Manchmal wird auch $\mathbf{u}=(u,v,w)^T$ geschrieben.
$\lambda$ und $\mu$ (Schubmodul) sind Materialparameter die Lamé-Parameter genannt werden.

Statt $\lambda$ und $\mu$ werden oft auch der Elastizitätsmodul $E$ und die Querkontraktionsszahl $\nu$ angegeben.

### Materialkonstanten

\begin{align}
&E=\frac{\mu(3\lambda+2\mu)}{\lambda+\mu} &\nu=\frac{\lambda}{2(\lambda+\mu)}\\
&\lambda=\frac{E\nu}{(1+\nu)(1-2\nu)} &\mu=\frac{E}{2(1+\nu)}
\end{align}

# Exemplarischer FEM Prozess in 1D: Mit Vorgegebener Verschiebung

## Physikalische Modellierung

Zum Beispiel, Ermittlung der Verschiebung, Spannung und Dehnung eines Zylinders. Das einfachste Modell hierzu ist das "Zugstab-Modell". Hier können die gesuchten Größen analytisch berechnet werden. Deswegen kann anhand dieses Beispiels die Finite Elemenete Methode gut illustriert werden.

\begin{align}
-\frac{\partial }{\partial x} \cdot E \frac{\partial u}{\partial x}&= f &x\in (0,2)\label{inner}\\
u(0) & =u_{Dirichlet0} \\
u(2) & =u_{Dirichlet2}
\end{align}

## Preprocessing: Diskretisierung/Vernetzung

Aufteilung des Berechnungsgebietes in einzelne "Elemente", in diesem Fall Linien. In diesem Fall in 5 Punkte (Knoten): $$0, \frac{1}{2}, 1, \frac{3}{2}, 2 $$
Die Elemente haben in diesem Fall die gleiche Länge: $\frac{1}{3}$. Das muss nicht so sein.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib as mpl
from matplotlib import ticker
import numpy as np
from scipy import integrate
from fem.funktionen import *
import seaborn as sns
mpl.rcParams["figure.figsize"] = (20,7)
mpl.rcParams['lines.linewidth'] = 4
mpl.rcParams['lines.markersize'] = 11

#sns.set()
farben = sns.color_palette()
plt.style.use('seaborn-darkgrid')
```

```python
x=np.linspace(0,2,1000)
ax=plt.axes()
ax.tick_params(labelsize=15)

ax.plot(np.array([0,1/2.,1,3/2.,2]), np.array([0,0,0,0,0]), label='Elemente', color='black')
ax.scatter(np.array([0,1/2.,1,3/2.,2]), np.array([0,0,0,0,0]), label='Knoten',color='red')
ax.tick_params(labelsize=15)
#ax.set_axis_off()
ax.axes.get_yaxis().set_visible(False)
plt.legend(loc='best',fontsize=15)
```

## Solving

### Berechnung der Steifigkeitsmatrix

Ansatz (Nach Ritz): $$u_{fem}=u_0 N_0(x)+u_{\frac{1}{2}} N_{\frac{1}{2}}(x)+u_1 N_1(x)+u_{\frac{3}{2}} N_{\frac{3}{2}}(x)+u_2 N_2(x)$$

Die Formfunktionen $N_i$ sind so definiert, dass sie an ihrem eigenen Knoten 1 sind und an anderen Knoten 0. Die Koeffizienten der Knoten $u_i$ sind konstant und unbekannt. Bei FEM geht es im "solving" darum, diese Koeffizienten $\boldsymbol{u}=(u_0, u_{\frac{1}{2}},u_1,u_{\frac{3}{2}},u_2)^T$ zu berechnen. Sind die Koeffizienten bestimmt, ist die Approximation der Verschiebung $u_{fem}$ bestimmt!

In einem nächsten Schritt wird mit etwas Mathematik (Analysis und lineare Algebra) eine "Steifigkeitsmatrix" $\boldsymbol{K}$ hergeleitet und eine entsprechende rechte Seite $\boldsymbol{b}$. Am Ende des Tages steht dann ein lineares Gleichungssystem, dass numerisch gelöst wird: $$K \boldsymbol{u}=\boldsymbol{b}$$

Um konkret die Steifigkeitsmatrix zu berechnen, müssen konkrete Formfunktionen $N_i$ gewählt werden, z.B. "Hütchenfunktionen" (Lagrange-Polynome erster Ordnung). Im Fall des aktuellen Beispiels:

\begin{align}
N_0 =
\begin{cases}
1-2x & x\in [0,\frac{1}{2}) \\
0 & \, \text{sonst}
\end{cases}
\end{align}

\begin{align}
N_2 =
\begin{cases}
2(x-\frac{3}{2}) & x\in [\frac{3}{2},2) \\
0 & \, \text{sonst}
\end{cases}
\end{align}

Für $i=1,\frac{1}{2},\frac{3}{2}$

\begin{align}
N_i=
\begin{cases}
2(x-(i-\frac{1}{2})) & x\in [i-\frac{1}{2},i) \\
1-2(x-i)& x\in [i,i+\frac{1}{2}) \\
0 & \, \text{sonst}
\end{cases}
\end{align}

```python

ax=plt.axes()
ax.plot(x,Nh(x,0), label=r'$N_{0}$')
ax.plot(x,Nh(x,1/2.), label=r'$N_{\frac{1}{2}}$')
ax.plot(x,Nh(x,1), label=r'$N_{1}$')
ax.plot(x,Nh(x,3/2.), label=r'$N_{\frac{3}{2}}$')
ax.plot(x,Nh(x,2), label=r'$N_{2}$')
ax.tick_params(labelsize=15)
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.85, box.height* 0.85])
plt.legend(loc='center left', prop={'size':15}, bbox_to_anchor=(1, 0.5))
plt.show()
```

Um die Steifigkeitsmatrix zu bekommen wird die Gleichung $-\frac{\partial u}{\partial x} \cdot E \frac{\partial u}{\partial x}= f$ mit einer beliebigen Funktion $v$ multipliziert und es wird partiell integriert:
$$-E\int_{0}^{2}\frac{\partial }{\partial x}\frac{\partial u}{\partial x}vdx=\int_{0}^{2}fvdx$$

$$E\int_{0}^{2}\frac{\partial u}{\partial x}\frac{\partial v}{\partial x}dx-\frac{\partial u}{\partial x}v\rvert_{0}^{2}=\int_{0}^{2}fvdx$$

Dann werden die Verschiebung $u$ und die Testfunktion $v$ mit Linearkombinationen der Formfunktionen approximmiert und in die Gleichung eingesetzt.
\begin{align}
u_{fem}&=u_0 N_0(x)+u_{\frac{1}{2}} N_{\frac{1}{2}}(x)+u_1 N_1(x)+u_{\frac{3}{2}} N_{\frac{3}{2}}(x)+u_2 N_2(x)\\
v&=v_0 N_0(x)+v_{\frac{1}{2}} N_{\frac{1}{2}}(x)+v_1 N_1(x)+v_{\frac{3}{2}} N_{\frac{3}{2}}(x)+v_2 N_2(x)
\end{align}

Um nach dem Einsetzen mit den Rechenregeln für Matrizen und ihrer transponierten die Steifigkeitsmatrix zu erhalten, werden $u_{fem}$ und $v$ kompakter als Matrixprodukt geschrieben:

\begin{align}
u_{fem}&=\boldsymbol{N}^T\boldsymbol{u}\\
v&=\boldsymbol{N}^T\boldsymbol{v}
\end{align}

wobei

\begin{align}
\boldsymbol{u}=(u_0, u_{\frac{1}{2}},u_1,u_{\frac{3}{2}},u_2)^T\\
\boldsymbol{v}=(v_0, v_{\frac{1}{2}},v_1,v_{\frac{3}{2}},v_2)^T\\
\boldsymbol{N}=(N_0, N_{\frac{1}{2}},N_1,N_{\frac{3}{2}},N_2)^T
\end{align}

Im Folgenden ist es wichtig zu wissen, dass für zwei Matrizen $\boldsymbol{A},\boldsymbol{B}\in \mathbb{R}^{n\times 1}$ gilt:

\begin{align}
\left(\boldsymbol{A}\boldsymbol{B}\right)^T=\boldsymbol{B}^T\boldsymbol{A}^T\\
\end{align}

In dem aktuellen Beispiel ist dazu noch wichtig, dass $$\left(\boldsymbol{N}^T\boldsymbol{u}\right)^T=\boldsymbol{N}^T\boldsymbol{u}$$

Gerne testen!

Mit etwas linearer Algebra erhählt man dann

\begin{align}
\boldsymbol{v}^T\left(E\int_{0}^{2}\frac{\partial \boldsymbol{N}}{\partial x}\frac{\partial \boldsymbol{N}^T}{\partial x}dx-\frac{\partial \boldsymbol{N}}{\partial x}\boldsymbol{N}^T\rvert_{0}^{2}\right)\boldsymbol{u}=\boldsymbol{v}^Tf\int_{0}^{2}\boldsymbol{N}dx
\end{align}

Da diese Gleichung für alle Testfunktionen gelten soll, kann $\boldsymbol{v}^T$ weggelassen werden und man erhält das gesuchte Gleichungssystem

\begin{align}
\left(E\int_{0}^{2}\frac{\partial \boldsymbol{N}}{\partial x}\frac{\partial \boldsymbol{N}^T}{\partial x}dx-\frac{\partial \boldsymbol{N}}{\partial x}\boldsymbol{N}^T\rvert_{0}^{2}\right)\boldsymbol{u}=f\int_{0}^{2}\boldsymbol{N}dx
\end{align}

oder

$$\boldsymbol{K} \boldsymbol{u}=\boldsymbol{b}$$

 mit

\begin{align}
\boldsymbol{K}&=E\int_{0}^{2}\frac{\partial \boldsymbol{N}}{\partial x}\frac{\partial \boldsymbol{N}^T}{\partial x}dx-\frac{\partial \boldsymbol{N}}{\partial x}\boldsymbol{N}^T\rvert_{0}^{2}\\
\boldsymbol{b}&=f\int_{0}^{2}\boldsymbol{N}dx
\end{align}

Um die Steifigkeitsmatrix konkret zu berechnen um das Gleichungssystem numerisch zu lösen müssen die oberen Integrale berechnet werden und die Randbedingungen berücksichtigt werden.

Ableiten, ausmultiplizieren und integrieren ergibt:

\begin{align}
E\int_{0}^{2}\frac{\partial \boldsymbol{N}}{\partial x}\frac{\partial \boldsymbol{N}^T}{\partial x}dx&=
\begin{pmatrix}
2 & -2 & 0 & 0 & 0 \\
-2 & 4 & -2 & 0 & 0 \\
0 & -2 & 4 & -2 & 0   \\
0 & 0 & -2 & 4 & -2 \\
0 & 0 & 0 & -2 & 2
\end{pmatrix}\\
\frac{\partial \boldsymbol{N}}{\partial x}\boldsymbol{N}^T\rvert_{0}^{2}&=
\begin{pmatrix}
1 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 0 \\
0 & 0 & 0 & 0 & 1
\end{pmatrix}\\
f\int_{0}^{2}\boldsymbol{N}dx&=f\begin{pmatrix}
\frac{1}{4}  \\
\frac{1}{2}  \\
\frac{1}{2} \\
\frac{1}{2} \\
\frac{1}{4}
\end{pmatrix}
\end{align}

Damit ergibt das Gleichungssystem $$K \boldsymbol{u}=\boldsymbol{b}$$

\begin{align}
E
\begin{pmatrix}
1 & -2 & 0 & 0 & 0 \\
-2 & 4 & -2 & 0 & 0 \\
0 & -2 & 4 & -2 & 0   \\
0 & 0 & -2 & 4 & -2 \\
0 & 0 & 0 & -2 & 1
\end{pmatrix}
\begin{pmatrix}
u_0  \\
u_{\frac{1}{2}}  \\
u_{\frac{1}{2}} \\
u_{\frac{1}{2}} \\
u_2
\end{pmatrix}=
f\begin{pmatrix}
\frac{1}{4}  \\
\frac{1}{2}  \\
\frac{1}{2} \\
\frac{1}{2} \\
\frac{1}{4}
\end{pmatrix}
\end{align}

oder, wenn die Elementlänge $\frac{1}{2}$ invertiert (d.h. $2$) ausgeklammert wird

\begin{align}
2E
\begin{pmatrix}
\frac{1}{2} & -1 & 0 & 0 & 0 \\
-1 & 2 & -1 & 0 & 0 \\
0 & -1 & 2 & -1 & 0   \\
0 & 0 & -1 & 2 & -1 \\
0 & 0 & 0 & -1 & \frac{1}{2}
\end{pmatrix}
\begin{pmatrix}
u_0  \\
u_{\frac{1}{2}}  \\
u_{\frac{1}{2}} \\
u_{\frac{1}{2}} \\
u_2
\end{pmatrix}=
f\begin{pmatrix}
\frac{1}{4}  \\
\frac{1}{2}  \\
\frac{1}{2} \\
\frac{1}{2} \\
\frac{1}{4}
\end{pmatrix}
\end{align}

Da in dem aktuellen Beispiel $u_0=u_{Dirichlet0}$, $u_2=u_{Dirichlet2}$ vorgegeben sind, muss das im oberen Gleichungssystem noch berücksichtigt werden. Dafür gibt es mehrere Möglichkeiten (siehe z.B. [Fish, Belytschko, 2007]). Ein weit verbreitetes Verfhren (robust) wird verständlich von [Rieg, Hackenschmidt,2009] vorgestellt. Dabei werden die Zeilen und Spalten von $u_0$, $u_2$ in jeweilige Einheitsvektoren verwandelt und von der rechten Seite die mit $u_{Dirichlet0}$ und $u_{Dirichlet2}$ skalierten Spalten der ursprünglichen Steifigkeitsmatrix (außer die Einträge die $u_0$ und $u_2$ treffen würden) abgezogen. Anschließend werden mit $u_{Dirichlet0}$ und $u_{Dirichlet2}$ skalierte Einheitsvektoren auf die rechte Seite addiert.

\begin{align}
\begin{pmatrix}
1 & 0 & 0 & 0 & 0 \\
0 & 4E & -2E & 0 & 0 \\
0 & -2E & 4E & -2E & 0   \\
0 & 0 & -2E & 4E & 0 \\
0 & 0 & 0 & 0 & 1
\end{pmatrix}
\begin{pmatrix}
u_0  \\
u_{\frac{1}{2}}  \\
u_{\frac{1}{2}} \\
u_{\frac{1}{2}} \\
u_2
\end{pmatrix}=
f\begin{pmatrix}
0 \\
\frac{1}{2}  \\
\frac{1}{2} \\
\frac{1}{2} \\
0
\end{pmatrix}
-
\begin{pmatrix}
0  \\
-2E  \cdot u_{Dirichlet0}  \\
0 \\
-2E \cdot u_{Dirichlet2}  \\
0
\end{pmatrix}
+
\begin{pmatrix}
u_{Dirichlet0}  \\
0  \\
0 \\
0 \\
u_{Dirichlet2}
\end{pmatrix}
\end{align}

### Lösen des Gleichungssystems

Das lineare Gleichungssystem wird dann numerisch gelöst (siehe z.B. [Rieg, Hackenschmidt,2009]). Im folgenden werden konkrete Werte genommen und die mit FEM berechnete Funktion $u_{FEM}$ wird mit der analytischen Lösung $$u_{ana}=\left(\frac{u_{Dirichlet2} - u_{Dirichlet1} }{2}+\frac{f(2-x)}{2 E}\right)x+u_{Dirichlet1}$$ verglichen.

Als Beispiel sei der Stab aus Baustahl und $2 m$ lang, mit einem Querschnitt der einen $10 cm$ Radius hat, an einem Ende an der Decke befestigt und am unteren Ende auch fest. Der Stab wird durch Schwerkraft nach unten gezogen.
Für das numerische Beispiel sei das E-Modul $E=2\cdot 10^{11} Pa$, $u_{Dirichlet1}=0 m$, $u_{Dirichlet2}=0 m$, $f=7850\cdot 9.86\frac{N}{m^3}$.

```python
emod=200 #stahl
#emod=0.1 # gummi
#emod=0.01 # styropor, Dichte theoretisch auch anpassen
A=np.pi*(10/100.)**2
Laenge=2.

E=emod*10**9
u0=0
u2=0
gewicht=(7850)*(Laenge*A)
kraft=gewicht*9.8066

f=kraft/(Laenge*A)

K=2*E*np.array([[0,-1,0,0,0],
            [-1,2,-1,0,0],
            [0,-1,2,-1,0],
            [0,0,-1,2,-1],
            [0,0,0,-1,0]])

K0=2*E*np.array([[(2*E)**(-1),0,0,0,0],
            [0,2,-1,0,0],
            [0,-1,2,-1,0],
            [0,0,-1,2,0],
            [0,0,0,0,(2*E)**(-1)]])

rhsKorrektur=-K.T[0]*u0-K.T[4]*u2+u0*np.array([1,0,0,0,0])+u2*np.array([0,0,0,0,1])
F=f/(2.)*np.array([0,1,1,1,0])+rhsKorrektur

U=np.linalg.solve(K0,F)

#print(rhsKorrektur)
```

Ansatz (Nach Ritz): $$u_{fem}=u_0 N_0(x)+u_{\frac{1}{2}} N_{\frac{1}{2}}(x)+u_1 N_1(x)+u_{\frac{3}{2}} N_{\frac{3}{2}}(x)+u_2 N_2(x)$$

```python
u=U[0]*Nh(x,0)+U[1]*Nh(x,1/2.)+U[2]*Nh(x,1)+U[3]*Nh(x,3/2.)+U[4]*Nh(x,2)

def uAna(y):

    wert=((u2-u0)/2.+(f*(2-y))/(2.*E))*y+u0

    return wert

ax=plt.axes()
ax.tick_params(labelsize=15)
ax.plot(x,uAna(x), label='Analytisch', color=farben[0]),
ax.plot(x,u, label='FEM', linestyle='--', color=farben[1])
ax.scatter(np.array([0,1/2.,1,3/2.,2]), uAna(np.array([0,1/2.,1,3/2.,2])), label='', color=farben[1])
ax.scatter(np.array([0,1/2.,1,3/2.,2]), np.array([0,0,0,0,0]), label='Knoten', color='black')
ax.tick_params(labelsize=15)
box = ax.get_position()
ax.set_xlabel(xlabel='x in m',fontsize=15)
ax.set_ylabel(ylabel='u in m',fontsize=15)
plt.legend(loc='best',fontsize=15)
print(max(uAna(x)))
```

## Berechnung der Kenngrößen: Postprocessing

Nach dem Solving-Schritt ist eine FEM-Approximation $u_{FEM}$ der Verschiebung $u$ bekannt. Meistens ist die eigentliche Kenngröße jedoch nicht die Verschiebung, sondern eine Größe, die sich aus der Verschiebung berechnet. In der linearen Elastizität z.B. die Dehnung oder die Spannung.

\begin{align}
\sigma & =E \epsilon &\text{ Hook-Gesetz: Spannung-Dehnung}\\
\epsilon & = \frac{\partial u}{\partial x} &\text{ Dehnung-Verschiebung}
\end{align}

Meistens wird aus der Dehnung und Spannung noch eine Vergleichsdehnung bzw. Vergleichsspannung berechnet (z.B. von Mises Vergleichsdehnung bzw. Vergleichsspannung). In 1D ist das nur der Betrag:

\begin{align}
\sigma_V & =\sqrt{\sigma^2} &\text{ Vergleichsspannung}\\
\epsilon_V & = \frac{1}{1+\nu}\sqrt{\epsilon^2} &\text{ Vergleichsdehnung}
\end{align}

Wobei $\nu=0.3$ bei Stahl; siehe [Issler et. al, 2003].

Zusätzlich kann auch die Dehnungsenergie als Integrale Postprocessinggröße ausgewertet werden.

$$\Pi=\frac{1}{2}A\int_{0}^{2}\epsilon\sigma dx$$

```python
def dehnung(y):

    wert=(u2-u0)/2.+(f*(2-2*y))/(2.*E)

    return wert

def spannung(y):
    wert=E*dehnung(y)
    return wert

ax=plt.axes()
ax.tick_params(labelsize=15)
ep=round(max(abs(dehnung(x))),10)
ax.plot(x,abs(dehnung(x)), label=r'$\epsilon_V (max. {0})$'.format(ep), color=farben[0])

ax.tick_params(labelsize=15)
ax.set_xlabel(xlabel='x in m',fontsize=15)
ax.set_ylabel(ylabel=r'$\epsilon_V$',fontsize=15)
plt.legend(loc='best',fontsize=15)
```

```python
ax=plt.axes()
ax.plot(x,abs(spannung(x)), label=r'$\sigma_V (max. {0})$'.format(max(abs(spannung(x)))), color=farben[1])

ax.tick_params(labelsize=15)
ax.set_xlabel(xlabel='x in m',fontsize=15)
ax.set_ylabel(ylabel=r'$\sigma_V$ in Pa',fontsize=15)

plt.legend(loc='best',fontsize=15)
```

Ist das Ergebnis noch physikalisch, ist die Spannung noch im linearen Bereich? $$\sigma_V=7.74\cdot10^{-5}MPa<200 MPa$$

Dehnungsenergie in Joule (J) $$\Pi=\frac{1}{2}A\int_{0}^{2}\epsilon\sigma dx$$

```python
0.5*A*integrate.simps(dehnung(x)*spannung(x), x)
```

    0.00031029505836870585

```python

```