R mlr3verse: Tasks, Learners, Evaluation, and Tuning

This English version was machine-translated from the Chinese original. Technical terms may require verification.

Preparatory knowledge and overview

mlr3

mlr3  Packages and broader mlr3verse A common, object-oriented and scalable framework is provided for R language return, classification and other machine learning tasks.

At the most basic level, the unified interface provides training, testing and evaluation of algorithms for many machines. It could go further by over-parameter optimization, computing conduits, model interpretation, etc.

mlr3 and scikit-learn  caret  tidymodels With similar overall goals

mlr3 It is designed to provide greater flexibility than any other ML framework. mlr3 A simple method of using advanced functionality is still available. Although... tidymodels In particular, it makes it easier to carry out simple ML tasks. mlr3 But it's better for senior ML.

mlr3 The data box should be secured.

Examples

Now, we're going to use two examples of what we're looking at later.

A simple decision tree.

## 训练
library(mlr3)
task = tsk("penguins")
split = partition(task)
learner = lrn("classif.rpart")

learner$train(task, row_ids = split$train)
learner$model
预测
prediction = learner$predict(task, row_ids = split$test)
prediction
评估
prediction$score(msr("classif.acc"))

More complex examples

library(mlr3verse)

tasks = tsks(c("breast_cancer", "sonar"))
glrn_rf_tuned = as_learner(ppl("robustify") %>>% auto_tuner(
    tnr("grid_search", resolution = 5),
    lrn("classif.ranger", num.trees = to_tune(200, 500)),
    rsmp("holdout")
))
glrn_rf_tuned$id = "RF"
glrn_stack = as_learner(ppl("robustify") %>>% ppl("stacking",
    lrns(c("classif.rpart", "classif.kknn")),
    lrn("classif.log_reg")
))
glrn_stack$id = "Stack"
learners = c(glrn_rf_tuned, glrn_stack)
bmr = benchmark(benchmark_grid(tasks, learners, rsmp("cv", folds = 3)))
bmr$aggregate(msr("classif.acc"))

2 required packages

We need to add software to R in order to make the ML more objects-oriented. PackageR6 Meanwhile, in order to process a lot of data more efficiently, we introduceddata.frameImprovementsdata.table

R6

R6 It's one of the most recent examples of object-oriented programming, and it's identical to the other object-oriented languages.

$new() It's an initialization method for creating R6-type objects.

foo = Foo$new(bar = 1)

We'll use it.FooClass created one.fooObject and set parameters Yes.mlr3 We've made a lot of progress in creating what we need.

And for objects that have a variable state, we also offer a way to modify it.$

foo$bar = 2

We visited the variable state of the object and modified it.

Besides, we certainly have a method (methods) that allows users to check the status of objects, retrieve information or perform changes to the internal state of objects. For example, for learning machines $train() The method is to modify the internal state of the learning device. R6The object, of course, has a way to give something else, and the way back is quite free, and we can, of course, enter multiple ways at once, and then call in, like,

Foo$bar()$hello_world()

Object FirstFoo It's done. bar And then we run the way they return. hello_world

Finally. R6The target.environments The direct value is quoted instead of creating a new object as follows:

foo2 = foo
foo2 = foo$clone(deep = TRUE)

The former does not create new objects, but quotes.foo Yeah.foo2The change is the same as the direct pair.fooModify Other OrganisercloneMethod See Article 2

data.table

It's right.RMediumdata.frame And that's why we're here.mlr3Use it.

He's on basic grammar rules.data.frameThere's no difference. It's an extension, but it's based entirely on the syntax rule of R.

data.table Also use semantics of quotes, which require use copy() Cloning data.table

Practical applications

mlr3 Including some important practical procedures that are essential to simplify codes in the mlr3 ecosystem

Sugar Functions Sugar

mlr3 Most objects can be created by a facilitative function called a auxiliary function or a sugar function. They provide a shortcut to common code customary terms and reduce the number of codes that users must prepare. For example: lrn("regr.rpart") , return the learner without creating a new R6 object

SugarFunctions is designed to cover most users, and complete only when building custom objects or extensions R6 Backend knowledge

mlr3 to be standardized according to agreement mlr_<type>_<key>

Dictionaries

mlr3 Use the dictionary to store values in the R6 category dictionaries usually accessed through the sugar function where objects are retrieved from the dictionaries For example: lrn("regr.rpart") It's...  mlr_learners$get("regr.rpart")Packaging, so from mlr_learners A simpler way to load the decision tree learner

The dictionary groupes a large number of clusters of objects so that they can be easily listed and retrieved, for example. as.data.table(mlr_learners) lrn() You can view available learners in loaded packages

mlr3viz

mlr3viz Including all the drawing functions of mlr3 and some of the ggpllot2 functions used. We use theme_minimal() To unify our aesthetics, but with all ggplot As with output, users can be completely customised It's...

mlr3viz Extension fortify and autoplot They're used for common use. mlr3 Output, including Prediction 、 Learner and BenchmarkResult Object

I understand. mlr3viz The best way to do this is through experiments; load the bag, see where mlr3 Run on objects autoplot What happens sometimes? The drawing type is recorded on the corresponding manual page and can be accessed through ?autoplot.<class> , for example, by running ?autoplot.TaskRegr to find different types of drawings for returning tasks

Design principles

Here's what the developers are saying.mlr3The principles of design, knowledge of them, help us develop better code habits and understand.mlr3Basic composition logic

• Object-oriented programming: we embrace R6 Clean, object-oriented design, object status changes and references
• Table data: Embrace data.table Its first-class computing performance and table data as a structure that can easily be further processed
• Uniform table input and output data formats: greatly simplified API
• Defense programming and type security: all user input is used checkmate  Inspection
• Reduction of dependency relationships: main mlr One of the maintenance burdens is to keep up with the changing learning tool interface and the many software packages on which it relies. What we need. mlr3 The significantly fewer software packages make installation and maintenance easier. We still provide the same function, but it's broken down into more packages with fewer dependencies.
• Calculates and indicates separation. mlr3 Most of the ecosystem packages focus on processing and converting data, applying ML algorithms and computations, and visualization of data and results mlr3viz Available

Data and Basic Modelling

In this chapter, we will present the basic building blocks of machine learning. mlr3 Object and Correspond R6 Class. These building blocks include data (and methods for creating training and testing sets), machine learning algorithms (and their training and prediction processes), machine learning algorithms through over-parametric configuration and assessment measures to assess the quality of projections

This chapter will be ours.mlr3verseThe rest of this book will be based on the basic elements seen in this chapter.

Our presentation will begin with the fundamentals of return and then gradually expand to the classification issue, which is the core of our monitoring studies.

Tasks

The task is to include the object of data (usually tables) and additional metadata (metala); the additional metadata contains the name of the target characteristic for monitoring machine learning The information is automatically extracted when required, so users do not have to specify the target for each training model

Build Task

mlr3 Yes. mlr_tasks  Dictionary Include some predefined machine learning tasks To get jobs from the dictionary, use tsk() Function and return value to new variable Run without any arguments tsk() Lists all the jobs in the dictionary, which also applies to other sugar functions

#查看字典中存储的预先定义的任务
mlr_tasks

#建立一个任务 我们是从字典中提取的
tsk_mtcars = tsk("mtcars")
#打印任务简报
tsk_mtcars
#查看字典中存储的预先定义的任务
tsk()

To create your own return task, you need to construct a new one TaskRegr Example. The simplest way to use a function as_task_regr()Will data.frame Type of object converted to return task by passing it to target Parameters to specify target characteristics.mtcarsDataset

#引入数据 提取其中的一部分作为我们需要的数据集 然后简单展示
data("mtcars", package = "datasets")
mtcars_subset = subset(mtcars, select = c("mpg", "cyl", "disp"))
str(mtcars_subset)

用数据集建立一个回归任务 其中预测目标是 'mpg' id 是任务的简述 后面绘图会用它称呼我们的任务
tsk_mtcars = as_task_regr(mtcars_subset, target = "mpg", id = "cars")

as_task_regr() Various types of data boxes are acceptable, includingdata.frame data.table tibble It's very compatible.

Special as_task_regr() In many cases, the UTF-8 code name is not accepted.

For the mission, we can use it.mlr3vizIt fits directly with the mlr3 object, and it's all automatic.

library(mlr3viz)
autoplot(tsk_mtcars, type = "pairs")

Retrieving data

We've learned how to create jobs to store data and metadata, and now we'll know how to retrieve stored data.

You can use a variety of fields to retrieve metadata about tasks like

• The names of the functional and target columns are stored separately in $feature_names and $target_names Inside.
• Available $nrow and $ncol Retrieval dimensions
• Task column has only one character value, the row is marked by the only natural number (known as line ID). Yes. $row_ids Field Access
• The data contained in the task can be passed $data() Visit, it returns one. data.table object. There are options. rows and cols Parameters to specify a subset of data to be retrieved

## 回报维数 也就是行数（观测数）列数（特征数）
c(tsk_mtcars$nrow, tsk_mtcars$ncol)

#回报功能列和目标列名称
c(Features = tsk_mtcars$feature_names,
  Target = tsk_mtcars$target_names)
#回报行ID head用于限定返回前几个元素（默认6个）
head(tsk_mtcars$row_ids)

When we filter some lines, the lines change, but the lines don't change.

task = as_task_regr(data.frame(x = runif(5), y = runif(5)),
  target = "y")
task$row_ids
## 返回 1 2 3 4 5
task$filter(c(4, 1, 3))
task$row_ids
## 返回 1 3 4

The data contained in the task can be passed $data() Visit, it returns one. data.table object. There are options. rows and cols Parameters to specify a subset of data to be retrieved Default lines are searched with line ID

## 返回全部数据子集
tsk_mtcars$data()

根据行ID以及列名称返回数据子集
tsk_mtcars$data(rows = c(1, 5, 10), cols = tsk_mtcars$feature_names)
这就起到了用行号检索数据子集的效果
tsk_mtcars$data(rows = task$row_ids[2])

Task changes

It works by modifying the mission after it was created.$data()The difference is... It changed the mission directly.

Use $select() Sub-assembly by feature (column), with the desired feature name transmitted as a character vector, using $filter()Observation sub-unitation by using line ID as a digital vector These methods directly modified the mission. Because of the R6 quote, if you want to keep the original job at the same time, you need to $clone() R6 Example of semantic references

## 创建任务
tsk_mtcars_small = tsk("mtcars")
## 只选取一个列（特征） 它不可以移除target
tsk_mtcars_small$select("cyl")
## 选取下面的行
tsk_mtcars_small$filter(2:3)

To add additional rows and columns to the task, you can use them separately. $rbind() and $cbind()

tsk_mtcars_small$cbind(
  data.frame(disp = c(150, 160))
)

tsk_mtcars_small$rbind(
  data.frame(mpg = 23, cyl = 5, disp = 170)
)

Learner Learners

Introduction to the learner

Category Learner The object provides a unified interface for many commonly used machine learning algorithms in R. mlr_learners Dictionary contains mlr3. We will discuss the available learning tools later; Now we're going to use the tree-returner as an example. LearnerInterface. As with tasks, you can use a single sugar function to access learners from the dictionary, as in this case lrn()

## 查看可用的学习器
mlr_learners
lrn()

#从字典中建立一个学习器 作学习器的简要报告（因为没存储）
lrn("regr.rpart")

The learning device is at the core of the algorithm and, unlike the mission, the learning device has nothing to do with the training data, and in most cases we only have to call on the already existing learning device, and we only need to customize the new learning device when we design the new algorithm from the bottom, so it's enough to use it from the dictionary.

All Learner Object contains the following metadata, which can be seen in the output of the summary report of the learner

• $feature_types : Characteristic types that can be handled by learners
• $packages : Use the software that the learners need to install Package
• $properties : learner attributes, for example, “missings” properties mean that models can process lost data, while “importance” means that it can calculate the relative importance of each characteristic
• $predict_types : Type of prediction that the model can make
• $param_set : Available Super Parameter Set

A full-fledged machine learning experiment, with learners going through two stages:

• Training: training Task Passed to learners $train() function, which trains and stores models, i.e. learning relationships between features and targets
• Forecast: New data, probably different divisions of the original data set, passed to trained learners $predict() Methodology for projecting target values

The method of training and forecasting is a learning device, not a mission.

Training

By Usage $train() The method is to transfer the task to the learner to train the model.$modelMedium

## load mtcars task
tsk_mtcars = tsk("mtcars")
## load a regression tree
lrn_rpart = lrn("regr.rpart")
## 训练学习器 任务和学习器均已经给出
lrn_rpart$train(tsk_mtcars)
## 查看训练出的模型 具体怎么理解可以参考模型的help 我们的帮助是可以对对象使用的
lrn_rpart$model

In many cases, we want not to use all the raw data to train models, and here we're going to introduce a simple division that corresponds to what we're talking about in machine learning theory, machine learning introductory and supervisory learning: setting aside.

partition() Function creates index set, which randomly divides tasks into two discrete sets: training set (67 per cent of total data by default) and testing set (remaining data)

## 划分数据 返回的是两个集合
splits = partition(tsk_mtcars)
splits

借助前面的划分来实现 row_ids让我们可以选择一部分行号进行训练
lrn_rpart$train(tsk_mtcars, row_ids = splits$train)

Projections

The prediction from a trained model is like data. Task Pass it to the training. LearnerYes. $predict() It's as simple as that.

Foundation projections

Because projections also require data, and according to machine learning habits, training and testing data will be combined.tasksThat's why we moved in.tasks Specify the forecast line number

prediction = lrn_rpart$predict(tsk_mtcars, row_ids = splits$test)

$predict() Method returns one from Prediction The object of succession will vary according to the object of our mission.row_ids Column corresponds to the line ID of the predicted observation. truthColumn contains real data taken from the task by the object (if available)response Column contains values projected by the model

Use as.data.table() The function can easily be Prediction Convert Object to data.table or data.frame Object

Special treatment of forecasted objects

Asmlr3verseIt also has its own special ways of doing things.

## 直接访问 如果访问的不是很多没必要调用数据框相关函数 语法也非常的自然
prediction$response[1:2]

#mlr3viz 为 所有继承Prediction类的对象提供了一个 autoplot() 方法
library(mlr3viz)
prediction = lrn_rpart$predict(tsk_mtcars, splits$test)
autoplot(prediction)

Forecast new data

The machine learns the habit of bringing all the data together into the tasks, but sometimes we do have a need to predict some new data, and at this point there's no need to re-establish the task. mlr3 takes this into account. $predict_newdata() That's it.

mtcars_new = data.table(cyl = c(5, 6), disp = c(100, 120),
  hp = c(100, 150), drat = c(4, 3.9), wt = c(3.8, 4.1),
  qsec = c(18, 19.5), vs = c(1, 0), am = c(1, 1),
  gear = c(6, 4), carb = c(3, 5))
prediction = lrn_rpart$predict_newdata(mtcars_new)
prediction

At this point,truthIt's all empty.row_idsContinue New Data Box Note that when predicting new data, make sure the column name andtasksIt's the same thing.

Change projection type

While individual values are the most common projection type in regression, they are not the only projection type. Some regression models can also give the prediction standard error at the same time as we do in the traditional linear regression.

In order to predict this, before training, we have to... LearnerRegr Yes. $predict_type Field changed from 'response' (default value) to "se"

We use it up there. "rpart" The learner does not support predicting standard errors, so in the example below we will use linear regression modelslrn("regr.lm")

## 导入需要的包并且建立新的学习器
library(mlr3learners)
lrn_lm = lrn("regr.lm", predict_type = "se")
## 训练与预测
lrn_lm$train(tsk_mtcars, splits$train)
lrn_lm$predict(tsk_mtcars, splits$test)

Hyperparameters

Learners A machine learning algorithm and its super-parameters are encapsulated and can be set by the user.

Superparameters may affect the mode of training or prediction of models and may require expertise to determine how to set them

We're going to be able to optimise the superparameters, and we're going to be able to talk about how to optimise them automatically, but in this chapter, we're going to focus on how to set them manually, and that's the basis for setting auto-optimizations behind us, and in actual machine learning, super-parameters are often manually set.

Parameters and Arguments Set

We've explained before that the learning machine should be used.$param_setVisits A description of the learner's hyperparameter As follows:

lrn_rpart$param_set

The output is one. ParamSet object, by paradox Package provided. These objects provide information about the super-parameters, including their names. id Type of data () class ), technical validity range of ultra-parameter values () lower 、 upper ) possible levels if the type of data is classified ( nlevels Default value from bottom package ( default ) and last set value ( value ）

class Inheritance is paradox the categories in which the parameters are determined and the possible values that it is possible to use.

Hyperparameter Class	Hyperparameter Type
ParamDbl	Real value (value)
ParamInt	Integer
ParamFct	Factor
ParamLgl	Boolean Type (T or F)
ParamUty	No type

In most cases, the super-parameters are properly initialized into the defaults that they should have, but in some cases they can be misinformed, i.e., when creating the learning device, the super-parameters are not in the situation that they should be, and at this point, they usually give a hint.bugIt's a development that avoids it as much as possible, but not entirely.

Fetch and set hyperparameter values

Now that we know how the super-parameter sets are stored, we can think about getting and setting them. Back to our decision-making tree, assuming we're interested in growing a deep one, 1 and the tree, also known as the decision stake, in which the data are divided into only two terminal nodes

There are several different ways to change this parameter. The simplest way is to transmit the name and new value of the super-parameter during the construction of the learning device. Here. lrn() Like

## 建立学习器的时候设置超参数
lrn_rpart = lrn("regr.rpart", maxdepth = 1)

返回那些非默认超参数的列表 本质上就是在超参数集上多了一层访问
lrn_rpart$param_set$values

Manual setting of super-parameters directly when constructing a learning device is the most practical method

Just now, we've introduced another way to access the super-parameter assembly. $valueStart to modify the hyperparameter

lrn_rpart$param_set$values$maxdepth = 2
lrn_rpart$param_set$values

There's only one thing we can do about it at a time.

lrn_rpart$param_set$set_values(xval = 2, cp = 0.5)
lrn_rpart$param_set$values

lrn_rpart$param_set$values Return one list But don't use new ones. listAnd the way to change the parametrics, he'll cause some of them to be erased, because we're always building them. list, and only include the hyperparameters you want to modify.

All modifications to the hyperparameters are subject to relevant cross-border checks to ensure the type of compliance.

Superparameter Dependence

More complex hyper-parameter space may contain dependency relationships, which occurs when the setting of one super-parameter is conditional on the value of another; An example of this is support for vector machines. lrn("regr.svm") I don't know. Fields $deps Return one data.table It's listed. Learner Overparameter dependency in

lrn("regr.svm")$param_set$deps

of which id The column indicates who depends on other super-parameters on The column tells us who's dependent. cond Column tells us what the deal is.

#访问cond列内容
lrn("regr.svm")$param_set$deps[[1, "cond"]]
lrn("regr.svm")$param_set$deps[[3, "cond"]]

CondAnyOf MeaningonIt's one of the numbers in the pool. CondEqual Meaningon Equals to a value It means our condition.

If the conditions for the relevant superparameters are not met, then Learner Error

Benchmark learners

Before we continue with the learning machine assessment, we will highlight an important learning tool. These are very simple or “weak” learners, referred to as benchmarks;

For the return, we have achieved the benchmark. lrn("regr.featureless") , it always predicts that the new value is the average (or median) of the target in the training data if robust Set Hyperparameter to TRUE ）

If a model works worse than a benchmark learner, then it's a bad model.

df = as_task_regr(data.frame(x = runif(1000), y = rnorm(1000, 2, 1)),
  target = "y")
lrn("regr.featureless")$train(df, 1:995)$predict(df, 996:1000)

Evaluation

Perhaps the most important step in the application machine learning workflow is to assess model performance. Without that, we will not know whether our training models can make very accurate predictions, whether they are worse than random speculation, or whether they are in between; and here is an example of our code, which can also be seen as a review of some of the previous code contents.

lrn_rpart = lrn("regr.rpart")
tsk_mtcars = tsk("mtcars")
splits = partition(tsk_mtcars)
lrn_rpart$train(tsk_mtcars, splits$train)
prediction = lrn_rpart$predict(tsk_mtcars, splits$test)

Evaluator

The quality of projections is assessed using measures that compare them with real data on monitoring learning assignments and Tasks and LearnersLike, mlr3 . The available measures are stored in the name mlr_measures , and can be used msr() Visits

## 访问评价器的字典
mlr_measures
msr()

Because the idea of evaluating a model is often fixed, our evaluators rarely need to create new ones.

mlr3 All measures achieved are defined mainly by three components

• Function to measure
• Whether lower or higher values are considered “good”
• Scope of possible values for measurement Besides this, an evaluator has some metadata like
• Measure any special properties
• Type of projection that measures can assess
• Measurement with any "control parameters"

If you look directly at an evaluator, you can get all the data you need.

measure = msr("regr.mae")
measure

Projection scores

To calculate model performance, we just have to call. Prediction object $score() The method and the measure we want to calculate as a single parameter conveys the facts. Let's go. Prediction It stores all the data we need to evaluate a model, including real values and projections. Value

prediction$score(measure)

All job types have default evaluators, for example, re-entry models using average MSE as default evaluators if we don't pass in$score() Parameters, then use the default evaluator

The evaluator evaluates only the test data, and we focus on generalization performance rather than the intended effects, as do other tests later.

By passing multiple ratingrs to $score() , multiple evaluations can be counted simultaneously

## 同时把多个评价器给了这个变量 然后一起传入
measures = msrs(c("regr.mse", "regr.mae"))
prediction$score(measures)

Other evaluations

mlr3 Measurements of the quality of modelling projections are also provided, not quantitative, but rather “meta-information” on models. These include:

• msr("time_train") - Time for training models.
• msr("time_predict") - Time taken to predict the model
• msr("time_both") - The total time spent on training models and forecasting.
• msr("selected_features") - Number of features selected for the model only if the model has a "selected features" attribute

One simple example:

measures = msrs(c("time_train", "time_predict", "time_both"))
prediction$score(measures, learner = lrn_rpart)

We put the learning device in together.$score() It's a special attribute of the evaluator.

For the number of model selection features there are

## 查看评估器的元数据
msr_sf = msr("selected_features")
msr_sf

In particular, there are two of these in the metadata of this evaluator.

• Parameters: normalize=FALSE
• Documents: references task, references learner, references model That is, this evaluator has parameters that can be set up for direct reference to evaluation parameters. Hyperparameters All methods are the same.
• normalize Superparameter specifies whether the selected number of returns should be standardized according to the total number of features
• Properties Tells us that this assessor needs a mission, a learning machine, to travel together.

Displays the code for using this assessor as

## 设置了评估器参数
msr_sf$param_set$values$normalize = TRUE
## 调用了评估器 它需要任务和学习器
prediction$score(msr_sf, task = tsk_mtcars, learner = lrn_rpart)

Return Experiment

Research. mlr3 We'll suspend all the above in a short experiment to assess the quality of our projections.

Independent review of the code below, understanding usage, and learning to expand.

library(mlr3)
set.seed(349)
## 任务构建和划分 并没有自建任务
tsk_mtcars = tsk("mtcars")
splits = partition(tsk_mtcars)
## 加载学习器 这是基准学习器
lrn_featureless = lrn("regr.featureless")
## 加载学习器 这是决策树
lrn_rpart = lrn("regr.rpart", cp = 0.2, maxdepth = 5)
## 加载评估器 两种评估方法
measures = msrs(c("regr.mse", "regr.mae"))
## 对两个学习器训练 使用训练数据
lrn_featureless$train(tsk_mtcars, splits$train)
lrn_rpart$train(tsk_mtcars, splits$train)
## 对两个学习器预测 使用预测数据 同时对预测的结果进行评价
lrn_featureless$predict(tsk_mtcars, splits$test)$score(measures)
lrn_rpart$predict(tsk_mtcars, splits$test)$score(measures)

You'll notice that our learning tools and measurements are available. "regr." Prefix, which is a convenient way to remind us that we are dealing with a return mission and that it is necessary to use learning devices and metrics built for a return.

In the next section, we'll use mlr3 It's just a slight change to consider the classification task.

Classes Classif

The classification issue is a model that predicts a discrete, disaggregated target rather than a continuous, numerical volume. For example, predicting the species from the physical characteristics of penguins will be a classification problem because there is a defined group of species

mlr3 Ensure that the interface of all tasks is as similar (if not identical) as possible, so focus only on differences that make classification a unique machine learning issue

We'll start by implementing one of theReturn ExperimentA very similar experiment to demonstrate the similarities between regression and classification.

Then we will discuss the differences between tasks, learners and projections, and then the threshold, which is a method specific to classification.

Classification test

Here's the code.

library(mlr3)
set.seed(349)
## 构建任务 这里我们的任务还是直接用已有的 建立一个新的预测任务的代码与回归存在的差异后面来解释 划分了集合
tsk_penguins = tsk("penguins")
splits = partition(tsk_penguins)
## 加载一个分类学习器 它是基准学习器
lrn_featureless = lrn("classif.featureless")
## 加载一个分类学习器 还是决策树 但是是分类专用
lrn_rpart = lrn("classif.rpart", cp = 0.2, maxdepth = 5)
## 加载分类评价器
measure = msr("classif.acc")
## 训练
lrn_featureless$train(tsk_penguins, splits$train)
lrn_rpart$train(tsk_penguins, splits$train)
## 预测并评价
lrn_featureless$predict(tsk_penguins, splits$test)$score(measure)
lrn_rpart$predict(tsk_penguins, splits$test)$score(measure)

Classification Tasks

The classification task is from TaskClassif The object of succession, except for the target variable, which is a factor type, is very similar to a return mission.

Filtered mlr_tasks Dictionary View mlr3 sort tasks predefined in

as.data.table(mlr_tasks)[task_type == "classif"]

Available as_task_classif Create your own category task

as_task_classif(palmerpenguins::penguins, target = "species")

mlr3 In support of two types of classification tasks: dual classification, where the result could be one of two categories, and multiple classifications, where the result could be one of three or more categories

We can see in the mission's brief report all the relevant attributes and use the most natural habits to access them.

An important difference between these tasks is that the binary classification task is named $positive , which defines the " positive " category. In the binary classification, since there are only two possible categories, as is customary, one is referred to as the “positive” category and the other as the “negative” category. Category

## 加载数据
data(Sonar, package = "mlbench")
## 建立tasks
tsk_classif = as_task_classif(Sonar, target = "Class", positive = "R")
## 查看正类
tsk_classif$positive
## 修改正类
tsk_classif$positive = "M"

Although the choice of categories is arbitrary, they are essential to ensure that the results of models and performance indicators are interpreted as intended - as demonstrated when we discuss thresholds and ROC indicators

Finally, it's available. autoplot.TaskClassif Draw

library(ggplot2)
autoplot(tsk("penguins"), type = "duo") +
  theme(strip.text.y = element_text(angle = -45, size = 8))

Classification Learner

From LearnerClassif The classification learners have almost identical interfaces with regression learners;

But the possible predictions in the classification are not the only ones."response" That's the type of predictive observation."prob" Projections of the probability vectors of observation for each category (or a lateral probability) response The default is the highest predictive probability. Category

Classification Evaluator

Classification measures (categorys) MeasureClassif the same interface as the regression measure.

But we found that the task type of the classification is divided into two categories and multiple categories, and the predictive type of the classification is divided into probability and class predictions, and they all relate to the gap in the evaluator, and we need to make a choice based on looking at all the evaluators in the first place.

as.data.table(msr())[
    task_type == "classif" & predict_type == "prob" ]

The first part limits the type of task for the evaluator:classifStill?regr The second part limits the type of projection.probStill?response

The example of the code is that the whole interface is the same.

measures = msrs(c("classif.mbrier", "classif.logloss", "classif.acc"))
prediction$score(measures)

Projections in the classification

PredictionClassif There are two important differences between objects and their regression simulations.

• Add field first $confusion
• Next to add method $set_threshold()

They wouldn't have been.

prediction

The code directly accesses them. They're all special amounts of classification problem predictions.

Confusion matrix

Confusion matrix is a popular way of showing in more detail the quality of classification (response) forecasts by seeing whether the model is good at categorizing observations in a given category (in error)

For binary and multi-classification, confusion matrix stored in PredictionClassif object $confusion Field Access Code

prediction$confusion

On the theoretical interpretation, we can look at the introduction to machine learning and supervision of learning: calibration rate, full rate and F1.

Specifically, we can visualize the graphics from the confusing matrix.

autoplot(prediction)

Threshold threshold

Another issue raised by classification as compared to returns is the issue of thresholds;

Default response The projection type is the highest predictive probability category, and if the maximum probability is not the only, that is, multiple categories are projected to have the highest probability and are then randomly selected from these categories;

In the binary classification, this means that if the projected category is more than 50%, the positive category will be selected, otherwise the negative category will be selected

This value of 50 per cent is referred to as the threshold, which may be useful if there is a class imbalance (when a class is over- or under-centralized), or if there are different costs associated with the class, or if only if there is a preference for a class “over” predicting.

It's easy to set a threshold in a category II problem.

prediction$set_threshold(0.7)

At this point, there's just...prob>0.7That's when it's supposed to be positive.

In multiple categories, the working principle for threshold processing is to start with each n Class allocates a threshold by dividing the predicted probability of each category by these thresholds to return n And at this point, the threshold still reflects the preferences that we choose, and the bigger the threshold, the more we deviate from it.

Yes. mlr3 ♪ Medium, it's through ♪ $set_threshold() Pass naming list to achieve

library(ggplot2)
library(patchwork)

tsk_zoo = tsk("zoo")
splits = partition(tsk_zoo)
lrn_rpart = lrn("classif.rpart", predict_type = "prob")
lrn_rpart$train(tsk_zoo, splits$train)
prediction = lrn_rpart$predict(tsk_zoo, splits$test)
before = autoplot(prediction) + ggtitle("Default thresholds")
new_thresh = proportions(table(tsk_zoo$truth(splits$train)))
new_thresh
prediction$set_threshold(new_thresh)
after = autoplot(prediction) + ggtitle("Inverse weighting thresholds")
before + after + plot_layout(guides = "collect")

It's usually called reverse weighting.

Taskbar Roles

Now that we have described regression and classification, we will briefly return to the task; the role is the most important metadata that learners and other objects can use to interact with the task; there are seven roles:

• "feature" : function for prediction
• "target" Target variable to predict
• "name" : row name/observation label, e.g., for mtcars This is... "model" Columns
• "order" : For right $data() variables for sorting returned data; using order()
• "group" : variable used to keep observations together during redistribution
• "stratum" : Layered variables during re-sampling
• "weight" : Observation weight. Only one numerical column can have this role

feature and target We've talked about it before.Tasks stratum and group We'll introduce it later in the section. We're not going to give you details. name , it's mainly used for mapping, and it's almost always the bottom data. rownames()

Use $set_col_roles() Update Column Roles When the column is updated, it will not be used as another column, which means that each column has only one column.

order

Yeah."order" Role Data sorted according to this column ♪ When we run ♪ $data() It is no longer used as a feature, but rather to rank observations according to their values. This metadata will not be passed to learners

df = data.frame(mtcars[1:2, ], idx = 2:1)
tsk_mtcars_order = as_task_regr(df, target = "mpg")
## 初始排序
tsk_mtcars_order$data(ordered = TRUE)

根据列 idx 进行排序
tsk_mtcars_order$set_col_roles("idx", roles = "order")
tsk_mtcars_order$data(ordered = TRUE)

weight

weights Column roles are used to weight data points differently; in classification tasks with serious category imbalances, a more weighted minority category may increase the predictive performance of the model for that category

Example code:

cancer_unweighted = tsk("breast_cancer")
summary(cancer_unweighted$data()$class)

add column where weight is 2 if class "malignant", and 1 otherwise
df = cancer_unweighted$data()
df$weights = ifelse(df$class == "malignant", 2, 1)
create new task and role
cancer_weighted = as_task_classif(df, target = "class")
cancer_weighted$set_col_roles("weights", roles = "weight")
compare weighted and unweighted predictions
split = partition(cancer_unweighted)
lrn_rf = lrn("classif.ranger")
lrn_rf$train(cancer_unweighted, split$train)$
  predict(cancer_unweighted, split$test)$score()
lrn_rf$train(cancer_weighted, split$train)$
  predict(cancer_weighted, split$test)$score()

In this example, the weighting increases the overall performance of the model; not all models can handle the weights of the task, so please check the attributes of the learners to ensure that this role is used as expected

Supported learning algorithms

mlr3Support many learning algorithms; these are mainly mlr3 、 mlr3learners and mlr3extralearners Packages are available; of course, the newer packages usually contain some of the newer algorithms.

mlr3

mlr3 The list of learning devices included is deliberately small, thereby reducing reliance on other packages;

• Featureless learners ("regr.featureless"/"classif.featureless") Use as a benchmark learner
• Debug learners ("regr.debug"/"classif.debug"For code debugging
• Classification and regression trees (also known as CART: "regr.rpart"/"classif.rpart"It's also known as CRAT.

mlr3learners

mlr3learners The package contains a series of algorithms chosen by the mlr team.

• Linear "regr.lm" ) and logic ( ) "classif.log_reg" ♪ Back
• The punishment is a broad linear model in which the punishment is either used as a super-parameter (para. "regr.glmnet" / "classif.glmnet" ), or optimise automatically ( "regr.cv_glmnet" / "classif.cv_glmnet" ）
• Weighted$k$Near Neighbors "regr.kknn" / "classif.kknn" ）
• Kriging / Gaussian process regression（ "regr.km" ）
• Linear "classif.lda" ) and secondary ( ) "classif.qda" Other Organiser
• PARK Soo Bayes "classif.naive_bayes" ）
• Support vectors "regr.svm" / "classif.svm" ）
• Gradient Enhancement "regr.xgboost" / "classif.xgboost" ）
• Return and classification of random forests "regr.ranger" / "classif.ranger" ）

View learning algorithms

Normally, we would take all available learners and convert them to a data frame and briefly review their format.

learners_dt = as.data.table(mlr_learners)
learners_dt

Generated data.table It contains a large amount of metadata that are very useful for identifying learners with specific attributes

Lists all learners who support classification questions:

learners_dt[task_type == "classif"]

Several conditions are filtered, listing all retrogressors that can predict standard errors:

learners_dt[task_type == "regr" &
  sapply(predict_types, function(x) "se" %in% x)]

Evaluation and Benchmarking

Monitoring machine learning models can only be deployed in practice if they have good generalization capabilities, so accurate estimates of generalization performance are essential for many aspects of machine learning applications and research, which will be an important basis for our selection in multiple models and for over-parametric adjustments;

We know that the use of the same data to train and test models is a bad strategy and that it is simply impossible to solve the problem of alignment. In the previous section, we've introduced partition() It divides the data sets into training data (for training models) and test data (for testing models and estimating generalization performance)Partition() Description This is called holdout strategy, and it will be the beginning of this chapter. We will then consider a more advanced strategy for assessing generalization performance.

A common misunderstanding is that holdout and other, more advanced, resampling models can prevent over-formulation, and in fact these methods simply make it visible because we can evaluate training/test performance separately. He allowed us to make almost impartial estimates of general errors.

Holdout Policy

An important objective of ML is to learn a model that can then be used to predict new data. In order to make the model as accurate as possible, we would ideally use as much data as possible to train it. However, the data are limited and, as we discussed, we cannot train and test models on the same data.

In practice, one usually creates an intermediate model that is trained on a subset of available data and then tested on the remaining data. The performance of the intermediate model obtained by comparing model predictions with real data is an estimate of the generalization of the final model. And finally, we get intermediate model information and superparameter information to train models on all data, which is the result of our final output.

The holdout strategy is a simple way of creating a division between training and testing data sets. Ideally, training data sets should be as large as possible so that intermediate models represent the final model as possible, while test data sets should be as large as possible to allow accurate estimates of general error.

Based on experience, two thirds of the data are usually used for training and one third for testing, as this provides a reasonable balance between deviations and differences in generalized performance estimates

We've already introduced the code he needs to use. Down

tsk_penguins = tsk("penguins")
splits = partition(tsk_penguins)
lrn_rpart = lrn("classif.rpart")
## 在训练集上训练 测试集上预测
lrn_rpart$train(tsk_penguins, splits$train)
prediction = lrn_rpart$predict(tsk_penguins, splits$test)
## 对测试集的预测结果进行评分
prediction$score(msr("classif.acc"))

When dividing data, observation values must be confused to remove any information coded in the data sorting. Because... tasks Establishing the data used is likely to be based on some regular data, which is also a common practice in data collection. But it will affect the division of our tests and training sets.

partition() And all the restampling strategies discussed below will automatically and randomly divide the data to prevent any bias, to ensure that our training in models, projections, and generalized error estimates are valid.

Many performance indicators are based on “decompositionable” losses, which means that they calculate differences between projections and real values first at the observation level, and then summarize the loss values in the test set into single value fractions.

In fact, we have a more sophisticated assessment strategy, which is a non-dissolved performance measure, and we'll talk about it later.

Resampling Policy

The Resampling strategy repeats all available data into multiple training and testing sets. One of the repetitions corresponds to the other. mlr3 Resampling itseration or re-sampling Subaru.

The panoramic performance is ultimately estimated by aggregating the performance scores of multiple retraces.

By repeating the data split process, the data points can be reused for training and testing, making it more efficient to use all available data for performance estimates. In addition, a large number of reclassifications can reduce the difference in fractions, resulting in more reliable performance estimates. This means that performance estimates are unlikely to be affected by “unfortunate” fragmentation.

We can generally think that Resampling's strategy provides a better general error estimate than the Holdout strategy described earlier. But at the same time, he'll bring more performance costs because we trained and tested multiple models.

Resampling Policy Theory

CV

A very common strategy is k-fold cross-validation. It randomly divides the data into$k$A non-overlapping subset, called discounts; $k$The models are always there.$k-1$Collapse training, with the remaining fold being used to test data; repeat the process until each fold is accurately implemented as a test set. Finally, a summary of performance estimates for each fold is usually sought in mean. CV ensures that each observation is used only once in the test concentration, thus effectively using available data for performance estimates

$k$Common values are 5 and 10, which means that each set will consist of 4/5 or 9/10 of raw data.

CV has several variants, including repeat k-clip cross-certification (where k-cV is repeated) and a cross-certification (LOO-CV), with a discount equal to the number of observations, resulting in a test set consisting of only one observation per discount

Theory can be used to refer to the introduction to machine learning and supervisory learning: cross-certification.

Subsampling and Bootstrapping

Subsampling randomly selects the data of the given ratio (commonly 4/5 and 9/10) for use in the training data set, where each observation in the data set is extracted from the original data collection and does not need to be replaced. The model is trained on this data and tested on the remaining data, and this process is repeated$k$Minor

Bootstrapping, the strategy is self-help.

Policy Selection

Resampling strategy choices usually depend on the specific tasks at hand and the objectives of the performance assessment, but there are some empirical rules.

If available data are small ($N)<$500, with a large number of duplicate cross-checks that can be used to keep performance estimates low (10 times and 10 times as a good starting point)

The LOO-CV were also recommended for these small sample quantities, but the estimated cost is very high (except in exceptional cases where a shortcut exists) and is in violation of a fairly high range of instincts. At the same time, he has problems with the unbalanced binary classification task.

For $500.<N<$5,000 range, usually 5-10 CV

Bootstrapping has become less common because repeated sampling can cause problems in machine learning algorithms.

Later, we'll give you details of how these Resampling strategies are implemented in R.

Create Resampling Policy

All achieved Resampling policies are stored in mlr_resamplings Dictionary

as.data.table(mlr_resamplings)

params Shows the parameters of each Resampling policy that can be constructed from behindResampling Modify Parameters When Object itersColumn shows the default number of Resampling iterative times we do not normally need to adjust

Resampling Object can pass the policy "key" to the sugar function rsmp() To construct, for example.

rsmp("holdout", ratio = 0.8)

It's built. holdout We modified the default ratio from two thirds of the training set to four fifths of the training. Set

From Resampling The calibration and evaluation of the parameters of the object of succession. Let's go. Resampling The grammatical rule that you're constructing is simply to replace the SugarFunction with rsmp()

## three-fold CV
cv3 = rsmp("cv", folds = 3)
## Subsampling with 3 repeats and 9/10 ratio
ss390 = rsmp("subsampling", repeats = 3, ratio = 0.9)
## 2-repeats 5-fold CV
rcv25 = rsmp("repeated_cv", repeats = 2, folds = 5)

We can do this manually, but this operation is cumbersome and often ineffective.

Resampling objects for practical learning

partition() The function accepts the task automatically dividing the training and testing set for us and returns the line numbers; of course, Resampling objects should have the same function;

resample() Function accepts given Task 、 Learner and Resampling object to run a given Resampling policy. resample() Repeat the assembly model on the training set, predict it on the corresponding test set and store it in ResampleResult object, the object contains all the information needed to estimate the generalized performance

rr = resample(tsk_penguins, lrn_rpart, cv3)
rr

We changed the process of learning the front learners while we were at Resampling, while changing the training and prediction steps.

Of course, we still need an evaluator to evaluate it.

## 返回在每次迭代中的性能
acc = rr$score(msr("classif.ce"))
acc[, .(iteration, classif.ce)]

聚合多次迭代 给出更加常用的结果
rr$aggregate(msr("classif.ce"))

By default, most measures will use macro averages (the average of scores directly to each test set) to aggregate fractions, but we can specify micromeans (he considers different sizes of each test set) to aggregate fractions.

## 这就是采用了微平均值 需要直接修改我们的评价器
rr$aggregate(msr("classif.ce", average = "micro"))

Through Query Measure object $average Fields can find the default type of aggregation method

Visual reordering results, available autoplot.ResampleResult() function. Histograms can be used to measure intuitively the variance of the intermediate performance of the rearranged trajectories, while box charts are usually used to compare multiple studies in parallel Device

## 训练模型 返回ResampleResult对象
rr = resample(tsk_penguins, lrn_rpart, rsmp("cv", folds = 10))
## 使用 autoplot 函数 和前面一样 他为各种 mlr3 对象绘图 此时可以选择绘图的类型
autoplot(rr, measure = msr("classif.acc"), type = "boxplot")
autoplot(rr, measure = msr("classif.acc"), type = "histogram")

ResampleResult Object

We changed the learning function of the learning device to use the Resampling strategy, so the result of the study became a ResampleResult object.

We discussed how to use ResampleResult objects to calculate general errors, but ResampleResult objects cannot be used for this purpose only.

We can use it. $predictions()The method of obtaining a corresponding forecast for each resource Prediction List of Objects The target.Prediction

## 返回结果是一个列表 里面含有迭代次数个元素
rrp = rr$predictions()

By default, intermediate models produced in each Resampling policy trajectories are discarded after predicting steps to reduce ResampleResultMemory consumption of objects (the greatest effect of which is performance measurement)

But we can go through the settings. store_models = TRUE Configure resample() function to maintain the proposed intermediate model. And then, through $learnersi$model Visit every model trained in a particular recovery trajectories, where i It means no. iOther Organiser

rr = resample(tsk_penguins, lrn_rpart, cv3, store_models = TRUE)
## 得到各个学习器 后面的$model查看学习器的模型
rr$learners[[1]]$model

We'll be able to access all the information about the model, if we need it.

Custom Resampling

Self-defining Resampling is something that might be needed. mlr3 It provides the appropriate way.

Custom holdout For information

rsmp_custom = rsmp("custom")

resampling strategy with two iterations
train_sets = c(1:5, 153:158, 277:280)
rsmp_custom$instantiate(tsk_penguins,
  train = list(train_sets, train_sets + 5),
  test = list(train_sets + 15, train_sets + 25)
)
resample(tsk_penguins, lrn_rpart, rsmp_custom)$prediction()

Customcv For information

tsk_small = tsk("penguins")$filter(c(1, 100, 200, 300))
rsmp_customcv = rsmp("custom_cv")
folds = as.factor(c(1, 2, 1, 2))
rsmp_customcv$instantiate(tsk_small, f = folds)
resample(tsk_small, lrn_rpart, rsmp_customcv)$predictions()

Layers and Layers

Use taskbar roles to group or layer observations according to specific columns in the data

Grouped Resampling

In longitudinal studies, measurements are made from the same body at multiple points of time. If we do not group these data, we may overestimate the model's ability to extend to unknown individuals, since observations of the same individuals may be present at the same time in the training set and in the concentration of testing.

"group" Column roles allow us to specify columns in the data that define the group structure for observation. At this point in the construction of Resampling, the folding of each observation becomes a folding of groups.

rsmp_loo = rsmp("loo")
tsk_grp = tsk("penguins")
tsk_grp$set_col_roles("year", "group") rsmp_loo$instantiate(tsk_grp)

Stratified Sampling

A layer sample ensures that one or more discrete features of the training set and test concentration will have a distribution similar to that of the original mission covering all observations; this ensures that multiple and iterative estimates are accurate in cross-checking;

Different from grouping, can be used "stratum" Column roles are layered according to multiple discrete characteristics. In this case, the layer will be formed by each combination of the layers, as follows:

tsk_str = tsk("penguins")
## 设定 species 同时作为分层用的`"stratum"` 列 和"target"列
tsk_str$set_col_roles("species", c("target", "stratum"))
rsmp_cv10$instantiate(tsk_str)

Benchmark Test Benchmarking

Benchmark tests in machine learning are learning devices that compare different tasks.

When comparing multiple learning devices on a single mission or on more than one similar task, the main purpose is usually to rank learning devices according to predefined performance measures and to determine the best learning instrument for the given task.

When multiple learners are compared on multiple assignments, the main purpose is often less simple than before. For example, an in-depth understanding of the performance of different learners in different data situations, or the existence of certain data attributes that significantly affect the performance of some learners (or some over-parameters of learners).

Since baseline tests usually consist of many assessments that can operate independently of each other, mlr3 The possibility of automatic parallelization is thus provided. In this section, we present the most extensive baseline tests used and discuss more complex benchmarking issues later.

Benchmark

mlr3 The baseline experiment is used. benchmark() It's done, it's only run to each task and learner separately resample() , then collect the results. The re-sampling strategy provided will be automatically exemplified for each assignment to ensure that all learners are compared with the same training and testing data

It's obvious that for benchmarking, we need to introduce multiple tasks, multiple learners, possibly multiple Resample methods, so the code has

## 建立两个任务
tasks = tsks(c("german_credit", "sonar"))
## 建立三个学习器
learners = lrns(c("classif.rpart", "classif.ranger",
  "classif.featureless"), predict_type = "prob")
## 建立一种 Resample 方法
rsmp_cv5 = rsmp("cv", folds = 5)

构造benchmark()方案并审阅
design = benchmark_grid(tasks, learners, rsmp_cv5)
head(design)

It's essentially just a design. data.table , if you want to delete a particular combination, you can modify it, even without it benchmark_grid() Create from scratch in a function

Then you can pass the constructed baseline design to benchmark() Run the experiment. The result is one. BenchmarkResult Object:

bmr = benchmark(design)
bmr

Because benchmark() It's just... resample() The extension we can use again $score() or $aggregate() This is how we look at the results.

bmr$score()[c(1, 7, 13), .(iteration, task_id, learner_id, classif.ce)]

bmr$aggregate()[, .(task_id, learner_id, classif.ce)]

We don't have a rigorous statistical hypothesis test here, so we have to be careful to draw conclusions about which model is better.

BenchmarkResult Object

Object BenchmarkResult Multiple ResampleResult Collection of Objects

We can extract the BenchmarkResult objectResampleResult Object has

rr1 = bmr$resample_result(1)
rr1

And then if you need more detailed access, you can pass.ResampleResult Object Codes in order to achieve access

In addition, as_benchmark_result() It can also be used to direct objects from ResampleResult Convert to BenchmarkResult 。c() Available for grouping multiples BenchmarkResult Object

bmr1 = as_benchmark_result(rr1)
bmr2 = as_benchmark_result(rr2)

c(bmr1, bmr2)

The BenchmarkResult object also has an exclusive visualization method, which gives boxplot to compare the effects of multiple algorithms.

autoplot(bmr, measure = msr("classif.acc"))

Assessment of the binary taxonomyr

We're here.Classification Evaluator Projections in the classification It's about taxonomic evaluation; now we're going to look into it.

In the theoretical part of machine learning, we're introducing some of the knowledge that machine learning leads and supervises learning: performance measurement, simple re-reading and code realization.

mlr3measures Package allows you to use the following: confusion_matrix() Function calculates several common measures based on confusing matrices

mlr3measures::confusion_matrix(truth = prediction$truth,
  response = prediction$response, positive = tsk_german$positive)

Draw ROC curves needs

autoplot(prediction, type = "roc")

It's not a common measure derived from a confusing matrix.

prediction$score(msr("classif.auc"))

Draw a PRC curve (precision-recall rate curve) that requires

autoplot(prediction, type = "prc")

Finally, we consider the relationship between thresholds and indicators.

autoplot(prediction, type = "threshold", measure = msr("classif.fpr"))
autoplot(prediction, type = "threshold", measure = msr("classif.acc"))

These visualizations are perfectly usable. ResampleResult BenchmarkResult 对象 替换原本的 Just do it.

Hyperparameter Optimization

Starting with this chapter, we're working on three consecutive chapters on how to upgrade learning machines; including the most basic automatic over-parameter regulation; and on further modulation methods and feature engineering; that's what we're doing.mlr3Then we started learning how to build a better model.