DaniFrame/AFRLA-instance-level-results

Assessors For Regression: Loss Analysis - Instance Level Results

AFRLA - Instance Level Results is a collection of predictions at the instance/example level for eleven different regression tasks tested on 255 tree-based models (also called "base systems"). The aim of this dataset is to provide example-level results to train assessor models to predict performance of the tree-based models.

The dataset

The dataset presents eleven sections (one per regression task), with varying degrees of performance, difficulty and characteristics from the original tasks. Every one of the 255 models was trained on a subset of the dataset used for every task, and the results shown here are the test (never-before-seen by the models) predictions. Each subset has:

• —An instance identifier indicating the instance nº from the test set. This is just an identifier and it is not usually employed for training assessors, although in some occasions it may be useful for other analysis.
• —The original task features, the features used by the models to learn the task. Along with the instance identifier, they fully describe a test example.
• —The model features, descriptors of the 255 models. Mainly:
• —The model used (XGBoost, Random Forest, Decision Tree...)
• —Hyperparameters such as the maximum depth, number of estimators if applicable...
• —Profiling metrics such as training time, inference time or memory usage

These metrics are not recorded per example, but rather per model (that is, if the inference time is 1.2 ms, the model predicted the entirety of the test dataset in that time, instead of just that example), and are then casted for each example. As such, they fully describre a model.

[image]

Partitions and versions

The sections are already partitioned into a predefined train-validation-test split for training assessors. Assessors need a particular kind of partitioning (mainly stratified by instance identifier to avoid contamination), so that's why the subsets are given.

The main branch contains the unaltered datasets, keeping the original values of the task and model characteristics, whereas the normalised branch contains the datasets properly normalised (numerical features are centered and scaled and categorical features are transformed into dummies).

Original tasks

<table> <thead> <tr> <th style="text-align: center; width: 35%;">Dataset</th> <th style="text-align: center;">#Feat.</th> <th style="text-align: center;">#Inst (test).</th> <th style="text-align: center;">Cat.</th> <th style="text-align: center;">Num.</th> <th style="text-align: center;">Domain</th> </tr> </thead> <tbody> <tr> <td style="text-align: left;">Abalone</td> <td style="text-align: center;">8</td> <td style="text-align: center;">4177</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Biology</td> </tr> <tr> <td style="text-align: left;">Auction Verification</td> <td style="text-align: center;">8</td> <td style="text-align: center;">2043</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Commerce</td> </tr> <tr> <td style="text-align: left;">BNG EchoMonts</td> <td style="text-align: center;">10</td> <td style="text-align: center;">17496</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Health</td> </tr> <tr> <td style="text-align: left;">California Housing</td> <td style="text-align: center;">8</td> <td style="text-align: center;">20640</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Real State</td> </tr> <tr> <td style="text-align: left;">Infrared Thermography Temperature</td> <td style="text-align: center;">33</td> <td style="text-align: center;">1020</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Health</td> </tr> <tr> <td style="text-align: left;">Intrusion detection</td> <td style="text-align: center;">4</td> <td style="text-align: center;">182</td> <td style="text-align: center">No</td> <td style="text-align: center">Yes</td> <td style="text-align: center">Computer Science</td> </tr> <tr> <td style="text-align: left;">Life Expectancy</td> <td style="text-align: center;">21</td> <td style="text-align: center;">2938</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Health</td> </tr> <tr> <td style="text-align: left;">Music Popularity</td> <td style="text-align: center;">14</td> <td style="text-align: center;">43597</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Music</td> </tr> <tr> <td style="text-align: left;">Parkinsons Telemonitoring (<i>motor</i>)</td> <td style="text-align: center;">20</td> <td style="text-align: center;">5875</td> <td style="text-align: center;">No</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Health</td> </tr> <tr> <td style="text-align: left;">Parkinsons Telemonitoring (<i>total</i>)</td> <td style="text-align: center;">20</td> <td style="text-align: center;">5875</td> <td style="text-align: center;">No</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Health</td> </tr> <tr> <td style="text-align: left;">Software Cost Estimation</td> <td style="text-align: center;">6</td> <td style="text-align: center;">145</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Yes</td> <td style="text-align: center;">Projects</td> </tr> </tbody> </table>