deepcarskit.evaluator

deepcarskit.evaluator.abstract_metric

class deepcarskit.evaluator.base_metric.AbstractMetric(config)

Bases: object

AbstractMetric is the base object of all metrics. If you want to

implement a metric, you should inherit this class.

Args:

config (Config): the config of evaluator.

calculate_metric(dataobject)

Get the dictionary of a metric.

Args:

dataobject(DataStruct): it contains all the information needed to calculate metrics.

Returns:

dict: such as {'metric@10': 3153, 'metric@20': 0.3824}

smaller = False

class deepcarskit.evaluator.base_metric.LossMetric(config)

Bases: deepcarskit.evaluator.base_metric.AbstractMetric

LossMetric is a base object of loss based metrics and AUC. If you want to implement an loss based metric, you can inherit this class.

Args:

config (Config): The config of evaluator.

metric_info(preds, trues)

Calculate the value of the metric.

Args:

preds (numpy.ndarray): the scores predicted by model, a one-dimensional vector. trues (numpy.ndarray): the label of items, which has the same shape as preds.

Returns:

float: The value of the metric.

metric_need = ['rec.score', 'data.label']

metric_type = 2

output_metric(metric, dataobject)

used_info(dataobject)

Get scores that model predicted and the ground truth.

class deepcarskit.evaluator.base_metric.TopkMetric(config)

Bases: deepcarskit.evaluator.base_metric.AbstractMetric

TopkMetric is a base object of top-k metrics. If you want to implement an top-k metric, you can inherit this class.

Args:

config (Config): The config of evaluator.

metric_info(pos_index, pos_len=None)

Calculate the value of the metric.

Args:

pos_index(numpy.ndarray): a bool matrix, shape of n_users * max(topk). The item with the (j+1)-th highest score of i-th user is positive if pos_index[i][j] == True and negative otherwise. pos_len(numpy.ndarray): a vector representing the number of positive items per user, shape of (n_users,).

Returns:

numpy.ndarray: metrics for each user, including values from metric@1 to metric@max(self.topk).

metric_need = ['uc', 'rec.topk']

metric_type = 1

topk_result(metric, value)

Match the metric value to the k and put them in dictionary form.

Args:

metric(str): the name of calculated metric. value(numpy.ndarray): metrics for each user, including values from metric@1 to metric@max(self.topk).

Returns:

dict: metric values required in the configuration.

used_info(dataobject)

Get the bool matrix indicating whether the corresponding item is positive and number of positive items for each user.

recbole.evaluator.collector

class deepcarskit.evaluator.collector.CARSCollector(config)

Bases: recbole.evaluator.collector.Collector

The collector is used to collect the resource for evaluator. As the evaluation metrics are various, the needed resource not only contain the recommended result but also other resource from data and model. They all can be collected by the collector during the training and evaluation process.

This class is only used in Trainer.

eval_batch_collect(scores_tensor: torch.Tensor, interaction, positive_u: torch.Tensor, positive_i: torch.Tensor)

Collect the evaluation resource from batched eval data and batched model output. Args:

scores_tensor (Torch.Tensor): the output tensor of model with the shape of (N, ) interaction(Interaction): batched eval data. positive_u(Torch.Tensor): the row index of positive items for each user. positive_i(Torch.Tensor): the positive item id for each user.

deepcarskit.evaluator.evaluator

class deepcarskit.evaluator.evaluator.Evaluator(config)

Bases: object

Evaluator is used to check parameter correctness, and summarize the results of all metrics.

evaluate(dataobject: recbole.evaluator.collector.DataStruct)

calculate all the metrics. It is called at the end of each epoch

Args:

dataobject (DataStruct): It contains all the information needed for metrics.

Returns:

dict: such as {'hit@20': 0.3824, 'recall@20': 0.0527, 'hit@10': 0.3153, 'recall@10': 0.0329, 'gauc': 0.9236}