fairical.metrics

Helpers to evaluate scikit-learn metrics at arbitrary thresholds.

Module Attributes

UtilityMetricsType	Supported utility metrics type for pareto front estimates.
FairnessMetricsType	Supported fairness metrics type for pareto front estimates.
MinMaxFairnessMetricsType	Supported min-max fairness metrics type for pareto front estimates.

Functions

calculate_metric(metric, y_true, y_score, ...)	Entry-point function to calculate arbitrary (supported) metrics.
parse_metric(name)	Parse and validate a string supposed to carry a metric name.
should_minimize(metric)	For a given metric, tells if it should be minimized or maximized.
supported_metrics()	Generate a comma-separated list of supported metrics.

fairical.metrics.UtilityMetricsType

Supported utility metrics type for pareto front estimates.

alias of Literal[‘fpr’, ‘tpr’, ‘tnr’, ‘fnr’, ‘roc_auc’, ‘prec’, ‘rec’, ‘avg_prec’, ‘f1’, ‘acc’, ‘bal_acc’]

fairical.metrics.FairnessMetricsType

Supported fairness metrics type for pareto front estimates.

alias of Literal[‘dpd’, ‘dpr’, ‘eod’, ‘eor’]

fairical.metrics.MinMaxFairnessMetricsType

Supported min-max fairness metrics type for pareto front estimates.

alias of Literal[‘minmaxd’, ‘minmaxr’]

fairical.metrics.parse_metric(name)

Parse and validate a string supposed to carry a metric name.

Valid metric names are the ones listed in UtilityMetricsType, FairnessMetricsType (followed by a “+<attr>”), or MinMaxFairnessMetricsType (followed by a “+<util>+<attr>”), where <attr> corresponds to the protected attribute being measured by the fairness metric, and <util> corresponds to the UtilityMetricsType to be used to measure min-max fairness difference or ratios.

Parameters:

name (str) – The string to be validated.

Return type:

Union[Literal['fpr', 'tpr', 'tnr', 'fnr', 'roc_auc', 'prec', 'rec', 'avg_prec', 'f1', 'acc', 'bal_acc'], tuple[Literal['dpd', 'dpr', 'eod', 'eor'], str], tuple[Literal['minmaxd', 'minmaxr'], Literal['fpr', 'tpr', 'tnr', 'fnr', 'roc_auc', 'prec', 'rec', 'avg_prec', 'f1', 'acc', 'bal_acc'], str]]

Returns:

The parsed metric.

Raises:

ValueError – If the metric expressed in name is invalid.

fairical.metrics.should_minimize(metric)

For a given metric, tells if it should be minimized or maximized.

Currently, “fpr”, in the utility side, “minmaxd” on the min-max fairness metrics, or any other fairness metric should be minimized. All others should be maximized.

Parameters:

metric (str) – Metric name.

Return type:

bool

Returns:

True, if the metric should be minimized (instead of maximized). False otherwise.

Raises:

ValueError – If the metric is invalid.

fairical.metrics.supported_metrics()

Generate a comma-separated list of supported metrics.

Return type:

list[str]

Returns:

A comma-separated list of supported metrics.

fairical.metrics.calculate_metric(metric, y_true, y_score, thresholds, sensitive_attributes=None)

Entry-point function to calculate arbitrary (supported) metrics.

This function works as an entry-point to the metric calculation submodule. It can calculate arbirary (supported) metrics provided input information for a system, consisting of ground-truth, scores, thresholds and (optionally) sensitive features.

Parameters:

• metric (str) – The metric to calculate.

• y_true (Sequence[int]) – True binary labels (0 or 1).

• y_score (Sequence[float]) – Predicted continuous scores or probabilities.

• thresholds (Sequence[float]) – Threshold values at which to binarize y_score (\(score >= threshold\) implies sample is classified as positive).

• sensitive_attributes (Optional[Mapping[str, Sequence[int | str]]]) – Group membership for each sample, according to protected attribute. Only required if metric is a fairness metric. Each entry in the input dictionary should match the order of samples in y_true and y_score. When metric refers to a particular sensitive attribute, it should be a key in this dictionary.

Return type:

list[float]

Returns:

The metric over all considered thresholds.

Raises:

ValueError – In case of unknown metrics.