Command-line Apps¶
The command-line apps provided implement the workflow in a simplified manner (for full control, directly use the Python API).
fairical¶
Assess adjustable demographically fair Machine Learning (ML) systems.
Usage
fairical [OPTIONS] COMMAND [ARGS]...
evaluate¶
Evaluate systems for adjustable fairness.
This command takes an arbitrary number of JSON files describing ML system solutions and estimates the Pareto front (i.e. non-dominated solutions according to an arbitrary number of utility and demographic fairness metrics), and outputs indicators that characterize the trade-off between all metrics.
The JSON files must be formatted as described in the documentation.
Usage
fairical evaluate [OPTIONS] SYSTEM...
Options
- -r, --radar <radar>¶
Creates radar plot of system indicators and saves it to file
- -d, --deduplicate, -D, --no-deduplicate¶
If we should prune (deduplicate) similar solutions.
- -t, --table-format <table_format>¶
Output table format.
- Default:
'simple'- Options:
asciidoc | colon_grid | double_grid | double_outline | fancy_grid | fancy_outline | github | grid | heavy_grid | heavy_outline | html | jira | latex | latex_booktabs | latex_longtable | latex_raw | mediawiki | mixed_grid | mixed_outline | moinmoin | orgtbl | outline | pipe | plain | presto | pretty | psql | rounded_grid | rounded_outline | rst | simple | simple_grid | simple_outline | textile | tsv | unsafehtml | youtrack
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Arguments
- SYSTEM¶
Required argument(s)
Examples:
fairical evaluate system1.json system2.json
radar.pdf):fairical evaluate system1.json system2.json --radar radar.pdf
pareto¶
Compute pareto frontiers of systems.
This command takes an arbitrary number of JSON files describing ML system solutions and displays the Pareto front (i.e. non-dominated solutions according to an arbitrary number of utility and demographic fairness metrics) on the terminal.
A scatter plot showing the dominated and non-dominated solutions can be created by passing the –plot flag.
The JSON files must be formatted as described in the documentation.
Usage
fairical pareto [OPTIONS] SYSTEM...
Options
- -d, --deduplicate, -D, --no-deduplicate¶
If we should prune (deduplicate) similar solutions.
- -p, --plot¶
If we should plot the solutions
- --hide-ds¶
If true, hide the ds points for a-priori data from the plot.
- -o, --output-path <output_path>¶
Directory where to store system solutions.
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Arguments
- SYSTEM¶
Required argument(s)
Examples:
fairical pareto system-1.json system-2.json
Evaluate adjustable fairness display the pareto front in table format on the terminal, and generate a plot showing the dominated and non-dominated solutions.
fairical pareto system-1.json system-2.json --plot
solve¶
Find operating-modes (a.k.a. solutions) for ML systems.
This module defines the solve command, which loads one or more system score
output files in JSON format, validates them against a predefined data model, and
computes fairness-related indicators such as accuracy and equalized odds. It outputs
JSON files containing solutions (or operating-modes) following the chosen metrics
space, and thresholds.
Input JSON files must be formatted as described in the documentation.
Usage
fairical solve [OPTIONS] SYSTEM...
Options
- -d, --deduplicate, -D, --no-deduplicate¶
If we should prune (deduplicate) similar solutions.
- Default:
True
- --ds, --nds¶
This flag only concerns a priori analysis. If –ds is set, then only applies only dominated solutions to input score set, if –nds, only applies non-dominated solutions. If neither of these flags is set, then apply all solutions from the input solution set during analysis.
- -o, --output-path <output_path>¶
Required Directory where to store system solutions.
- Default:
PosixPath('.')
- -m, --metric <metric>¶
Metric to consider for solution evaluation. Values are considered in the order they are provided forming the axes of a potentially multi-dimensional surface where dominated and non-dominated solutions are evaluated. Valid values are: fpr, tpr, tnr, fnr, roc_auc, prec, rec, avg_prec, f1, acc, bal_acc, dpd+<attr>, dpr+<attr>, eod+<attr>, eor+<attr>, minmaxd+<util>+<attr>, minmaxr+<util>+<attr>, where
<attr>refers to attributes in the provided sensitive attribute dictionary, and<util>refers to any supported utility metrics. This option is ignored for a priori analysis.- Default:
'acc', 'eod+gender'
- -T, --thresholds <thresholds>¶
If set, then run for a fixed number of equally-spaced thresholds in the interval [0.0, 1.0]. Here, you only specify the number of thresholds to use, with the minimum being 2. If you want explicitly set the thresholds, you should use the API directly. If unset, then let scikit-learn calculate the number of sensible thresholds based the scores and changes on the ROC or PR-curve depending on utility metrics chosen.
- -v, --verbose¶
Increase the verbosity level from 0 (only error and critical) messages will be displayed, to 1 (like 0, but adds warnings), 2 (like 1, but adds info messags), and 3 (like 2, but also adds debugging messages) by adding the –verbose option as often as desired (e.g. ‘-vvv’ for debug).
- Default:
0
Arguments
- SYSTEM¶
Required argument(s)
Examples:
fairical solve scores/system1.json scores/system2.json
Compute and store a-priori solutions for system1, using the thresholds from the non-dominated solutions of system2:
fairical solve scores/system1.json@solutions/system2_nds.json