Further Examples

This example reproduces the empirical findings reported at Özbulak et al. [OJdTF+25], for an empirical setup using 3 metrics (one for utility, and two other for demographic fairness for different attributes).

Note

Obtaining sample data

This example requires you download sample data for two example ML systems. You can download them from these links:

• system_1.json

• system_2.json

On the example below, we assume you downloaded/copied these files into a directory named empirical.

import numpy

import fairical.scores

thresholds = list(numpy.linspace(0, 1, 11, dtype=float))
metrics = ["eod+race", "eod+gender", "acc"]

scores1 = fairical.scores.Scores.load("empirical/phn_hgf/system_1.json")
scores2 = fairical.scores.Scores.load("empirical/phn_hgf/system_2.json")

sol1 = scores1.solutions_a_posteriori(metrics, thresholds).deduplicate()
sol2 = scores2.solutions_a_posteriori(metrics, thresholds).deduplicate()

indicators = {"system_1": sol1.indicators(), "system_2": sol2.indicators()}
print(fairical.utils.make_table(indicators))

The resulting table is shown below:

  System    RELATIVE-ONVG    ONVGR    UD    AS    HV    Area
--------  ---------------  -------  ----  ----  ----  ------
system_1             0.86     0.35  0.70  0.16  0.76    0.28
system_2             1.00     0.46  0.77  0.15  0.73    0.35

To plot a graphical representation of this table, do the following:

import fairical.plot
fig, ax = fairical.plot.radar_chart(indicators)
fig.savefig("radar-empiric.svg")

This code should generate a plot like the following:

The corresponding 3-D Pareto plot can be obtained with:

nds_ds = {
    "system 1": sol1.non_dominated_solutions(),
    "system 2": sol2.non_dominated_solutions()
}

fig, ax = fairical.plot.pareto_plot(nds_ds)
fig.savefig("pareto-empiric.svg")

This code should generate a plot like the following:

Already with 3 (metric) dimensions, it becomes difficult to analyze the estimated Pareto front (non-dominated solutions). You can animate the pareto plot to better visualize the system surfaces in 3 dimensions:

from matplotlib.animation import FuncAnimation

def update(frame):
    ax.view_init(elev=10, azim=frame)
    return (fig,)

animation = FuncAnimation(fig, update, frames=360, interval=20)
animation.save("pareto_animation.mp4", fps=30, extra_args=["-vcodec", "libx264"])

The generated animation for these systems is shown below: