pyhealth.metrics.fairness#

fairness_metrics_fn is also importable directly from the top-level pyhealth.metrics package (from pyhealth.metrics import fairness_metrics_fn), not just from this submodule.

pyhealth.metrics.fairness.fairness_metrics_fn(y_true, y_prob, sensitive_attributes, favorable_outcome=1, metrics=None, threshold=0.5)[source]#

Computes metrics for binary classification.

User can specify which metrics to compute by passing a list of metric names. The accepted metric names are:

  • disparate_impact:

  • statistical_parity_difference:

If no metrics are disparate_impact, and statistical_parity_difference are computed by default.

Parameters:

  • y_true (ndarray) – True target values of shape (n_samples,).

  • y_prob (ndarray) – Predicted probabilities of shape (n_samples,).

  • sensitive_attributes (ndarray) – Sensitive attributes of shape (n_samples,) where 1 is the protected group and 0 is the unprotected group.

  • favorable_outcome (int) – Label value which is considered favorable (i.e. “positive”).

  • metrics (Optional[List[str]]) – List of metrics to compute. Default is [“disparate_impact”, “statistical_parity_difference”].

  • threshold (float) – Threshold for binary classification. Default is 0.5.

Return type:

Dict[str, float]

Returns:

Dictionary of metrics whose keys are the metric names and values are

the metric values.

Both disparate_impact and statistical_parity_difference raise ValueError if either the protected or unprotected group has zero instances – the favorable-outcome rate is undefined for an empty group, so this is always an error rather than a value (e.g. 0 or NaN) that could silently poison a downstream average across folds/seeds.

pyhealth.metrics.fairness_utils.disparate_impact(sensitive_attributes, y_pred, favorable_outcome=1, allow_zero_division=False, epsilon=1e-08)[source]#

Computes the disparate impact between the the protected and unprotected group.

disparate_impact = P(y_pred = favorable_outcome | P) / P(y_pred = favorable_outcome | U)

Parameters:

  • sensitive_attributes (ndarray) – Sensitive attributes of shape (n_samples,) where 1 is the protected group and 0 is the unprotected group.

  • y_pred (ndarray) – Predicted target values of shape (n_samples,).

  • favorable_outcome (int) – Label value which is considered favorable (i.e. “positive”).

  • allow_zero_division – If True, use epsilon instead of 0 in the denominator if the denominator is 0. Otherwise, raise a ValueError.

Return type:

float

Returns:

The disparate impact between the protected and unprotected group.

Raises:

ValueError – If either group has no instances at all (this is always an error, regardless of allow_zero_division – there is no meaningful epsilon substitute for a group we have zero information about), or if the unprotected group’s favorable-outcome rate is exactly 0 and allow_zero_division is False.

Examples

>>> import numpy as np
>>> from pyhealth.metrics.fairness_utils import disparate_impact
>>> sensitive_attributes = np.array([0, 0, 1, 1, 1])
>>> y_pred = np.array([1, 0, 1, 1, 0])
>>> disparate_impact(sensitive_attributes, y_pred)
1.3333333333333333
pyhealth.metrics.fairness_utils.statistical_parity_difference(sensitive_attributes, y_pred, favorable_outcome=1)[source]#

Computes the statistical parity difference between the the protected and unprotected group.

statistical_parity_difference = P(y_pred = favorable_outcome | P) - P(y_pred = favorable_outcome | U) :type sensitive_attributes: ndarray :param sensitive_attributes: Sensitive attributes of shape (n_samples,) where 1 is the protected group and 0 is the unprotected group. :type y_pred: ndarray :param y_pred: Predicted target values of shape (n_samples,). :type favorable_outcome: int :param favorable_outcome: Label value which is considered favorable (i.e. “positive”).

Return type:

float

Returns:

The statistical parity difference between the protected and unprotected group.

Raises:

ValueError – If either group has no instances at all. Unlike disparate_impact, a favorable-outcome rate of exactly 0 for a non-empty group is not an error here (it’s a legitimate value for a difference, e.g. 0 - 0.3 = -0.3).

Examples

>>> import numpy as np
>>> from pyhealth.metrics.fairness_utils import statistical_parity_difference
>>> sensitive_attributes = np.array([0, 0, 1, 1, 1])
>>> y_pred = np.array([1, 0, 1, 1, 0])
>>> statistical_parity_difference(sensitive_attributes, y_pred)
0.16666666666666663
pyhealth.metrics.fairness_utils.sensitive_attributes_from_patient_ids(dataset, patient_ids, sensitive_attribute, protected_group)[source]#

Returns the desired sensitive attribute array from patient_ids.

Parameters:

  • dataset (BaseDataset) – Dataset object (must implement get_patient(patient_id) returning a Patient with a "patients"-typed demographic event).

  • patient_ids (List[str]) – List of patient IDs.

  • sensitive_attribute (str) – Sensitive attribute to extract.

  • protected_group (str) – Value of the protected group.

Return type:

ndarray

Returns:

Sensitive attribute array of shape (n_samples,).

Examples

>>> import polars as pl
>>> from datetime import datetime
>>> from pyhealth.data import Patient
>>> event_df = pl.DataFrame({
...     "patient_id": ["patient-0", "patient-1"],
...     "event_type": ["patients", "patients"],
...     "timestamp": [datetime(2020, 1, 1), datetime(2020, 1, 1)],
...     "patients/gender": ["F", "M"],
... })
>>> class ToyDataset:
...     def get_patient(self, patient_id):
...         return Patient(
...             patient_id=patient_id,
...             data_source=event_df.filter(pl.col("patient_id") == patient_id),
...         )
>>> sensitive_attributes_from_patient_ids(
...     ToyDataset(), ["patient-0", "patient-1"], "gender", "F"
... )
array([1., 0.])