Calibration and Uncertainty Quantification#

This module provides post-hoc calibration methods and prediction set constructors for uncertainty quantification in healthcare AI models. All methods can be combined with any trained PyHealth model.

Overview#

Model Calibration Methods adjust predicted probabilities to better reflect true confidence levels:

Prediction Set Constructors provide set-valued predictions with coverage guarantees:

  • LABEL: Conformal prediction with bounded error

  • SCRIB: Class-specific risk control

  • FavMac: Value-maximizing sets with cost control

  • CovariateLabel: Covariate shift adaptive conformal

  • ClusterLabel: K-means cluster-based conformal prediction

  • NeighborhoodLabel: Neighborhood Conformal Prediction (NCP)

Getting Started#

New to calibration and uncertainty quantification? Check out this complete example:

Browse all examples online: https://github.com/sunlabuiuc/PyHealth/tree/master/examples

  • Example: examples/covid19cxr_conformal.py - Comprehensive conformal prediction workflow demonstrating:

    • Training a ResNet-18 model on COVID-19 chest X-ray classification

    • Applying conventional conformal prediction with LABEL

    • Using covariate shift adaptive conformal prediction with CovariateLabel

    • Comparing coverage guarantees and efficiency between methods

    • Understanding when to use each method based on distribution shift

This example shows the complete pipeline from model training to uncertainty-aware predictions with formal coverage guarantees.

Module Contents#