Software

Research software I build and maintain: libraries for auditing and developing machine learning models responsibly, together with the datasets and benchmarks behind my papers.

Open-source Libraries

  • Virny

    Virny is a Python library for in-depth profiling of model performance across both overall and disparity dimensions. It decouples model auditing into subgroup metric computation, disparity metric composition, and metric visualization, so it serves both model development and post-deployment monitoring. It is easily extensible, works with custom datasets and model types, and ships with an interactive tool, VirnyView, that streamlines responsible model selection and generates nutritional labels for ML models. Distributed under the 3-clause BSD license.

  • VirnyFlow

    VirnyFlow is a design space for responsible model development. Rather than returning a single best model like conventional AutoML frameworks, it lets data scientists define their own evaluation protocol and optimization criteria — including fairness, stability, and uncertainty — experiment across all stages of a pipeline, and iteratively refine models against the constraints of their problem. The system brings multi-objective Bayesian optimization, cost-aware multi-armed bandits, query optimization, and distributed parallelism together in a modular architecture built for human-in-the-loop workflows. On six real-world benchmarks it outperforms state-of-the-art AutoML systems in both optimization quality and scalability.

Datasets and Benchmarks

  • Shades-of-Null

    Shades-of-Null is an evaluation suite for responsible missing value imputation. It applies state-of-the-art imputation techniques to popular fairness benchmark datasets under evaluation settings that go beyond the usual Rubin assumptions: multi-mechanism missingness, where several missingness patterns co-exist in the data, and missingness shift, where the mechanism changes between development and deployment. Imputers and downstream models are scored with a holistic set of metrics covering predictive performance, fairness, and stability. The suite is extensible, so researchers can add custom datasets and new imputation techniques, and it stores experiment results in a database (MongoDB in our experiments).