Premier Advises NQF on Artificial Intelligence in Quality Measurement

Premier submitted comments to the National Quality Forum (NQF) in response to its draft report on Artificial Intelligence (AI) in Healthcare Quality Measurement. In its report, NQF offers initial guidance and recommendations for developing, selecting and implementing AI-enabled quality measures in programs like value-based purchasing and accreditation. Premier draws upon our experience with tech-enabled quality measurement and improvement programs to advocate for:

• Validation of underlying training and test data, any Large Language Model or Natural Language Processing engine, and any model outputs;
• Validation that ensures AI models contain the appropriate volume and diversity of data to produce meaningful and valid outputs;
• Double-blind validation of models, where an adversarial third-party team assesses a model against a baseline without knowing which set of outputs were generated by the AI model;
• Ongoing surveillance to identify data draft over time; and
• Leveraging Machine Learning Operations (ML Ops) to automate model output surveillance and flag any outputs that fall outside of acceptable criteria.