An Evaluation of Metrics Used by the Performance-Based Research Fund Process in New Zealand

Abstract

The New Zealand Performance-based Research Fund (PBRF) applies a unique set of metrics to assess researchers and rank disciplines and universities. The process involves giving a ‘raw score’ to individual researchers and then assigning them to one of four Quality Evaluation Categories (QECs), used to derive Average Quality Scores (AQS). This paper evaluates the properties of these metrics and argues that QEC thresholds influence the final distribution of assessments. The paper also demonstrates that the derivation of AQSs depends on the weights assigned to each QEC and the distribution of portfolios. The method used to determine the raw scores also has an independent effect on the distribution of scores. The number of researchers included in evaluations also influences the rankings. The paper compares how research rankings of New Zealand universities would vary if alternative summary measures, based on the raw scores rather than QECs, were used to evaluate peformance.