Board accountability for quality and patient safety is widely accepted but the science for how to measure it is immature, and differences between measuring performance, identifying hazards, and monitoring progress are often misunderstood. Hospital leaders often provide scorecards to assist boards with their oversight role yet, in the absence of national standards, little evidence exists regarding which measures are valid and useful to boards to assess quality improvement. The authors describe results of a cross-sectional board study, identifying the measures used to monitor quality. The measures varied widely and many were of uncertain validity, generally identifying hazards rather than measuring rates. This article identifies some important policy implications regarding boards' oversight of quality and acknowledges existing limits to how we can measure quality and safety progress on the national or hospital level. If boards and their hospitals are to monitor progress in improving quality, they need more valid outcome measures.