Measurements of effectiveness (sometimes called 'outcome-based measures') are only useful if they provide a balance between accuracy and precision requirements. Such measures can indicates what a program is accomplishing and whether results are actually being achieved. They can also help decision-makers by providing them information on how resources and efforts should be allocated over time to maximize effectiveness. Finally, they can keeps program partners focused on the key goals of an endeavor. Such measures are most likely to be successful when they satisfy the following objectives:

• Quality over quantity: Measures should be relevant to the results an endeavor is intended to achieve. This generally argues for quality over quantity, with a focus on a limited number of well-designed measures.
• Importance to decision-making: Measures should provide information that helps make decisions.
• Transparency: Measures should be understandable by all stakeholders, including the intentions behind the measurements that are to be addressed, and their effectiveness in practice
• Feasibility: Measurements should be feasible to collect, but should not be chosen just because collecting the measurement is the path of least resistance.
• Relevance: A measure must be meaningful to managing performance, and not be selected just because you have data readily available. If you can't control an outcome due to randomness or noise, why measure it.
• Collaboration: Stakeholders responsible for measures should have a say in crafting them and a commitment to acting on what they communicate.

Defining such measures is thus as much an art as a science, and collecting and reporting such measures on an ongoing basis can draw critical resources from other important activities. As a result, leveraging measurement best practices is highly recommended. One good set of those resources is here.

Since there is no substitute for having a good example to follow, here is one of IHI's sample definitions for a measure they use to track the incidence of medical harm through the Rate of Inpatient Adverse Events in a hospital setting:

This measure is defined as the rate of adverse events (AEs) that cause harm to the patient, based on a review of a representative sample of hospitalized patients’ medical records. The IHI Global Trigger Tool for Measuring AEs allows organizations to conduct a retrospective review of patient records using triggers (or clues) to identify possible AEs. The use of triggers to identify AEs is an effective method for measuring the rate of harm from medical care in a health care organization over time. The IHI Global Trigger Tool defines an adverse event as an injury or harm to the patient related to (or from) the delivery of care. This measure is reported as the number of Adverse Events (AEs) per 1,000 Patient Days. Computation is as follows:

• AEs per 1,000 Patient Days = (Total number of AEs / Total length of stay for all patient records reviewed) * 1,000
• The frequency of collection for this data is to be monthly; the method for measuring this parameter will be as follows:
  • Use the detailed instructions in the IHI Global Trigger Tool to review all 20 charts. The complete IHI Global Trigger Tool, including rules and methods for reviewing, is available on IHI’s website.
  • Submit the reviewed data to the data repository by xx day of each month.
  • The aggregated data will be reported by xx day of each month as Adverse Events per 1,000 Patient Days per month on a line graph with added annotations as appropriate to indicate changes in measurements, data population, or factors which would impact representation of results.

This definition demonstrates that it is not practical to measure every intervention in a broad-based campaign such as IHI's, as the collection costs would be unacceptably high. As an alternative, statistical data analytics and a standardized tool are used to draw meaningful conclusions from a subset of actual results. The resulting measures are statistically valid and can be aggregated with others as long as the samples themselves are carefully selected (as indicated above). Note how only a random sample of 20 charts across the entire network can be used to produce meaningful data.

Note also that such measures may be sufficiently accurate, they will not be precise enough for other questions to be answered, such as to allow enumeration of all adverse impacts. Despite this limitation, in IHI's case, such measures of effectiveness are providing useful information for its intended purpose, which is to build momentum and support for their follow-on campaigns. In parallel, different and more frequently collected measurements could help in local situations to isolate the root causes of the problems themselves.

An excellent site which highlights the challenges in managing and deploying useful measurements can be found here.