Evaluation Tools

Formal Formative Evaluation Tools

There are many existing questionnaires that may be useful for evaluating the process and outcomes of a collaboration between public health and primary care. The selection of the appropriate tool is based on the purpose of the evaluation.

One example is the Partnership Self-Assessment Tool, or PSAT, which is an easily administered tool developed and tested by researchers in the USA. To explore more information on the validity and reliability of the PSAT tool, visit www.lmgforhealth.org

The PSAT measures the overall synergy of a collaborative partnership, the effectiveness of a partnership’s leadership, the efficiency of a partnership, the effectiveness of a partnership’s administration and management, and the sufficiency of a partnership’s financial and non-financial resources. Completing the PSAT has been shown to be a valuable exercise for collaborations and can be used on an ongoing basis.

Case Example: Using the PSAT Tool

A public health and primary care collaboration involved community service partners to provide joint youth health services and programs. Six months post commencement they asked all key collaboration partners to complete the tool. PSAT requires an 80% response rate for reliable and valid results.

After everyone completed the 15 minute questionnaire, scores were tallied and results were reviewed in a joint meeting with all partners. Time was taken to discuss the results and to gain a deeper understanding of everyone’s perspectives. Below is a chart with the collaboration’s results displayed on a kiviat graph.

Partnership Self-Assessment Tool (PSAT)

Synergy: Synergy
Effectiveness: Effectiveness of Partnership’s Leadership
Efficiency: Efficiency of Partnership
Administration: Effectiveness of Partnership’s Administration and Management
Non-Financial Resources: Sufficiency of Non-Financial Resources
Capital Resources: Financial and Other Capital Resources

Based on the results in the Kiviat graph, all aspects of this collaboration were located in the PSAT work zone. The players in the collaboration determined that they needed further formative evaluation to explore the problem areas in more depth. Nevertheless, they also were able to determine some immediate areas for adjustment in the way they worked together in the collaboration.

Conducting the PSAT and reviewing the results proved to be a great strategy to increase effective communication among all partners in the collaboration.

Interviews
Using individual and focus group interviews is a common way to collect qualitative information.

Individual interviews are usually conducted with key informants, who are likely to have an important viewpoint on what is being evaluated. These would include community partners.

Focus groups are an efficient way of collecting information from a number of different people at the same time and can capture group interaction.

Formal formative evaluations include both qualitative and quantitative methods such as:

• Interviews
• Focus Groups
• Surveys using existing tools, for example:
  • Partnership Self-Assessment Tool (PSAT)
  • The Team Climate Inventory

Informal Formative Evaluation

Formative evaluation can also be informal. This is frequently used in evaluating the structures and processes within public health and primary care collaborations.

Strategies
One strategy is to design meetings to include opportunities for partners to discuss and provide feedback on the collaboration. Skilled meeting facilitation is important so that all partners have an opportunity to share their perspectives. This simple inexpensive approach to evaluation can be a powerful quality improvement tool.

“So, we’re not perfect. However, we keep improving. In our monthly meetings we evaluate what we’ve done, where we’ve been, and how it’s worked.”

Summative Evaluation Tools

There are many ways to collect data for summative evaluation of the outcomes of a collaboration.  As with formative evaluation, the selection of tools depends on the focus of the evaluation. Patient or client satisfaction surveys are commonly used but many other strategies are possible.

For example:
In a case where tobacco reduction is the goal of the collaboration, data from the electronic medical record can be used to track the number of patients accessing the collaborative service.

Another goal may be to ensure that there are evidence-based smoking cessation guidelines implemented by practitioners (e.g., the 5 As – Ask, Advise, Assess, Assist, and Arrange). To evaluate this goal, chart audits could be conducted to measure the application of some of the 5 A’s (e.g., referrals to smoking cessation programs).

In a collaboration where the goal is to improve 18 month baby assessments, chart audits can be carried out to determine uptake of assessment tools.

Some Markers of Successful Collaborations