25 February 2014
Imagine this situation: A team of doctors were trained in palliative care by their UK partners in a one-week course. There were 30 trainees in the group and the partnership plans to train another cohort of the same size. The trainee doctors received a certificate of competence at the end of the course if they performed well enough in the final written assessment.
Once they completed the course, the doctors returned to their places of work in the hospitals and clinics both in peri-urban and urban areas. For many of the doctors, they will be the most senior or only clinician at their place of work and they will also be the only ones responsible for delivering palliative care.
What can the partnership do to gather data on the doctors’ practice in palliative care once they have returned to their places of work? In this scenario, the trainees are spread widely geographically making it difficult for those responsible for M&E to visit each of them.
The doctors do not have an appraisal system or a senior colleague who can provide objective feedback on their practice. The doctors do keep practice logbooks but it may not be feasible to retrieve data from all of those logbooks.
The partnership does not have enough funds or supervisors available to visit each doctor in their place of work to assess their skills, and in any case, how thorough an assessment can be made in just one visit? So the partnership faces various issues in data collection but they do need to come up with ways to gather data on practice that are reasonable given the context they work in.
This scenario is typical of many health partnerships’ experience of monitoring change in a rigorous way: they have limited staff, time and funds available to monitor and evaluate each health worker’s performance yet they still need to gather evidence that could demonstrate the impact of the training on the health worker’s practice. Given the context that health partnerships work in, there is a strong case for using a self-assessment tool, such as an online or paper questionnaire.
The obvious issue is the lack of objectivity or external verification of the claims made in a questionnaire and indeed the aim should be to combine it with other sources of information (e.g. clinical records) yet the questionnaire can overcome some of the difficulties health partnerships face in data-gathering to provide interesting insights to trainees’ experiences in their own places of work.
How can you design a questionnaire for use in your own work? I asked THET’s community of practice for examples of self-assessment tools that they use in their projects.
Interestingly, I received several examples of workshop evaluations, which looked at things like knowledge gained, course relevance, and overall satisfaction with the training, all gathered at the end of the course.
I had far fewer examples of questionnaires that trainees complete further down the line on if, or how, they applied their training. Although any questionnaire must be tailored to the specific techniques taught, the following generic topic areas are a useful starting point for designing your self-reporting tool:
Lastly, what response rate would you be satisfied with? It is unlikely that you will get 100% of trainees to complete and return the questionnaires so make it as user-friendly as possible (test it out on some colleagues before you distribute it), consider creating it online if local bandwidth allows – THET uses the online tool Survey Monkey because it is easy to use and has useful reporting functions – and review your M&E plans to determine other data-gathering tools so that you are not reliant on just the questionnaire for data on change in practice.
If you have an example questionnaire that you would be happy to share with us, please send it to email@example.com or post it on the community of practice. I am also interested to hear about people’s experience of using self-reporting to gather evidence.