Comparing apples with pears?

Over a bowl of pumpkin curry in one of Melbourne’s lovely restaurants, I asked her about her first impression in regards to the implementation of evidence-based youth care interventions in Australia. As I soon found out, trying to define how ‘Australia’ works in regard to this topic is, unsurprisingly, impossible. Expecting Australia as a whole country to have one way of doing things is like expecting the whole of Europe to act as one. With my interest in treatment integrity, this realisation triggered the question of how one should consider differences between and within countries in respect to the measurement of treatment integrity.

The next week I was asked to present at a seminar on the topic of Creating effective support systems for establishing and maintaining treatment integrity, at the University of Melbourne. However, at the seminar I found myself in the midst of experts from 4 different countries with all having an interest in cross country comparison of measurement of treatment integrity. We soon found ourselves talking about measurements instead of support systems. Specifically the threshold scores of treatment integrity levels, or to put is otherwise, at what score therapist are considered to be treatment integer and positive outcomes can be expected. Developers of a treatment integrity instrument will have the threshold score defined and some interventions use these scores for (re)certification purposes of therapists. With a lot of interventions used in different countries the question arises; are the threshold scores the same in different countries and should they be?

The threshold score for treatment integrity of Multisytem Therapy (MST), a youth care intervention spread across many countries, is set at .61 following American norm scores. Therapists in the Netherlands do not always manage to reach this threshold score. There are very many ways of trying to explain this. Let’s just say that one of them is now considered to be that Dutch people that score the treatment integrity of therapists on the instrument of MST (the TAM-R) tend to do this differently to the Americans. This consideration implies that there is a country/culture difference in how people score on a Likert scale ranging from ‘not at all’ to ‘very much’, which is also commonly used in other treatment integrity instruments.

Without putting the Dutch away as passive, I do think they are not very likely to score everything in the extremes on a Likert scale. We have the saying ‘just act normal, that is weird enough’ for a reason. The Americans in that respect might just be more likely to score extreme. They either totally love or totally hate things, which is my totally not scientific impression. I am therefore very interested in the outcomes of the study on country/cultural differences in the scoring of the TAM-R. If it is found to be true that there are significant differences, then in my opinion the set threshold score should not be used in the Netherlands. Not to forget that we might also want to reconsider the international comparisons of treatment integrity scores that have been made up to that point, as they could be improper.

Question being: Is comparing treatment integrity scores across countries equal to comparing apples with pears on the notion that they are both fruit?

Dit blogartikel is eerder gepubliceerd op http://paulinegoense.wordpress.com/.