Adaptation of instruments developed to study the effectiveness of psychotherapeutic processes

Shushanikova, A.A. National Research Tomsk State University, Tomsk, Russia

Lukyanov, O.V. National Research Tomsk State University, Tomsk, Russia

Introduction

Currently in psychological science there is a need for models and instruments to measure and demonstrate the effectiveness of psychological interventions. Not only are there evidence-based schools of psychotherapy, but also naïve and eclectic “schools” which irresponsibly proclaim the effectiveness of their approaches. An important issue is also the “schoolism” of therapists who are more concerned with asserting the success of their particular school than with helping clients.

In our research, we adapted and applied instruments developed by John McLeod and Mick Cooper, based on the principle of assessments of significant events, to demonstrate the effectiveness of therapeutic practices. McLeod and Coo­per pioneered a pluralistic approach to therapy, and their findings are presented in a number of articles and books (e.g., Cooper & McLeod, 2011). The data we collect­ed indicates how therapeutic practices were subjectively evaluated by different par­ticipants (e.g., by a client and a therapist). Then their assessments were compared, and on the basis of qualitative and quantitative correlations, we developed “process maps” (Cooper, 2004; Cooper & McLeod, 2011; Watson et al., 2012), showing the direction and the range of efficiency and potential of therapeutic activities. There have been no previous studies of this kind in Russia.

Methods

Adaptation of the Russian versions of the instruments was carried out in three stag­es. After receiving consent from J. McLeod and M. Cooper, we obtained their initial data and several translations were made and reviewed by psychologists fluent in Russian and English. Based on the expert reviews, the wordings were chosen that most accurately expressed the essence of each statement on the forms. The experts were professors at Tomsk State University and practicing psychologists working in the city of Tomsk.

In the second stage, we studied the use of the adapted instruments with a sam­ple of 40 individuals undergoing psychotherapy (existential group psychotherapy, individual therapy, and art therapy). Men and women aged 18 to 50 participated in the research.

In the third stage, we interviewed the participants and therapists who had filled out the forms (the instruments). The purpose of the interviews was to determine whether the process of filling out the forms was useful, and whether the statements on the forms were clear or needed revision. We also requested and gathered sugges­tions for improvements of the instruments.

As required by the ethics of psychological research, all participants and clients provided their informed consent.

The instruments were intended to gather and to analyze qualitative data, to reveal meanings of events, and to understand relations between therapeutic ac­tivities. The reliability of the data and the appropriateness of the instruments are determined by the extent to which they allow us to clarify what is helpful in psy­chotherapy.

Description of the instruments

The set of instruments developed within a pluralistic approach (Cooper & McLeod, 2011) facilitates not only study of the specific structure of the therapeutic process, but also of the dynamic process of the therapy; evaluation of the effectiveness of the methods (and, indirectly, of the therapeutic approach); verification of hypotheses about the activities of the client and therapist; and formulation of new hypotheses.

We used the following instruments: “Goals Form”, “Goals Assessment Form”, “Post-Session Form”, and “Therapy Personalization Form”.

Goals Form, Goals Assessment Form

In the pluralistic approach within which the instruments have been developed, the basic starting point in therapy is the client’s goals. There is good evidence that people are more productive when they work towards specific, and relatively chal­lenging, goals (Locke & Latham, 2002). Goals can be defined — the changes that the client intends to achieve — and they can be represented in a hierarchy, ranging from a quest for the higher meaning of life to the most specific, situational goals. Clients may have multiple, discrete goals for therapy, often involving interpersonal relations or specific problems and symptoms. In the pluralistic approach, it is espe­cially important to encourage clients to discuss and clarify what they expect from therapy in the early stages of the process. To do this, clients fill in the “Goals Form”, identifying two or three issues/goals/problems for which they are seeking help. Then, on a scale from 0 to 9, the client indicates: “how much this issue bothers him at the moment (over the last week)”; “how important this issue/goal/problem is for him in relation to his life as a whole”; and “how much progress he expects to make on this issue during therapy”.

In the process of therapy, clients are invited to assess their goals using the “Goals Assessment Form”, which helps to evaluate the effectiveness of the work, to review issues/goals/problems, and to formulate new, more urgent goals. Here the client specifies the issues/goals/problems for therapy and assesses their level of achievement on a scale from 0 to 7. In our studies, goal assessment was carried out in the middle of therapy (intermediate evaluation) and at the end of therapy (final evaluation).

Our research shows that the “Goals Form” and “Goals Assessment Form” make it possible to evaluate not only the general meaning of the goals, but also the inten­sity of the stated issues/goals/problems; the urgency of the problems; and the level of expectation of results.

due to the identification and evaluation of their goals, the clients better rec­ognize and understand their emotions and psychological processes, and feel more confident and able to make decisions. The very task of writing down (stating) one’s goals for therapy is therapeutic and encourages the client to better understand him­self in psychotherapy, his problems and the situations he confronts.

Identification of goals for therapy and goals assessments during therapy pro­vide abundant material for new hypotheses about the effectiveness and the thera­peutic alliance between the client and therapist. To exemplify this, we present a comparative analysis of the use of these instruments in existential group therapy. We used the “Goals Form” and “Goals Assessment Form”. More detailed informa­tion is presented in Lukyanov & Shushanikova (2015).

To develop process maps, we used the clients’ reflexive reports. A phenomeno­logical interpretation of the texts was conducted using the A. Giorgi method (1997) (interpretation of a single text). The procedure identifies “meaning units” by break­ing down participants’ statements into small chunks expressing a single meaning. These “meaning units” were assigned a label to allow organization of the data set.

Labels remained as close to the participants’ actual words as possible. All the texts that have the same meaning form small chunks expressing a single meaning. The “meaning units” are then interpreted (put into acceptable professional language), which results in formulation of a brief phenomenological description (understand­ing) of the reflected process. The topics and meanings that occurred sporadically were combined with others, if possible, as was recommended by Hill, Thomson, and Williams (1997).

Therapy Personalization Form

From a pluralistic standpoint, the choice of the method for therapy is especially important, as it may have therapeutic value for the client. The authors of this ap­proach, along with some other researchers (e.g., Maria Bowens) suggest using the “Therapy Personalization Form” to refine and clarify with the client whether the applied therapeutic method matches the client’s needs. The form consists of 20 scales in which clients indicate what changes they would like to see in their therapy, on a scale from 0 (“Just right for me”) to 5. This form allows the therapist to adjust the process of therapy and make it as effective as possible and in compliance with client’s individual preferences and needs.

In our study, we explored the effectiveness of existential therapy. Analysis of the “Therapy Personalization Forms” showed that clients’ preferences for therapy generally corresponded with the approach of their therapists.

In some cases (mainly in individual therapy), clients noted that there were no scales on the form describing the methods of art therapy or therapy of creative self-expression. Therefore, we consider it reasonable to add these scales to the form.

Our study showed that the “Therapy Personalization Form” was more effective in individual psychotherapy, where the therapist and client together could adjust the process and choose methods based on the client’s feedback about helpful thera­peutic effects and helpful activities.

Post-Session Form (PSF)

The PSF was developed by Cooper and McLeod as a tool for gathering information about significant events that benefit therapy and contribute to therapeutic changes.

Significant events research has its origins in studies of group therapy (Bloch & Reibstein, 1980) and in the research into helpful events in individual therapy conducted by Elliott (1984, 1985). Timulak (2010) carried out a meta-analysis of key themes in client and therapist accounts of helpful events in psychotherapy, us­ing clients’ and therapists’ reflexive reports. He found that client reports focused on contributions to the trusted therapeutic relationship (e.g., reassurance, feeling understood, and personal contact) and to in-session outcomes (e.g., insight, relief, behavioral change, new feelings, and empowerment), that is, on events which help clients to “understand” the problem. Timulak (2010) also found that perspectives on what is significant in therapy differ between clients and therapists, and that sig­nificant events seem to be therapeutically productive.

In many of the studies reviewed and conducted by Timulak (2010), the Helpful Aspects of Therapy (HAT) form (Elliott, 1993; Llewelyn, 1988) was used to collect client and therapist accounts of significant events. This form invites the client or therapist to describe one or more helpful and/or hindering event that occurred during the most recent therapy sessions in which he or she took part. The plu­ralistic framework for therapy developed by Cooper and McLeod (2011) suggests improving the method for analyses by distinguishing among helpful client activities, helpful therapist activities, and helpful therapeutic effects. The aim of this distinc­tion is to distinguish between the meanings of therapist and client activity and to develop a more nuanced understanding of the multiple pathways through which therapeutic change can come about, and also to create a framework in which the client’s contribution to change (Bohart & Tallman, 1999) can be brought to the fore. The HAT makes use of relevant elements of the Brief Structured Recall interview guide (Elliott & Shapiro, 1988) in a self-report format.

The original “Post-Session Form” consists of two parts. In the first part (rating), the participants are asked to assess the session (or the therapeutic day, as in some of our cases) by answering four questions: 1) How helpful or hindering to you was this session overall? 2) How do you feel about the session you have just completed? 3) How much progress do you feel you made in dealing with your problems in this session? 4) To what extent can you say about yourself, “In this session something shifted for me. I saw something differently or experienced something in a new way”? The clients needed to answer with one of the choices offered.

The second, main part of the form (reflexive reports) asks clients to describe their actions during the therapy process: 1) two or three examples of helpful actions by the client “Please write down something that you did in this session that felt partic­ularly helpful to you?” 2) two or three examples of therapist activities: “Please write down what your therapist did, if anything, that helped you to do this”; 3) and then to describe the therapy process: “Please write down why you felt that this activity was helpful — what did it achieve” (helpful effects). Then the clients are requested to describe a second and third example of helpful effects, helpful client activities, and helpful therapist activities. In the final part of the form, the clients are welcome to write down something that happened in the session that felt unhelpful and to sug­gest ways in which this session could have been more helpful.

According to our research, filling in this form was especially meaningful to the clients because it allowed them to reflect on each session about their personal commitment and involvement in the process, as well as the involvement and ef­fectiveness of the therapist. In response to participants’ feedback (both clients’ and specialists’) about the research and the “Post-Session Form”, we added a section for reflexive reports about the most vivid (positive or negative) experiences during the therapy session, thoughts, notes, comments, questions, topics for further discus­sions. We also added a section for self-efficacy assessments in therapy, which con­sists of four scales: sincerity, honesty, effectiveness, and involvement, where clients are invited to give themselves an excellent, good, fair, or poor grade.

Thus, the final version of the adapted instrument consists of four parts: 1) evalu­ation of the therapy session as a whole; 2) examples of useful client activities, thera­pist activities, helpful effects, hindering effects, and desired effects/actions; 3) reflex­ive reports, comments, and ideas about the therapy; 4) evaluation of self-efficacy.

Comparison of the results of two cases of therapy performed according to two different therapeutic approaches, using “Post-Session Forms”, indicates that in both cases the instrument allows the researcher to obtain qualitative data accurately reflecting the dynamics of the therapy and its effectiveness, due to clients’ reports about significant events. Table 1 shows the clients’ core meaningful themes and examples of helpful effects in the therapy, revealed during PSF processing by the phenomeno­logical method. The table shows two cases of existential group psychotherapy.

In Case 1, due to the specificity of existential group therapy, helpful effects were formulated in the context of mental and emotional recovery and coping with chal­lenges. In Case 2, “psychological travel” — meaning psychological renewal, broad­ening of one’s horizons (discovery of new landscapes), harmonization with nature, and intensification of creative living — came to the fore.

Table 1. Core meaningful themes and descriptions of helpful therapeutic effects.

Cases	Most frequent correlations and descriptions
	Helpful therapeutic effects	Description and examples of clients’ responses	No. of participants who mentioned this effect	Percentage of total participants
Case 1. Existential group therapy (“Intensive Therapeutic Life”, as conducted by Dr. A.A. Alekseichik)	1. Authentic / genuine living of life;	As a helpful effect, clients mentioned their authentic and genuine experience during the group. This implies being active, responsible, accepting, honest, etc.; The life was genuine, with no pretentions and illusions; I couldn’t live a miserable life in the group.	8	42.1
	2. Gaining new resourceful experience and more holistic / complete understanding (of the situation/ process/ challenges in life and myself in them);	Clients expressed the view that in the group therapy they gained new experiences, which was very helpful and gave them new resources: I focused on other people’s feelings, which I hadn’t done for a long time; I started to consider questions about the simplicity of life and gratitude.	14	52.6
	3. Humility, determination, willingness to change;	Becoming more humble helped the clients to accept some events as being outside their control and to understand those things in life they can change: I became more humble and it was wonderful; I feel I can change some things.	6	31.5
	4. Considerate attitude in words and actions;	Being actively involved in the life of the group helped the client to be more attentive to what they and other participants say and do, because their words and actions have effects on others (positive or negative): I understood I could not ignore him anymore.	6	31.5
	5. Understanding of oneself and others;	The therapy process allowed the clients to meet their true selves, which helped them to understand others as well: I understood myself better; I can understand people better now.	7	36.8
	6. A sense of the reality of life;	The therapy process intensified reality, the present moment, encouraged clients to act and change at that very moment:What helped me is that feeling of the reality of life; Everything was real – me, my actions, and my words.	5	26.3
	7. Belief in possible change;	The therapy allowed the clients to gain hope for and belief in possible change in life; change seemed to be more achievable:I hope I can change things in my life; I gained hope for change.	8	42.1
	8. Changes in relation to problems;	During the therapy many different problems described by clients concerning relationships, feelings, etc., were gradually losing their force; they were not as dramatic as they were to begin with:My problems seem to be so insignificant now; now I can stop talking about my problems and can open myself for new experiences.	7	36.8
Case 2. “Psycho­logical travel” group (including existential group therapy, art therapy, and landscape therapy)	1. Psychological renewal, rest, feeling of being filled with life;	The therapy led to psychological renewal, helped clients to have a very refreshing rest and to gain a feeling of being filled with life: I see life in a new way; I am filled with joy and energy.	11	84.6
	2. Exploration of individuality and creativity;	The therapy process involved lots of activities to reveal one’s creative potential as fully as possible. Clients experienced those activities as very helpful: I was impressed with how talented I am; I started to paint and it was so beautiful and easy.	8	61.5
	3. Satisfaction, enjoyment, inspiration;	“Psychological travel” brings new experiences and inspires people to make their lives more exciting and creative: The helpful effects for me are: inspiration, strength, joy.	5	38.5
	4. Being with other people, mutual support, participation, care, attention;	In their “travel”, clients could interact with each other and with the therapists as much as they wanted:I always had a feeling of support and care from other members and the therapists; it was very helpful to be with other people who share my ideas and beliefs.	10	76.9
	5. Understanding of oneself in space (landscape, nature);	Landscape therapy implies understanding of oneself in a particular area, which a person can find some connection to, gain some good feeling from etc.:I understood that that whole area was my home for those days; I have a better connection with nature now.	9	69.2
	6. Personal activity and going forward;	The therapy encourages clients to be involved in all the processes and activities, which gives them a helpful and resourceful feeling of going forward:My activity didn’t take energy from me but gave it to me; I was tired, but absolutely happy that I didn’t stop and kept doing it.	10	76.9
	7. Joy of communing with nature and the members of the group;	Clients reported the joy they experienced as being very helpful and favored therapeutic changes: What was very helpful is that I enjoyed being there with those people; pleasure from being with nature and good people.	12	92.3

Discussion

Using reflexive forms and developing process maps were mainly aimed at improv­ing the style of therapeutic work. However, as our results show, in systematic stud­ies, qualitative data obtained by using these instruments (forms) may contribute to describing psychotherapy as a living space in the modes of relations and psycho­therapy as a process of forming specific relations (relationality).

Our adaptation and modification of the instruments, relying on the results of our foreign colleagues’ research, showed the instruments to be beneficial for moni­toring the effectiveness of psychological assistance; for giving responsibility both to professionals who are offering assistance and to clients who are relying on it; and also for studying the effectiveness of therapeutic methods (approaches).

The reliability of the instruments is shown by the fact that the information gained allows us to formulate more precise and clear hypotheses. Reflection about identifying goals for therapy and goals assessments provides therapists with abun­dant material for forming new hypotheses about the therapeutic alliance. The in­terpretations of the data obtained using the “Goals Form” and “Goals Assessment Form” may be diverse. For example, data about the fact that clients change their goals might be understood as an indication that they are becoming more psycho­logically healthy, showing the ability to “evolve” as a result of psychotherapy. At the same time a client’s prolonged focus on the same goals might show the develop­ment of relations in therapy, such as enhancing the therapeutic alliance, or building trust in the therapeutic approach or goals of the approach.

Most of the hypotheses generated by analysis of the reflexive data, of course, re­fer to the process of therapy itself and have practical value for the therapeutic pro­cess. In the more systematic approach towards research of this type, more detailed phenomenological interpretation of the results and comparative analysis may lead to the formulation of theoretical hypotheses as well. The study of relations and meanings of emerging therapeutic relations (relationality) against the background of various therapeutic approaches, the forms of work of different therapists, etc., allows us to present psychotherapy not only as a form of the therapist’s self-expres­sion, neither in the form of models reducing life to techniques or mechanisms, nor in a theoretical concept, but rather a full life, including consistent approaches as well as assumptions of situational decisions. Hopefully this will also make it pos­sible to overcome the struggle for assertion by therapists of their own approach to the detriment of the development and fullness of therapeutic reality; to overcome the antinomy between subjectivism and objectivism that currently exists on the basis of relativism, and that in the future will be based on pluralism and ever more intense and vivid discourses and approaches.

Reflexive forms allow us to study and better understand not only therapists’ relational qualities (Watson et al., 2012; Lukyanov & Shushanikova, 2015), but also clients’ relational qualities.

By registering helpful activities as well as noticing favorable therapeutic effects, clients recognize and understand their emotions and psychological processes bet­ter; they feel more confident, able to influence the course of events and to take responsibility for what is happening, and they are also better able to make deci­sions. This gives them energy, determination, and a sense of genuine presence and belonging. The very task of formulating and writing down activities and the effects to which they lead is therapeutic and promotes a better understanding of what is happening, and the client’s involvement in the process. As the relationship between the therapist and the clients strengthens, therapy becomes richer mentally and emotionally and shows multiple variations depending on the situational content. This contributes to a full involvement of clients in life of the group and in their own lives, as they become psychologically closer (forming bonds), and it increases their potential for “renewal” and “recovery”.

Conclusion

The study of client and therapist activities and helpful effects in the process of ther­apy is significant for further research because it reveals client activity, authenticity, responsibility, self-development, and involvement in the process of therapy that enforce the therapeutic effect. Involvement of clients in the therapeutic process can also show that changes can occur through a variety of therapeutic methods and therapist actions identified by clients as most helpful, but in some cases they might not be consistent with the therapist’s initial assumptions, and can also encourage the search for other therapeutic pathways.

Limitations

The key limitation of this study is the generalization of its findings, given the small sample. Another limitation is the small group of experts who reviewing the effec­tiveness of the adapted instruments.

With the use of process maps, we could visually illustrate many processes of change in clients. Therapists can use these visual tools to improve the process of therapy. However, the main problematic aspect in developing process maps is the low frequency of some combinations of responses. In Case 1, the number of re­sponses received during three therapeutic days is different, since not all the par­ticipants filled out the forms every day. Clients did not always describe helpful therapeutic effects, but always described helpful activities (both by the client and therapist). We assume that clients could not describe/identify/name the effects, or were not aware of them.

Further research could add to demonstrating the effectiveness of the adapted instruments and give a more thorough understanding and hopefully an explana­tion of the effectiveness of different types of therapy.

Acknowledgements

The authors would like to thank the participants and professionals who supported and participated in the research, and our colleagues for creating useful instruments for exploring effectiveness of therapy. The research was conducted with the support of the Mendeleyev Science Foundation at the National Research Tomsk State Uni­versity. The registration number of the research is 8.1.71.2015.

References

Bloch, S., & Reibstein, J. (1980). Perception by patients and therapist of therapeutic factors in group psychotherapy. British Journal of Psychiatry, 137, 274–278. doi: 10.1192/bjp.137.3.274 

Bohart, A.C., & Tallman, K. (1999). How clients make therapy work: The process of active self heal­ing. Washington, dC: American Psychological Association. doi: 10.1037/10323-000

Bowens, M., & Cooper, M. (2012). Development of a client feedback tool: A qualitative study of therapists’ experiences of using the Therapy Personalisation Form. European Journal of Psychotherapy and Counselling, 14. doi: 10.1080/13642537.2012.652392.

Castonguay, L.G., Boswell, J.F., Zack, S.E., Baker, S., Boutselis, M.A., Chiswick, N.R., ... Holt­forth, M.G. (2010). Helpful and hindering events in psychotherapy: A practice research net­work study. Psychotherapy: Theory, Research, Practice, Training, 47, 327–344. doi: 10.1037/ a0021164

Cooper, M. (2004). Counselling in schools project: Evaluation report. Glasgow: Counselling Unit, University of Strathclyde. Retrieved from http://www.strath.ac.uk/departments/counsunit/research/cis.html

Cooper, M., & McLeod, J. (2011). Pluralistic counselling and psychotherapy. London: Sage, 2011.

Elliott, R. (1984). A discovery-oriented approach to significant events in psychotherapy: Inter­personal Process Recall and comprehensive process analysis. In L. Rice & L. Greenberg (Eds.), Patterns of change (249–286). New York: Guilford Press.

Elliott, R. (1985). Helpful and non-helpful events in brief counselling interviews: An empiri­cal taxonomy. Journal of Counselling Psychology, 32, 307–322. doi: 10.1037/0022-0167 .32.3.307

Elliott, R. (1993). Helpful aspects of therapy form. Retrieved from: http://www.experiential­researchers.org/instruments/elliott/hat.pdf

Elliott, R., & Shapiro, D.A. (1988). Brief Structured Recall: A more efficient method for studying significant therapy events. British Journal of Medical Psychology, 61, 141–153. doi: 10.1111/ j.2044-8341.1988.tb02773.x

Giorgi, A. (1997). The theory, practice and evaluation of the phenomenological activity as a qualitative research procedure. Journal of Phenomenological Psychology, 28, 235-260. doi: 10.1163/156916297X00103

Hill, C.E., Thomson, B.J., & Williams, E.N. (1997). A guide to constructing consensual qualita­tive research. The Counselling Psychologist, 25, 517–572. doi: 10.1177/0011000097254001

Llewelyn, S.P. (1988). Psychological therapy as viewed by clients and therapists. British Journal of Clinical Psychology, 27, 223–237. doi: 10.1111/j.2044-8260.1988.tb00779.x

Locke E.A., & Latham, G.P. (2002). Building a practically useful theory of goal setting and task motivation: A 350-year odyssey. American Psychologist, 57(9), 705-717. doi: 10.1037/0003­066X.57.9.705

Lukyanov O.V., & A.A. Shushanikova A.A. (2015). Relational-oriented goals in existential group therapy. Manuscript submitted for publication.

Sundet, R. (2009). Therapeutic collaboration and formalized feedback: Using perspectives from Vygotsky and Bakhtin to shed light on practices in a family therapy unit. Clinical Child Psy­chology and Psychiatry, 15, 81–95. doi: 10.1177/1359104509341449

Timulak, L. (2010). Significant events in psychotherapy: An update of research find­ings. Psychology and Psychotherapy: Theory, Research and Practice, 83, 421–447. doi: 10.1348/147608310X499404

Watson, V. C., Cooper, M., McArthur, K., & McLeod, J. (2012). Helpful therapeutic processes: Client activities, therapist activities and helpful effects. European Journal of Psychotherapy and Counselling, 14 (1), 77-89. doi: 10.1080/13642537.2012.652395