Busy PhD Holidays // Experimenting with social impact evaluation frameworks
I've just returned from my holiday break in the Scottish Borders. I dedicated my time-off to a participatory video project with the Alchemy Film Festival - a perfect opportunity to analyse and test some of my recent literature review.
Participatory Video and Social Change is something I had previously researched as a part of my undergraduate final project in 2012. Since then, I have had loads of opportunities to develop my skills as a project facilitator in diverse settings. As a digital media practitioner, I worked with stakeholders of different ages and ethnic backgrounds, investigating different problems. Most of the time, I navigated through these projects relying on my intuition and my 'inner voice' leading the way. My projects were never prescriptive, which in a way, allowed participants to experiment with various methods and come up with original and unique ideas.
My recent project with Alchemy Film Festival, focusing on mental health stigma, was slightly different though.
This time I had my PhD Literature Review lenses on. They allowed me to analyse the media co-creation processes from a new angle.
As some of you might already know, for the next three years, I'm on the hunt for the perfect social impact evaluation toolkit. One of my research aims is to find an accessible framework, which can aid digital practitioners M&E challenge during participatory community projects. During my recent project with the Alchemy Film Festival, I was once again faced with this problem.
How do I capture the social impact of this community media project? How can make M&E process participatory? How can I ensure that my M&E isn't in the way of our creative work?
I decided to mix some of the impact evaluation methodologies (the ones I had already come across). Video4Change is one of the most interesting resources, researching the importance of impact in the fields of film and digital storytelling. I'm fascinated with their current publications focusing on the Participatory Impact Pathways Analysis (PIPA) . This particular framework has its roots in the development field, and began functioning in the mid 2000s. It engages participants in "a structured participatory process, promoting learning and providing a framework for 'action research' on processes of change" (Alvarez, Sophie, et al., 2010).
What is great about PIPA, is that it allows everyone involved in the project (stakeholders, facilitators, staff and the potential beneficiaries) to co-construct the ideal vision for the project and its possible 'impact-pathways'. This collective, inclusive approach towards M&E and project planning, is something that really appeals to my idea of participatory approach.
My only problem with PIPA is the fact that it requires 3 days to be fully completed. Knowing that most of my workshops are super dynamic and have time limitations, I had to find a shortcut.
For my 5-day Alchemy Film Festival's workshop, I needed something that can be easily embedded into the creative process. Something that the participants could complete within 15-20 minutes.
The key elements of PIPA framework are : the problem tree, vision, network maps and logic models. For the purpose of my workshop, I decided to borrow the notion of the problem tree.
"The workshop begins with participants developing a problem tree that links the project goal framed in terms of a challenge or problem to what the project is actually going to do. The approach used for developing the problem tree is based on work by Renger and Titcombe (2003). The problem tree helps participants clarify the key problems / opportunities their projects are addressing, and the outputs (things others will use) that their projects need to produce." source
In a way, I deconstructed 'the problem tree' and played around with its different elements. I decided to divide our M&E into three different sections:
My challenge : Embedding M&E into a 5-day participatory video workshop (2.5 hours daily). Here how I tested my M&E shortcut.
The Problem Tree
Our initial M&E exercise took place at the beginning of the first 2.5h session. We spent 15-20 minutes discussing the problems we'd like to tackle with our video story. We thought about how our stories can influence the wider problem of mental health stigma. Additionally, we spoke about our impact on the organisation ( all stakeholders are regular clients of the partnering mental health charity).
This initial exercise provided us with a vision for our collective project's goals - both external and internal. Interestingly, some of the key project themes appeared during the initial discussion. The group used these in their later storytelling work.
Participants Journey - Postcards Exercise
Apart from capturing the collective of impact, I was also interested in the personal perspectives of the changes occurring during the co-creation process. I asked the participants to write a postcard to themselves - describing their hopes and goals for our 5-day workshop. The postcard was placed in their personal envelops. This was followed by a daily feedback note after each session. The folders provided me with some fascinating personal narratives of the workshop.
Reflecting on the outcomes : Returning to The Problem Tree + Thinking about the Most Significant Change
During the final day of the project, we spent about 15 minutes discussing our initial analysis of 'the problem tree'. Participants discussed their goals and vision, and compared it with the project's outcomes. Interestingly, most of the aims were achieved. The ones that were not, provided as with a bank of ideas for next creative project.
The final evaluation exercise included the Most Significant Change analysis. This particular M&E framework was developed by Rick Davies as part of his PhD focusing on the monitoring and evaluation of a rural development program in Bangladesh. The Most Significant Change aims to collect the stories from all parties involved in the process. I used this method in many of my workshops, mainly recording stories/interviews at the end of the project.
This time, most of the stories I collected were quite short and not extremely detailed. However, I still think this exercise worked well to conclude the evaluation process. Having some time to reflect on your personal journey is a great idea to conclude the workshop. It made everyone ( including myself) to think about the entire process once again and identity the key changes that occurred. Some of the participants found their folders/envelops really helpful with this task.
So here it is, my M&E mix and match.
I am really glad I had a chance to take my some of my literature review into the field already.
This is an early stage of my PhD project, but I feel like this 5-day workshop gave me a glimpse of something I might be doing in my 2nd year.
As a digital community practitioner, I really enjoyed having an actual 'framework' that I could refer to in my practice. Overall, I am very positive about combining intuitional work with academic methodologies. Surprisingly, I doesn't feel restrictive and provide me with loads of interesting insights into my work.
I'm yet to learn more about social impact evaluation - this the next topic on my literature review to-do-list.