Are you a digital community practitioner and not sure how to measure the impact of your projects? Are looking for a perfect solution to your dig-evaluation problem? Sadly, there isn't one.
Although this blog post will not provide a straight forward answer to the above questions, it might might still be a good idea to read on, as you will find some useful links at the bottom of the text.
Ironically, a digital/online search for digital-resources to evaluate your digital-projects might often leave you very confused. After my first attempt to find a perfect evaluation tool in 2014 , I was not only shocked but also a bit disappointed. There was so much information available, yet none it in an accessible and user-friendly format. Additionally, I was no longer sure about what ought to get measured - literacy skills, creativity, empowerment, development, resilience?
While looking up "digital-projects-evaluation (or measurement) online again in 2017, I got multiple results varying from Digital Skills, Digital Competencies, Digital Capabilities, Digital Competence to Digital Literacy. To make things worse, I later reviewed some of the frameworks and realised that for example the GoON Basic Digital Skills Framework (now substituted with an updated version, which has just been published on 10th May - the Essential Digital Skills ) proved to be difficult to use by some of the organisations involved in the UK wide digital inclusion project funded by Carnegie Trust #NotWithoutMe in 2017 (you can read more about this project on my blog here).
To learn more about how (and if) these online resources are being used, I carried out 20 interviews with digital community practitioners (UK) about their experiences and attitudes towards project evaluation.
Results? Only 4 people could provide me with some frameworks/models. I could also note similar themes of confusion 'emerging' from my data of "not knowing what to look for" or "how to capture/measure/analyse it".
11 out of 20 the digital community practitioners (mainly working in the youth sector) said that there is an existing anxiety in the filed when it comes to the digital projects - nobody knows how to define the digital element (e.g is it an email, social media, film club, podcast or playing video games?). That is why, there are even more problems when it comes to evaluation, as hardly anybody knows what type of evidence to look for when it comes to digital project's evaluation.
Unsurprisingly, digital practitioners just want to get on with their work and produce important and empowering initiatives for their communities. That is why, many admitted that in the context of digital project evaluation the priority is to quite frankly "give funders what they want" ( the question here is: do funders really know what they want or need in the context of digital skills/literacy/competence and so on?).
After each interview, I felt that each of the digital community practitioner that I spoke to were keen to explore their options. I would get asked two key two questions: Could you send us some methods or toolkits to measure or analyse digital projects? Are there any practical guides on how to involve participants into evaluation process too?
Acknowledging that:
1) "Frameworks and models are like toothbrushes - everybody has one and no one wants to use someone else's" (as I have recently I learned from Cliff Manning on Twitter :))
2) and “no single tool or method that can capture the whole range of impacts or that can be applied by all” (Dufour, 2015:5).
I decided to create a resource page, where a range of publications relating to digital projects and their evaluation are available.
I'm using this as my "living literature review" and am keen to share it with anyone that might need information on digital literacy/ digital skills frameworks. In addition, I have a section focusing on youth participatory evaluation resources, which is a key element of my PhD research.
The page can be accessed here is still work in progress. I realise that much could be done to improve accessibility and description. However, in my defence, this is just a starting point and I hope to take this idea forward over the coming months (and perhaps after my graduation next year?). Hope you'll find some of this useful in the context of your work. Please note that there is a mixture of academic and non-academic resources, so some of them might be more user-friendly than other.
Resources now available on my page - please email me your feedback or any missing links.
