​'Stories from behind the scenes' tracked a profile group as part of an evaluation in 2013. Here's an inside design look.

Thinking about Story the Future and the ILO Knowledge Academy I'm helping out with in October, I've been rooting through the archives for good examples of story gathering, curation, sense-making and virtual collaboration that produces actionable knowledge and insight.

One piece of work from five years back that comes to mind that I've not thought about for a while. It's an evaluation of the second phase of the Behind the Scenes programme conducted for the British Council in 2013, with David Gunn and Andrew Holden. The British Council client team particularly wanted to track the impact of cultural capacity building workshops on attendees in different countries over a period afterwards. This posed particular challenges around language, attention, and around the ability to notice your own experiences, let alone convey them to others, and trace back to moments that might have influenced you.

The nature of the theme also posed questions. The transfer of expertise from experts at the top of their game gave the British Council convening power to bring people together, both to learn from the expert, and to learn from and build lasting local relationships with each other. So the collaborative capital, if you could call it that, mattered as much as a long-lasting impact as any other capacity that might have been built. And this was an elusive datapoint. How would we even know? How would participants even know or recognise that the spark had been at that moment, or influenced by that moment? We were also curious about whether it was possible to build a community between participants that might outlast the experience of what was, essentially, just a one or two day workshop, and a community of strangers from very different countries and cultures.

We decided to run the post-workshop group as a blog with a series of questions and forms and images posted over time, inviting people share experiences and memories, in their own languages if they preferred, that we could than pattern into our data sets. It wasn't scientific, and it wasn't hugely active, but people did join in, and it did yield some surprising and useful data to supplement our dataset. In particular, breaking questions down and humanising them, creating the sense of an exchange between participants in different countries, and reincorporating experiences that are being shared, to create a conversational dynamic virtually is a technique worth deepening And that act of noticing by us, working out loud if you like, somehow gives permission to people to become more aware of their own experiences and able to access them. Inviting people to hop forwards and backwards in the questions, able to arrive at any point and join in, is something I like about the design, looking back on it.

Here, from the archives, are a few extracts from the blog, to give you a flavour, and a page extracted from the report. I caveat this by saying that I have not kept in touch with how things have unfolded since, so this extract is shared in the spirit of sharing narrative method, and that's how the reader should approach it. I'm curious again, mind!

Background

Behind the Scenes is a new initiative being piloted in 2013, which is designed to build on and rapidly expand the range of capacity building / professional development programmes undertaken by the British Council. While many of those activities have revolved around the creative processes involved in making a new artwork, this project – as its title suggests – will instead address various aspects of the ‘backstage’ infrastructures that need to be in place for any new artistic ventures to be successful.

Between January and June 2013, the Behind the Scenes stage two pilot is forecast to have delivered workshops in two regions covering four subject areas:

• lighting design (1 & 2)
• museum management
• audience development and strategic marketing (1 & 2)
• producing and programming (1 & 2)

The British Council worked with leading UK practitioners to devise and deliver the programme. Workshops took place in Armenia, Azerbaijan, Croatia, Georgia, Ghana, Macedonia, Montenegro, Nigeria, Russia, South Africa, Ukraine, Kazakhstan

A little bit more about our approach to the evaluation
Our evaluation team are a trio with background in cultural policy and practice, strategic consulting and evaluation. We have worked with the British Council team to develop a way of evaluating that

• feels practical and hands-on
• uses our roles as observers to identify small, personal, stories and moments that paint a bigger picture
• works with the the hosts, participants and practitioners who have been involved to build a bigger pictures with several perspectives in it.

For Behind the Scenes, we sketched out an evaluation plan at the outset that looks like this:

Introduction to the Behind the Scenes Profile Group

We start with a big thank you

This is where we start on the evaluation journey together. Thank you for committing to being part of the Behind the Scenes Profile Group between June and September 2013. We really appreciate it and will do our best to make it easy, fun and interesting.

Please don’t worry if you are not joining us right at the beginning. We know that some people are still to have their workshops, and all of you are very busy. We have designed this journey so that you can join at any time and will find it straightforward to join in and contribute.

Let’s start with a reminder of who we all are.

Who we are

We are David, Andrew and Victoria. We find out the stories of people’s experiences and use them to help projects move forward.You may have met one of us at a workshop or course hosted by the British Council Arts team over the past few months. The British Council have commissioned us to assess the impact of Behind the Scenes during 2013 and make strategic recommendations for the future.

On behalf of the British Council, we’d like to thank you for agreeing to be part of a small international group of Behind the Scenes participants who will help us evaluate the design, impact and potential of the programme, and give you the opportunity to influence its future direction.

Who you are

You are a participant of one or more of the Behind the Scenes workshops which have so far included:

• audience development and strategic marketing
• lighting design
• producing and programming
• museum management.

For some of you the workshop will be quite recent and for others it was earlier in the year.

Your commitment

You are willing to think about what you’ve gained from the experience and to share your thoughts and stories. You can spend 10 minutes every two weeks from June to August on this, ending with a longer interview of around 30 minutes in August. The commitment of time should be a maximum of 90 minutes over three months.

Watching your own learning like this is great professional practice in general, as well as helping the British Council.

What we’ll do together

From June to August we’ll build on the questionnaire you filled in at the end of the workshop to send you all a simple question approximately every two weeks. Each question will explore a different angle, for example:

• Practical: what skills and tools have you been able to take back and use at work?
• Creative: have you generated new ideas and insights?
• Feelings:  do you have new confidence in an approach, for example?
• Collaboration: do you have new networks and projects with others as a result of having attended?
• Memory: what sticks in your mind from the experience, looking back?

If you would like to start straight away, you can go to the first question here and then come back and read the rest of this introduction later.

There are two main reasons we want to evaluate in this way:

• We know from experience that learning sometimes takes some time to work its way into practice, so three months feels like a good time period to be thinking about things together.
• Some of you said you would enjoy collaborating with each other on this, and seeing each other’s thoughts and stories, so we will use this blog to invite you to  keep in touch with each other and share your experiences with us and with each other.

We would only like to set two conditions about how you contribute:

• Emphasise your own experiences and examples, rather than just general comments – the more the specific, the better.
• Be thoughtful about how you comment on contributions by others. We’d love to see conversations growing, and we have found that this happens best when comments are very warm.

Thank you.

Translation

We have decided that we will conduct this impact assessment in English, as there may be people from several different countries sharing their experience and this seems like the most open way.

We know that some of you are able to help each other out with translation, and we don’t mind at all about polished comments, we’d just like to hear from you all, even if it’s just a very short comment with a link to a Facebook page or somewhere else where you are writing and talking about your work.

Please don’t be put off sharing! We do want to hear from everybody who has something to share, and our impression is that there is a lot of enthusiasm for some of the skills and tools that have been shared, as well as for the new contacts and networks that have been made. We would like to hear about all of that.

How it works

There will be five short questions posted approximately every two weeks: two in June, two in July and one in August.

We will post each question and then share it with you and ask you to respond by commenting on the question. The only thing you have to do at this stage is to take a moment to make sure you record the link to this page. Don’t worry too much, though, because we will email the questions directly to you, with the link in them, every two weeks.

At the end of August, we would like to have a longer telephone interview or email exchange with you, in which we can talk in a bit more depth about reflections and recommendations and any stories you have to share.

Sharing materials

Nothing tells a story like a picture a video or a sound clip from an interview or a project, so if you are collecting any materials, for example on your smartphone, we’d love it if you shared them with us. Anything you think is interesting – even a  picture of a lighting design drawn on a piece of paper tells a story. We imagine that this could feel a bit like you sharing postcards with each other, which might be fun and interesting, and help us find out new things to look at as we go along. Please just share them with us by email for now – you can find our contact details here – and we will find the best way to share these with you all as part of the final report.

If you are a keen Twitter user, you could post tweets and images using the hashtag #scenestories as one way of sharing things. If you look on the right hand side of this page, you’ll see everything being tweeted with that hashtag, and it’s very easy for you to just add a tweet there, or you can go to your own twitter account and tweet from there.

You might have other suggestions for sharing too, so do make sure you tell us about them.

......

Here's just one question, by way of illustration. We mixed up forms, and prompts, and imagery from the workshops we'd observed, and mixed in answers from previous questions. We also tried to start with the easier questions, perhaps easier: practical impacts, followed by creative, collaborative, felt. And finally memorability. That sequence was carefully chosen. Here's the collaboration question.

Question three: people, friendships, networks, happenings

Lots of your Behind the Scenes evaluation forms said that one of the experiences you most valued was the chance to spend time together, meeting new colleagues and building fresh relationships. So Question Three is about collaborative and network impact. If you’d not already done so, before you answer this question, please do spend a few moments answering Question One (creative impact) and Questions two (practical impact). And if you’d like a reminder of the background, there’s information here.

Theme: People, friendships, networks, happenings

This time, instead of asking you to fill in a form to send to us, please do write your response to the blog as a short comment that tells us something about about

• people you met at a Behind the Scenes workshop, new networks and friendships
• an exchange of ideas that resulted from the meeting and the new things that is leading to
• any case studies, tools and ideas from the workshop which have helped you in working together to bring something new about between you

Ripple effect

Finally, after we'd coded our data and arranged it in patterns, we sought to convey a ripple effect out from the pebble dropped into the experiences, creative ideas, connections and feelings of participants. Here's a page from the report: