Whenever we meet a new or prospective client, we spend a lot of time chatting with them about their requirements and needs. Often, they have a lot of questions for us too, so Kate sat down to chat with Beckie and asked her some of the questions we get most frequently asked when chatting with clients about evaluation.
Have we missed anything? What would you ask? Do let us know.
Do we need to involve beneficiaries in the evaluation process?
“Ideally, yes. We are not going to be able effectively measure the impact you’re creating for beneficiaries if we don’t involve them in the evaluation process. The Charity Spark model is to take a participatory approach, unless there’s a really good reason not to. That goes from inception through to reporting and dissemination. What this means is that we involve service users and beneficiaries in everything from developing the wording used in the surveys, all the way through to how we’re going to tell people about our findings.
This is really important, because while we can do this just with frontline staff if they know service users really well, sometimes, even then, there are things that don’t come out until we start speaking with service users.
Typically, we pull together a small group of beneficiaries and they act as our advisory group throughout the evaluation. Sometimes we don’t do this and that might be because we can get enough service user input from the methodology we’ve chosen which is a very short, phased approach. But generally, at some point, in every evaluation, the service user will be involved, whether that’s being interviewed, responding to surveys or being part of a focus group, it is so important to include the experience of the beneficiaries, and we really value raising their voice above and beyond anything else in the evaluation process.“
What type of evaluation do you do?
“Our team is experienced in lots of different types of evaluation. The important thing for us is that we choose the theory, the type and the approach of evaluation, and that we tailor that to the project. For example, where there’s a very systems-, place-based, network approach to a project, we’ll be looking at doing an evaluation that suits that style of innovative learning, of continuous learning and of responsive evaluation – often Developmental Evaluation. We might use tools that vary from most significant change technique, through to semi-structured interviews through to photo-journaling and creative evaluation tools. If we are evaluating at the end of a project, we will take a completely different approach. We will always tailor both our overarching approach to the evaluation and the tool set that we use, the methodology that we choose, to your project, your situation and circumstance. That is important because each approach and each methodology have strengths and weaknesses and we want to make sure that what we are using is going to provide a realistic representation of what is going on in your project and the impact it is creating. We often use multiple approaches, for example we might combine developmental evaluation with a process evaluation component with a value for money component, and that will use different methods and tools.”
How do you deal with vulnerable clients and service users?
“We have two evaluators on our team who are trained in a trauma informed approach, Beckie and Andrea. It is also something that we will be training the rest of our team in as well. Beckie and Andrea are both very experienced at dealing with vulnerable clients and have a lot of experience working with frontline services and are trained as support workers themselves. Andrea is a mental health support worker with MIND and Beckie is a volunteer asylum seeker support worker in Southampton. Understanding the impact of evaluation on those participating is something that’s really important to us at The Charity Spark and we want to know that we are not going to create any harm for anyone who’s participating in evaluation. We will also always work really closely with the organisation to make sure that our work is fully risk assessed and managed.“
How much does evaluation cost?
“As you can imagine, that’s a very difficult question to answer, and we don’t mean to be evasive! It really depends on the size of the project, the timescales, the amount of resources that need to go into it and the deliverables that you want. For example, there is a big difference between producing written reports or a video report, all of these variables can affect the price of a project.
The best thing for you to do is really clearly think about what you want to achieve from the evaluation, all of the following will affect the cost of your project:
- What sort of deliverables will you want?
- How many people do you reach at the moment? (This will impact how many people we need to involve.)
- What is the purpose of the evaluation? Is it for internal programme development (which it should always partly be) or for a specific funder?
- Is it programme specific or a wider, whole organisation evaluation?
All of these things will all impact the amount of our time that we need to put in.
What we always do promise is that we price transparently. When we do a quote for you, it will be broken down really clearly by activity type, the amount of days of our time you’re getting, it will include the costs of expenses like proofreading and design of final reports, our travel expenses and any costs that we need to include such as reimbursing volunteers and beneficiaries who are taking part. You will be able to see clearly how we have got to the final figure.
If you already have a budget in mind, please do share that with us. We can build a proposal that will show you what you can get for your money. We’ll be very clear with what is and isn’t included for the money that you have available.”
Do you travel?
“Yes. All of our evaluations have a tailored approach and that will probably mean most evaluations will have an in-person component where we travel to meet your beneficiaries and your staff members. That travel and all expenses will always be included in the budget. It is important to us to get face-to-face with clients whenever we can. However, we have done projects which have been entirely online (hello COVID) and with the online tools at our disposal now we can even do focus groups and workshops online. Generally, we take a flexible approach, we think about who we’re trying to involve, how beneficiaries might respond best, and then we build the plan to suit them.”