The first step to conducting an evaluation is to identify what you are going to evaluate. Without knowing what you want to find out, good evaluation is impossible.

You should have identified what you are hoping to achieve from the residential during its planning. If you have used the logic model or theory of change approach to planning suggested earlier, and outlined in the Planning Tools resource on the Learning Away website, you will have identified:

• The learning needs i.e. the particular learning problems or opportunities that could be addressed by providing a residential.
• The longer-term aims or impacts you would hope to achieve, once back at school, as a result of providing the residential.
• The medium-term outcomes likely to achieve your long-term aims i.e. the changes/benefits/learning you would hope to see as a result of your residential.
• The outputs i.e. the quantifiable changes that you hope to see as a direct result of your residential.

Evaluation will tell you if the residential has achieved what you hoped. Your evaluation should therefore be planned to help you find out whether you achieved the aims, impacts, outcomes and outputs based on your learning needs.

In order to identify what you want to find out, and therefore plan your evaluation approach, it is useful to ask yourself a simple question. A good way to frame the question you wish to answer is to fill out the blanks in the following sentence:

I would really like to know if the residential had an impact on  _________________   (aims) and  _________________  (outcomes) for my class/year group/school.

For example, the external evaluation of Learning Away focused on trying to find out whether residentials had an impact on the following:

• Boosting progress and attainment
• Improving knowledge, understanding and skills
• Improving students’ engagement with their learning
• Fostering deeper student-teacher and student-student relationships
• Improving resilience, self-confidence and wellbeing
• Boosting cohesion and sense of belonging
• Offering opportunities for student leadership and co-design of learning
• Improving students’ transition experiences
• Enabling teachers to widen and develop their pedagogical skills.

As suggested earlier, to evaluate the impact and effectiveness of a residential it may be necessary to collect some baseline data or evidence before the residential takes place; this should give you a point of reference against which you can measure the changes that have occurred as a result of the residential.

The Learning Away schools collected two types of baseline data/evidence in order to determine the measure(s) against which impact could be judged. Where appropriate to their evaluation ‘question’ (see above), they used their own available quantitative data sets, which included attainment and progress as well as behaviour and attendance data. For example, pre- and post-residential quantitative data was provided by three secondary schools to evaluate the impact of their residentials on on student achievement. Behaviour and attendance data was provided by primary and secondary schools hoping to demonstrate the impact of their residentials on students who were at risk because of poor attendance, disengagement and possible exclusion from school.

All of the students and staff taking part in Learning Away residentials also completed pre-residential surveys to enable before and after (both short- and long-term) comparisons to be made. Examples of these pre-residential surveys can be found in the toolkit’s Resource Library.

Don’t forget that if you want to track the changes that have taken place to individual students pre- and post-residential, you will need to identify their data in some way. If using online surveys (see below), this is best done by providing each student with some form of personal identification code or number. It is good practice to do this rather than asking for actual names as it gives clear permission for participants to be more honest with their answers. It also means that all data collected in this way can be easily anonymised.

It may be appropriate to carry out some evaluation while on your residential. These are two possible approaches:

1. Plan, Do, Review

To evaluate the effectiveness of their programme and its activities, some of the Learning Away schools carried out a form of daily evaluation during the residential itself using a Plan, Do, Review Cycle. For this process students make plans, carry them out, and reflect on what they have done. In doing so, they learn to take initiative, solve problems, work with others, and accomplish their goals: their residential becomes even more purposeful and focused.

• Plan – think about the aims, purposes and goals for the programme/activities to ensure it meets the needs of the group.
• Do – implement and adapt the programme/activities in response to individuals, the group and situation (best done with input from the group).
• Review and evaluate the programme/activities to ensure maximum learning, impact and effectiveness (again best done with input from the group).

This approach can only really be used to discover short-term impacts and plan changes to improve efficiency and effectiveness. However, findings can add evidence to any final evaluation report.

2. Participant observation

Participant observation is when the evaluator ‘joins in’ with the group s/he is studying to gain a deeper insight into their experiences. It can be an effective means of exploring and assessing the impacts of the residential on your students. Participant observation is carried out during the residential. See below for details of how to carry out participant observation.

If a residential has impact on those involved, the impacts and benefits could be short and long term. In order to find out whether your residential had longer-term impacts it is well worth considering how you might collect this information some time after the residential is over.

Students involved in Learning Away residentials were asked to complete pre- and post-residential surveys, along with longer term follow-up surveys to capture their views on the impact of Learning Away. Post-residential surveys were completed back at school, shortly after the residential had ended. Longer-term follow-up surveys were carried out one to two terms after the residential was over at an appropriate time i.e. the students involved were still in school (not always possible for residentials taking place later in Year 6, for example). The staff involved in delivering Learning Away residentials completed pre- and post- surveys. Other staff, not directly involved in delivering the residentials, were also asked to complete surveys, some weeks later, for individual students where Learning Away was felt to have had a significant impact.

Focus groups were carried out a few weeks after the residential, once those involved had completed their post-residential surveys and had time to reflect on their experience.

See below for a section on how to use surveys and focus groups to collect evaluation data.

As outlined above, Learning Away found that asking those involved in residentials (students and staff) and those who didn’t take part but also saw the impacts of these residentials (other staff and parents/carers) to complete surveys was a very useful and effective way of collecting both quantitative and qualitative evaluation data.

This section of the toolkit gives examples of three different surveys and suggests ways in which surveys may be carried out. Many more survey examples can be found in the Resource Library, with lots of additional questions for different residential ‘themes’, age groups and participants.

1. Online or paper (or both)?

Surveys can be completed using paper copies or online, using free online survey tools such as Survey Monkey. There are benefits to using online surveys. They give students additional IT opportunities; data can be collected more quickly and easily; and responses are automatically stored in a survey database, providing simpler handling of data. Significantly, online surveys can also be much quicker and easier to analyse. Data is instantly available so graphs, tables, etc. can be created for you in real time by the online tool. Data can easily be transferred into specialised statistical software or spreadsheets when more detailed analysis is needed.

All of the Learning Away schools used online surveys for Key Stages 2, 3 and 4 students, and staff. However, they found it more practical and effective to ask parents, when appropriate, to complete paper surveys which were sent home with the students as not all families had access to the internet at home.

2. Designing survey questions for different age groups/key stages

It is important to produce different surveys age groups/key stages, with shorter and fewer open-ended questions for younger students. Findings and feedback from the pilot phase of the Learning Away evaluation also suggested that Key Stage 1 pupils found it too difficult to complete surveys and comment on how they might have changed as a result of the residential. We have therefore recommended different ways of collecting evaluation evidence with this age group (see Evaluating with Early Years and Key Stage 1 below).

3. Short term and long term impact questions

As suggested earlier, you may want to design surveys to collect both short- and longer-term evidence of impact. Examples of these different types of survey/questions can be found in the toolkit’s Resource Library.

4. Simple post-residential survey – three examples

You can find examples of three different simple surveys here. These questions can be cut and pasted to create your own surveys.

KS 2 Survey Example – Resilience, Confidence & Wellbeing

KS 3 Survey Example – Resilience, Confidence & Wellbeing

KS 4 Survey Example – Attainment, Progress, Knowledge & Understanding

5. More survey questions – for students, staff and parents/carers

Explore the toolkit’s Resource Library if you would like to use survey examples/questions that cover the following areas:

Key Stage 2 pupils – examples of different pre-, post- and longer-term surveys/questions covering these themes:

• Attainment
• Knowledge, skills and understanding
• Engagement with learning
• Relationships – teacher/student and student/student
• Resilience, confidence and wellbeing
• Cohesion and sense of belonging
• Student leadership, co-design and facilitation
• Transition

Key Stage 3 and 4 students – examples of different pre-, post- and longer-term surveys/questions covering these themes:

• Attainment
• Knowledge, skills and understanding
• Engagement with learning
• Relationships – teacher/student and student/student
• Resilience, confidence and wellbeing
• Cohesion and sense of belonging
• Student leadership, co-design and facilitation
• Transition

Staff – examples of different post-residential survey questions covering:

• Impact on students
• Impact on teacher/student relationships
• Impact on pedagogy and skills

Parents/carers – post-residential survey

• Impact on their children

These example questions can also be cut and pasted to create your own pre-, post- and longer-term surveys, as needed or appropriate to your evaluation.

Don’t forget to give each participant their own personal identification code/number if you want to track the changes that have taken place to individuals pre- and post-residential. Space for them to give their PIN should therefore be added as you create your surveys.

6. Writing up your data

The data collected by surveys can be written up in a number of ways. The document below suggests and gives examples of three simple options:

• Tables
• Infographics
• Illustrative quotes (using free text responses from both surveys and focus groups)

Surveys – Wring Up Suggestions

Don’t forget… photographs taken on the residential will add colour and context to your write-up.

1. Why use focus groups

Focus groups (or group interviews) can capture multiple views and opinions on events. They enable participants to develop their thoughts, based upon their interactions with others and are a useful way of collecting additional evaluation data as they:

• Act as a further opportunity for reflection and learning for both staff and students.
• Are a great way of getting more in-depth views about your residential from students and staff.
• Help students and staff explore issues in more detail.
• Help to verify survey findings, at the same time as adding depth and colour to these.
• Can be a fantastic source of examples of transformational change, as well as quotes, that bring your evaluation evidence alive.
• Help you to share the impact of your residentials with others in an engaging way that they can relate to – everyone loves a story!

2. How to run a focus group – the process

Focus groups are easy to run, but there are a few basics to get right.  These are our top tips:

a) Before the focus group

• So that data remains impartial, focus groups are voluntary – staff and students need to be asked to attend if they would like to, not required, and can leave at any point.
• Think about how many people would be a good number – too big and some people won’t speak, too small and you lose the flow of conversation and ideas that make focus groups so valuable.  During Learning Away, we found that around six people worked well in a group, whether staff or students (although bigger can work with staff if you want to include everyone who went on the residential).
• When thinking about which students to involve, there is a range of options. You could, for example:
  • Choose a group randomly
  • Use the ‘snowball’ approach where you ask two or three students and ask them to bring a friend
  • Choose a group specifically based on the focus of your conversation.  If you’re not choosing people at random, the group should be representative of your school population.
• If you are choosing a group specifically, go for participants who will bring you a range of views – not just those who are going to say everything was great.  You need people who are going to be good reporters i.e. those who will tell you what they really think, not what they think you want to hear.
• In our experience, doing focus groups with Key Stage 1 was not effective for collecting the data that we were looking for, so we developed a specific evaluation tool for this age group (see below).
• For students in Key Stages 2 to 4, we found that a mix of activities and conversation worked well rather than solely conversation (see the Activities section for further details).  Whilst our Key Stages 5 and staff focus groups just used conversation, you could also choose to use a mix of activities and conversation with these groups too.
• People need to know what they are coming to so that they can decide whether they want to take part or not, so being clear with them about: the purpose of the focus group; how they will be recorded; how anonymous their input will be; and what will happen to the data is important.  This is called ‘informed consent’ and we have provided an example of what the information to gain this consent might look like which you can find in the resources section of the toolkit.
• Choose a limited number of areas for discussion. 
You can’t cover everything in a focus group, so decide on the specific areas you would really like to find out more about and plan your questions around these.
• When planning the venue, find a neutral space in the school where you won’t be disturbed – you won’t want to have the flow interrupted once it gets going.  Making the room welcoming for participants can make a difference – consider using round tables (and providing refreshments always goes down well).
• We found that student focus groups tended to last about 45 minutes, and staff groups a bit longer – allowing around an hour for these seemed to be about right.
• You will also need to plan how you are going to record the group.  You could use a video camera, dictaphone or smartphone. Most smartphones have good enough microphones to pick up a small group if you place them on a table in the middle of the group, but test this first to make sure. Also, you may find that placing a dictaphone or smartphone on a notebook rather than directly onto a table gives you a better sound quality.
• Make sure you have all of your resources ready – we found that both primary and secondary students liked to have a range of colouring pens to use, and that the thin felt pens were the most versatile as they allowed for both drawing and writing.

b) During the focus group

• At the beginning, make sure that:
  • All participants knows who you are and why the group is being held
  • Everyone is clear about the information for informed consent and check everyone is happy to be there and be recorded – also make it clear again that the group is voluntary and people can leave at any time
  • The group understands that everyone’s views are equal, even if they are different (so allowing everyone time to speak will be important).
• Give a brief outline of what the group will be asked to do (e.g. activities, answering questions) and an idea of how long the focus group will last.
• Have all of your activity materials ready and to hand so that you can maintain the flow.
• Do a quick recording check to make sure your voice recorder is working and is in the right place to pick everyone’s voice up.
• By all means clarify things when you need to, and probe when you want to know more, but make sure you don’t lead – keep your questions open and neutral.
• Make sure everyone’s voice is heard and valued. Group dynamics may mean that some voices tend to be heard more than others are. Facilitation skills are important to keep the balance of voices right.
• Say thank you at the end – seems obvious but it goes a long way!

3. Activities

Activities are a great way of focusing thoughts and generating reflection and conversation about both general feelings about a residential and specific learning.  You can find some examples of activities we used during the action research phase of Learning Away in toolkit’s Resources Library.  We have separated it into two sections, one for Key Stage 2 pupil activities and another for Secondary student activities. Special school staff can choose which activities will work with their students.

These are just examples; you may have existing resources in school or ideas of your own that you can use with very effective results.

4. Analysing and writing up your data

This section is not intended as advice about analysing your data in an academic sense; it is intended as guidance for getting what you need as a busy staff member from your data in the most effective and time-efficient way possible.

• If you are lucky enough to have transcribing software, use it!  You can then use different coloured highlighters to pick out transcribed text relating to (for example) views about specific topics, examples of change, and useful quotes.
• If you don’t have transcribing software, you will need to decide whether you are going to transcribe every word of the focus group, or make a summary.  Whilst a summary wouldn’t reach rigorous academic research standards, it will probably be more than good enough for your purposes.
• For a summary, we would suggest listening to the audio recording and making typed notes under each question heading.  (Typed notes mean you will be able to return to the document and cut and paste different parts of it at any point in the future, which we found invaluable.)
• It can be useful to outline significant themes using a mind-map.
• Also for a summary, make sure that:
  • Everyone’s views are captured in your write-up (e.g. five of the six students said that they learnt about working as a team from the den-building activity; four of the six students said that they felt more confident about solving angles-related maths problems).
  • You note particularly interesting examples of change.
  • You transcribe verbatim any strong quotes that you will be able to use in presentations or case studies.
• Begin the writing process, using quotations from interviews, interwoven with 
interpretation.

Participant observation is when the evaluator ‘joins in’ with the group s/he is studying to gain a deeper insight into their experiences. It can be an effective means of exploring and assessing the impacts of the residential on your students. It can help you get a truer picture; what students say they do or feel when answering survey questions or talking to you in a focus group may not always be the same as what they really do or feel. Participant observation can therefore be a fairly simple additional evaluation approach. The information gained from doing this may be analysed on its own, or it may help you to contextualise focus group and survey data. This is our suggested six-step approach to carrying out participant observation:

• Identify the group of students you want to observe. It will be difficult to observe all the students on your residential.
• Design your approach. Participant observation can be overt (the participants know they are part of a study) or covert (they are unaware they are being observed).
• Decide how you will record your observations. This may involve taking photographs, using a notebook, and/or recording interactions using a voice recorder or video camera.
• Carry out the observations.
• These observations can generate a wide variety of data, so begin to organise and categorise your data. Highlight themes as they emerge. 
It will be easier if you begin this process early on.
• Explore and interpret the data by reflecting on what themes emerge most prominently and significantly.

Evaluating outcomes for children in Reception or Key Stage 1 requires an innovative and creative approach. Older children may be able to articulate why activities worked well or why they did not, or talk about the effect activities had on them. Younger children are less likely to be able to provide this information through evaluation methods such as surveys, focus groups, interviews and diaries.

The Evaluating the impact section of the  Early Years and Key Stage 1 resource on the Learning Away website provides both a simple evaluation process and a more enhanced evaluation model (inspired and informed by Alison Clark and Peter Moss’ Mosaic Approach). The Mosaic Approach is a creative framework for listening to young children’s voices through a range of practical activities.

1. Blob trees

Many Learning Away schools found Blob Trees very useful as they developed their residential programmes and carried out their evaluations. A Blob Tree is a tool designed to help students think and talk about their feelings within a specific context, for example their feelings about themselves and others before and after their residential experience. They can also be used therefore to stimulate discussion during focus groups with Key Stage 2 and 3 students. For further information and to purchase downloadable versions of this tool go to www.blobtree.com.

2. Diamond 9 activity

The Diamond 9 tool is an evidence collection activity designed to help people collectively explore several issues by prioritising them collaboratively. It supports a focussed discussion in a relatively short space of time. It is an interesting tool to use with students during a residential as part of your evaluation (or with students as you co-design the residential itself). It can offer insights into how your students see their learning, what aspects of their learning context they think makes it effective, and will help you to consider what aspects of the residential were particularly helpful for achieving the aims of the residential.

A Diamond 9 tool helps people to prioritise and categorise key factors related to the research questions. The most important factors are placed towards the top of the Diamond 9. The least important factors are placed towards the bottom. Factors of equal importance are placed in the same row. Each factor can be colour coded for further sophistication. Download a Diamond 9 sheet from the Learning Outside the Classroom website here.

As an evaluation activity the Diamond 9 tool works particularly well during the later stages of a residential.  For example, you can ask students to take photos of their learning environment/activities/context and use a Diamond 9 format to rank these to establish what or who best supported their learning. The ranking part of the activity can happen either during the residential (if you have printing facilities in your residential setting or if you use ‘instant’ cameras)  back at school, where the pictures can provide a good stimulus for an evaluation conversation.

If your residential has a focus on improving engagement with learning, behaviour or attainment and progress (e.g. to improve performance and help achieve higher than predicted GCSE grades at Key Stage 4 or improve literacy scores in Year 6), it will be important to collect additional pre- and post-residential quantitative data. This evidence will support and add value to that collected by student surveys. You may therefore want to collect

• Attendance data
• Behaviour data
• Appropriate attainment and progress data

If you want to find out whether any improvements have been sustained over the longer term, don’t forget to build the use of longer-term data into your evaluation plan. This will also help you to consider how you reinforce and embed the learning that happened on the residential once you return to school.