ARBs in the ASSESSMENT MIX

Sally Boyd

In this article, two recent case studies are used to give examples of how Assessment Resource Bank (ARB) resources, along with other assessment tools, are being used for both formative and summative purposes in a primary and intermediate school. Staff at both these schools engage in a detailed analysis of their formative practices, and the data gained from summative assessments, to ensure that the results of assessments inform learning and contribute to improving classroom practice.

In June 2003, as part of the ARB contract with the Ministry of Education, NZCER researchers visited Upper Hutt School and Takapuna Normal Intermediate School for a day. They interviewed a number of staff (mostly frequent ARB users) about assessment practices at their school, and collected examples of ARB use.

The two schools

Established in 1864, Upper Hutt School is a contributing Decile 7 school located north of Wellington. The school has approximately 320 students, 12 full-time teachers, and 4 part-time teachers. Takapuna Intermediate, on Auckland’s North Shore, was established in 1970. It is a Decile 10 school with approximately 620 students, 21 classroom teachers, and 6 specialist art and technology teachers.

The assessment mix

Teachers and school managers at both schools said they felt they were “over-assessing” students. To address this concern, both schools are currently engaged in a process to refine their assessment programme:

…it [the review] had to come as teachers were overloaded…A key issue for teachers is manageability – in the past we have been guilty of overkill. We have been working towards cutting down the workload and refining and refocusing on what is important. A huge part of that is student feedback.

As part of this process, these schools are both involved in whole school Assess to Learn (AtoL) contracts with local providers. The focus of these contracts includes:

•&&&&a review of each school’s assessment focus

•&&&&an increased emphasis on the key areas of literacy and numeracy

•&&&&the further development of formative assessment practices

•&&&&the development of more child-centred classroom practices and an increase in student involvement in decision making about learning

•&&&&for Takapuna Intermediate, a move towards electronic reporting, to enable more systematic use of classroom data to meet schoolwide reporting needs.

As a result of these contracts, some teachers reported that they have changed their practice, while others have affirmed their beliefs about teaching:

The formative assessment contract made a huge difference to me personally – it’s altered the way I plan as a teacher. I used to go to the resource room and select resources…now I look at the resources and ask, “Do they teach the children what I want them to learn?”

The contract has given that [formative assessment] a real emphasis. Before it was ad hoc – now it is really formalised [teachers are aware of when they are using formative assessment practices] …It fitted in really well with my philosophy and affirmed the notions [I had about] children negotiating criteria, reflecting, and recognising the next steps for their learning.

Along with other assessment tools, ARB resources are being used in the drive to improve assessment practices at both schools. ARB tasks are used in various ways to provide feedback to students about their learning, and are used for schoolwide assessments, the results of which inform classroom programmes.

The assessment mix at Upper Hutt School

Upper Hutt School has a strong schoolwide assessment focus. It uses a number of externally developed assessment tools, including ARB resources, to assist in this programme. Other tools used include School Entry Assessment (SEA), the Supplementary Tests of Achievement and Reading (STAR), National Exemplars,¹ and National Education Monitoring Project (NEMP) tasks (see Flockton and Crooks, 1999; Crooks and Flockton, 2002).

The principal noted that over the last few years, results in literacy and numeracy have been improving. Staff consider that these changes can to a large extent be attributed to the information they gain from the assessment programme, which is used to plan next step teaching.

As part of this programme, a timetable is produced each year which outlines the key assessments to be undertaken in each curriculum and essential skills area. Schoolwide information is collected in the key areas of literacy and numeracy. This information is used to identify strengths and weakness and priorities for teaching. As the principal noted: “every review tells us something, and we change something”.

As part of this programme, ARB resources are used in two main ways at the school: to support formative assessment in writing, and to develop schoolwide tests in mathematics. These two uses are described below.

Schoolwide formative uses of ARB English resources at Upper Hutt School

In 2002, as part of the AtoL contract, the staff had a schoolwide focus on writing skills. They made use of writing tasks and exemplars from the ARBs and the National Exemplars Project. As part of this focus, the ARB poetic writing task, “The Worst Haircut Ever” (WL3238 – see box), was completed by all middle and senior school students at the start of the year.

Using the ARB poetic² and transactional³ writing levels-based scoring guides, teachers assessed students’ writing for surface and deep features. (For more information about these scoring guides, see Croft, McDowall, and Rapson, 2002.)

The senior literacy teacher looked at all the writing samples, and noticed discrepancies in teachers’ interpretations of students’ use of surface and deep features. As a result, teachers met to discuss examples of their students’ writing. A shared understanding of what they were looking for, and a common language to talk about writing features, emerged from these meetings. Teachers reported that these discussions had helped to focus their approach to teaching surface and deep features.

Teachers used the annotated exemplars of students’ work from the ARB resources, “Should Animals be Kept in Zoos?” (WL3655), and “The Giant Egg” (WL3249), as models for students, to show them the features to look for when they edited their work. “The Giant Egg” and one of the associated student exemplars are shown in the boxes.

Students were also given a version of a student’s text, typed up without the teacher’s annotations, to practise editing. One teacher described how she uses examples of students’ work from her class, her own models, and the models from “The Giant Egg” to help students to develop editing skills. She discussed how much her students enjoy using other students’ work.

During 2002, students completed the persuasive writing task, “Should Animals be Kept in Zoos?” (WL3655). At the end of the year, another task was used as a comparison to assess growth in writing skills, and the impact of the schoolwide focus on writing. In the junior school, a similar exercise was conducted with persuasive writing exemplars from a National Exemplars Project trial.

All the staff we talked to were extremely positive about this focus on writing skills, and discussed how rewarding it had been, both for students and for themselves. These teachers outlined how using the ARB levels-based scoring guides and the annotated examples of students’ work had made marking across the school more consistent. They also considered that this in-depth focus on writing, and being able to view students’ work from other schools, had raised their expectations of what students could do, and had increased students’ motivation and interest in writing.

Schoolwide mathematics assessment at Upper Hutt School

For the last four or five years, all Year 3 to 6 students at Upper Hutt School have completed a schoolwide mathematics test in number (computation/problem-solving), and all Year 4 to 6 students have completed a test in place value. These tests are repeated each year, to allow comparisons of the cohorts over time. The principal used ARB resources and some NEMP tasks (see Crooks and Flockton, 2002) to construct the tests.

The number test is developed from the ARBs, using a search tailored to specific objectives, and the difficulty level and diagnostic information available on the ARB scoring guides. Resources are selected that reflect the school’s current focus, and cover the range of difficulties present on the ARB scoring guides. Level 2 and Level 3 number tests are developed, containing approximately 20 questions; about 55 percent are “easy”, 30 percent “moderate”, and 15 percent “difficult” or “very difficult”.

A scoring guide for each test is developed from the ARB scoring guides. The overall percentage correct for each student is recorded, along with the percentage of students getting each question correct. This information is commented on, and a summary is presented to teachers. It includes:

•&&&&percentage correct for individual students, and by room, gender, ethnicity, year level, and total cohort

•&&&&percentage correct for the cohort in comparison with previous years

•&&&&students showing “flair and ability” or “requiring support and guidance”

•&&&&information about the questions that students found the most difficult

•&&&&teaching priorities for later in the year, based on an analysis of the areas that students found easy or difficult

•&&&&information about strategies to use in teaching to the areas students found difficult.

The results are used to evaluate the level students are working at. Individual results are discussed with students, and the information from these assessments is reported to parents. The summaries are presented to the Board of Trustees, and comparisons with the ARB performance data are made, to show how students at Upper Hutt School compare with other students nationally.

The assessment mix at Takapuna Intermediate

Like Upper Hutt School, Takapuna Intermediate also has a strong schoolwide assessment programme, which contains both formative and summative components. The science and mathematics teams make extensive use of ARB resources. Other externally developed tools used for assessments in other curriculum areas include the STAR and Progressive Achievement Tests (PATs), the Essential Skills Assessments: Information Skills, NEMP (see Flockton and Crooks 1999), and National Exemplars tasks.

At Takapuna Intermediate, ARB resources are used in two main ways in the schoolwide programme: as performance-based tasks in science, and to develop entry and exit tests in science and mathematics. These uses are described below.

Schoolwide formative uses of ARB science resources at Takapuna Intermediate

The science curriculum team review their schoolwide science plan each year. They have instigated a number of schoolwide initiatives which use ARB resources. One of these is a focus on “fair testing”, which is a key area in the science plan. To support teachers in teaching and assessing this area, the science curriculum team has produced a “fair testing” kit which includes ARB tasks, tasks designed by teachers, and equipment. Students complete at least five pen and paper or performance tasks. Some of the tasks provide scaffolding for students; for example, MW5673 requires students to test different makes of paper towels, to see which is the best at mopping up spills. Other tasks require students to design their own “fair test” and discuss their plan with their teacher, for example, MW5670, part of which is shown in the box

Some of the tasks in the “fair testing” kit are used as teaching and learning activities, and others as assessment tasks. The results of the assessment tasks are recorded, and students’ work is included in their portfolio.

Schoolwide assessments at Takapuna Intermediate

As part of the assessment programme, Year 7 entry and Year 8 exit assessments are completed by students in a number of key areas, including mathematics, science, and English. In the second week of term one, the new cohort of Year 7 students complete science and mathematics entry tests. Later in term one, students complete STAR and PAT tests to give information about their development of reading and listening skills. Students also complete exit assessments at the end of Year 8.

Pre and post tests in science

ARB tasks are used by the Head of Department (HOD) Science to develop the Year 7 entry and Year 8 exit tests (these have been completed by all students for the last two years). The HOD Science noted that before the entry and exit assessments began to be used, there was no collation of schoolwide data on achievement in science, and little information about what science was being taught at the school, or how. The criteria⁴ used to select ARB resources for these tests are that they:

•&&&&fit within the key science areas selected by the school and with the National Exemplars project Doing Science Matric (Draft)⁵

•&&&&cover contexts used in the school programme – for example, students at Takapuna Intermediate have been engaged in a cross-curricula unit on bicycles, so the science context chosen covered wheels, cogs, and motion

•&&&&use New Zealand contexts, for example, native bush

•&&&&have diagnostic information

•&&&&are resources at level 3 (for the Year 7 test) and level 4 (for the Year 8 test)

•&&&&include questions that have a range of ARB difficulty levels, with the bulk having a difficulty level of “moderate”

•&&&&include a range of resource types, that is, selected response and short and long answer questions.

In 2003, for the first time, students will complete the same science test at the start and end of the year. Teachers plan to report the end of year results by each year cohort, strand, gender, and ethnicity. Summaries will also be produced for teachers and school management by class, gender, and curriculum strand (within each class).

Pre and post tests in mathematics

For the last three years, ARB and NEMP (see Crooks and Flockton 2002) tasks have been used by the HOD Mathematics to develop Year 7 and 8 entry and exit tests. These tests have been refined each year. In 2003, students in the two year levels will complete the same test at the start and end of the year. The tests include ARB resources that assess the same skills, but at different levels; for example, the Year 7 test uses resources at level 3, and the Year 8 test uses level 4, to assess students’ ability to calculate the mean, median, and mode of a distribution.

The first two sections of the tests (basic facts and the four operations) are NEMP tasks. The third section includes two number, one statistics, one algebra, and one measurement word problem from the ARBs. The ARB word problems that are selected incorporate diagnostic information, are mostly short answer, and have a range of different layouts and contexts.

The average score for each year cohort is recorded, and summary graphs are produced for the three main sections of the test, in order to enable trends to be more visible. Summaries are also produced for teachers and school management, by class, gender (within each class), and ethnicity.

The HOD Mathematics noted that each year, the tests have provided information to improve programmes. For example, previous test results highlighted that both Year 7 and Year 8 students were having difficulties with division. As a result, the HOD suggested some changes to the teaching programme, and sourced resources to support the teaching of division.

Use of the information gained from the schoolwide programme at Takapuna Intermediate

The results from both the formative and summative elements of the schoolwide assessment programme are used in a cyclical way to improve teaching and learning programmes, and for reporting, as shown in the box.

To contribute to the improvement of the teaching programme, resourcing, and reporting, Takapuna Intermediate is currently using, or has plans to use, information from the schoolwide entry and exit assessments in mathematics and science for a variety of other purposes:

•&&&&To gather information about each new cohort of Year 7 students, so teachers can group students according to their learning needs, as well as identify those who need extension or extra support.

•&&&&To identify overall strengths and weaknesses. The diagnostic information available on the ARB scoring guides is used to suggest priorities for next step teaching, and resources are identified to assist teachers to teach to these priorities.

•&&&&To provide information about the curriculum levels students are achieving at, so as to support school resourcing at particular levels.

•&&&&To refine teaching programmes and select key areas to focus on.

•&&&&To compare delivery effectiveness between the integrated, composite, and traditional classes.

•&&&&To report to students on their achievement and provide information for their portfolios.

•&&&&To compare the achievement of Takapuna Intermediate students with students nationally. This is achieved by comparing Takapuna Intermediate students’ results with ARB difficulty level information and NEMP data.

•&&&&To provide an analysis of growth over time. This is achieved by comparing students’ end of year results with the achievement targets set by the school.

•&&&&To provide data on student achievement for reporting to parents, school management, the Board of Trustees, and the Education Review Office.

Other uses of ARB resources in the two schools

At both schools, ARB resources from all three curriculum areas are used by teachers for a variety of other purposes, some of which are described below:

•&&&&At Upper Hutt School, senior teachers select ARB tasks from the English, mathematics, and science banks to include in units to assess key areas. From the ARB scoring guides, teachers select the main features they are assessing against, and produce a coversheet to attach to each completed task to show students’ progress. This assessment is shown to students and their progress is discussed.

•&&&&As part of the integrated studies programme at Takapuna Intermediate, students are given the opportunity to suggest areas which they are interested in including in the teaching programme. In 2003, students identified “the future” and “genetics” as areas they wanted to learn more about. The science curriculum team then used the ARBs to find resources that matched students’ interests and their teaching priorities. To provide teachers with resources about “the future”, a set of three ARB tasks about rockets – testing the fuel, designing rockets, and rocket flight (MW6338, PW3665, PW3666) – were made into a booklet and given to all teachers.

•&&&&At Upper Hutt School, students use the ARB scoring guides for self and peer assessment. First, the teacher leads a discussion about the key features of the scoring guide and the language used, to ensure that students have a shared understanding of the criteria they are looking for. Students then assess their own or others’ work, and discuss their decisions, either with each other or with their teacher.

•&&&&At Takapuna Intermediate, ARB resources are used as extension tasks or as teaching resources to support students working at level 2. For example, level 2 and level 3 word problems are used to support students identified as having difficulties in mathematics. Resources such as PW3651 (which assesses observational skills) are used with the ESOL reception class as integrated science and written language tasks.

•&&&&At both schools, ARB tasks are used: in students’ portfolios and to discuss students’ progress with parents; as models for practical tasks and new contexts in science or mathematics; to provide professional development for new teachers, so as to strengthen their understanding of core curriculum areas; and as teaching resources, for example, as starter activities for group work, as worksheets, or as homework.

Using the ARBs when planning

The amount of use of the ARBs varied widely among staff at both schools. Some teachers used the banks frequently and were very positive about the resources. Others had looked at the banks once or twice and/or had other assessment tasks they preferred to use. On the whole, most of these users were either senior teachers who selected resources and passed them to other teachers, or new teachers who had been introduced to the banks while training. Most of the teachers we talked to preferred to access the ARBs at home while they were doing their planning.

The frequent users at Takapuna Intermediate described how they looked at the banks every time they planned units of work. Some teachers had a group of websites they browsed to get resources or ideas when they planned (the ARBs, English Online,⁶ New Zealand Maths: problem-solving,⁷ and TKI⁸). The teachers at Upper Hutt School noted that they returned to use their favourite resources. Like the teachers at Takapuna Intermediate, they were also constantly searching for new resources.

As teachers usually had a particular area in mind when they accessed the ARBs, they therefore tended to use searches which combined strand, level, and keyword to find resources directly related to this area. Teachers outlined how the search menus made it very easy for them to find the resources they wanted:

[Using the ARB search screen] I can search specifically for a level, and I like the way the fields are set up for searching so I can narrow it down – like the strand or whatever I am interested in.

Benefits of ARB use

The main benefit teachers at both schools derived from using the ARBs was access to new resources which are used to develop and refine their schoolwide and classroom assessment programmes. Teachers commented that these resources would have been time-consuming to develop themselves. The teachers at both schools discussed how the ARB resources provided new and novel ideas for activities and contexts to use in the classroom.

At Takapuna Intermediate, ARB resources played an important role in assisting teachers to gather information on new cohorts of students. Teachers were very positive about the confidence ARBs use gives them with regard to their schoolwide programme, and the way the diagnostic information included on many resources helps them in targeting their next teaching steps and priorities for resourcing:

They are so handy, they are ready made and so easily accessible…they are already in levels for our curriculum…the fact that they have next steps – diagnostics. …You know it is a valid site and you are using valid information which is going to assist in learning.

They [the ARB resources] come up with heaps better ideas than I could!… I like the way they are presented [with the graphics].

At Upper Hutt School, teachers commented on how the ARBs gave their assessments more credibility, gave them more confidence to interpret the achievements of their students and plan their next teaching steps, gave them a wider perspective on student achievement, and helped them focus on the needs of their students:

You can pick and choose bits and pieces from the ARBs that fit your particular objectives. The ARBs provide a better match with the learning objectives that we are focusing on… They give a “ring of confidence” I probably wouldn’t have had a reading response [comprehension] sample [for students’ portfolios] until the ARBs came along.

At both these schools, getting the assessment mix right is an ongoing challenge for staff. Consequently the staff at both schools give assessment, and its relationship to teaching and learning, a lot of thought and attention. They are engaged in a programme to examine and use the range of assessment strategies and tools available, in an effort to find the right mix for their school.

One of the key features in how these two schools approach assessment is a strong emphasis on using both formative and summative schoolwide assessment results to feed into next step teaching and to improve teaching practice. For example, the diagnostic information available on ARB scoring guides is used to focus the classroom programme, and the ARB scoring guides are a starting point for discussions about what constitutes a “surface” or “deep” feature in written language.

Another key feature is the way in which assessment information is used to report to a range of audiences, for example, to provide information to include in students’ portfolios, or to show parents and the Board of Trustees how students are progressing.

Further reading

To read the full case studies, go to: http://arb.nzcer.org.nz/nzcer3/casestudies.htm

To use the ARB website, go to: http://arb.nzcer.org.nz/nzcer3/casestudies.htm

To read more about the range of ways in which ARB resources can be used for schoolwide assessment, see: Croft, C. (1999). School-wide assessment: Using the Assessment Resource Banks. Wellington: New Zealand Council for Educational Research.

References

Croft, C., McDowall, S., & Rapson, G. (2002). Levels-based assessment of writing: Scoring guides from the Assessment Resource Banks. set: Research Information for Teachers, 1, 31-37.

Crooks, T., & Flockton, L. (2002). Mathematics: Assessment results 2001: National Education Monitoring Project. Wellington: Ministry of Education.

Flockton, L., & Crooks, T. (1999). Writing: Assessment results 1998: National Education Monitoring Project. Dunedin: Educational Assessment Research Unit, University of Otago.

Notes

1&&&&http://www.tki.org.nz/r/assessment/exemplars/index_e.php

2&&&&http://arb.nzcer.org.nz/nzcer3/ENGLISH/WRITTEN/scoringg/poeticwriting.htm

3&&&&http://arb.nzcer.org.nz/nzcer3/ENGLISH/WRITTEN/scoringg/transactionalwriting.htm

4&&&&To read more about the criteria to use when developing tests see: Croft, C. (1999). School-wide Assessment: Using the Assessment Resource Banks Wellington: New Zealand Council for Educational Research

5&&&&http://www.tki.org.nz/r/assessment/exemplars/sci/science_matrix_e.php

6&&&&http://english.unitecnology.ac.nz/

7&&&&http://www.nzmaths.co.nz/

8&&&&http://www.tki.org.nz/e/tki/