Pilots

Stay on top of the latest developments with our blog on the homepage.

Piloting of the Assessment Resource Project

Piloting

Phase 1

Phase 2

Piloting

The Assessment Resource Project ran two separate piloting phases to test out the materials. The first phase was done entirely at Bow Valley College with learners in a variety of literacy, academic upgrading and post secondary careers classes. The issues that arose from the first phase were addressed and the revised versions of the tests were used in the second phase. This second phase was completed with a variety of partner organizations across Canada who work with adult learners.

Phase 1: Local Pilot

• Test A (3 tests)
• Test B (all 5 tests)
• Test C (all 5 tests)
• Test D (all 5 tests)
• Test E (3 tests)
• Test F (all 5 tests)

We approached the pilot in two ways. We worked with one instructor to embed the tests of Test B into the curriculum. This provided her learners and us the opportunity to see their progress from the pre-test through a practice test to the post-test. For all the other tests, we administered them one at a time with individual classes.  Due to delays in development and challenges in scheduling piloting opportunities at the end of the term, we revised our aim for Test E and Test A. For Test E we chose to only pilot 3 of the 5 tests. Test A was piloted concurrently at Bow Valley College during the national pilot.

We returned to each of the classes who wrote a test with individual feedback forms for all the learners and then conducted a group debriefing. This gave us the opportunity to ask the learners about their experiences with the test and the feedback and gave the learners the chance to express any concerns we may not have thought to ask.

The feedback from both learners and instructors was overwhelmingly positive. Learners are immediately seeing the connection between the texts and questions in the text and their day-to-day lives. This connection has sparked an interest in continuing to use the resource we are developing. Instructors have also commented on their learners’ level of interest and engagement. One instructor who has used 3 tests (from a single level) with her learners said students progressed much more quickly with their skill development than she is used to seeing. An issue with timing arose from the piloting of Test E and Test F; most learners commented that there was too much reading to complete the test in the given one-hour timeframe.

The piloting process resulted in a number of changes to the resource that were implemented before the second phase of the piloting began. These changes included:

• improved wording of questions
• altering confusing graphics
• fixing typos
• switching the balance of texts in a test
• reordering the texts in a test to alternate between prose and document texts
• reducing the number of texts per test from 8 to 6 for Test E level and Test F level
• reducing the number of questions per test from 30 to 20 for Test E level and Test F level
• increasing consistency with layout choices
• removing and replacing questions and texts that were deemed unsuitable

Phase 2: National Pilot

• Test A (all 5 tests)
• Test B (all 5 tests)
• Test C (all 5 tests)
• Test D (all 5 tests)
• Test E (all 5 tests)
• Test F (all 5 tests)

In the quest for piloting partners, we were looking for a variety of organizations and a wide geographic distribution. This was done to make sure the resource was appropriate to use in a variety of contexts including community programs, workplace programs and colleges. Additionally, our intension is that the resource represents the everyday reading of learners across Canada; by piloting in cities and towns from various provinces and territories we were able to gather feedback as to whether or not we reached this goal. After receiving interest from many different people, we partnered with eight other organizations. All partner organizations has a minimum of one person attend a training session on how to administer the test and participate in the pilot. This training included: how to select and prepare learners, how to administer the test, how to mark the tests, how to provide feedback to the learners, history of the project, details about the segmentation used for the test and how it corresponds to the International Adult Literacy Survey (IALS) framework, and how to gather and submit feedback for the pilot. The details of all partner organization who participated in the pilot are outlined in the chart below.