Key Question :What is the Punjab Examination Commission?

Brief Answer: It is an autonomous body set up by the Punjab Government to administer annual examinations to administer annual examination in the Punjab to grade 5 and grade 8.

Key Question : What was wrong with the old system?

Brief Answer: Analysis of Old exam system indicated that:

Link to extended response: Ken Vine MISSION REPORT FEB 2006

Key Question : What does the new examination system try and achieve?

Brief Answer: The new examination system will serve as a database to provide advice to a wide variety of stakeholders within the education sector. It will provide advice to the Minister, District superintendents, Curriculum developers, Schools, parents and students. More specifically its objectives include:

Link to extended response: Ken Vine MISSION REPORT FEB 2006

Key Question : Who sets the Examination Papers and what format do they take?

Brief Answer: A committee of subject specialists representing, teachers, Punjab Text Book Board, and Universities has been appointed and given the responsibility of setting examination papers under the coordination of the Operations Manager of PEC. There are three steps in this process. The first is to develop a Subject design matrix to specify the content and question difficulty to be included in the examination. The second is to prepare a Pilot examination that is tested using approximately 2000 students. Finally, the annual examination paper is prepared, along with a similar ‘model paper’ for sharing with schools.

For grade 5 each exam paper is 1 hour long and consists of some multiple choice questions and some open response questions

Key Question : How is the analysis of the student responses under taken? Is it any different from the traditional “mark” given to students?

Brief Answer: While we do provide a “mark” for students, we also try and give greater meaning to this mark. The traditional mark system is what is technically referred to as a “norm referenced” system that gives us a mark and a rank for each student, but tells us very little about what it is the student can do. That is, what outcomes the student has acquired. The new system of analysis attempts to link outcomes to groups of students in what is known as a “criterion referenced” system.

The approach taken is known as Rasch Modeling, and we use Quest software to perform the analysis following the entry of student responses into an electronic database.

Link to extended response: Analysis Presentation Lahore

References: Adams R.J. & Khoo, S-T. (1996) Quest: The interactive test analysis system, Version 2. Camberwell: ACER.

http://www.rasch.org/
Bond and Fox

Key Question : We understand that PEC is introducing a new conceptual framework to support the design of examination papers and the design of marking schemes for the open-ended questions.

Brief Answer: Yes, this is so. Traditionally, test developers have used Bloom’s Taxonomy to encourage the use of questions of different difficulty. While Bloom has served us well and will continue to do so, there is some literature now available that suggests it does not describe student’s cognitive growth. (For example see Hattie article in the links below). If we ask “knowledge” or an “evaluation” question (Bloom’s terms) we can still expect a variety of qualities of responses from a cohort of students. In an attempt to classify these response qualities and the make explicit the differences between them we have introduced a conceptual framework that has grown out of the work of Piaget, and was developed by Biggs and Collis in 1982 known as the SOLO (Structure of Observed Learning Outcomes) Taxonomy. This conceptual framework has undergone considerable development since 1982, but thus far we are applying the relatively simple original version

Link to extended response:
SOLO Presentation
Solo Taxonomy: A Theoretical Framework
Hattie Critique of Bloom
Web site
http://www.tedi.uq.edu.au/downloads/Biggs_Solo.pdfsearch=%22SOLO%20taxonomy%22

References Biggs, J. and Collis, K. 1982. Strucure of Observed Learning Outcomes. Academic Press.

Key Question : How the grades of A, B, C, D and E determined

Brief Answer: We calculate the mean and standard deviation of all the students. We then take a band, 1 standard deviation wide about the mean and students lying in this band are awarded a grade C. A Band one standard deviation wide above the grade C band is the Grade B band. Students above grade B are awarded grade A. In a similar manner, students below Grade C are awarded grades D and E

Link to extended response: Subject Variable Map

Key Question : Why do you use transformed scores rather than raw scores?

Brief Answer: There two level at which we can respond to this question. The first is a highly technical. The Rasch model converts raw scores, which are an ordinal scale, to a interval level scale using logits as a unit of measurement of student ability and question difficulty. Initially these scores are located on a scale with a mean near 0 and range of approximately +4 to –4. To make the scores more meaningful to the non-statisticians amongst us, we use the Quest software to convert the logit scale to a range of zero to 50. This transformation retains the essential psychometric properties of the original logit scale, and the mean and standard deviation will depend on the original mean and standard deviation.

Link to extended response: http://rasch.org/rmt/rmt32b.htm

Key Question : What will the student reports look like, and will they include competency statements?

Brief Answer: The student reports will contain personal details of the students, subject marks and grades. It is hoped that in the near future the outcomes that the grades represent will be included on the report.

Link to extended response: Sample Student Report

Key Question : Who can get information from the PEC and how do they go about getting that data?

Brief Answer: A policy on this matter is being developed. This is not a straight forward matter as there are issues of confidentiality involved.

However there are a set of standard reports being developed that will automatically be provided to students and parents, schools, district offices and specialist units within the Ministry of Education

Key Question : How can my organization find out more about the work of PEC. Does PEC provide training for appropriate organisation