Can we make the assumption that C@S data is a population?

Notes from Pip Arnold

At any level I would accept either C@S or NZ as being an acceptable population for data from C@S.  When we first started we got ourselves tied up in knots around having the year and NZ and C@S.  Almost more important is that the students recognise that the particular cohort that has been sampled from is year 11 for example, or secondary school students.  In fact… NZ girls and boys is not as precise as Yr 11 girls and boys, but better NZ Yr 11 girls and boys.

I think we should be having the discussion at year 12 about the C@S database and how it is a subset of all students in yrs4-13 and how there may be some biases, so you can get overall population stats and discuss if they match expectations.  For example, there are usually more girls than boys, and there are usually more junior students than senior students.  The schools come from a broad range of deciles so that isn’t really an issue.  In terms of representing NZ at a particular year level and for a particular gender I think that C@S is pretty good.  However if you were looking at teenagers (without the gender split etc) you might have a bit of an issue because of the weighting to girls and junior secondary.    Most of the time the students are looking at a year level and selecting boys and girls separately.

To be fair the annotated exemplar below reflects the thinking at the time and the absolute best population, but I would also accept all those I listed.   i.e. yr 11 NZ, yr 11 C@S, yr 11 NZ C@S, yr 11 NZC@S 2009, yr 11C@S 2009, (and probably yr 11 also though with this one I would give the student an opportunity to tidy up).

So I don’t agree that Yr 11 NZ C@S 2009 is the only correct population, there are at least four other versions that would be acceptable.  If you want to get down and dirty into describing target populations it would be better to use other populations.

The following material is share by Pip Arnold.  It is part of her thesis which she is currently writing.  The copyright for this material rests with Pip.  Feel free to share but please acknowledge the source.  Queries can be directed to Pip at pip@karekareeducation.co.nz July 2012.

“Descriptions of populations

In the initial classification for summary and comparison questions, both had the top category as being the “good” question plus the population.  As student pre- and post-test responses were analysed from the second (2008) and third (2009) teaching experiments it became clear almost immediately that the “super” category of population was not going to work.  Students who had similar types of questions had a wide range of populations.  For example, in the 2008 post-test 22 of the 24 students posed an investigative question about one group being taller than another group. Aside from the variation in the question format, 14 different population or group descriptors were used.  The descriptors fell into three main categories: (1) boys and girls; (2) various combinations of age groups; and (3) Year 11 boys and girls.  Table 6‑1 lists the descriptions used and the number of students who used the description and the sample came from the 2007 New Zealand (NZ) Census At School (C@S) database and only year 11 students had been selected.

Table 6‑1 Population descriptors in 2008 post-test responses for questions about “being taller”

Sorted by	Group or population descriptors	Number
Boys and girls	Sample from Boys and girls, 2007 NZ C@S	1
	Boys and girls	1
	Boys and girls, 2007C@S	1
	Boys and girls, 2007 C@S	2
Ages with and without boys and girls	Sample from 15 yr olds and 16 yr olds, 2007 NZ C@S	1
	15 yr olds and 16 yr olds	1
	16 yr old girls and 15 yr old boys	2
	15 yr olds and 16 yr olds, 2007 C@S	1
	15 yr old girls and 15 yr old boys, 2007 NZ C@S	2
Year 11 boys and girls	Yr 11 girls and yr 11 boys	1
	Yr 11 girls and yr 11 boys, 2007 C@S	2
	NZ yr 11 boys NZ yr 11 girls	1
	Yr 11 girls and yr 11 boys, NZ C@S	2
	Yr 1 girls and yr 11 boys, 2007 C@S	4

Within the three broader categories there are multiple ways that students can phrase a descriptor and this is based around whether they acknowledge that the population is New Zealand students, and that a particular Census At School database has been sampled from.  It could be possible to make a fine graded scale for population descriptors, but pragmatism and what would be useful to teachers and students meant that fewer categories were better than more.

Initially there seemed to be three clear categories: (1) boys and girls, this is very general and could mean all boys and all girls in the world; (2) New Zealand boys and girls is better, but generally the data came from a sub group of New Zealand boys and girls, giving (3) New Zealand year level(s) boys and girls as the target.   The three categories are:

1.      Broad student population, for example, boys, girls, students.

2.      Broad New Zealand student population, for example, New Zealand boys, New Zealand students.

3.      Actual New Zealand student population, for example, New Zealand Year 10 students, New Zealand Year 11 students, New Zealand secondary school girls.

However, as can be reasonably expected student responses did not fall nicely into the three categories.  Where for example did Year 11 boys and Year 11 girls fit.  Clearly it is more specific than New Zealand students, but it doesn’t specify New Zealand.  An additional category was needed between broad New Zealand student population and actual New Zealand student population.  This category covers any age or year group that could reasonably be assumed from the year group that was selected for the sample.  For example, as well as accepting year 11 students (2008 pre- and post-tests and 2009 pre-test), 15 year olds and 16 year olds were also accepted; as well as accepting secondary students (2009 post-test and 2011 pre- and post-tests), any combination of years 9-13, ages 13-18 and teenagers were accepted as well.

The other two categories that occurred and that didn’t fit within these four were: (1) the rare times when students went broader than boys and girls and used either no population descriptor, for example, what are typical heights, or used male and female or people; and (2) students who specifically or inadvertently posed their investigative question about the sample, for example, what are typical heights of these year 11 students, what are typical heights for year 11 students sampled from the 2007 NZ C@S database.

The year and the Census At School database did add a further dimension, as can be seen from the examples in Table 6‑1, but in the end these were not considered a requirement.  It may have been better in hindsight to have just said the sample was of New Zealand secondary students, or New Zealand year 11 students.  If a student wrote 2007 C@S, this was considered to be equivalent to having written New Zealand.

Ultimately there were six population categories that are considered as part of the overall question classification.  These categories held through analysing the questions posed in the fourth teaching experiment and are relevant for both summary and comparison questions and would be relevant for relationship questions too.  The final six population categories are:

1.      Referring to the sample.

2.      Broad population, not specifying students.

3.      Broad student population, for example, boys, girls, students.

4.      Broad New Zealand student population, for example, New Zealand boys, New Zealand students.

5.      Any relevant student population that can be generalised about from the actual New Zealand student population used, for example, year 11 students, teenagers, secondary school girls.

6.      Actual New Zealand student population, for example, New Zealand Year 10 students, New Zealand Year 11 students, New Zealand secondary school girls.

In the teaching and learning the focus was always on using the category six type populations, but in assessment situations category five would be an acceptable population, but not category four as there needs to be some recognition of the limiting factor of the specific age group selected in the sample, the fact that they are New Zealand students is not enough by itself.”