My 2c on all this.

Long ago, I took Psych 101 at Cornell.  With several thousand other=20
students.  Needless to say, the tests were all multiple choice.  The=20
professor had been teaching it for years and had developed a large pool=20
of multiple choice questions so that this semester's test#2 would have=20
(mostly) different questions than last semester's test#2, etc.

However, for every test some new questions would be developed and put=20
in.  And even though this was the late 80s they did in fact have=20
computers back then and the professor used statistical analyses to check=20
how good the new questions were.

For example: if there was an unusually wide spread of wrong answers,=20
i.e. if the correct answer was (c), but there were a substantial number=20
of people who answered (a), (b) and (d), or the correct answer was (c)=20
but lots of people answered (b), then that is a clue that the question=20
is badly written and/or the teaching material related to what the=20
question was testing wasn't very good.

You can go further of course, and see what students who did well on the=20
"tried and true" questions did with the new ones.  If they did well,=20
then (at least they) could understand them well.  If not, that's a=20
pretty big clue the question(s) need improvement.

The professor would mention after each test if there was a question that=20
was "thrown out" because the statistics showed is was confusing.

If you think about it a little, having several thousand questions and=20
thousands of students semester after semester could yield some=20
interesting long-term data.

I remember very little of Psych 101, but nearly 20 years later, this=20
nifty approach to iterative development stuck with me.

It was "big-data analysis" back when the hot computer to have for=20
research was the Mac SE.  :)

J



Dwayne Reid wrote:
> At 02:36 AM 1/5/2015, William \"Chops\" Westfield wrote:
>
>> I'm actually pretty happy that people are arguing over subtleties in
>> my proposed answers, rather than thinking that the questions are poor :-=
)
>
> This is a very interesting discussion and I greatly appreciate it.
>
> I mentioned that I have already provided criticism for their existing
> pool of questions.  Much of that has been discussed here: poorly
> worded questions leading to ambiguity, more than one possible correct
> answer, too narrow a focus (questions relating to experience that
> only a fraction of people might have), etc.
>
> I also appreciate the lists of questions provided.  I'm going to take
> those with me as examples of questions that may want to be included.
>
> Please keep the suggestions coming!
>
> dwayne
>
--=20
http://www.piclist.com/techref/piclist PIC/SX FAQ & list archive
View/change your membership options at
http://mailman.mit.edu/mailman/listinfo/piclist
.