Got a number of things to say about testbanks, including a sort of “justification” for my use of them and several comments about how I use them. But I thought I’d post a “tidbit” about a specific testbank system. In a way, it’s more “practical” than other things I have to say about testbanks and practical things fit more in the “tidbit” concept I’m using here. I even thought about doing a tutorial. But tutorials are time-consuming and I’m not sure it’d be more useful or useful to more people than this less-structured post.

EZ Test Online (EZTO) is, as you might expect, an online system to facilitate test creation. More specifically, it’s the online version of the EZ Test program developed by McGraw-Hill to distribute the testbanks which accompany its textbooks as “anciliary material.” Almost all of the features of the standalone version of EZ Test are available in EZTO and the overall user experience is quite similar. Haven’t used the offline version in a while but, if I remember correctly, it also used a Web browser (though it could be used without an Internet connection; basically, it was using the local machine as a simplified webhost).

Obvious advantages of the online version over the offline one:

• You can work from any machine.
• You can share testbanks.
• You can have students take tests directly in EZTO.

Obvious disadvantages:

• You need an Internet connection.
• In some circumstances, it might be slightly less secure than an offline program.
• In some ways, you’re sharing data with McGraw-Hill (though their privacy policy doesn’t really specify much about this).

In my case, EZTO makes a good deal of sense. I tend to use “the cloud” as much as possible, these days, and EZTO is convenient in that sense.

In a way, I wish McGraw-Hill were to go further with EZTO. As with Learning Management Systems, I can easily imagine some connection with the “social Web.” Of course, most people would prefer testbank items to remain protected, hidden, and private. But I can still imagine the sharing of testbanks between instructors to become similar to sharing Web content generally.

In fact, I’m slowly coming to terms with a notion of “transparent testbanks.” Basically, it would mean that all potential questions would be available to the wide public, instead of being restricted to instructors. It’s an extreme case, and one which I think most instructors would think is crazy. But I think it might actually make it easier to control such things as cheating, as counterintuitive as it may sound. Even though I’m typically in favour of radical transparency, this is really an edge case and I’m not that attached to it. But knowing that many exam questions are already shared among students, putting all questions in the open (and, one would suspect, making sure that some very similar questions would be part of those testbanks, to prevent mere memorization of the answers) could have actually make this practice of private sharing of questions more impractical.

After all, there are already some sample questions available in diverse places, including on open access websites created by textbook publishers. These questions are typically meant to help students prepare for exams and quizzes. Using the same questions on exams may have the effect of encorcing rote learning. But if there are enough questions in a centralised testbank which serves both as a practice and as a source for exam questions, the result may actually be to avoid using questions which were already seen by individual students.

But, again, I’m not pushing this model. I’m just wondering about the effects. Part of the reason is because of things like information about the “Rorschach test” having been publicised. Haven’t had to deal with this issue directly and what I’ve heard about it was rather minimal, but the fact that test items can now be shared easily means that we need to think about the effects of sharing. After all, even if one is to limit testbank access to instructors of a given course (with all the necessary processes of verification and validation), it only takes one hacked account to make the whole thing insecure. And even the most careful (and even paranoid) of instructors can be the victim of some form of hacking, including “social engineering.” So the assumption that testbank items can be protected clashes with the ease of “leaking content” online. A transparent testbank is a way to flood potential cheaters with an overwhelming number of potential questions, which is similar to disinformation strategies currently used to protect some private information, including trade secrets.

Even in the case of a “transparent testbank,” I imagine much metadata to remain private. In this I include success rates for each question, answers given by individual students, and comments about use of the question. Other pieces of metadata could potentially be shared too, including “source information” about each question (chapter or page number in a textbook), question type, and even aggregate ratings. These can make testbanks look like so much “user-generated content” online. Which could have some advantages.

I didn’t originally plan to make this tidbit about those issues, but I’ve been thinking about them a bit and it’s probably better to put them “out there.”

What I did plan to talk about is how I use EZTO.

So, in some courses I teach, I do use McGraw-Hill textbooks which are accompanied by testbanks in EZ Test format. In fact, I just finished creating an exam using such testbanks.

This is for an online course using Moodle. Students take the actual exam through Moodle. Students have also been taking reading quizzes, with questions from the same testbanks. The midterm exam focuses on understanding. Students have a full week to take the exam (i.e., there’s no set schedule for everyone to take the exam at the same time). In this context, it seems necessary for each student to receive a different set of questions as other students in the class (otherwise, shared questions would guarantee a near-perfect grade for every student who sees them). With individual versions of the exam, risks related to questions being shared are still a matter of concern, but the effects are much less damaging. (In fact, I haven’t noticed any significant effect from questions being shared.) All questions are multiple-choice and they are graded automatically in Moodle.

The key thing, here, is that I want to have a big pool of questions for the exam (to make sure that very little overlap exists between two versions of the exam) yet I want to make sure the questions are as appropriate as possible. The testbanks I use have many questions which are really about rote learning, and that’s not the kind of exam I want to give. I’m also trying to get a nice balance in terms of the types of questions asked. Otherwise, I’d just dump all the testbanks in Moodle and select the appropriate number of questions from the whole set of testbanks.

I’ve used several methods in the past but I think I found the most useful one in this context. There are likely other methods which work well for other people and/or other contexts. But it’s the one I’m likely to use, for the time being.

Here’s what I did… (This is where a tutorial would make sense!)

• In EZTO:
  • Uploaded all the testbanks to EZTO (they’re separated by chapter).
  • Created new tests for each question category (according to the testbank): definitions, perspectives, etc.
  • For each test by question category, added questions of that question category from all the chapters. In other words, collected all the questions about definitions from each chapter by using “Select Questions from Banks” with each chapter-based testbank.
  • In each question category testbank, created a new category to select questions.
  • Assigned that new category to questions which seemed appropriate for the exam.
  • Created new testbanks for the selected questions from each question category (again, using the “Select Questions from Banks,” but using the new category to select the questions).
  • Checked and edited each testbank of selected questions by question category.
  • Exported each bank of selected questions by question category to Blackboard format.
• In Moodle:
  • Created new question categories and new folders to reflect the exported EZTO testbanks (e.g., a folder and question category for questions about “definitions”).
  • Uploaded the files exported by EZTO (“bboard.zip”) into the new folders.
  • Unzipped these Blackboard export files to get multiple files which are all called “res00001.dat” (that’s why it was so important to separate in folders).
  • Imported questions from each “res00001.dat,” making sure the appropriate category was selected.
  • Created a “quiz” activity, with a description of the exam, some information about the process, and some settings (can only be taken once, time is limited to 60 minutes, etc.).
  • Editing the exam, chose each question category and added a number of random questions from that category.
  • Changed the total on which grades will be calculated (60 points total for 60 questions).
  • Previewed the exam.
  • Made the exam visible to students.
  • Announced the exam.

A fairly convoluted process, but one which provides me with remarkably appropriate results. Actually, other attempts I’ve made, in the past, were even more convoluted. For instance, noticing that several questions revolved around the same concept or issue, I would create subcategories from which only one question was randomly picked. It was very time-consuming and the rule of only having one question per issue or concept was self-imposed. Even with a lot of cautious work, it’s almost impossible to get each version of the exam cover exactly the same topics in exactly the same depth as every other version. Given a sufficient number of questions, one can do as if these things even out. Not ideal. But reasonable in this context.

Selecting questions is neither difficult nor that time-consuming. A lot of questions are easy to eliminate, using a few “rules” I have about exam questions (including “no question about factoids from the text”). There are enough questions in the testbanks I use that I need not worry about having enough for the exam. For this midterm, I had more than 800 questions in the relevant testbanks. I selected a bit less than half of these questions as being appropriate for the exam. The exam contains 60 questions. So, basically, I threw away more than half of the testbank questions and ended up with six times the number of questions for each version of the exam. Unless I’m mistaken, the chances that two students have almost identical versions of the exam are low enough to be insignificant (in a group of 25 students). And, though the questions come from a testbank, they’re appropriate for this specific group.

Much of this could be done through another testbank platform. And, of course, the questions could come from a testbank I’ve created. But, in this case, the quality of the questions I selected is sufficient for me to be rather satisfied. I’ve created other exams through similar methods in similar contexts and the results seemed rather appropriate. If problems arise, it’s still possible to work through issues without losing my sanity.

If I sound like I’m justifying myself, it’s probably because I am. There’s something strange about using testbanks, especially when exam versions are randomised. I wouldn’t use the same strategy in other contexts but this is a course for non-major given purely online. It’s also a course for which the textbook really is the main source. And it so happens that the testbanks are rather decent, containing a good number of questions that I find very appropriate.

The parts which are specific to EZTO mostly have to do with how questions are selected and exported. This is a situation in which I wish I had access to ratings for each question, and some way to bulk process the kind of question I don’t want. (For instance, I exclude all questions which have “All of the above” or “None of the above” as the correct answer.) I’d also want to get some way to assess the overall difficulty of the exam so that I could tweak it up or down depending on what seems to be needed.

I also feel like I’m getting too close to standardized tests. In fact, I keep thinking about the GRE and TOEFL (the two standardized tests I’ve taken). I used to react strongly against them but it sounds as if I were getting closer to them, in this context. Exams in my other courses differ significantly from that model, thanks in part to open questions. And I could technically apply some of the same principles in this case, though it’d make things more difficult for everyone. Yet the reasons I use randomized questions from testbanks have to do with the conditions in which I teach this course, much beyond convenience and time commitment.

In such a context, McGraw-Hill’s EZ Test Online makes some sense. It’s “good enough” as a solution to a given task in a very specific set of conditions.

I wouldn’t use it too extensively, though.