This presentation describes two attempts to employ different online services for recruiting research subjects. Subjects were to engage in an experiment to evaluate teaching behavior. Specifically, the study is an extension of previous research on the abilities of observers to distinguish between high and low performing teachers, as defined by their history of improving student learning. In previous published experiments subjects from a variety of backgrounds evaluated short teaching samples presented in both video with audio and audio only where it was established that, regardless of background, observers were no better than chance at correctly placing teachers in the high and low groups. However, there was considerable agreement on their choice of teachers. The present study aimed to extend this work to the use of lesson transcripts, where subjects were randomly presented with either audio or transcript lesson samples and asked to judge whether the given teacher was in the high or low group. Whereas previous experiments had invited subjects into a research lab to make judgments in person, the present study experimented with online subjects recruited through Amazon’s Mechanical Turk (Mturk) and through a UK company called Prolific. We compare the two procedures including the recruitment process, the cost, and the quality of the results we obtained. We provide examples and a discussion on the advantages and disadvantages of each procedure, and of using online versus live subject recruitment.

Authors:
Michael Strong, Texas Tech University, United States

About the Presenter(s)
Dr. Michael Strong is a Research Scientist at the College of Education, Texas Tech University. His general interest is research on teachers and teaching. His current projects are related to teacher evaluation and the use of AI.

See this presentation on the full schedule – Saturday Schedule

Bookmark