FAST Method to Positive Behavior Support - Phase I Report
Martin R. Sheehan, Ph.D., Principal Investigator
1 R43 HD044336-01A1
Project Period: 2/01/2004-7/31/2004
Program Title: The FAST Method: Reducing Problem Behaviors in the Classroom, Grades 7-12
Summary
The idea behind The FAST Method was to develop staff training materials that would present clear, easily understood behavior intervention strategies for responding effectively to challenging student behavior.
Through a grant (*Grant # 1 R43 HDO44336-01A1) from the National Institute of Child Health and Human Development, an IRIS Media research and development team, along with expert consultants, gathered data, developed a video, tutorial, assessment material and print materials based on the data, and tested the effectiveness of the program.
To test program effectiveness, 18 secondary-school teachers used the FAST method video and 18 teachers were given written material on functional analysis and positive behavior support. After using the materials, teachers in both groups completed an assessment to evaluate their mastery of the material.
When compared to a control group, study participants had a higher degree of confidence in their mastery of the material and were more enthusiastic about the video-based training than were participants who used written material. Of particular significance for busy teachers was the finding that FAST method users were able to master concepts in far less time than were control group members.
Thirty-six teachers completed the study. Eighteen teachers used the FAST method materials (intervention group: IGN), and 18 used written materials on positive behavior support and functional assessment (control group: AT).
Project Aims
This Phase 1 project had the following as the primary goals: 1) to develop content materials, with consultant review, for the first IMM module, TheFast Method; 2) to conduct formative evaluation of IMM materials and finalize development of IMM materials based on consultant review and formative evaluation; 3) to produce IMM materials and print materials, develop outcome instrument, and conduct performance tests; 4) to conduct a pilot test of the effectiveness of the program with a sample of 40 secondary school teachers.
Research Results and Significance
All four specific aims were accomplished. The final product is in Appendix B. As will be discussed in more detail below, teachers not only learned the FAST material in an efficient manner, but were also very positive in how they rated the product.
Project Changes
After the Phase I project was funded, but before we started to work on the Phase I tasks, Dr. Dan Baker, who was the P.I., took a faculty position as an Assistant Professor of Pediatrics at the University of Medicine and Dentistry of New Jersey. The Research Associate on the project, Dr. Martin R. Sheehan, took over the role of P.I. Dr. Sheehan has extensive experience in working with schools in the area of behavior management and was also very involved in the original writing of the grant application.
Develop content for an IMM module that simplifies the functional assessment process.
Project development staff worked on the content, design, and usability of an IMM module, which was titled The Fast Method. The IMM program, based on current research in effective principles of behavioral interventions, consisted of a video training program; an interactive tutorial; a web-based, interactive self-assessment; and supplementary print materials. The goal of the developers was to create a demonstration-driven instructional experience that was engaging to participants and would make a clear, simplified presentation of skills. IRIS staff members who were involved in the content development consisted of the Principal Investigator, the Research Associates, the Instructional Designer, and the Media producers.
Conduct formative evaluation of IMM materials and finalize development of IMM materials based on consultant review and formative evaluation.
Two focus groups were conducted to determine the program?s efficacy, utility, and value. Two focus groups, Group A and Group B, attended by the Principal Investigator and Research Associate, were moderated by each respectively, with the other taking observational notes. Following the focus group sessions, project staff met to debrief and analyze the information collected in the groups. The results of this meeting were written up by the Research Associate and shared with the project staff and scriptwriters on the desired direction of content development.
Focus Group A consisted of 10 urban-metropolitan teachers who discussed teachers' needs for behavior support, discussed the program?s utility potential, offered feedback on the FAST concept and script ideas, and generated video vignette scenarios. The teachers? demographics broke down into the following categories: a) 9 female, 1 male; b) 8 Caucasian, 1 Hispanic, 1 declined to state; c) had 1-14 years' teaching experience; and d) 6 had at least a master?s level degree. Participants discussed a variety of student behavior problems and contexts, which became the foundational scenarios for FAST instruction (e.g., working with the disengaged student, responding with relationship-building, demonstrating sensitivity to students? cultural differences, and dealing with power struggles.) The participants? skills in behavior support varied greatly, but all agreed that the FAST concept could provide useful support to teachers, particularly new teachers and those who don?t have strong student-behavior management skills. When asked if they would view such a program, all the teachers said they would be interested.
Focus group B consisted of 6 teachers in the Eugene metropolitan area who reviewed the FAST script content for efficacy, utility, cultural appropriateness, and interest. The 6 teachers were: a) 3 male and 3 female; b) 4 Caucasian, 1 Hispanic, and 1 declined to state; c) 2 white, 1 Native American, 2 African American, and 1 declined to state; d) had 7-30 years teaching experience; and e) all but one had at least a masters' level degree. All teachers found the dialogue and interactions between the characters to be realistic and believable, as well as culturally sensitive and appropriate. Participants? responses were used to conduct a formative evaluation of the product. Suggestions from participants included specific points to emphasize, wording to be changed, and overall content/pacing. Overall, 100% of participants found the program to be "very useful" or "extremely useful" to teachers in general, teachers who need help with student discipline, and "teachers like you." This group also gave feedback on the outcome instrument (see below in Task 5).
Script development. After the focus groups, the Principal Investigator and Media producers collaborated on scripts for the components of the IMM module that aimed to simplify the functional assessment process.
Further formative evaluation. The FAST content and video were reviewed for formative evaluation by the Project Consultants?2 expert consultants in education and 5 consulting secondary-level educators. Consultants were overwhelmingly positive in their feedback of the FAST script, citing the script?s engagement aspects, its straightforward presentation and easy-to-implement strategies, and its cultural-appropriateness. In addition, the Consultants? detailed feedback was used to further refine the program and address technical concerns about the use of the web and streaming video.
Produce IMM and print materials, develop outcome instrument and conduct performance tests.
Produce Phase IMM and print materials. The production team was responsible for producing the Phase I interactive multimedia program, The FAST Method, an eDVD which consisted of a video training program, an interactive tutorial, and interactive web assets. An eDVD or enhanced-DVD is a hybrid technology that can be viewed on a home DVD player or used interactively on a computer with DVD-ROM and Internet access. In its interactive form, the DVD disk provides a user with seamless access to interactive web assets and the World Wide Web. A supplementary print manual was also developed. Samples of The FAST Method eDVD are included in Appendix B (Product).
Video Production. The 17-minute video is built around three principal situational vignettes that present problem behaviors in the classroom and offer teachers opportunities for analyzing the functional goals of behavior and arriving at an appropriate response. Utilizing a diverse cast of students and teachers, the video presents a strategy (the FAST method: Find, Assess, Select, and Track) that teachers can employ to objectively identify and respond to student misbehavior. Using an array of techniques (dramatic storyline, quick-paced editing, action, and skill-building routines), the video presents viewers with multiple examples and opportunities to learn how to solve common student problem behaviors quickly and effectively. The project team, with assistance from Dr. Jeffrey Sprague and five consulting secondary level teachers, developed the materials for the curriculum.
Interactive Tutorial . The tutorial material, available on VHS, DVD, and eDVD, summarizes principles and provides problem-solving sequences on the three behavioral goals addressed in the Phase I program: power, attention, and avoidance. Illustrative video clips of student behavior show students and scenarios that are different than those used in the video program or self-assessment to ensure that viewers have opportunities to generalize skills and knowledge from the instructional video program to other contexts. Due to media limitations, the VHS version is not interactive; viewers observe scenarios and are presented with appropriate analysis/responses. DVD and eDVD versions allow viewers to select responses to questions related to live-action video examples of student behaviors.
Self-Assessment Tool. The self-assessment uses three video scenarios to test users? grasp of concepts and their ability to generalize skills and knowledge. All scenarios are new to viewers and do not appear in any other material in the program. Users? responses are scored and explanatory material is presented when answers are incorrect. Users have the option of reviewing the related video scenario with added explanatory narrative if they have not done well on the particular questions.
Interactive Web Assets. FAST Online provides a combination of information and instructional resources. For users who do not have access to a computer with a DVD-ROM drive, the self-assessment tool described above is available at FAST Online. In addition, printable summary charts and a FAST method quick reference card are available in PDF format. Links to appropriate resources are also included.
Print Materials . Print materials for the Phase I FAST method program include a viewer?s guide, implementer?s guide, and a FAST method quick reference card. The 18-page Viewer?s Guide summarizes background and principles of the FAST method and provides detailed information on each of the three behavioral goals covered in the program. Charts, diagrams, and photos are used extensively to make information as clear and memorable as possible. The implementer?s guide is designed to quickly inform administrators and trainers about the assets in The FAST Method program and provide guidance in how to access them. The quick reference card shows the four steps of the FAST method on a small card that teachers can keep handy to cue themselves about appropriate responses.
Development of an Outcome Instrument.
The Principal Investigator, Research Associate, and Project Methodologist developed an online, video-enhanced outcome instrument that presented common classroom problem behaviors using three video and three written stimulus scenarios. The scenarios were accompanied by a series of questions measuring the teacher's ability to analyze the situations using strategies from the intervention materials. To develop the scenarios, we began by asking teachers in Focus Group A (see above) about the types of situations and interactions most common and most difficult for them (e.g., girls who talk back). Teachers were encouraged to provide specific examples and to work in small groups to develop a written outline of a problem interaction. This information was later used by the development team in drafting the assessment scenarios. Teachers in Focus Group B reviewed these scenarios and offered feedback on the relevance, accuracy, and cultural appropriateness of each scenario. Following this, scripts were revised.
The final assessment instrument consisted of 3 video vignettes and 3 brief written vignettes. Two vignettes demonstrated avoidance situations, two demonstrated power situations, and two demonstrated attention-seeking situations. Following each vignette, teachers responded to six questions. Five of the six questions were open-ended, requiring written answers; the sixth question asked teachers to rate, on a Likert-type scale, how confident they were that their analysis of the situation (Intervention group) or testable hypothesis(Alternate Treatment group) was correct. The five open-ended questions were different for each group and reflected the different content provided to each group. Teachers in the Intervention group were asked: 1) Describe the problem behavior; 2) What did the teacher feel and do? 3) How did the student respond?; 4) What's the goal of the student's behavior?; and 5) How could the teacher respond appropriately to the student? Teachers in the Alternate Treatment group were asked: 1) What is the problem behavior?; 2) What is the antecedent to the problem behavior?; 3) What is the consequence for the problem behavior?; 4) Why do you think the behavior is occurring?; and 5) What might the teacher do differently? Note: After the experimental test was completed, the video vignettes were included in a release version of the DVD and are listed in the main menu as ?Self Assessment.? In the evaluation version of the test, open-ended answers are required; for the self-assessment version on the DVD, the questions have been rewritten in a multiple choice format. For the written vignettes, see Appendix A.
Answers from the open-ended questions were scored by two research assistants using a consensus methodology. The Principal Investigator developed criterion answers for each question, based on the different content for the two groups. The research assistants were trained on both sets of materials and the assessment scenarios. Each research assistant scored each written answer independently. They then met to compare scores; any differences were resolved by discussion to reach a consensus; if they couldn't reach a consensus, the Principal Investigator resolved the disagreement. Only two answers had to be resolved by the Principal Investigator. Answers were scored either 0 (incorrect) or 1 (correct), except in the case of Intervention group answers to question 2 (What did the teacher feel and do?). Since two responses were called for, a score of .5 was given for partial answers. Scores were computed as the sum of correct answers for each scenario.
Performance Test.
Beta testing of FAST materials was conducted with 5 teaching experts in Oregon and California, all of whom were experienced secondary teachers with advanced degrees in Education. Beta testers viewed the FAST video program and used the online FAST assessment, then responded to an interview of open-ended and multiple choice questions. Beta testers responded very positively to the FAST program?s content and accessibility (using a Likert scale with 1 as lowest and 5 highest in overall reaction to the program: high quality content 4.6, ease of use 4.2, satisfaction 4.8). Two of the five (40%) users noted problems playing the DVD. One respondent commented that difficulty was likely due to the age of the computer/system used to view the program; another mentioned the way the opening sequence developed made it difficult to discern which action to choose. Three users (60%) reported technical problems with the online (website) portion of the program (video-enhanced) assessment: One had to restart their system a number of times, another had difficulty getting the video to play (user thought difficulty might be related to Quicktime video-viewing program), and the third reported problems loading on Windows Media Player. Feedback on technical issues led IRIS staff to consider additional computer platform and web-based access issues, and results were incorporated into the final FAST product.
Conduct a pilot test of the effectiveness of the program with a sample of 40 secondary school teachers.
To demonstrate the feasibility and effectiveness of this project, we conducted a pilot test using a two-group (Intervention/Alternate Treatment) post-only design. The aim of this activity was to examine whether teachers in the Intervention Group would demonstrate a greater mastery of the materials, use less time to master the materials, and rate the materials higher on consumer acceptance factors than teachers in the Alternate Treatment group. Teachers in the Intervention group received the FAST materials (a DVD containing The FAST Method presentation, an interactive tutorial, and a viewer?s guide); teachers in the Alternate Treatment group received written materials (Chapters 2 and 3 from Building Positive Behavior Support Systems in Schools [Crone and Horner, 2003]).
The initial sample pool consisted of 44 secondary school teachers from 11 different states. These teachers were randomly assigned to one of two groups, an Intervention group (IG:N=22) and an Alternate Treatment group (AT:N=22). After receiving the materials through the mail, teachers were instructed to log in to a website when they had completed the program/reading. On the website they completed an outcome assessment, consisting of questions related to usage of the materials, consumer reactions to the materials, and a test assessing their ability to apply their knowledge by analyzing common problem-behavior situations.
Approximately 2 weeks following the online assessment, teachers were asked to respond to an email questionnaire containing five questions: 1) Have you used any of the techniques you learned from the materials? 2) If yes, can you give an example? 3) Have you gone back to review any of the materials? 4) Have you shared the materials with friends or colleagues? 5) Do you have any other comments you'd like to share with us?
Teachers in both groups were paid $125.00 -- $100.00 for completing the online assessment and $25.00 for completing the email questionnaire.
Recruitment .
Participants for the study were recruited using a variety of methods. The Principal Investigator and Research Associate used professional contacts to announce the study. We also conducted online searches for school addresses and email addresses of principals and teachers and selected likely participants from the IRIS Media database. Principals were sent flyers and emails announcing the study, and teachers were contacted directly through email. In total, we sent over 900 emails and 60 flyers to likely participants in 14 states ( Oregon , Colorado , Pennsylvania , New York , Illinois , Arizona , Michigan , Alabama , California , Arkansas , Nebraska , Hawaii , and New Mexico ).
Interested teachers responded via email or phone to arrange a time for a screening/intake phone interview. To be eligible for the study, teachers had to have a high-speed internet connection, access to a DVD player, and no previous training or experience using functional analysis and positive behavior support techniques. After determining eligibility, an audio-taped verbal informed consent was obtained. Eligible teachers were then randomly assigned to either the Intervention or Alternate Treatment group.
Sixty-five teachers responded to the recruitment efforts. Twenty one were not eligible for the study (12 had pre-existing knowledge of functional analysis/positive behavior support, 2 had no high-speed internet access, 1 did not have access to a DVD player, and 6 contacted us after the deadline for participation). Of the 44 teachers who began the study, 36 completed the assessment (IG N=18; AT N=18), an 18% attrition rate. One teacher of the eight who failed to complete the assessment had technical difficulties; despite our best efforts, we could not overcome those difficulties. Of the remaining seven, we identified reasons why four didn't complete the assessment (e.g., personal problems, scheduling conflicts, etc.); three simply never logged into the website and did not respond to reminder emails we sent.
To determine whether there were any systematic differences between teachers who completed the study and those who didn't, we examined demographic variables ranging from gender, ethnicity, and race to years of teaching experience. The only significant difference (Chi square(1)=6.13; p=.01) we found was that six of the seven non-completers were high school teachers. There was no significant difference in completion rates between the two groups.
Sample Characteristics.
Chi square and ANOVA were used to examine whether there were significant demographic differences between the two groups. No differences were found. The two groups displayed similar percentages of high school and middle school teachers (IG-56% middle school teachers: AT-57% middle school teachers) and were similar in gender distribution (IG-78% female, AT-75% female). Both groups were primarily non-Hispanic (IG -100%: AT -91%) and Caucasian (IG -87% Caucasian, 4% African-American, 9% Native American/Caucasian; AT -95% Caucasian, 5% Native American/Caucasian). It should be noted that these ethnic and racial distributions approximate the national racial/ethnic distributions of secondary school teachers, who are 95% non-Hispanic, 86% Caucasian, 6.5% African-American, 1% Native American or Alaska Native, and 1.3% Asian American, Native Hawaiian or other Pacific Islander (Kleiner & Farris, 2002). The two groups were also equivalent on educational background (IG -9% BA/BS, 26% post-graduate work, 50% MA/MS; AT -9% BA/BS, 24% post-graduate work, 48% MA/MS) and years of teaching experience (IG -mean 10.6 years (s.d. 8.2); AT -mean 10.7 years (s.d. 7.8). These teachers taught many different academic subjects; over 32 different subject areas were mentioned, ranging from math, English, and social studies to computers, vocal music, and 3D animation. They had access to high-speed internet connections at both home and school (14% only at home, 46% only at school, and 41% at both home and school). Most had DVD players on either a home or school computer; 32% had a set-top player and players on both their home and school computers, 22% had both set-top and DVD players on a school computer, and 21% had both set-top players and DVD players on home computers, while only 25% had only a set-top player.Results.
Table - Teachers' Mastery
| FAST | Alternative Treatment | |||
|---|---|---|---|---|
| Mean | s.d. | Mean | s.d. | |
| Vignette 1 | 4.3 | 0.9 | 3.7 | 1.0(a) |
| Vignette 2 | 3.0 | 1.2 | 3.4 | 1.4 |
| Vignette 3 | 3.2 | 101 | 3.8 | 1.0 |
| Vignette 4 | 4.2 | 0.9 | 3.8 | 0.9 |
| Vignette 5 | 4.0 | 1.2 | 2.8 | 1.2(b) |
| Vignette 6 | 4.1 | 1.1 | 3.7 | 1.0 |
| (a) = p<.10 (b) = p<.01 | ||||
Mastery. Teachers? mastery of the material was measured by their scores on the six assessment vignettes. Each vignette was followed by 5 open-ended questions requiring teachers to analyze a problem behavior situation and describe what would be an appropriate response by a teacher. Scores for each vignette were computed by summing the number of correct answers to the 5 questions. The scores on the individual vignettes were highly inter-correlated (e.g. .47 to .70), so a MANOVA was used to examine differences between the two groups on scores from the assessment test. The overall effect for Group was significant (F(6,27)=27, p=.002). Univariate ANOVAs revealed that on one vignette, the Intervention group had a significantly higher score than the AT group (F(1,32)=8.76, p=.006), and scores on three of the remaining five vignettes were higher for the Intervention group, though none of these differences were statistically significant. Table 1 displays the mean score for each group.
Table 2 - Teachers' Confidence
| FAST | Alternative Treatment | |||
|---|---|---|---|---|
| Mean | s.d. | Mean | s.d. | |
| Vignette 1 | 4.2 | 0.6 | 3.3 | .86(a) |
| Vignette 2 | 3.7 | 1.0 | 3.4 | 1.0 |
| Vignette 3 | 3.8 | 0.7 | 3.6 | 0.6 |
| Vignette 4 | 4.0 | 0.6 | 3.4 | 1.2 |
| Vignette 5 | 4.3 | 0.6 | 3.3 | 1.1(b) |
| Vignette 6 | 4.1 | 0.5 | 3.3 | 1.1(c) |
| (a) = p<.001 (b) = p<.01 (c) = p<.05 | ||||
Confidence . After answering the assessment questions for each vignette, teachers were asked to rate, on a scale of 1 to 5, where 1= "Not at all", 3 = "So-so" and 5 = "Very", how confident they were in their analysis. Since these scores were moderately (.48) to highly (.80) correlated, MANOVA was used to examine differences between the groups. The overall effect for Group was significant (F(6,27)=2.66, p=.037). Univariate ANOVAs revealed that teachers in the Intervention group had significantly higher scores in their confidence ratings for 3 vignettes: Vignette 1 (F(1,32)=13.01, p=.001); Vignette 5 (F(1,32)=8.64, p=.006), and Vignette 6 (F(1,32=6.13), p=.019). Means on the remaining 3 vignettes were also higher for the Intervention group, though none of these differences attained statistical significance. Table 2 displays the means.
Table 3 - Program Efficiency
|
FAST
|
Alternative Treatment
|
|
| 0-1 hour |
22%
|
0%
|
| 1-2 hours |
72%
|
50% |
| 2-3 hours |
6%
|
44%
|
| 3+ hours |
0%
|
6%
|
Efficiency. Teachers in both groups reported how much time they spent accessing/reading the materials, using a Likert-type scale where 1 = "1 hour or less", 2 = "1-2 hours", 3 = "2-3 hours", 4 = "More than 3 hours.? ANOVA was used to examine differences between the two groups. A significant difference (F (1,34)=14.58, p<.001) was found. The mean for teachers in the Intervention group was 1.83 (s.d. .66); for teachers in the AT group the mean was 2.55 (s.d. .62). 94% of teachers in the Intervention group spent no more than 2 hours using the FAST materials, while only 50% of teachers in the AT Group spent less than 2 hours on the material.
Consumer Acceptance.
Information about consumer acceptance of the two sets of materials was gathered at two times. As part of the final outcome assessment, teachers responded to open-ended and Likert-type questions related to consumer acceptance factors; all teachers completed these questions. In the email follow-up assessment questionnaire administered two weeks later, both open-ended and binary questions were asked; 75% (27) completed the email questionnaire (14 in the Intervention group and 13 in the AT group).
Responses to these questions fall into 5 broad categories of consumer acceptance: (1) Stimulation (Did the material get and keep teachers' interest?), (2) Comprehension (Was the material easily comprehensible?), (3) Acceptability (Was the material relevant and useful?), (4) Ownership (Did the audience identify with the messages and did the messages respond to the interests and priorities of the audience? Did the audience feel the language and images reflected their everyday situations?), and (5) Persuasiveness (Was the material sufficiently convincing to achieve attitudinal and behavioral changes?). In addition, teachers were asked which parts of the materials they found difficult to understand and for any suggestions they had about how to improve the materials. ANOVA and Chi Square were used to analyze differences between the two groups on the quantitative variables; thematic analyses were conducted on the qualitative data.
Stimulation . Although a quantitative analysis was not conducted on the stimulation category, an examination of teacher responses indicated a difference between the two groups. While the majority of participants in the AT group found the content ?interesting? or ?good,? several participants noted that the material was ?very academic,? ?not user-friendly,? and text heavy. Intervention group participants were more enthusiastic about their video-based materials. Their responses tended to be more positive, noting they ?loved it,? were ?impressed with the simplicity,? and found it ?a great, easy-to-implement approach.? Regardless of group, teachers seemed to have interest in what they determined was a useful, well-thought out program. In the email questionnaire, teachers were asked whether they'd gone back to review the materials. Only 14% of the AT Group versus 62% of the Intervention group said yes ( Chi Square (1)=16, p<.001).
Comprehension . One Likert-scaled item was related to comprehension: "How much did the program/materials help you understand the function of common problem behaviors?" No significant difference was found between the two groups (Intervention group mean 3.38, s.d. .92; AT mean 3.5, s.d. 1.04). However, 56% of the teachers in the Intervention group responded "Quite a bit" or "Very much" while only 44% of teachers in the AT Group had similar answers. The thematic analysis confirmed this finding. Overall, participants in both groups found the programs to be understandable. Intervention group participants overwhelmingly found the program to be simple and easy to use. Several mentioned that the FAST acronym helped to remind them of the steps to addressing the behavior more effectively, and others mentioned the emphasis on using verbs to label the behavior, not the child.
Acceptability. One Likert-scaled item was related to acceptability: "Would you recommend this program to other teachers?" No significant difference was found between the two groups (Intervention mean=3.9, s.d 1.05; AT mean 4.0, s.d. 1.02). In the Intervention group, 67% of the teachers responded "Probably Yes" or "Definitely Yes"; in the AT Group, 62% responded similarly. In the email questionnaire, teachers were also asked whether they'd shared any of the material with friends or colleagues. Responses in both groups were similar; 50% of the AT group and 46% of the Intervention group said yes.
Ownership. One of the Likert-scaled items was related to ownership: "How realistic were the classroom scenes on the DVD/examples in the reading?" No significant difference was found between the two groups (Intervention mean=4.0, s.d. 1.02; AT mean 4.11, s.d. .90). 72% of teachers in the Intervention group and 77% of teachers in the AT group responded "Quite a bit" or "Very realistic.?
Persuasiveness. Two Likert-scaled items were related to Persuasiveness: "How much did the program change your ideas about how to deal with problem behavior?" and "Do you think you'll use any of the ideas in the program?" No significant differences were found between the two groups. On the first question, the mean for the Intervention group was 2.83 (s.d. .98); for the AT group, the mean was 2.83 (s.d .92). 28% of teachers in the Intervention group responded "Quite a bit" or "Very much" while 17% of teachers in the AT Group had similar responses. On the second question, the mean for the Intervention group was 4.2 (.s.d. .98); for the AT Group, the mean was 3.94 (s.d .87). 83% of teachers in the Intervention group responded "Probably Yes" or "Definitely Yes"; 72% of teachers in the AT Group had similar responses. In the email questionnaire, teachers were asked whether they'd used any of the techniques they learned in the materials. While participants in both conditions reported that the program led them to change the way they thought about and responded to problem behavior, 92% of participants in the Intervention group, as compared to 79% in the AT group, were motivated toward change in their thinking and behavior.
Discussion.
Teachers report that they feel overwhelmed by classroom misbehavior and find it one of the most challenging and disturbing aspects of their jobs (Kulinna, Cothran, & Regualos, 2003). Despite the problem and prevalence of student misbehavior, teachers feel ill-equipped to deal with it (Martin, Linfoot, & Stephenson, 1999). There is a significant need to mobilize teachers to deal effectively with problem behaviors, in order to optimize the classroom setting for all students and to improve the future well-being of the students with problem behaviors. Our goal was to create an engaging, demonstration-driven instructional experience with clear, simplified presentation of skills teachers need to deal with everyday classroom misbehavior. The results from our evaluation of the FAST Method demonstrate that this interactive DVD presented a novel approach to analyzing and intervening with common problem behaviors, and was more effective at educating teachers than recently published text-based materials designed expressly to be used by educators in a school setting. Teachers participating in the interactive multimedia FAST training learned more in substantially less time, and were more confident in their ability to apply their knowledge. While there were few statistically significant differences on the consumer acceptance factors, there was a clear trend for teachers to respond more positively to the FAST program. They saw it as more stimulating and persuasive, revisited the materials, and were enthusiastic about the content and sharing the program with their colleagues. In the following section, results of each of the outcome measures are discussed in light of the consumer satisfaction feedback received from teachers.
Mastery.
A unique feature of our assessment was the use of 6 situational video vignettes as a quasi-behavioral outcome measure to assess teachers? ability to analyze in-class problem behavior and to respond appropriately across a range of simulated real-life situations. Using this assessment, teachers in the IG group showed higher mastery of the materials than those in the AT group. The FAST training is strengthened by the inclusion of video modeling, a central feature of the program. The need for effective skill-based training is clear: teachers strongly desire explicit information about useful methods of addressing misbehavior (Martin et al., 1999). This need was substantiated by feedback from participants in both groups who found the material on addressing challenging behaviors to be interesting and informative. Teachers receiving skill-based training, however, were considerably more enthusiastic about the training medium than teachers receiving commonly used text training. Teachers commented on the realism of the video vignettes and the immediate applicability of the training materials. For example, one teacher commented, ?I thought the role-playing situations were very accurate and helpful for the classroom teacher to see how the program can be applied for the various student behaviors.? Other teachers appreciated the authenticity of the materials: ?The video clips (vignettes) were very authentic and provided a concise ?illustration? of each type of behavior, as well as effective and ineffective teacher responses,? and ?The tape felt ?real? and not hired actors.?
Of course, teachers must be able to transfer their mastery of knowledge and skills using hypothetical vignettes to the actual classroom. Several teachers commented that the FAST method was already improving their effectiveness in handling misbehavior. A middle-school teacher recounted a recent interaction: ?An 8 th grade female who likes to draw teachers into a power struggle?was eating candy in class (no food/drinks outside the cafeteria). I simply looked at her, said the rule, and asked her to throw it out, then turned my back and continued with the class. She muttered under her breath but threw it out!!!? Teachers commented that the simple method of identifying a student?s goal helped guide their responses to student misbehavior: ?I have memorized the three goals of misbehavior. I quickly assess what a student is doing using those three goals: power, attention or avoidance. Once I can determine why the student is acting the way they are, I can understand and defuse the problem in a timely manner.? Another teacher noted, ?I have an attention-seeking student who thrives on inappropriate and excessive teacher attention-getting tactics. I have tried to ignore his tactics, and pay special attention to him when he is on task. It has had a good effect so far.? In addition to the three goals, teachers responded positively to the use of action verbs to describe student behavior rather than labeling a student: ?I really liked using the action-verbs to describe the behavior so that it separates it from being against the student, but rather what the student is doing.?
In contrast, many teachers participating in the AT group found that the text-based training was unwieldy. Some concerns were expressed by teachers in the AT group regarding comprehension of the written materials. Several participants noted that those materials, which relied more heavily on academic language and text, were not user friendly, or was ?wordy and overwhelming.?
Confidence.
Teachers in the Intervention group showed a significantly higher level of confidence in their ability to assess misbehavior than those in the AT group. Teachers? sense of confidence in using the FAST approach appeared to be related to the simplicity of the method and immediacy of results. One teacher commented, ?This program is simple to understand and simple to use. You can administer the techniques quickly and see results. It gives easy-t- use methods and skills that don?t require hours of practice. Why haven?t we been taught this before? This product is so simple?and it works! The very next day after watching the video I tried it with my 8 th grade class. The results were amazing. I can?t praise this product and technique enough, and I?ve been teaching since the ?70s. I?ve told everyone they HAVE to try this.? Clarity of program information increases teachers? confidence in using new methods (Merriam & Cafferella, 1999). Teachers noted that the FAST method was clear and easy to understand: ?The program is easy to understand and implement. I thought the role-play situations were very accurate and helpful for the classroom teacher to see how the program can be applied for the various student behaviors.?
Efficiency.
Training for teachers in the Intervention group took substantially less time than for teachers in the AT group. Although this effect could be viewed as an artifact of the different forms of the materials (reading obviously takes longer than viewing a DVD), it is a reflection of the true state of affairs: most training material on this topic is available only in print form.
Several teachers in the Intervention group commented on the efficiency of the program and ease of use. For example, one teacher commented, ?I think this is a wonderful product that is easy to learn and then implement in the classroom immediately. I found the materials with this program to be very clear and easy to understand. I read the manual, watched the video, and went to school the next day able to handle behavior problems in a much more effective manner.?
Efficiency in teacher training is essential, particularly in the area of student misbehavior. Teachers do not have the time to engage in complex and time-consuming training. They need simple, efficient training that yields quick results. As noted in the Commercial Application section, teachers identify time constraints due to increasing responsibilities and a lack of skill in working with challenging students as key factors in job stress, job dissatisfaction. A significant reason they choose to leave the teaching profession is that they feel they are required to work with challenging students (AARP Knowledge Management, 2003, http://research.aarp.org/general/exodus.pdf).
While participants in the Intervention group were, by and large, enthusiastic about the training materials, they also offered some helpful feedback about how the product could be improved. A few teachers in the Intervention group noted that the treatment process wouldn?t work with students with extremely challenging behaviors. The FAST program states that it is specifically for the most common 80% of student misbehavior; however, we recognize the need for materials to address more intractable student behaviors. Our Phase II training will address strategies for high incidence/high intensity behaviors by creating intensive individualized supports for those students who may need a more powerful intervention.
Teachers were engaged by the video exemplars, and asked for more. Several teachers in each group noted that they wanted more practice and case-study scenarios. As noted earlier, video can be a powerful tool in skill-building programs. In Phase II, we will significantly increase the number of video exemplars.
Finally, teachers wanted more extensive training for intervening after identifying a student?s goals. The primary goal of the Phase I training was to equip teachers with a method for assessing student misbehavior. The proposed Phase II module, student-specific positive behavior interventions, builds on teachers? ability to assess student misbehavior by providing extensive training in appropriate intervention.
Concluding Comments.
Our goal in Phase I was to address the critical need to equip teachers with strategies for dealing with student misbehavior in the classroom, by creating an engaging, efficient, and effective training program. Evaluation results confirm that the multimedia FAST Method is a refreshing and innovative approach to analyzing and intervening with common student misbehavior. The materials were more effective than text-based materials specifically created for use by teachers in the classroom. Teachers using the FAST Method quickly mastered the program material and felt a greater sense of confidence in their ability to apply what they learned. Teachers were very enthusiastic about the program content and offered some constructive feedback that will help shape the development of materials in Phase II. In Phase II, we will create materials that address teachers? needs for more intensive skill-building in intervening with student misbehavior (student-specific positive behavior interventions), including additional video exemplars, and we will address the need for training in strategies for high-intensity behaviors by creating intensive individualized supports for those students who need additional support.
References used in the research for this program.
Assor, A., Kaplan, H., & Roth, G. (2002). Choice is good, but relevance is excellent: Autonomy-enhancing and suppressing teacher behaviours in predicting student's engagement in school work. British Journal of Educational Psychology, 72, 261-278.
Baker, D. J. (1998a). Outcomes of behavior support training to an agency providing residential and vocational support to persons with disabilities. Journal of the Assoc. for Persons with Severe Handicaps, 23, 144-148.
Baker, D. J. (1998b). Effects of video-based staff training with manager-led exercises in residential support. Mental Retardation, 36, 198-204.
Baker, D. J., & Feil, E. G. (2000). A self-evaluation by agencies providing residential support of their capacity to support persons with disabilities and challenging behaviors. International Journal of Disability, Development, and Education, 47, 171-181.
Balkin, R. S., White, G. L., Boday, K. J. (2002). Do classroom behavior, access to materials, and teaching experience influence teachers leaving the profession? Arkansas Educational Research & Policy Studies, 2(2), 65-79.
Barrett, E.R., & Davis, S. (1995). Perceptions of beginning teacher?s in-service needs in classroom management. Teacher Education and Practice, 11(1), 22-27.
Blasé, J. J. (1986). A qualitative analysis of sources of teacher stress: Consequences for performance. American Education Research Journal, 23(1), 13-40.
Borg, M. G., & Riding, R. J. (1991). Towards a model for the determinants of occupational stress among schoolteachers. European Journal of Psychology of Education, 6, 355-373.
Browers, A. & Tomic, W. (2000). Disruptive student behavior, perceived self-efficacy, and teacher burnout. Paper presented at the Annual Meeting of the American Psychological Association (108 th, Washington, D.C., August 4-8, 2000)
Bryne, B. (1994). Burnout: Testing for the validity, replication, and invariance of causal structure across elementary, intermediate, and secondary teachers. American Educational Research Journal, 31, 645-673.
Carr, E. G. (1994). Emerging themes in the functional analysis of problem behavior. Journal of the Applied Analysis of Behavior, 27, 393-400.
Carr, E. G., Dunlap, G., Horner, R. H., Koegel, R. L., Turnbull, A. P., & Sailor, W. et al. (2002). Positive behavior support: Evolution of an applied science. Journal of Positive Behavior Interventions, 4, 4-16.
Carr, E. G., & Durand, V. M. (1985). Reducing behavior problems through functional communication training. Journal of Applied Behavior Analysis, 18, 111-126.
Carr, E. G., Levin, L., McConnachie, G., Carlson, J. I., Kemp, D. C., & Smith, C. E. (1994). Communication-based intervention for problem behavior: A user?s guide for producing positive change. Baltimore: Paul H. Brookes.
Chin, J. P., Diehl, V. A., & Norman, K. L. (1988). Development of an instrument measuring user satisfaction of the human-computer interface. Proceedings of SIGCHI 1988, ACM/SIGCHI, 213-218.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. (2nd ed.) Hillsdale, NJ: Lawrence Erlbaum Associates.
Costenbader, V. & Markson, S. (1998). School suspension: A study with secondary students. Journal of School Psychology, 36, 59-82.
.Cowick, B., Storey, K. (2000). An analysis of functional assessment in relation to students with serious emotional and behavioral disorders. International Journal of Disability and Education, 47(1), 2000.
Crone, D. & Horner, R., Building Positive Behavior Support Systems in Schools, Guilford Press, 2003.
Davis, C. A. (1998). Functional assessment: Issues in implementation and applied Research. Preventing School Failure, 43, 34-36.
Deci, E.L., & Ryan, R. M. (1985). Intrinsic motivation and self-determination in human behavior. New York: Plenum.
Deci, E. L., & Ryan, R. M. (1991). A motivational approach to self: Integration in personality. In R. Dienstbier (Ed.), Nebraska symposium on motivation: Vol. 38. Perspectives on motivation (pp. 237-288). Lincoln , NE : University of Nebraska Press.
Deci, E.L., & Ryan, R.M. (1995). Human agency: The basis for true self-esteem. In M.H. Kernis (Ed.), Efficacy, agency, and self-esteem (pp. 31-51). New York : Plenum.
Deci, E. L., & Ryan, R. M. (2000). The "what" and "why" of goal pursuits: Human needs and the self-determination of behavior. Psychological Inquiry, 11, 227-268.
Deci, E. L., & Ryan, R. M. (Eds.), (2002). Handbook of self-determination research. Rochester, NY: University of Rochester Press.
Deci, E.L., & Ryan, R. M. (2002). The paradox of achievement: The harder you push, the worse it gets. In J. Aronson (Ed.) Improving academic achievement: Contributions of Social Psychology (pp.59-85). New York : Academic Press.
Deci, E. L., Schwartz, A. J., Sheinman, L., & Ryan, R. M. (1981). An instrument to assess adults' orientations toward control versus autonomy with children: Reflections on intrinsic motivation and perceived competence. Journal of Educational Psychology, 73, 642-650
Deci, E. L., Vallerand, R. J., Pelletier, L. G., & Ryan, R. M. (1991). Motivation and education: The self-determination perspective. The Educational Psychologist, 26, 325-346.
Dodge, K.A. (1993). Social cognitive mechanisms in the development of conduct disorder and depression. Annual Review of Psychology, 44, 559-584.
Dunlap, G., Kern, L., dePerczel, M., Clarke, S., Wilson, D., et al. (1993). Functional analysis of classroom variables for students with emotional and behavioral disorders. Behavioral Disorders, 18, 275-291.
Engelmann, S., & Carnine, D. (1982). Theory of Instruction. NY: Irvington Press.
Fisher, D. L., & Fraser, B. J. (1983). A comparison of actual and preferred classroom environment as perceived by science teachers and students. Journal of Research in Science Teaching, 20, 55-61.
Foster-Johnson, L., & Dunlap, G. (1993). Using functional assessment to develop effective, individualized interventions for challenging behavior. Teaching Exceptional Children, 25(3), 44-50. Fox, J., Conroy, M., & Heckaman, K. (1998). Research issues in functional assessment of the challenging behavior of students with emotional and behavioral disorders. Behavioral Disorders, 24, 26-33.
Freeman, R. L., Britten, J., McCart, A., Smith, C., Heitzman-Powell, L., Baker, D., et al. (1999). Module 1 Foundations of positive behavioral support. Lawrence: University of Kansas UAP, Center for Research on Learning. Available: Onlineacademy.org.
Gable, R. A. (1999). Functional assessment in school settings. Behavior Disorders, 24, 246-248.
Gable, R. A., Quinn, M. M., Rutherford, R. B. Jr., & Howell, K.(1998). Addressing problem behaviors in schools: Use of functional assessments and behavior intervention plans. Preventing School Failure, 42, 106-119.
Gardill, M.C., DuPaul, G.J., &Kyle, K.E. (1996). Classroom strategies for managing students with attention-deficit/hyperactive disorder. Intervention in School and Clinic, 32, 89-94.
Graham,S. (1998). Social motivation and perceived responsibility in others: Attributions and behavior of African-American boys labeled as aggressive. In J. Heckhausen & C.S. Dweck (Eds.), Motivation and self-regulation across the lifespan (pp. 137-158). New York: Cambridge University Press.
Graham, S., Hudley, C., & Williams, E., (1992). Attributional and emotional determinants of aggression among African-American and Latino young adolescents. Developmental Psychology, 28(4), 731-740.
Gresham, F.M., Quinn, M.M. Restori, A. (1999). Methodological issues in functional analysis: Generalizability to other disability groups. Behavioral Disorders, 24(2), 180-182.
Horner R.H., & Carr, E.G. (1997). Behavioral support for students with severe disabilities: Functional assessment and comprehensive intervention. Journal of Special Education, 31, 84-104.
IDEA, (2004). Individuals with Disabilities Education Improvement Act. H.R. 1350. Signed into law on Dec. 3 rd, 2004.
Kleiner, A. & Farris, E. (2002). Internet access in U.S. public schools and classrooms: 1994-2001. Washington, DC: U.S. Department of Education, National Center for Education Statistics. (NCES 2002-018).
Koegel, L. K., Koegel, R. L., & Dunlap, G. (Eds.) (1996). Positive behavioral support: Including people with difficult behaviors in the community. Baltimore: Paul H. Brookes.
Kulinna, P. H., Cothran, D., Regualos, R. (2003). Development of an instrument to measure student disruptive behavior. Measurement in Physical Education and Exercise Science, 7(1), 25-41.
Maag, J.W. & Kemp, S. E. (2003). Behavioral intent of power and affiliation. Remedial and Special Education, 24(1), 57-64.
Market Data Retrieval. (2002). New from MDR. Received July 25, 2002 from, Market Data Retrieval@dnb
Martin, A.J., Linfoot, K., Stephenson, J. (1999). How teachers respond to misbehavior in their classroom. Psychology in the Schools, 36 (4), 347-358.
Merriam, S. B. &. Capperella, R. S. (1999). Learning in adulthood (2nd Ed.). San Francisco: Jossey-Bass.
Moos, R.H., & Trickett, E.J. (1987). Classroom Environment Scale manual (2nd ed.). Palo Alto, California: Consulting Psychologists Press.
Nelson, R., Roberts, M. L., & Smith, D. J. (1998). Conducting functional assessments. Longmont, CO: Sopris West.
O?Neill, R. E., Horner, R. H., Albin, R. W., Sprague, J. R., Storey, K., & Newton, J. S. (1999). Functional Assessment and program development for problem behavior: A practical workbook. Pacific Grove, CA: Brooks/Cole.
Reeve, J. (1993). The face of interest. Motivation and Emotion, 17, 353-375.
Reeve, J. (1998). Autonomy support as an interpersonal motivating style: Is it teachable? Contemporary Educational Psychology, 23, 312-330.
Reeve, J. (2002). Self-determination theory applied to educational settings. In E. L. Deci & R. M. Ryan?s (Eds.), Handbook of self-determination research (pp. 183-202). Rochester, NY: Rochester University Press.
Reeve, J., & Nix, G. (1997). Expressing intrinsic motivation through acts of exploration and facial displays of interest. Motivation and Emotion, 21, 237-250.
Reeve, J., Bolt, E., & Cai, Y. (1999). Autonomy-supportive teachers: How they teach and motivate students. Journal of Educational Psychology, 91, 537-548.
Reeve, J., Jang, H., Hardré, P., & Omura, M. (2002). Providing a rationale in an autonomy-supportive way as a strategy to motivate others during an uninteresting task. Motivation and Emotion, 26, 183-207.
Reeve, J., Hyungshim, J., Carrell, D., Soohyun,J., & Barch,J. (2004). Enhancing students? engagement by increasing teachers? autonomy support. Motivation and Emotion, 28(2), 147-169.
Repp, A., & Karsch, K. G. (1994). Hypothesis-based interventions for tantrum behaviors of persons with developmental disabilities in school settings. Journal of Applied Behavior Analysis, 27, 21-31.
Royer, E. (1995). Behaviour disorders, exclusion and social skills: Punishment is not an education. Therapeutic Care and Education, 4 (3), 32-36.
Skinner, E., & Edge, K. (2002a). Self-determination, coping, and development. In E. L. Deci & R. M. Ryan (Eds.). Handbook of self-determination research. (pp. 297-337). NY: University of Rochester Press.
Skinner, E., & Edge, K. (2002b). Parenting, motivation, and the development of children?s coping. In L. J. Crockett (Eds.). Agency, Motivation, and the Life Course. (pp. 77-143). Lincoln : University of Nebraska Press.
Skinner, E. A., & Belmont, M. J. (1993). Motivation in the classroom: Reciprocal effects of teacher behavior and student engagement across the school year. Journal of Educational Psychology, 85, 571-581.
Skinner, E. A., Edge, K., Altman, J., & Sherwood, H. (2003). Searching for the structure of coping: A review and critique of category systems for classifying ways of coping. Psychological Bulletin, 129(2), 216-269.
Stage, S.A. & Quiroz, D.R. (1997). A meta-analysis of interventions to decrease disruptive classroom behavior in public educations settings. School Psychology Review, 26,333-368.
Sterling-Turner, H.E., Robinson, S.L., Wilczynski, S.M. (2001). Functional assessment of distracting and disruptive behaviors in the school setting. School Psychology Review, 30(2), 211-226.
Vansteenkiste, M., Simons, J., Lens, W., Sheldon, K. M., & Deci, E. L. (2004). Motivating learning, performance, and persistence: The synergistic effects of intrinsic goal content and autonomy-supportive contexts. Journal of Personality and Social Psychology, 87, 246-260.
Virzi, R. A. (1990). Streamlining the design process: Running fewer subjects. Proceedings of the Human Factors Society, 34 th Annual Meeting, 291-294.
Virzi, R. A. (1992). Refining the design test phase of usability evaluations: How many subjects is enough? Human Factors, 34, 457-468.
Witt, J.C., Daly, E.M., Noell, G. (2000). Functional Assessments: A Ste-by-Step Guide to Solving Academic and Behavior Problems. Longmont, Colorado: Sophris West.
