Systematic Supervision for Elementary School
Stephen G. Smith, M.S., Principal Investigator
1 R43 MH067364-01
Project Period: 07/02/2003-3/31/2004
Program Title: Systematic Supervision for Elementary School: Creating a Safe and Positive Playground
Summary
Systematic Supervision was used by playground supervisors in three elementary schools in a study that looked at their use of four supervisory skills before and after receiving Systematic Supervision training.
Participants were served over a number of weeks to obtain baseline data about their use of the skills before receiving training, and then were observed after the training.
After the training, participants' use of the skills analyzed in the study (movement, scanning, positive contact and positive reinforcement) showed a significant increase:
- Planned and purposeful movement: 73%
- Scanning (long-distance observation: 137%
- Positive contact: 163%
- Positive reinforcement: 23%
In addition, the rate of problem student behaviors decreased six percent, and staff rated the program highly.
Project Aims
To address the incidence of problem behaviors in schools, this Phase I project focused on developing research-based, supervisory and behavioral support training for elementary school staff (i.e., administrators, teachers, counselors, and classified staff). The Phase I effort specifically targeted playgrounds, because their open, unstructured environments with large indistinct boundaries, high student-to-staff ratios, and typically untrained classified personnel, produce a challenging environment for delivery of effective supervision and behavior supports. This goal was achieved through the development of a multimedia program, which is currently being made widely available, that presents and supports effective, evidence-based supervisory and behavioral support. The training and materials produced by this project were developed and evaluated by the Principal Investigator in collaboration with the Institute on Violence and Destructive Behavior (IVDB) at the University of Oregon through their established school staff training program, BEST Behavior (BEST). Dr. Sprague, co-director of the Institute was the primary consultant on this effort.
Phase 1 had four primary goals: 1) to conduct an analysis of program content designed to train school staff in effective playground supervision; 2) to conduct focus groups with school administrators, risk management specialists, teachers, and playground supervisors to ensure that program content would be compatible with the school environment and would be sensitive to the needs of typical school districts and parents, 3) to develop multimedia scripts and navigational design for the multimedia program in collaboration with the Project Consultant, 4) to produce a program on playground supervision and master the multimedia content to DVD; and 5) to evaluate program efficacy using a wait-list control group design with three subject groups.
All projects activities have been completed satisfactorily. The Systematic Supervision training program was successfully completed. The program is now available in release form and consists of the following components in an attractively presented package (See Appendix A):
- Creating a Safe and Positive Playground, multimedia staff training on DVD and VHS, 35 minutes.
- Participant Workbook, 21-page companion guide to the multimedia training.
- Administrator Program Guide, full-color foldout providing guidelines for training, implementation and maintenance of safe playground practices.
- Common Area Behavior Log, a form for tracking problem behaviors (also available on the web, www.lookiris.com).
- Play By the Rules, a DVD for elementary students on maintaining a safe/positive playground.
Preliminary user responses for Systematic Supervision have been exceptionally strong. Between 50 to 100 percent of the participants requested to order the program following presentations by IVDB training specialists, during BEST Behavior (Sopris West) school-wide behavior support trainings. Press releases about the availability of the program also generated interest. A recent release about the Systematic Supervision programs has generated interest and requests from school districts around the nation, including an initial order of 40 from the Texas Behavior Support Initiative.
Conduct an Analysis of the Content and Design Training Materials.
The project's development staff (Principal Investigator, Project Consultant, and Media Developers) met in a series of work sessions to examine the basic principles of Systematic Supervision in order to determine the scope and sequence of the training, identify the core skills which would be presented, and discuss what strategies would be used for presenting these skills. It was determined that the program, Systematic Supervision, would focus on the basic techniques and methods of active supervision as used in school common areas: (a) movement by staff, (b) scanning by staff, (c) effective and efficient positive reinforcement, (d) effective and efficient delivery of consequences for inappropriate behavior, (e) basic behavior support and intervention and, (f) monitoring and data-based decision-making. The above issues were examined within the context of widely-accepted principles of instructional design for developing a multimedia group training program for professional development leading to a preliminary design. Following this, the project development team developed a preliminary script which could be sent to focus group participants.
Conduct Focus Group With School Administrators, Behavior Specialists, Teachers & Playground Supervisors.
We conducted a focus group attended by 10 educators who had been recruited by the Principal Investigator and Project Consultant through their professional contacts. The participants included a principal, a speech language specialist, child development specialist working in a rural school, a family center coordinator with experience as a playground aide, a graduate school student in social work who had been an elementary school teacher, a behavior skills specialist, a 7-8 th grade teacher, a middle school principal, a middle school ESL teacher (former playground aide), 5-6 th grade teacher, & 7 th grade teacher. Participants were selected to attain a representative distribution of experience, ethnicity, race, and gender. Focus group participants were mailed a copy of the Systematic Supervision playground script and were asked to review ensuring that program content was relevant, compatible with the school environment, sensitive to the needs of typical school districts, and respectful to parents. The meeting was held at the downtown offices of IRIS Media and participants were paid $70 for two hours of their time.
The focus group began with a brief description of the research project and an explanation of the meeting?s purpose. The participants were addressed as experts in their roles, and their feedback on the validity, clarity, relevance, and importance of the proposed content was sought. They were encouraged to share experiences from their own lives that illustrated the relevance of the content or, conversely, propose alternative material. The Project Consultant and Research Assistant also attended the meeting, and detailed notes of participants' comments were kept. Feedback centered on reactions to the script. Participants agreed that the content was appropriate and the playground scenes seemed realistic. They appreciated the use of positive and negative examples and the focus on behavior rather than speculation about kid attitudes. The focus group provided other valuable feedback on segmenting the training, length of individual scenes, what language though common to a playground would be inappropriate in a training program, and on the tone of the presentation. Focus group participants stressed the need to make the training engaging, in light of a 'tough' audience who may feel they already know how to manage effectively or who may resent changing their style.
The group also discussed the need for designing the training package with enough flexibility to use it at in-service meetings, group presentations by a trainer, in teacher-training classes, and for individual viewing. This would require interactive navigational menus and controls to enable viewers to pause for discussion and be able to access sequences as needed. The group also stressed the need to have the program available in a linear VHS format. Other recommendations were that optimum program length was 30 minutes, that a printed workbook or guide be included in the program, and that printable forms be available on the web. The group also stressed the need for motivating staff to adopt the strategies. The group agreed that the motivation for administrators was in having a well-managed playground and that the aides? motivation for improving their skills was that it would make their jobs easier. An important suggestion referenced the need for some systematic teaching of playground rules for kids due to the great variability in the way kids were taught the basic rules, ranging from no systematic teaching of rules to programmed sessions throughout the school year.
Develop Scripts & Navigational Design of the Multimedia Program in Collaboration with Project Consultant.
Upon evaluating focus group feedback, the development team determined that the multimedia training package would contain: a) an interactive DVD presentation for school staff and playground aides on applying systematic supervision principles toward creating a safe and positive playground (a VHS version of the program would be included in the packet for the convenience of schools or individuals who didn't use DVD), b) a Participant Workbook that summarizes the key concepts from the training presentation and serves as a discussion guide, c) an Administrator's Program Guide (both print and web-based) that provides guidelines on presenting, training, implementation, monitoring, and maintenance of safe playground practices, d) a Common Area Behavior Log, a printout available on the web that allows playground supervisors to track students who present problem behavior, e) Play By the Rules, a DVD presentation for elementary-school students on their role in maintaining a safe and positive playground. This last piece was added after at the suggestion of the focus group. The development team then completed work on these five components and received valuable input and review from Dr. Sprague, the Project Consultant.
Produce a Program on Playground Supervision.
DVD Production . The Media Producer was responsible for the production of the multimedia program, the bulk of which consisted of video sequences. For this purpose a substantial cast of actors to play teachers, playground supervisors and students was needed. Forty students and eight adults were hired for the production. Careful attention was paid to ensuring diversity of race, ethnicity, gender, and disability. The program was filmed at three different schools and playgrounds. A crew consisting of a Production Coordinator, Camera Operator, Audio Recordist, and Grip took part in the four-day shoot. A fifth day was devoted to shoot the on-camera sequences in which the Principal Investigator provided the training narrative. The program was shot on the Sony DVCAM professional digital video format and digitally edited on Pinnacle Liquid Silver CCIR 601 editing system with serial digital interface. The Graphics Designer prepared graphics and animations. The Principal Investigator and other project staff viewed digital offline versions of the program at key points in the post-production process so that revisions to the program were possible. Original music for introducing the program, enlivening sequences and for use at transitional points was commissioned. A key step was the production of a layered sound track consisting of dialog and natural sounds, narration, music, and sound effects which was mixed and 'sweetened' using the ProTools audio editor. The DVD master was authored using SONIC Solution's DVD Producer with SD1000 hardware MPEG2 encoder. In-house staff produced test duplicates to ensure that the interactive menu functions performed properly and all of the video and text sequences played without glitches. Once we had assurances of the quality and stability of the product, we duplicated 20 copies to use in the evaluation and for use by Principal Investigator and Project Consultant.
Print Materials . Print materials for Systematic Supervision program include a Participant Workbook and an Administrator's Guide. The Participant Workbook, a 21-page companion guide, recaps key concepts from the multimedia presentation and provides examples of systematic supervision principles, worksheet pages which allow trainees to explore how systematic supervision principles can be applied in their school, and a resource section with practical examples of behavioral expectations. A reproducible behavior tracking log sheet is also included, as well as posted on the Systematic Supervision page section of the IRIS website for free download. The Administrator's Guide is a full-color foldout providing guidelines for training, implementation and maintenance of safe playground practices. The guide leads trainers through the process of planning for, presenting and following up on a training session. Resources where administrators can find more information on the topic are included. In addition, a reproducible supervision checklist is included that allows administrators to evaluate a supervisor's skill level. This form can also be used by supervisors to evaluate their own performance. The material from this guide is also available for free download from the IRIS website.
Study design and data collection
The study was conducted in three elementary schools located in Oregon . Participating schools were recruited from available elementary schools in Northwestern Oregon school districts. The schools were evaluated and invited to participate by the Principal Investigator, with input from the Project Consultant and Media Developer. The district-level administration from which the participating schools came were advised of the nature and scope of the study, presented with prior human subjects approval for the study activities, and were asked for, and tendered, written permission for the eligible schools to participate. The schools were public elementary schools with the widest, proportional, and/or most representative possible ethnic, racial, and gender staff and student community available. All schools had the requisite 2 nd, 3 rd, 4 th, and 5 th student grade levels. None of the schools had participated in active supervision based training within the previous 3 years or were currently in the process of active supervision based training. All participating schools had a Principal committed to program completion. All participating schools solicited passive parental consent from the parents/guardians of all students (only one school had a parental refusal to participate ? a single 2 nd grade student in school 1. As a result, no 2 nd graders were observed during recess at that school). No other criteria for limiting participation was considered. Any school meeting all of stipulations was considered for participation.
A wait-list treatment-control group design was used for the evaluation study. The order in which participating schools began intervention was randomly assigned. Both multiple baseline and pre/post approaches were used across participating schools to analyze the data and evaluate possible effects of the training on increasing the active supervision behaviors of the participating supervisors and reducing problem behaviors during recess for 2 nd, 3 rd, 4 th, and 5 th grade students.
Data were collected by direct observation using pencil and paper. Observers were trained using video taped examples, discussion, and on site sessions to a 90% or better inter-observer agreement before baseline and intervention data collection began. The various key indicator behaviors of the participating supervisors and students on the playground during lunch recess were observed. The design tracked the essential behaviors of effective supervision presented in the program along with the concurrent student behavior.
Supervisor behavior data was coded in terms of; a) rate of purposeful movement, b) rate of scanning, c) rate of positive interaction with students, d) rate of all other (excepting positive) interactions with students, e) rate or interaction with groups of students as opposed to individuals (positive vs. all others), and f) rate of use of punishers to respond to problem behavior (Appendix B)
Student problem behavior data was coded in terms of: a) rate of hands on others, rough play, play fighting; b) rate of arguing (each individual involved), c) rate of disrespect or defiance towards peer and adults; d) rate of not following rules or misuse of equipment, etc; e) rate of teasing, name-calling, or offensive language; and f) rate of threatening, harassment, or fighting, etc. (Appendix C)
Our original study design included the identification and observation of individual students nominated by their teachers as exhibiting externalizing or internalizing behavior outside the norm. This study component attempted to determine what effect the intervention (SystematicSupervision training) would have on those students typically associated with increased behavioral concerns and supports. As implementation of the study took place, it became apparent that this data would not likely yield much value in comparison to cost and mechanics of implementation. At the same time, we had added a significant cost item by developing a video on playground rules for students as recommended by our focus group. These issues were discussed with the grant officer, Dr. Steinberg, who agreed that the individual student component could be dropped.
Intervention Procedures and Independent Variables.
The independent variables consisted of the provision of Systematic Supervision training to all supervision staff in the schools participating in the study. The participating schools could be characterized as middle to lower middle class in terms of SES and rates of free and reduced lunch eligibility. School 1 best fit an example of a ?rural/suburban? elementary school, while Schools 2 and 3 could be termed ?Urban/Near-urban.? The training was presented using the electronic/digital media program developed as part of this project (i.e., training in systematic supervision techniques, training in methods of behavioral management, positive behavior support, and instruction and training in methods of data collection and data analysis).
The Systematic Supervision training was presented to all participating school personnel in a one and a half hour inservice session. Associated written training and reference materials were distributed during the in-service. The training was introduced by the school administrator using the associated training materials (included as part of the training package). The Principal Investigator addressed questions, concerns, or comments about the training, both before and after the session. A short staff discussion and strategy session was suggested, but not mandated, as part of the training process following the end of the program presentation. A group, or all-staff, training strategy was employed for three main reasons: a) it represents the most cost-efficient way to train all staff at a time when school budgets are tight, b) when all members of a staff train together there is a greater likelihood that they will develop and maintain consistency and fidelity of implementation, and c) when all staff train together there is a greater likelihood of individual staff buy-in for the program and effort in making it succeed.
Students were taught the expected and appropriate behavior set out in the Systematic Supervision program through viewing the ~8 minute Play by the Rules video produced as a support piece for the supervisor training program. This video activity was not originally envisioned as part of the project, but the need for it became rapidly apparent during production since student behavior and changes in student behavior are integral to outcomes and outcome measurement. Play by the Rules was presented to all students in each of the participating schools. It was shown one time to students attending school on the days it was shown on a classroom by classroom basis. It is expected that some students missed the presentation due to absence or competing activities and, due to transitions within the school during the day, some students may have watched the video more than once.
Data Collection.
Observation. Six observers were trained to 90% reliability for both supervisor and student observations using video tape and on-site training sessions. Observations took place during lunch recesses during a variety of normal activities (e.g., wall-ball, tetherball, unorganized games, basketball, swings, or gym equipment). Inter-observer reliability was calculated daily by a research assistant and the Principal Investigator checked the calculations and inter-observer reliability on a semi-daily (every one to three days) basis in order to monitor emerging trends in observer disagreement, observer bias, or observer drift. On-site baseline observations of supervisors and students (Phase A) began 4, 7, and 10 school days prior to delivery of training (schools 1, 2 and 3, respectively). Students and staff members were not informed as to the nature or reason for observational activities. Baseline (Phase A) observation was scheduled up to the day before training was delivered. Intervention observation of supervisors and students (Phase B) began on the next school day following presentation of the training and continued for 10 to 12 Phase B observation days for each school.
Dependent Variables. To assess the effects of the intervention, changes in social behavior and supervision activities and use of active supervision strategies were measured for students and supervisors. These fell into two basic groups: (a) all students on the playground (non-identified, exhibiting the entire range of typical behavior), and (b) all certified, classified, and other staff who supervised these students at recess during observation periods. Supervisor data was collected on six major categories of supervisor behavior (see attachment 1) related to the active supervision strategies and methods presented in the Systematic Supervision training program. Data on student behavior were collected in six discrete categories of functionally associated problem behavior (Appendix B) Direct observation methods described above were used. The observational instruments, methods for use, and data analysis techniques had been piloted in previous studies (Smith, Sugai, & Sprague, 2002).
Data were collected on an occurrence basis and calculated as a rate per minute for both student and supervisor behaviors. All data were expressed as a rate per minute, both aggregated and unaggregated. The participating students were 2 nd, 3 rd, 4th, and 5th grade students (except for school 1, in which 2 nd grade was not observed) who happened to be on the playground during observation sessions.
Rationale for Use of Multiple Baseline
Multiple baseline research, utilizing a sequential application A-B design, has not only been shown to be effective in isolating treatment effects (Baer et al., 1968; Barlow & Herson, 1984), but also: (a) can be applied to subjects and behavior as both individuals and groups, (b) allows for the application of the same experimental variable both within and across subjects, (c) allows the researcher to use each subject as its own control group and, (d) by using multiple subjects starting treatment phases at staggered intervals, the researcher can obtain a high degree of confidence in treatment effects.
There are several classic limitations to using a single subject design. These limitations revolve around the use of individuals as subjects and do not apply to using a single subject design with groups (Barlow & Herson, 1984). The main concern associated with its proposed use in this project lies in the difficulty of determining independence of behaviors using a single subject design. However, since schools present a large inventory of variables by nature, the problem of independence of behaviors and effects is moot. The proposed measurement activities encompass the widest possible range of both functionally independent behaviors as well as aggregate behavior.
Inter-observer Agreement
Inter-rater reliability percentage agreement was calculated by comparing each observational session?s construct scores obtained by the criterion and reliability observers and dividing the small number by the larger number and multiplying it by 100. The overall average inter-observer agreement for supervisor behavior was 89.5% over 56% of observation sessions with a range of 76% to 100%. The overall average inter-observer agreement for student behavior was 85.1% over 40% of observation sessions with a range of 50% to 100%.
Overall reliability on data entry and database analysis was 99.7% for 33.6% of sessions checked (range 89% - 100%). The poorest inter-observer agreement was obtained during observations at School 1. This is thought to be due to challenging supervisory staffing dynamics as well as an unpredictable, volatile, and highly variable school climate rather than problems with observer fidelity and the observation process. The situational confounds presented at School 1 will be described in greater detail in the following discussion section.
Rates of Supervisor Systematic Supervision Skills Use: Multiple Baseline and Pre/Post Analysis of the Data.
Direct observation of supervisor behaviors (see above) focused on the key supervision skill presented in the Systematic Supervision program. The results are presented by skill; a) movement, b) scanning, c) positive interactions with students versus all other interactions with students, and d) overtly punishing interactions with students as a consequence of problem behavior. The data is presented in graphs and tables with analysis and discussion following each behavior set.
A. Movement. Movement was considered to be any purposeful relocation on the part of the supervisor from one area of the playground to another in order to supervise student activity. The basic determinates of movement for coding purposes were change in position in order to: 1) observe various students or activities, 2) increase proximity to students or activities, 3) deal with problems or problem behaviors, and 4) meet job requirements. If the movement consisted of moving randomly within a functionally static area, no movement was coded.
The graphs in Figure 1 (Appendix D) indicate a wide variability in rate of movement and show contradictory results in terms of change in movement behavior on the part of the supervisors. The graph for school 1, in which the program training was challenging (see discussion section below), shows a flat trend and no real change in the intervention phase. School 2?s data indicates a steadily increasing rate of movement starting in the baseline and continuing into the intervention phase until sessions 13 through 15 where a high degree of unexplainable variability is evident. School 3 shows decreasing trends in both baseline and intervention phases with variability evident in both. If outlier baseline sessions 3 and 4 were not considered, the data for School 3 might indicate that the supervisors came away from the training using a much higher rate of movement that settled into a higher average rate of movement across the intervention phase.
Table 3 - Pre/Post Rate per Minute of Supervisor Movement (Aggregated)
|
School |
Pre |
Post |
% Change |
|
1 |
0.38 |
0.40 |
5% |
|
2 |
0.46 |
1.06 |
130% |
|
3 |
0.47 |
0.81 |
72% |
|
Mean |
0.44 |
0.76 |
73% |
When the data are aggregated into pre and post rates of movement (Table 3), it can be seen that schools 2 and 3 had a significant increase in supervisor movement. Due to the high concentration of variables (and their high level of volatility potential) presented on school playgrounds, this method of analysis may be better suited to these kind of data. Although not as rigorous, this type of analysis can be useful in highlighting some potential effects of training and knowledge-based interventions such as Systematic Supervision.
B. Scanning . Scanning was considered to be any purposeful attempt on the part of the supervisor to observe students or activities in an area of the playground outside their immediate proximal supervision area. The basic determinates of scanning for coding purposes were: 1) an overt attempt to observe distant students or activities, 2) reaction to sounds or actions at a distance from their immediate position, and 3) some evidence of a regular attempt to keep watch over a particular student or activity outside their immediate area. Looking or listening to students or activities in their immediate proximity did not constitute scanning and was not coded.
In Figure 2 (Appendix D), School 1 has a steadily increasing trend from baseline through intervention phase evidencing an apparent lack of intervention effect. Schools 2 and 3 again show a high degree of unexplainable variability. School 2 seems to indicate increasing trends in both baseline and intervention phases, while School 3 seems to experience an increase in rate of behavior from baseline to intervention phases, both phases seem to show decreasing trends, even when discounting the highly variable outlying data points. Once again, Schools 2 and 3 show higher overall rates of target behavior than School 1 due perhaps to the unique challenges to program implementation in that school and the school?s existing culture and climate with regards to the supervisory staffing.
Again, when these data are aggregated and presented as pre/post average changes in rate (Table 4), indications are that the participating supervisors in each school increased their scanning behavior after training. School 1 seems to have experienced the smallest effect in this regard while School 2 indicates a more significant increase (as it did with movement rates) and School 3 rather a more modest increase.
Table 4 - Rate per Minute of Supervisor Scanning: Aggregated Pre/Post
|
School |
Pre |
Post |
% Change |
|
1 |
0.63 |
0.72 |
14% |
|
2 |
0.66 |
1.50 |
127% |
|
3 |
0.71 |
0.96 |
35% |
|
Mean |
0.67 |
1.06 |
58% |
C. Rates of Positive Interactions with Student. One of the key strategies of Systematic Supervision is the use of high rates of general positive contact and the use of positive reinforcement for expected behaviors. These two conceptual strategies were operationally linked during observation and coding as ?Positive Contact.? Supervisor contacts with students were either characterized as ?positive contact? or ?all other.? The ?all other? category was defined as contact with a student that was: a) instructional, b) instructional and corrective of behavior, c) aversive or punishing, or d) neutral or directive. Positive and reinforcing contacts as defined in the Systematic Supervision program needed to be proactive as well as positive in nature. That means interactions needed to be originated by the supervisor, on their initiative, and not originally solicited from the supervisor by the student. This definition greatly restricted the ability of the supervisors to meet the program-mandated goal of fours positive interactions for each one of the ?all other? interactions as many supervisor/student interactions are student solicited. Likewise, the strategy of interacting positively with groups of students over individual students when possible or feasible is emphasized as a key supervision strategy. This strategy is critical if supervisors are to effectively and efficiently meet their goal of creating and maintaining a positive climate by achieving a 4 to 1 ratio of positive to all other interactions. Figure 3 (Appendix D) presents the rates of positive to ?all other? interactions with individual students, while Figure 4 (Appendix D) shows the rates for groups.
Table 5 - Rates of Supervisor Positives and all other interactions
|
School |
Pre |
Post |
% Change |
|
1-Positive |
0.32 |
0.46 |
44% |
|
1-All others |
2.80 |
2.77 |
-1% |
|
2-Positive |
0.39 |
0.33 |
-15% |
|
2-All others |
1.36 |
0.92 |
-32% |
|
3-Positive |
0.88 |
1.15 |
31% |
|
3-All others |
1.42 |
2.86 |
101% 102% |
|
Mean Positive |
0.53 |
0.64 |
21% |
|
Mean All Others |
1.86 |
2.18 |
17% |
The data from School 1 show slight, but steady increases in the rate of both kinds of interactions with students after the training, but more data would need to be collected to establish a meaningful trend. School 2?s data show no real effect across the phases in this representation and indicate a fairly low rate of overall interactions with students on the part of the supervisors. School 3?s data show a spike in rate of both positive and ?all other? interactions for the observation session following the training, much like it did rates of scanning, but the rate quickly returns to pre-training levels and doesn?t seem to maintain at a higher rate.
When viewed in an aggregated pre/post average (Table 5), these data present mixed results at the individual school levels. Supervisors in School 1 had a slight increase in positive interactions while staying at about the same rate of ?all-other? interactions. School 2 supervisors showed a decrease in both types of interaction categories, perhaps most significantly in the ?all-other? type. School 3 supervisors had an increase in both classes of interactions and the mean for all schools showed a slight increase in rates of positive and ?all-other? interactions in general. None of the school?s supervisors achieved a collective 4 to 1 rate of positive to ?all-other? interactions as suggested in the training program, but it is to be expected that this goal is more of a long-term objective and would be reached only gradually, most likely with repeated exposure to the training program and with specific encouragement from school administrators.
The data for rates of supervisor positive and all-other interactions with students are presented in Figure 4 (Appendix D). Once again, supervisors at School 1 show relatively low rates of overall interactions with students. In this case, there is an extremely low rate of positive interactions with groups of students across both phases just as there was with individual students in Figure 3 (Appendix D). School 2 presents highly variable data across phases, with an apparent increasing (though highly volatile) trend in both categories of interactions after training. School 3 shows highly variable baseline data set although both classes of interactions seem to follow a similar rate pattern. This would suggest that supervisor interaction with groups of students was reacting to a variable, or set of variables, affecting conditions on the playground that changed from session to session.
Overall Schools 2 and 3 had a much better ratio of positive to ?all-other? rate of interaction with groups of students than did School 1, but there was no clear trend indicating that progress was made toward achieving a 4 to 1 ratio as suggested in the Systematic Supervision training program.
Table 6 - Supervisor's Positive to All Other Interactions with Groups
|
School |
Pre |
Post |
%Change |
|
1-Positive |
0.17 |
0.00 |
- 100% |
|
1-All others |
2.53 |
2.20 |
- 13% |
|
2-Positive |
0.25 |
2.24 |
796% |
|
2-All others |
2.29 |
2.53 |
11% |
|
3-Positive |
1.39 |
2.37 |
71% |
|
3-All others |
2.35 |
2.19 |
- 7% |
|
Mean Positive |
0.60 |
1.54 |
157% |
|
Mean All Others |
2.39 |
2.31 |
- 3% |
When the data is presented as aggregated pre/post average (Table 6), supervisors at School 1 again show very low rates of positive interaction if any, and show no positive interactions at all during post intervention observations. School 2 and 3 indicate an increase in rates of positive interactions with groups of students, even discounting the outlier data points. School 2 once more shows an increasing trend (although highly variable) after training and School 3 again shows a spike in the rate of the target behavior (positive interactions with groups of students) immediately following the training. As before, this spike in rate just as quickly returns to pre-intervention patterns of behavior.
D. Rates of Use of Overt Punishers. Another of Systematic Supervision training?s key features is to encourage supervisors to use support-based, rather than punishment-based, consequences for student problem behavior. To determine if this strategy is being used after training, observers took data on the rate of delivery of overt punishers by supervisors. Overt punishers were defined as any consequence, delivered as a result of problem behavior, that: a) was delivered in an angry tone of voice, b) was not paired with an instructional or corrective component, c) might reasonably be expected to cause the student shame, embarrassment, or mental distress, and d) was delivered in a critical or personally demeaning manner. Figure 5 (Appendix D) shows the use of overt punishers by supervisors in each school.
School 1 data indicated a relatively high rate of overt punishing interactions with students averaging about 1 every ~4 minutes. This rate increased after training in the intervention phase although with an apparently decreasing trend. It may be that the supervisors at School 1 took the training message of increasing contact with students, but were not able to assimilate and apply the associated strategies of high rates of positive rather than all-other or punishing interactions. School 2 and 3, although showing variability in rates of punishing interactions in the baseline phase, evidence a reduction to zero during intervention phase observations.
Table 7 - Pre/Post Use of Punishers
|
School |
Pre |
Post |
% Change |
|
1 |
0.19 |
0.27 |
42% |
|
2 |
0.06 |
0.01 |
- 83% |
|
3 |
0.01 |
0.04 |
300% |
|
Mean |
0.09 |
0.11 |
22% |
Presented as aggregated pre/post average (Table 7), School 1 again shows an increase in punishing interactions after training. The data for School 2 and 3 illustrate the drawbacks in using pre/post design as the sole method of analysis. Although School 2 and 3 had a reduction to zero in the rate of punishing interactions, the pre/post would indicate that School 3 actually experienced an increase in behavior. Taken as a whole, Schools 2 and 3 did not evidence a very significant rate of use or punishment to begin with and the data might indicate that the training may have prompted further reduction in a seldom used strategy for dealing with problem behavior by simple exposure to the issue.
E. Rates of Overall Student Problem Behavior. A key outcome of the Systematic Supervision training program is the intended end result of reducing student problem behavior through supervisor use of the strategies presented. In order to try to determine the effect of the training on student problem behavior, observations of the entire range of typical problem behaviors were conducted concurrently with the supervisor observations and data collection. Student problem behaviors were defined as: a) hands on others, play fighting, rough play, b) arguing, c) disrespect/defiance, d) misuse of equipment/not following game rules, e) teasing, name calling, offensive language, and f) fighting/ assault/ aggression. Due to the tremendously high rates of these behaviors, especially the low-level ones such as hands on others, teasing, and arguing, these data were aggregated to obtain a global rate of all problem behaviors. In Figure 6 (Appendix D) the rates of all student problem behaviors show School 1 with an obviously increasing trend in problem behavior after training (intervention) and at a fairly high rate as well (mean 4.53 problem behaviors per minute, range: 2.29 to 8.21 ). School 2 had a decreasing trend in problem behavior in baseline, but after training observations indicated an increase and an increasing trend in problem behaviors. Visual analysis of these data would seem to indicate that most of the increase in average rate (Table 8) might be due to two outlier data points obtained on sessions 13 and 16.
Review of observer notes indicates that a substitute (not observed as a supervisor participant) may have been on the playground on each of those two days. This potential disruption in routine may have had some effect on overall student behavior. However, even with that taken into consideration, there is a slight increasing trend in the rate of student problem behavior after intervention. School 3 observations, although
Table 8 - Rates of All Student Problem Behaviors
|
School |
Pre |
Post |
% Change |
|
1 |
4.23 |
4.53 |
7% |
|
2 |
3.13 |
3.44 |
10% |
|
3 |
3.48 |
2.14 |
- 39% |
|
Mean |
3.61 |
3.37 |
- 5% |
moderately variable, evidenced an apparent decrease and decreasing trend in student problem behavior following training. Keeping in mind that overall rates of problem behavior at ~ 2.5 per minute or lower should be considered to be relatively low while rates ranging from ~ 6 per minute or higher might be considered to be moderate to high. The aggregated pre/post average figures show an overall decrease in student problem behavior of ~5% with Schools 1 and 2 indicating a 7 to 10 percent increase in the rate of problem behavior and School 3 accounting for the overall decrease with an individual decrease of 39%. In the discussion section that follows we will undertake further analysis of these data taken as a whole and make recommendations and proposals for further study and possible Phase II activities.
Discussion and Recommendations
It is readily apparent that the data from each of the schools in the pilot study vary widely in most cases, both within and between schools. The effect of intervention is very difficult to determine as a result of this and other factors, such as the large number of variables present in the environment, etc. One avenue of explanation is to examine the situational conditions present in each school that might have reasonably affected these outcome measures.
School 1 presented the most challenging environment for implementation:
- The Principal was unexpectedly hospitalized and replaced by a district employee unfamiliar to the school, school staff, project, program, and students during program implementation. Since principal participation is key to the success of the program, this factor alone bears consideration in light of School 1?s apparent failure to implement successfully in terms of outcome measures.
- Two of the four regular supervisors took ill for a significant portion of the study and missed the training and most of the observation sessions. One of the remaining two regular supervisors declined to participate, not informing the observers of this until the start of the first day of observations. This supervisor was also not observed as called for in the project methodology. One alternate substitute supervisor who had also attended the training was subsequently observed as part of the study. This was problematic as this supervisor had heretofore been engaged only as a part-time bus driver and had little experience in playground supervision.
- During the program presentation, the supervisor who declined to participate was observed to openly denigrate the program, directly stated his intention not to use the Systematic Supervision skills and expressed the opinion that the program was not useful or valid. This behavior could have intimidated the other participants and affected their level of involvement.
- School 1 supervisors consistently evidenced a punishing approach to behavior management and support. Behavior support systems based on punishment typically have the effect of increasing aggressive and problem behaviors in students.
- School 1?s playground was being renovated, a fact not made clear before the study started. This involved the installation of a major play structure and associated bark-filled play area. Much of this installation took place during the study period and was disruptive to student and staff routine as well as behavior.
When these conditional factors are considered, along with the fact that a school?s intervention required follow-up and support from the administrator to fully implement and maintain, it may help explain the failure of School 1 to evidence any positive change in either supervisor or student target behaviors. In fact, since the strategies presented in Systematic Supervision were actively rejected by School 1 supervisory staff, unsurprisingly School 1 showed an increase in overall rates of student problem behavior during this period. The pilot study data seems relatively clear ? supervisory staff at School 1 were either uncommitted to the program, unavailable to participate, or adversarial in their response to the training and practices. Rather than reflect positive change in terms of effective systematic supervision skills, the opposite occurred. Also not surprising is that in conjunction with the supervisor rejection of the skills presented in the program, a concurrent increase in student problem behavior was evidenced.
School 2 and 3 both had experienced Principals, although both were in their first year with their respective schools. In addition, the Principal at school 2 was responsible for being the Principal at another local elementary school half of the time. Once again, along with supervisory staff by-in, strong administrative leadership is key to successful program implementation, fidelity, and maintenance and this was strongly challenged by the situations of each of these principals. In each of these schools, at least one of the regular supervisors became unavailable to participate during the course of the study. As in School 1, whenever there is a new person or substitute on the playground or other school environment, student behavior can be affected, sometimes drastically. This inconsistency promotes behavioral unpredictability for students and can contribute to non-typical behavioral expressions masking any effect the supervision training may be having.
Other confounds presented in one or more of the study sites include, but are not limited to, highly variable weather from day-to-day (rain, sun, temperature, etc.), the presence on playground of non-typical people (subs, parent, the principal(s), extra staff, observers, etc.), altered lunch or recess schedules, special events before or after observation sessions, and presence or absence of various students and student sub-groups in the environment (double recess, good behavior field trips, etc.).
One direction for further effort was to revisit School 1 and replicate the study with new personnel, controlling to the extent possible the potentially disruptive effects present during the pilot study. This replication was completed in March, 2004 by the Principal Investigator through the auspices of the project consultant and the Institute on Violence and Destructive Behavior at the University of Oregon (see below).
Replication Study at School 1
Methodology.The evaluation study of the Systematic Supervisi
