Quality of Research
Review Date: December 2006
Documents Reviewed
The documents below were reviewed for Quality of Research. The research point of contact can provide information regarding the studies reviewed and the availability of additional materials, including those from more recent studies that may have been conducted.
Study 1Flay, B., Acock, A., Vuchinich, S., & Beets, M. (2005). Progress report of the randomized trial of Positive Action in Hawai'i: End of third year of intervention (spring 2005). Unpublished manuscript, Oregon State University, Corvallis. Study 2Flay, B., & Slagel, M. (2006). The Positive Action family program: A pilot randomized trial. Unpublished manuscript. Study 3Flay, B. R., & Allred, C. G. (2003). Long-term effects of the Positive Action program. American Journal of Health Behavior, 27(Suppl. 1), S6-S21.  Study 4Flay, B. R., Allred, C. G., & Ordway, N. (2001). Effects of the Positive Action program on achievement and discipline: Two matched-control comparisons. Prevention Science, 2(2), 71-89. Study 5Flay, B. R., Allred, C. G., & Ordway, N. (2001). Effects of the Positive Action program on achievement and discipline: Two matched-control comparisons. Prevention Science, 2(2), 71-89.
Supplementary Materials Description of Positive Action evaluation instruments
Positive Action, Inc. Levels of implementation plans. (2004). Twin Falls, ID: Author.
Outcomes
| Outcome 1: Academic achievement |
|
Description of Measures
|
Academic achievement was measured using several standardized instruments: the Hawaii Content and Performance Standards (HCPS) test, Florida Reading Test (FRT), Florida Comprehensive Assessment Test (FCAT; 4th grade), Norm Referenced Tests (NRT) of reading and math (8th grade), Florida Writes Test, High School Competency Test (HSCT) of communication and mathematics, Scholastic Aptitude Test (SAT), American College Testing (ACT) test, and TerraNova Comprehensive Test of Basic Skills.
|
|
Key Findings
|
Over 4 academic years, intervention schools had higher rates of math proficiency (26%) than did control schools (21%), a difference that represents a small effect size (Cohen's d = 0.34).
Over 4 academic years, intervention schools had higher rates of reading proficiency (52%) than did control schools (44%), a difference that represents a medium effect size (Cohen's d = 0.73).
Another evaluation reported that elementary schools that implemented Positive Action produced average FRT scores that were 40% higher than all other schools and 45% higher than comparable schools with students of similar socioeconomic status.
Middle schools with relatively high enrollment from elementary schools that received Positive Action ("PA primary schools") produced higher average standardized test scores in math and reading than comparable schools with relatively low enrollment from PA primary schools. Improvement in average test scores ranged from 10.8% to 20.6%, depending on the subject matter and the percentage of enrollees from PA primary schools.
High schools with relatively high enrollment from PA primary schools reported 11% higher average scores on the Florida Writes Test and 10% higher average SAT scores than high schools with relatively low enrollment from PA primary schools. Similar results were observed for the percentages of students who passed the communication section of the HSCT.
Two additional evaluations reported improvements in elementary schools participating in Positive Action as measured by State achievement tests in Hawaii and Nevada, respectively. Hawaii scores in math and reading improved an average of 52%, while Nevada scores on math, reading, and language improved by an average of 16%.
|
|
Studies Measuring Outcome
|
Study 1, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.8
(0.0-4.0 scale)
|
| Outcome 2: Problem behaviors (violence, substance use, disciplinary referrals, and suspensions) |
|
Description of Measures
|
Problem behavior was measured by self-reported substance use, violence, and sexual behavior (for grades 5 and above); school records of disciplinary referrals, suspensions, and incidents related to violence, disobedience, sexual behavior, crime, and use of alcohol, tobacco, and other drugs; and school records of incident reports of student-to-student and student-to-staff violence and weapons possession. In one study, measures also included school records of felonies, misdemeanors, school suspensions, and violations of department and school rules.
|
|
Key Findings
|
In one study, students from intervention schools were 30% less likely to have used alcohol, 78% less likely to have been drunk, and 100% less likely to have tried an illegal drug than students enrolled in primary schools without the intervention.
One study found a 61% reduction in the occurrence of any violent behavior among boys in intervention schools compared with those in primary schools without the intervention (p = .001). No statistically significant reduction of violent behavior was observed among girls, but the rate of any reported violence by girls in grade 5 was very low.
In intervention schools, 0.74% of boys reported voluntary sexual behavior, compared with 4.55% of boys at primary schools without Positive Action.
Over one school year, schools participating in Positive Action reported a suspension rate of 1.63 per 1,000 compared with 6.2 per 1,000 for control schools. Three years earlier, the two groups of schools respectively reported suspension rates of 1.36 and 1.46 per 1,000. The effect size was medium for this measure (Cohen's d = 0.63).
Middle schools with relatively high enrollment from PA primary schools had 52%-71% lower rates of problem behaviors such as drug use, violence, "disrespectful, disobedient, and, disorderly behaviors," and property crime, compared with middle schools with relatively low enrollment from PA primary schools.
High schools with medium enrollment from PA primary schools had 17%-50% fewer problem behaviors, including substance use, violence, sexual behavior, falsifying records, and suspensions, than did high schools with relatively low enrollment from PA primary schools. High schools with high enrollment from PA primary schools had 25%-63% fewer of these problem behaviors than did schools with low enrollment from these primary schools.
Rates of violence, including felonies, misdemeanors, and rule violations, among PA primary schools in Hawaii were 51%-79% lower than rates in comparable schools. Rates of student-to-student and student-to-staff violence in Nevada's Positive Action schools were respectively 87%-100% lower than the rates among comparable schools in Nevada.
|
|
Studies Measuring Outcome
|
Study 1, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.4
(0.0-4.0 scale)
|
| Outcome 3: School absenteeism |
|
Description of Measures
|
Schoolwide attendance was assessed using multiple measures: average daily absences, number of students absent during the school year, number of days absent (> 21 days) during the school year, average number of days absent, and absenteeism rate.
|
|
Key Findings
|
Four studies examined change in attendance in schools that used Positive Action. In one study, average per student daily absences in Positive Action schools decreased over 4 years while other schools maintained stable rates of daily absence, with students averaging 9.8 days absent in intervention schools versus 11 days absent in control schools during the 4th year of the study. This result represents a medium effect size (Cohen's d = 0.55). Three other studies did not find statistically significant differences in absenteeism between students in PA primary schools and other elementary schools.
In other analyses, middle schools with relatively high enrollment from PA primary schools had 75% less absenteeism than middle schools with relatively low enrollment from PA primary schools. Similarly, high schools with relatively high enrollment from PA primary schools had 8% less absenteeism than high schools with relatively low enrollment from PA primary schools.
|
|
Studies Measuring Outcome
|
Study 1, Study 3, Study 4, Study 5
|
|
Study Designs
|
Experimental, Quasi-experimental
|
|
Quality of Research Rating
|
2.5
(0.0-4.0 scale)
|
| Outcome 4: Family functioning |
|
Description of Measures
|
Family functioning was measured using 16 self-report items that assessed family conflict, family cohesion, and quality of parent-child bonding. The items were administered at pretest and posttest.
|
|
Key Findings
|
When compared with control families, families that received the Positive Action curriculum had more improved family functioning in all three areas. Effect sizes were small for family conflict (Cohen's d = 0.36), small for family cohesion (Cohen's d = 0.34), and medium for parent-child bonding (Cohen's d = 0.59).
|
|
Studies Measuring Outcome
|
Study 2
|
|
Study Designs
|
Experimental
|
|
Quality of Research Rating
|
2.2
(0.0-4.0 scale)
|
Study Populations
The following populations were identified in the studies reviewed for Quality of
Research.
|
Study
|
Age
|
Gender
|
Race/Ethnicity
|
|
Study 1
|
6-12 (Childhood)
|
50% Female
50% Male
|
55.8% Native Hawaiian or other Pacific Islander
16.1% Race/ethnicity unspecified
13.4% White
10.7% Asian
2.2% Hispanic or Latino
1.3% Black or African American
0.5% American Indian or Alaska Native
|
|
Study 2
|
6-12 (Childhood)
13-17 (Adolescent)
18-25 (Young adult)
26-55 (Adult)
|
50% Female
50% Male
|
Data not reported/available
|
|
Study 3
|
6-12 (Childhood)
|
50% Female
50% Male
|
55.3% White
22.3% Black or African American
18.3% Hispanic or Latino
4.1% Race/ethnicity unspecified
|
|
Study 4
|
6-12 (Childhood)
|
50% Female
50% Male
|
54.9% Native Hawaiian or other Pacific Islander
32.3% Race/ethnicity unspecified
12.8% White
|
|
Study 5
|
6-12 (Childhood)
|
50% Female
50% Male
|
53.6% White
22.9% Hispanic or Latino
18% Black or African American
5.5% Race/ethnicity unspecified
|
Quality of Research Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the Quality of Research for an intervention's
reported results using six criteria:
For more information about these criteria and the meaning of the ratings, see Quality of Research.
|
Outcome
|
Reliability
of Measures
|
Validity
of Measures
|
Fidelity
|
Missing
Data/Attrition
|
Confounding
Variables
|
Data
Analysis
|
Overall
Rating
|
|
1: Academic achievement
|
3.3
|
3.3
|
1.5
|
2.5
|
2.7
|
3.4
|
2.8
|
|
2: Problem behaviors (violence, substance use, disciplinary referrals, and suspensions)
|
2.1
|
2.2
|
1.5
|
2.5
|
2.7
|
3.5
|
2.4
|
|
3: School absenteeism
|
2.5
|
2.5
|
1.3
|
2.5
|
2.7
|
3.5
|
2.5
|
|
4: Family functioning
|
2.5
|
2.0
|
0.0
|
3.0
|
3.0
|
2.5
|
2.2
|
Study Strengths The researchers used appropriate matching variables when making comparisons between schools, and they demonstrated the equivalence of intervention and control schools. The authors also did an exemplary job of matching schools and attending to differences in feeder school rates in the analysis of middle and high school data in one study. The authors used strong analytic techniques.
Study Weaknesses Schools were not randomly assigned to the intervention in some of the studies. In one study, the data from one sample were retrospective, and extraneous variables may have affected the outcomes. Some of the data presented (e.g., disciplinary referrals, incidents of violence) were based on administrative records, which are often unreliable and inconsistent from school to school.
|
|
Readiness for Dissemination
Review Date: December 2006
Materials Reviewed
The materials below were reviewed for Readiness for Dissemination. The implementation point of contact can provide information regarding implementation of the intervention and the availability of additional, updated, or new materials.
Kits:
- Positive Action counselor's kit [includes manual, posters, manipulatives, and materials for six individuals]. (1998). Twin Falls, ID: Positive Action, Inc.
- Positive Action elementary climate development kit [includes leader's manual, music, posters, support staff and parent's manuals, calendars, tokens, stickers, balloons, ICU Box, words-of-the-week cards, "positive notes" note cards, and notepads]. (1989). Twin Falls, ID: Positive Action, Inc.
- Positive Action elementary orientation training workshop kit [includes manual and materials for 25 individuals]. (2004). Twin Falls, ID: Positive Action, Inc.
- Positive Action family classes instructor's kit [includes manuals and posters]. (2006). Twin Falls, ID: Positive Action, Inc.
- Positive Action family kit [includes manual, posters, music, manipulatives, and materials for six individuals]. (1995). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 7 instructor's kit [includes teacher manual, posters, music, manipulatives, and materials for 30 students]. (2007). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 2 instructor's kit [includes teacher manual, posters, music, manipulatives, and materials for 30 students]. (2007). Twin Falls, ID: Positive Action, Inc.
- Positive Action high school kit II: "Lives on the line" play instructor's kit [includes teacher manual, posters, music, manipulatives, and materials for 30 students]. (2007). Twin Falls, ID: Positive Action, Inc.
Manuals:
- Positive Action community manual. (2004). Twin Falls, ID: Positive Action, Inc.
- Positive Action conflict resolution manual. (2000). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 8 teacher's manual. (1990). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 5 drug education supplement teacher's manual. (1990). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 5 teacher's manual. (1982). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 4 teacher's manual. (2006). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 1 teacher's manual. (1982). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 6 teacher's manual. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 3 teacher's manual. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action high school orientation training workshop manual. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action kindergarten teacher's manual. (1986). Twin Falls, ID: Positive Action, Inc.
- Positive Action media training workshop manual. (2001). Twin Falls, ID: Positive Action, Inc.
- Positive Action middle school drug education supplement teacher's manual. (1990). Twin Falls, ID: Positive Action, Inc.
- Positive Action middle school orientation training workshop manual. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action ongoing training workshop manual. (2004). Twin Falls, ID: Positive Action, Inc.
- Positive Action secondary climate development manual. (2000). Twin Falls, ID: Positive Action, Inc.
Other materials:
- Positive Action academic standards alignment, California, grade 5. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action academic standards alignment, California, grade 4. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action academic standards alignment, grades 7 and 8 [CD-ROM]. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action academic standards alignment, Hawaii, grade 5. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action academic standards alignment, Hawaii, grade 4. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action academic standards alignment, Illinois, grade 5. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action academic standards alignment, Illinois, grade 4. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action evaluation services. (2007). Twin Falls, ID: Positive Action, Inc.
- Positive Action grade 4 instructor's kit training CD [CD-ROM]. (2005). Twin Falls, ID: Positive Action, Inc.
- Positive Action Materials Catalog
Readiness for Dissemination Ratings by Criteria (0.0-4.0 scale)
External reviewers independently evaluate the intervention's Readiness for Dissemination
using three criteria:
- Availability of implementation materials
- Availability of training and support resources
- Availability of quality assurance procedures
For more information about these criteria and the meaning of the ratings, see Readiness for Dissemination.
Implementation
Materials
|
Training and Support
Resources
|
Quality Assurance
Procedures
|
Overall
Rating
|
|
4.0
|
4.0
|
4.0
|
4.0
|
Dissemination Strengths Implementation materials are very detailed and include everything a potential user would need to implement the program. The program implementation plan includes sample 3-year plans along with key questions to guide the implementation planning process. The Family Kit is a very valuable component, with structured materials and activities that support and extend the school-based program. A wide variety of both initial and ongoing training and support is available to users. All the material focuses substantially on implementation fidelity, and a comprehensive evaluation guidebook is provided to directly support quality assurance.
Dissemination Weaknesses No weaknesses were identified by reviewers.
|
|
Costs
The cost information below was provided by the developer. Although this cost information may have been updated by the developer since the time of review, it may not reflect the current costs or availability of items (including newly developed or discontinued items). The implementation point of contact can provide current information and discuss implementation requirements.
|
Item Description
|
Cost
|
Required by Developer
|
|
Instructor kits: Pre-K through grade 12 kits, Grade 5 Drug Education Supplement Kit, Middle School Drug Education Supplement Kit, Bullying Supplement Kit (elementary or secondary)
|
$250-$460 each
|
Yes
|
|
Additional component kits: Climate Development Kit (elementary or secondary), Conflict Resolution Kit, Counselor's Kit, Family Kit, Family Classes Instructor's Kit, Parenting Classes Instructor's Kit, Community Kit
|
$85-$1,450 each
|
No
|
|
1- to 5-day, on-site orientation, ongoing/refresher training, training of trainers, or professional development training
|
$2,000 per site per day plus $1,000 per travel day and travel expenses
|
No
|
|
Off-site training (at another site's training)
|
$250 per participant per day
|
No
|
|
Webinar training
|
$250 per hour per site
|
No
|
|
Self-training kit
|
$250 each
|
No
|
|
Implementation design and monitoring consultation
|
$300 per hour
|
No
|
|
Evaluation services consultation
|
$400 per hour
|
No
|
|
Readiness consultation
|
$300 per hour
|
No
|
|
Professional development consultation
|
$300 per hour
|
No
|
|
Other consultation
|
$300 per hour
|
No
|
|
Implementation monitoring surveys (teachers complete 10-minute reports at end of each unit)
|
- Free to download/print from Web site, then administer and analyze the data, or
- $250 per teacher for online administration, with data analysis completed by Positive Action evaluator
|
No
|
|
Progress monitoring surveys (students report on use of Positive Action skills or teachers rate students)
|
- Free to download/print from Web site, then administer and analyze the data, or
- $2.50 per student for online administration, with data analysis completed by Positive Action evaluator
|
No
|
|
Outcome monitoring surveys (pre- and posttest surveys of students on effects of Positive Action on student behaviors)
|
- Free to download/print from Web site, then administer and analyze the data, or
- $5 per student for online administration, with data analysis completed by Positive Action evaluator
|
No
|
|
Process surveys (end-of-year teacher survey)
|
- Free to download/print from Web site, then administer and analyze the data, or
- $250 per teacher for online administration, with data analysis completed by Positive Action evaluator
|
No
|
|
Professional development kit
|
$350 each
|
No
|
|
Replications
No replications were identified by the developer.
|
|
|
