COLLECTING ACCOUNTABILITY INFORMATION
John G. Richardson
Extension Program Delivery and Accountability Leader
INTRODUCTION
Accountability is required for Extension programs and funding needs. This information must be provided in ways and means desired by the respective audiences. Program impacts and success stories provide information that is crucial for accountability purposes. In order to provide appropriate information to audiences at the local, state, and national levels, a planned program assessment process should be implemented for obtaining the needed information.
There are several means available for collecting program assessment information. For most accountability needs, one or a limited combination of assessment methods will suffice. Also, it should be kept in mind that every Extension activity, event, or program does not need to be formally evaluated in order to provide accountability information as needed. A well planned and implemented program assessment process, however, should be initiated and maintained continuously, since several assessment tools are available which can be used with ease to accomplish this goal. Regardless of the method or methods used, the Extension educator should strive to obtain levels of accuracy and sufficient documentation to support validity requirements in assessing program results and impacts. The assessment methods that follow should meet most accountability requirements. A review of these requirements may be made by referring to "Extension Accountability", a companion publication of the Department of Agricultural and Extension Education, AEE 96-02.
ASSESSMENT METHODS
Existing Records (records/data analysis)- This program assessment method is the use of information or records that already exist or can be obtained with minimal planning. Information available in public reports, such as census reports, available in libraries; health records from local or state health organizations; youth crime rates from police departments; school dropout rates from school boards; and other data bases, such as agricultural statistics, which are readily available from Department of Agriculture publications, can provide significant insight into changing situations and other factors that may be directly or indirectly influenced by Extension programs. For example, some information, such as fertilizer reports, can indicate changing levels of use and analysis adjustments by farmers and other users.
Observation- This means of gathering accountability information is the deliberate act of looking at some object, event, group, individual actions or behavior to obtain program assessment information. Collecting information via observation is usually easy to implement and can be quite efficient. Observation for program assessment should be well thought out and a planned observation guide developed in order for the observer to look for those aspects that will provide the program information needed. Informed and structured observations made by professionals and trained observers are generally acceptable as a valid means of providing program result information. A precaution in using this means of evidence gathering is that the assessment is based on the individual perceptions and knowledge level of the person who makes the assessment. Planned observation that includes more than one observer can be helpful in assuring the validity of assessments. Some examples of program assessment by observation may include visual assessments of waste application procedures following extensive waste management programs; a look at the application of Best Management Practices in Christmas tree production resulting from programs focused on water quality improvement; observing members of a group to see their reactions to information being presented, through non-verbal behaviors such as their body language; or assessing the increasing beauty of a community resulting from actions taken by participants in community beautification programs.
In these examples, the observation guide for waste application may include amounts of solids that can be seen, degree of water containment, equipment operation, and/or crop responses to applications. Obviously, for the Christmas trees, a list of the management practices would provide the needed guide for observing for those practices implemented. For the presentation, such things as body language, sidebar conversations, attentiveness, demonstrated enthusiasm, follow-up questions, or others, such as people leaving or remaining long past the presentation for further discussions among small groups, can provide effective observation assessments. A guide for the beautification observation may include such things as home and business repairs, up-dated or painted facades, lawns mowed, landscaping improved, litter removed, trees planted, or other factors that were included in the educational program.
Unobtrusive Measures- A means of evaluating that can provide specific information to the evaluator without any obtrusive presence needed in order to obtain information. For example, through this means, electronic instruments can make reliable measurements of noise levels of machinery when assessing worker protection programs to reduce noise; or records can be made of the increased number of requests for additional information on subject and the information Extension has available on the subject. This means of program assessment is especially useful when direct observation or available records are unavailable or impractical.
Content Analysis- A way of assessing effects of change through analysis of materials, levels of nutrients, moisture levels, or other factors that may be assessed through actual measurement of components of a specific entity. Assessment of soil fertility, tissue analysis, materials analysis, nutrient tests, fat levels of hamburger, sausage, and other products, and numerous other analytical applications, may be made of this means of evaluation. An example are the water tests used to determine any change in levels of silt and nutrients during and following educational programs focusing on water quality.
Testimonial- This program assessment tool is the act of an individual or group expressing, through verbal or written means, their direct reaction to, or experiences resulting from, Extension's educational program efforts. This program assessment means can be a favorite among Extension faculty for evaluating educational programs or activities when numerous compliments are being shared voluntarily by clientele. While voluntary testimonials are usually preferred, they may also be requested, especially when certain program results may not otherwise be available, or communication of results by program users is requested. When only voluntary testimonials are depended upon for obtaining accountability information, it may be wise to use this method as a supplement or reinforcement to other planned methods of program assessment. Examples of testimonials may include current 4-H members stating the positive impact 4-H has had on their lives, or adults expressing gratitude for adopting healthy eating practices as a result of Extension's nutrition education programs.
Interview- The direct contact of an individual with another for the purpose of obtaining information. This means of evaluation offers the opportunity to conduct planned assessment that is specific to program needs. The interview may be obtrusive and requires skill to complete satisfactorily, but can be an effective means of seeking answers to desired questions through direct personal contact. An interview guide should be developed which contains the specific questions for which answers will be requested. A precaution in this form of program assessment is that care should be taken to avoid imposition of one's own views on others in the interview process. Also, non-verbal behavior, such as facial expressions, body movements, such as shoulder shrugs, voice pitch, demonstration of anger or happiness, etc., of the respondent should be observed in order to assess any potential unspoken reactions that may be communicated.
Questionnaire- A survey instrument is used to obtain specific reactions or input from all members of an audience or a sampling of the targeted audience. This means of planned evaluation is used rather extensively in Cooperative Extension education program assessments, and in reality may be over used. Collecting accountability information by questionnaire may require excessive time and other resources to plan and implement, especially since other means are available which are acceptable to those who need or require such information. Several State Major Plan task forces are developing guides which should be valuable for use in creating locally oriented questionnaires. Some special benefits of a questionnaire are that even though it is intrusive, response is voluntary, and anonymity of the person completing the questionnaire is protected.
Anecdotal Information- Program accountability information that may be difficult to verify or quantify, but can provide insights into program outcomes and successes. This information may be obtained through various sources. It may or may not be attributed, and may be considered hearsay information. While this means of evaluating Extension programs may not be as reliable as some other forms due to its hearsay nature, its validity is usually supported through testimonials or clientele response. This information should be considered in an overall program assessment even though other, more definitive program assessment techniques, may be necessary to meet organizational needs. Often, an agent is informed by an acquaintance of overhearing a discussion at a garden supply center, for example, where several individuals were commenting on the outstanding educational programs provided by the agent. Obviously, this type of program feedback is important, and can provide a basis for follow-up assessments using more definitive methods, such as informal interviews with program participants.
Tests- Often, pre-tests and post-tests are used as program assessment tools. This form of evaluation can be used to assess the level of knowledge of individuals regarding specific subjects. For knowledge level testing, the pre-test determines the level of knowledge prior to the planned educational input, while the post-test will provide the knowledge level at the conclusion of the program. This means of program assessment is especially useful to assess the impact of a planned educational initiative that spans over an extended time period. This tool is sometimes used as a measure of the effectiveness of a meeting or other program delivery method. Since actual knowledge gained and retained is the goal of educational programs, short term memory may significantly influence the overall validity of a post-test that is administered immediately following program inputs. While a stimulating program presentation may result in an indicated high knowledge level gain, as shown by a post-test immediately at the conclusion of the meeting or presentation, the same test administered a few days or weeks later may have quite different results. Therefore, since the normal recall time for new information that is not experienced or integrated into the prior knowledge base is only a few hours, a post-test should be given days or weeks later in order to assess the actual knowledge change that has occurred.
References
Dick, W. and L. Carey. (1978). The Systematic Design of Instruction. Scott, Foresman and Company, Dallas, Texas
Frutchey, F. P., Bryan, D., Fessenden, J., Gallup, G., et. al., (1962). Evaluation in Extension. H. M. Ives and sons. Inc., Topeka, Kansas.
Gagne' R. M. (1965). Conditions of Learning. Holt, Rinehart, and Winston, Inc., New York.
Liles, R. T., R. D. Mustian, and J. Pettitt. (1988). Working With Our Publics, Module 2, The Extension Education Process. Raleigh, NC: North Carolina Agricultural Extension Service and Department of Adult and Community College Education, North Carolina State University.
Warner, P. and R. C. Maurer. (1984). Methods of Program Evaluation. Kentucky Cooperative Extension Service, University of Kentucky, Lexington, Kentucky.
Worden, P. and P. Neumaier. (1987). A Source Book for Program Evaluation and Accountability. Colorado State University Cooperative Extension, Fort Collins, Colorado.
For additional information contact the Department of Agricultural and Extension Education, 117 Ricks Hall, Campus Box
7607,
North Carolina State University, Raleigh, North Carolina 27695-7607
|| North Carolina State University || Cooperative Extension ||
|| Table of Contents ||
Publication Number AEE 96-03
PEER REVIEWED BY: CLYDE CHESNEY, DONALD COBB, RONALD JARRETT, JOY STATON AND KENNETH VAUGHN, NORTH CAROLINA COOPERATIVE EXTENSION SERVICE, AND DAVID JENKINS AND RONALD SHEARON, DEPARTMENT OF AGRICULTURAL EXTENSION EDUCATION, NORTH CAROLINA STATE UNIVERSITY.
Published by: NORTH CAROLINA COOPERATIVE EXTENSION SERVICE
Created June 6, 1996
Updated May 15, 1997
created by Michael Ebbs