1. Definition of Evaluation
NQAAHE, for example, adopt a generic approach but apply it to an agency activity:
Evaluation (also called review) is used as the general description of the activity of an EQA-agency [external quality assurance agency] for collecting data, information and evidence about the quality of an institution, the quality of parts of an institution or the quality of its core activities: education and/or research. (INQAAHE, 2001, p. 2)
Campbell and Rozsnyai (2002, p. 31):
Evaluation is a general term denoting any process leading to judgments and/or recommendations regarding the quality of a unit. (A unit is an institution, programme, discipline.) Evaluation can be an internal process – self-evaluation – or an external one conducted by external experts, peers, or inspectors.
The UNESCO definition
Evaluation: The general process of a systematic and critical analysis leading to judgments and/or recommendations regarding the quality of a higher education institution or a programme. Evaluation is carried out through internal or external procedures. In the United Kingdom, evaluation is also called review. (Vlãsceanu, 2004, p. 37)
The Hungarian Higher Education Act (2000 amendment) defines quality evaluation as follows:
Quality evaluation: compares professional objectives with the operation of the institution in respect of result, efficiency and quality on the basis of a system of indicators prepared together with the professional community concerned. (Szanto, 2003)
The Tempus definition is more geared towards an ongoing process of (internal) evaluation of quality:
Quality evaluation – constant evaluation of the possibility of meeting the set demands.
Wojtczak (2002) defines evaluation in the context of medical education as:
A process that attempts to systematically and objectively determine the relevance, effectiveness, and impact of activities in light of their objectives. Evaluation can be related to structure, process, or outcome. One can distinguish these various types: individuals evaluation (see below) and program evaluation.
Formative individual evaluation provides feedback to an individual (usually a learner) in order to improve that individual's performance. This type of evaluation identifies areas for improvement and provides specific suggestions for improvement serving as an educational tool.
Summative individual evaluation measures whether specific objectives were accomplished by an individual in order to place a value on the performance of that individual. It may certify competency or lack of competency in performance in a particular area.
2. The aim of Evaluation
There are two common purposes in educational evaluation which are, at times, in conflict with one another. Educational institutions usually require evaluation data to demonstrate effectiveness to funders and other stakeholders, and to provide a measure of performance for marketing purposes. Educational evaluation is also a professional activity that individual educators need to undertake if they intend to continuously review and enhance the learning they are endeavouring to facilitate.
At least two evaluation functions or purposes should be considered:
a. Formative evaluations provide information to improve a product or process. Formative evaluation of new instructional materials would ideally be conducted with experts and selected target audience members prior to full-scale implementation. Expert review of content by instructional designers or subject-matter experts may provide useful information for modifying or revising selected strategies. Learner review is the process of determining if students can use the new materials, if they lack prerequisites, if they are motivated, and if they learn. In both types of formative review, problems may be discovered such as as spelling errors, confusing organizational structures, confusing media interfaces and navigation, or a need for more illustrations and examples. These voiced concerns would lead to revised and improved instruction.
b. Summative evaluations provide short-term effectiveness or long-term impact information to decide whether or not to adopt a product or process. Summative evaluation can occur just after new course materials were implemented in full (i.e., effectiveness evaluation), or several months to years after the materials were implemented in full (i.e, impact evaluation). It is important to specify what decisions will be made as a result of the evaluation, then develop a list of questions to be answered by the evaluation. You may want to know if learners met your objectives, if the innovation was cost-effective, if the innovation was efficient in terms of time to completion, or if the innovation had any unexpected outcomes. Data could include quantitative results of the innovation including criterion-referenced assessments to determine how well students met specified objectives. Data could also include qualitative interviews, observations, and artifact analyses.
3. The type of Evaluation
There are two main categories of evaluation, formative and outcomes. (See Figure 1.)
a. Formative evaluation is used to develop or improve program designs, and includes three subtypes: (1) market assessment, which includes market characterization and baseline studies. Market assessment and characterization are similar to market research and shed light on market workings to aid in program design and evaluation. These studies look at “how a market operates, the key information points, information hubs, and how products flow” (California Public Utilities Commission 2004:30). Baseline studies may be more narrowly focused on a program and often appear as the first part of an impact evaluation for a program relying primarily on direct effects. (2) Potential studies, which are “studies conducted to assess market baselines and savings potentials for different technologies and consumer markets. Potential is typically defined in terms of technical potential, market potential, and economic potential” (National Action Plan for Energy Efficiency 2007:B- 4) Potential studies may feed into baseline studies. (3) Process evaluation, which is “A systematic assessment of an energy efficiency program for the purposes of documenting program operations at the time of the examination, and identifying and recommending improvements to increase the program’s efficiency or effectiveness for acquiring energy resources while maintaining high levels of participant satisfaction” (National Action Plan for Energy Efficiency 2007:B-4)..
b. Outcomes evaluation helps in determining program results, and includes four subtypes:(4) impact evaluation, which is “an evaluation of the program-specific, directly induced changes (e.g., energy and/or demand usage) attributable to an energy efficiency program” (National Action Plan for Energy Efficiency 2007:B-2). It involves estimating how much energy was saved due to program efforts (“direct effects”), and often also other program impacts, including “non-energy effects” such as avoided emissions, improved health or comfort, dollars saved by households, etc. Statistical analysis, preferably including economic and behavioral data, should be part of impact evaluation. Statistical analysis also often includes billing data,1 data collected from measurement and verification (M&V)2 activities (which are conducted at the facility or project level, not at the program level) and deemed savings data.3 The second subtype of outcomes evaluation is (5) market effects evaluation. This is “an evaluation of the change in the structure or functioning of a market, or the behavior of participants in a market, that results from one or more program efforts. Typically the resultant market or behavior change leads to an increase in the adoption of energy-efficient products, services, or practices” (National Action Plan for Energy Efficiency 2007:B-3). Strategies that lead to market effects are considered indirect because energy savings result from them indirectly. Market effects evaluation encompasses evaluation of education and training program, which are also indirect strategies to achieve energy savings. Evaluation of education and training programs focuses on “documenting the degree to which the programs are achieving their desired effects within the markets targeted by the program, which is educating and training people on energy efficiency” (National Action Plan for Energy Efficiency 2007:C-6). Finally, (6) cost-effectiveness evaluation, sometimes known as cost-benefit evaluation or analysis, quantifies the cost of program implementation and compares this with program benefits to determine whether the energy efficiency program is cost effective compared to other options (National Action Plan for Energy Efficiency 2007). Cost-effectiveness evaluation is usually not stand-alone, relying on the results of other types of evaluation to develop inputs for in the analysis.
4. The principle of Evaluation
5. The approaches of evaluation
6. Evaluation method
| Including assessments, a quality evaluation will likely utilize multiple other methods (e.g., interviews, focus groups, observations with rubrics). Further information on specific methods can be obtained from the resources below. Focus Groups
|
|
| Surveys and Questionnaires
|
|
| Interviews
|
|
| Delphi and Nominal Group Techniques
|
|
| Direct Observation, Taping
|
7. Evaluation Instrument
8. How is the process of evaluation
Process evaluations are at aimed at enhancing your current program by understanding it more fully. Process evaluations measure what is done by the program, and for whom these services are provided. Ideally, process evaluations assist in the identification of “active ingredients” of treatment, and assess whether a program is meeting accepted standards of care. In general, process evaluations pose questions in two areas: coverage and process.
Coverage
· What proportion of those who might need the service(s) actually used it/them?
· Has the service, or network of services served the intended clients?
· What were the demographic and clinical characteristics of clients?
· What proportion of clients completed treatment and what were the characteristics of those who dropped out?
Process
· By what route have clients entered treatment?
· How long was the waiting list and how has it been managed?
· What actually happened to clients in treatment and is this what was intended?
· Were treatment plans consistent with the results of assessment?
· What was the average length of stay or the average number of appointments kept?
· How were discharge plans developed?
· Were services within the community treatment network well-co-ordinated?
Coverage and process questions can be asked at different levels of treatment (see
Framework workbook): activity, service, agency, or system levels. Procedures for different levels will be discussed in detail.
Test your knowledge. Write down whether each of the following questions is a coverage question or a process question:
a) Is the program serving clients it was intended to serve?
b) How many direct contact hours does each client receive?
c) What is the average age of program participants from each referral source?
d) What percent of clients complete the program?
9. Who is the evaluator
The Evaluator's Role
Whether you decide on an external or internal evaluator or some combination of both, it is important to think through the evaluator’s role. With your staff and stakeholders, think through all of the potential evaluator roles and relationships and determine which configuration makes the most sense given your particular situation, the purpose of the evaluation, and the questions you are attempting to address. One important role to think through is the relationship between the evaluator and primary stakeholders or the evaluation team. Questions to consider include:
- Should this relationship be distant or highly interactive?
- How much control should the evaluator have over the evaluation process as compared to the stakeholders/evaluation team?
- How actively involved should key staff and stakeholders be in the evaluation process?
Depending on the primary purpose of the evaluation and with whom the evaluator is working most closely (funders vs. program staff vs. program participants or community members), an evaluator might be considered a consultant for program improvement, a team member with evaluation expertise, a collaborator, an evaluation facilitator, an advocate for a cause, or a synthesizer. If the evaluation purpose is to determine the quality or importance of a program, you might look for an evaluator with methodological expertise and experience.
Bloom's Taxonomy of Learning Domains
The Three Types of Learning
There is more than one type of learning. A committee of colleges, led by Benjamin Bloom (1956), identified three domains of educational activities:
- Cognitive: mental skills (Knowledge)
- Affective: growth in feelings or emotional areas (Attitude)
- Psychomotor: manual or physical skills (Skills)
Since the work was produced by higher education, the words tend to be a little bigger than we normally use. Domains can be thought of as categories. Trainers often refer to these three domains as KSA (Knowledge, Skills, and Attitude). This taxonomy of learning behaviors can be thought of as "the goals of the training process." That is, after the training session, the learner should have acquired new skills, knowledge, and/or attitudes.
The committee also produced an elaborate compilation for the cognitive and affective domains, but none for the psychomotor domain. Their explanation for this oversight was that they have little experience in teaching manual skills within the college level (I guess they never thought to check with their sports or drama department).
This compilation divides the three domains into subdivisions, starting from the simplest behavior to the most complex. The divisions outlined are not absolutes and there are other systems or hierarchies that have been devised in the educational and training world. However, Bloom's taxonomy is easily understood and is probably the most widely applied one in use today.
Cognitive Domain
The cognitive domain (Bloom, 1956) involves knowledge and the development of intellectual skills. This includes the recall or recognition of specific facts, procedural patterns, and concepts that serve in the development of intellectual abilities and skills. There are six major categories, which are listed in order below, starting from the simplest behavior to the most complex. The categories can be thought of as degrees of difficulties. That is, the first one must be mastered before the next one can take place.
Category | Example and Key Words |
| Knowledge: Recall data or information. | Examples: Recite a policy. Quote prices from memory to a customer. Knows the safety rules. Key Words: defines, describes, identifies, knows, labels, lists, matches, names, outlines, recalls, recognizes, reproduces, selects, states. |
| Comprehension: Understand the meaning, translation, interpolation, and interpretation of instructions and problems. State a problem in one's own words. | Examples: Rewrites the principles of test writing. Explain in one's own words the steps for performing a complex task. Translates an equation into a computer spreadsheet. Key Words: comprehends, converts, defends, distinguishes, estimates, explains, extends, generalizes, gives Examples, infers, interprets, paraphrases, predicts, rewrites, summarizes, translates. |
| Application: Use a concept in a new situation or unprompted use of an abstraction. Applies what was learned in the classroom into novel situations in the work place. | Examples: Use a manual to calculate an employee's vacation time. Apply laws of statistics to evaluate the reliability of a written test. Key Words: applies, changes, computes, constructs, demonstrates, discovers, manipulates, modifies, operates, predicts, prepares, produces, relates, shows, solves, uses. |
| Analysis: Separates material or concepts into component parts so that its organizational structure may be understood. Distinguishes between facts and inferences. | Examples: Troubleshoot a piece of equipment by using logical deduction. Recognize logical fallacies in reasoning. Gathers information from a department and selects the required tasks for training. Key Words: analyzes, breaks down, compares, contrasts, diagrams, deconstructs, differentiates, discriminates, distinguishes, identifies, illustrates, infers, outlines, relates, selects, separates. |
| Synthesis: Builds a structure or pattern from diverse elements. Put parts together to form a whole, with emphasis on creating a new meaning or structure. | Examples: Write a company operations or process manual. Design a machine to perform a specific task. Integrates training from several sources to solve a problem. Revises and process to improve the outcome. Key Words: categorizes, combines, compiles, composes, creates, devises, designs, explains, generates, modifies, organizes, plans, rearranges, reconstructs, relates, reorganizes, revises, rewrites, summarizes, tells, writes. |
| Evaluation: Make judgments about the value of ideas or materials. | Examples: Select the most effective solution. Hire the most qualified candidate. Explain and justify a new budget. Key Words: appraises, compares, concludes, contrasts, criticizes, critiques, defends, describes, discriminates, evaluates, explains, interprets, justifies, relates, summarizes, supports. |
Bloom's Taxonomy refers to a classification of the different objectives that educators set for students (learning objectives). The taxonomy was first presented in 1956 through the publication "The Taxonomy of Educational Objectives, The Classification of Educational Goals, Handbook I: Cognitive Domain," by Benjamin Bloom (editor), M. D. Englehart, E. J. Furst, W. H. Hill, and David Krathwohl. It is considered to be a foundational and essential element within the education community as evidenced in the 1981 survey "Significant writings that have influenced the curriculum: 1906-1981", by H. G. Shane and the 1994 yearbook of the National Society for the Study of Education. A great mythology has grown around the taxonomy, possibly due to many people learning about the taxonomy through second hand information. Bloom himself considered the Handbook, "one of the most widely cited yet least read books in American education."[1]
Key to understanding the taxonomy and its revisions, variations, and addenda over the years is an understanding that the original Handbook was intended only to focus on one of the three domains (as indicated in the domain specification in title), but there was expectation that additional material would be generated for the other domains (as indicated in the numbering of the handbook in the title). Bloom also considered the initial effort to be a starting point, as evidenced in a memorandum from 1971 in which he said, "Ideally each major field should have its own taxonomy in its own language - more detailed, closer to the special language and thinking of its experts, reflecting its own appropriate sub-divisions and levels of education, with possible new categories, combinations of categories and omitting categories as appropriate."[2]
Bloom's Taxonomy divides educational objectives into three "domains:" Affective, Psychomotor, and Cognitive. Within thetaxonomy learning at the higher levels is dependent on having attained prerequisite knowledge and skills at lower levels (Orlich, et al. 2004). A goal of Bloom's Taxonomy is to motivate educators to focus on all three domains, creating a moreholistic form of education.
Contents [hide] · 5 Notes |
[edit]Affective
Skills in the affective domain describe the way people react emotionally and their ability to feel another living thing's pain or joy. Affective objectives typically target the awareness and growth in attitudes, emotion, and feelings.
There are five levels in the affective domain moving through the lowest order processes to the highest:
The lowest level; the student passively pays attention. Without this level no learning can occur.
The student actively participates in the learning process, not only attends to a stimulus; the student also reacts in some way.
The student attaches a value to an object, phenomenon, or piece of information.
The student can put together different values, information, and ideas and accommodate them within his/her own schema; comparing, relating and elaborating on what has been learned.
The student holds a particular value or belief that now exerts influence on his/her behaviour so that it becomes a characteristic.
[edit]Psychomotor
Skills in the psychomotor domain describe the ability to physically manipulate a tool or instrument like a hand or a hammer. Psychomotor objectives usually focus on change and/or development in behavior and/or skills.
Bloom and his colleagues never created subcategories for skills in the psychomotor domain, but since then other educators have created their own psychomotor taxonomies.[3]
[edit]Cognitive
Categories in the cognitive domain of Bloom's Taxonomy (Anderson & Krathwohl, 2001)
Skills in the cognitive domain revolve around knowledge, comprehension, and critical thinking of a particular topic. Traditional education tends to emphasize the skills in this domain, particularly the lower-order objectives.
There are six levels in the taxonomy, moving through the lowest order processes to the highest:
Knowledge
Exhibit memory of previously-learned materials by recalling facts, terms, basic concepts and answers
§ Knowledge of specifics - terminology, specific facts
§ Knowledge of ways and means of dealing with specifics - conventions, trends and sequences, classifications and categories, criteria, methodology
§ Knowledge of the universals and abstractions in a field - principles and generalizations, theories and structures
Questions like: What are the health benefits of eating apples?
Comprehension
Demonstrative understanding of facts and ideas by organizing, comparing, translating, interpreting, giving descriptions, and stating main ideas
§ Translation
§ Interpretation
§ Extrapolation
Questions like: Compare the health benefits of eating apples vs. oranges.
Application
Using new knowledge. Solve problems to new situations by applying acquired knowledge, facts, techniques and rules in a different way
Questions like: Which kinds of apples are best for baking a pie, and why?
Analysis
Examine and break information into parts by identifying motives or causes. Make inferences and find evidence to support generalizations
§ Analysis of elements
§ Analysis of relationships
§ Analysis of organizational principles
Questions like: List four ways of serving foods made with apples and explain which ones have the highest health benefits. Provide references to support your statements.
Synthesis
Compile information together in a different way by combining elements in a new pattern or proposing alternative solutions
§ Production of a unique communication
§ Production of a plan, or proposed set of operations
§ Derivation of a set of abstract relations
Questions like: Convert an "unhealthy" recipe for apple pie to a "healthy" recipe by replacing your choice of ingredients. Explain the health benefits of using the ingredients you chose vs. the original ones.
Evaluation
Present and defend opinions by making judgments about information, validity of ideas or quality of work based on a set of criteria
§ Judgments in terms of internal evidence
§ Judgments in terms of external criteria
Questions like: Do you feel that serving apple pie for an after school snack for children is healthy? Why or why not?
Some critiques of Bloom's Taxonomy's (cognitive domain) admit the existence of these six categories, but question the existence of a sequential, hierarchical link.[4] Also the revised edition of Bloom's taxonomy has moved Synthesis in higher order than Evaluation. Some consider the three lowest levels as hierarchically ordered, but the three higher levels as parallel. Others say that it is sometimes better to move to Application before introducing concepts[citation needed]. This thinking would seem to relate to the method of problem-based learning.
No comments:
Post a Comment