Types of Evaluation Designs
True Control Group: Pre- Post-test Design
In this design, there are two groups, an experimental group
and a control group. Both have been randomly selected and both complete the
pre-test. Only the experimental group gets the intervention, then both groups
complete the post-test.
Step 1.
Identify people or groups, some of which could get the intervention.
Step 2.
Pre-test everyone.
Step 3.
Randomly assign some of the participants to either the control group or the
experimental group.
Step 4.
Deliver the intervention to the experimental group. The control group may
receive an alternative intervention or nothing at all.
Step 5.
Post-test both groups with the same instrument under the same conditions.
Example:
A Native non-profit wants to
test the curriculum content of an HIV education program they are developing for
group settings. They decide to do a formative evaluation on the curriculum
using a ?true control group? design. They recruit support from a substance
abuse provider in the community and ask to work with a group of clients for
about an hour. In a classroom setting, they ask everyone to complete a KABB on
HIV and risky behavior. They then randomly sort the group into either an
experimental or control group e.g. everyone counts off and all the ones are in
the control group and the twos are in the experimental group. The group level
HIV education program is delivered to the experimental group. The control group
gets free time or receives information on an unrelated subject. Both groups are
called back together and asked to complete the post-test.
True Control Group: Post-test Only
In this design, an experimental and a control group are
formed; however there is no pre-test. Instead, the experimental group gets the
intervention and then both groups are measured at the end of the program.
Example:
Health providers in a Native
Nation want to find a method for raising awareness about HIV and risky
behaviors in the adolescents in the community. They have identified two
programs that can be delivered in the classroom; however, there is a
considerable difference in the cost of the two programs in terms of not only
materials, but also staff time. Before making a final recommendation, they
decide to pilot each program in a different school setting. Since the goals and
objectives of both programs were there, both groups receive the same post-test
at the conclusion of the program year. Changes in the KABB of the students can
be attributed to their respective program.
Non-Equivalent Control Group: Pre- Post-Test
Design
In this design, two groups which are similar, but which were
not formed by random assignment, are measured both before and after one of the
groups gets the program intervention (experiment).
Step 1. Identify people who will
be getting the program.
Step 2. Identify people who are
not getting the program, but are other ways very similar.
Step 3. Pre-test both groups.
Step 4. Deliver the intervention
to the experimental group. The control group may receive an alternative intervention
or nothing at all.
Step 5. Post-test both groups.
Example:
A nurse, working with Native
women in an OB/GYN clinic wanted to see if her HIV education programs were
increasing awareness among pregnant women and new mothers. She asked a
colleague working with Native women at another OB/GYN clinic to help her. Women
in their second trimester of pregnancy in both clinics were asked to complete a
pre-test. Visits by women in the experimental clinic also included regular HIV
education messages, delivered during their routine clinic visits. At the
post-partum visit, six weeks after the birth of their child, participants at
both clinics completed the post-test. If the intervention proved successful,
the information could be used to train other nurses to deliver prevention
messages during the regular course of their work.
The Single Group Time Series Design
For this design, a survey is administered multiple times -
before, during, and after a program
Step 1. Select a program outcome
measure that can be used repeatedly.
Step 2. Decide who will be in the
experimental group. Will it be the same group of people measured many times, or
successive groups of different people?
Step 3. Collect at least three
measurements prior to the intervention and made at regular intervals.
Step 4. Check the implementation
of the intervention.
Step 5. Continue to collect
measurements at least through the duration of the program.
In this design, if all members of the experimental group are
measured, or if randomly selected members of the group are measured, it is
called a longitudinal time series. If successive groups are used, it is called
a successive groups time series.
Example:
A health educator is
concerned about medication adherence with her HIV+ clients. She suspects some have
been taking ?drug holidays?. She starts administering a short survey about
their self-reported drug adherence at each monthly visit. On the fourth visit,
she schedules extra time to review their medications and the benefits of drug
adherence to a longer, healthier life (the intervention). She notices on the
following visits, there has been an improvement in the number of days the
client was adherent to his/her drug regimen.
The Time Series with a Non-Equivalent
Control Group
Two groups that are similar but were not formed by random
assignment are measured at regular intervals before and after a program
intervention. (This is like the Single Group Time Series with the addition of a
non-equivalent control group.
Step 1. Identify an experimental
group.
Step 2. Select a program outcome
measure that can be collected repeatedly.
Step 3. Locate a control group
that is similar to the experimental group and from which you can collect
observations on the outcome measure.
Step 4. Collect at least three
measurements prior to the intervention.
Step 5. Check the implementation
of the intervention.
Step 6. Continue to collect
observations at the same regular intervals from both groups.
Example
Using the above example, the
health educator could work with a colleague at another clinic to collect
medication adherence data from her clients. This information could be compared
to determine if the intervention alone was responsible for improvements in
medication adherence, or if the change was also a factor of time.
Before and After Design
In this method of evaluation, only people who are
participating in the program get the pre- and post-test.
Step 1. Pre-test everyone in the
program.
Step 2. Deliver the intervention.
Step 3. Post-test the same
individuals.
This design does not provide any information about what
kinds of results might have occurred without the program and is the weakest in
terms of scientific rigor.
Example:
An HIV program coordinator
wants to know if her education program is changing basic knowledge about HIV
transmission routes. She develops a pre- and post test survey that everyone in
the program completes before and after completing the class. The scores are
compared for changes. She also decided to match the tests to strengthen the
reliability of the scores.
