Bias

There are other factors that influence the evaluation process that often goes beyond the design and implementation activities. Bias is one such factor. Bias occurs when the objectivity of the project has been jeopardized. It can happen in many ways and will skew the results of the program, ultimately impacting on the program?s effectiveness and diminishing its contribution to the community. For example, interviewers can insert bias by asking questions in such a way as to encourage one set of responses over another, or by ignoring cues to probe deeper into a given response. Participants may feel it is polite to say what they think an interviewer wants to hear, rather than reveal their true feelings or perceptions. And many questions relevant to HIV prevention deal with sensitive issues about sexuality and sexual behavior. These may cause participants to feel uncomfortable in revealing their truth. Staff that are asking evaluation questions about programs that they themselves are responsible for, may not report negative comments or concerns that would reflect poorly on their own (or their friend?s) work performance. To minimize these subjective biases, many groups elect to hire an independent evaluation team to work closely with them in the design, implementation, and analysis of their program evaluation.

Outside Consultants

In making a decision to hire expertise outside the community, seek out those who are willing to be active at every level of the evaluation process. Often universities have evaluation teams in their education or social work colleges, but other resources may also be available in the community. Often word-of-mouth is the best referral so ask other program directors how they handle evaluation and how satisfied they are with their team.

Once an evaluation team is identified, include them in the conceptualization phase of the project. In so doing, administrators can be assured that pertinent and relevant data is being collected from the outset. Nothing is more discouraging that to be well into the implementation and data collection phases of a program, only to realize that the information being collected doesn?t adequately answer core questions about the program?s performance, effectiveness, impact, and so on.

Budget

The growing importance of evaluation to a program?s longevity requires adequately funding evaluation activities. The cost of a good evaluation will range 5% - 10% of the total program?s budget. These expenses include consulting fees, internal staff support and indirect expenses (e.g. rent, utilities, insurance, workers comp, etc.), mileage, equipment, training, and meeting expenses (see focus groups).

Confidentiality/Anonymity

Confidentiality means than while personal information will be collected, it will be maintained in a secure place and is not accessible to the public, or any but a select number of employees at the organization collecting the data.

Anonymity means that there are no identifying markers to connect responses or tests results back to an individual or a community.

Working with HIV programs in Native communities requires hyper-vigilance regarding use of information and knowledge. Unfortunately, the fear of disclosure is high enough to keep people from seeking care, or at the least, seeking it in their own communities. Often, they will go to clinics in neighboring counties or towns to test for HIV. When collecting any type of sensitive information it is very important to explain how the information will be used, who will have access to it, and how anonymity or confidentiality will be maintained.

Institutional Review Boards (IRBs)

An IRB is an assurance to a funding agency that a program is in compliance with human subjects protection and regulations. Generally, IRBs are most pertinent to research projects; however, if over 9 people are involved in the data collection process, the project needs Office of Management and Budget (OMB) clearance. If a project needs an IRB is involves answering three questions:

1. Does the grant include research?

2. If yes, does the research study human beings?

3. If yes, does the study meet the regulatory definition of ?human subjects??

These questions come from the federal regulations for human research protection. The definition of ?research? in 45 CFR 46 is:

Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge. Activities that meet this definition constitute research for purposes of this policy, whether or not they are conducted or supported under a program that is considered research for other purposes. For example, some demonstration and service programs may include research activities. [45 CFR 46.102]

The key phrase in this definition is ?generalizable knowledge.? This usually (but not always) means publication of results. The decision about a project requiring an IRB is made by a funder or an institution (usually a university) through which the grant was awarded and is not made by the local researcher. If a program unaffiliated with a local university requires an IRB, approaching that institution and requesting assistance with the IRB process is often the easiest means toward this end. However, this generally requires that someone on faculty be a part of the project. Finally, there are some national organizations that provide IRB oversight for a fee. When in doubt, call the project officer listed on the award notice.

The IRB process was designed for the protection of human subjects. There is usually a committee charged with approving an IRB proposal. This group is called the Committee for the Protection of Human Subjects (CPHS).

Random Selection

Before we explore the different designs an evaluation can have, it is important to discuss the notion of control groups, random sampling, and matched pairs. A control group is a group that does NOT receive the intervention. It serves as a constant. The experimental group DOES receive the intervention. Both groups may participate in surveys or data collection of some sort, and scores between these two groups are compared to test for change in knowledge, behavior, beliefs, and so on. To make an evaluation as scientifically rigid and sound as possible ? which also supports the validity, accurateness, and applicability of the project - control groups and random sampling are very desirable. The difficulty comes in working with HIV prevention programs. Communities and educators do not want to withhold the intervention from anyone and this makes identifying a control group more difficult. Oftentimes, a group may have to serve as its own control, using a pre- and post-test evaluation design.

Random sampling protects the project against biases caused when participants are selected because of a particular characteristic. An example of how this would look in an HIV education program being delivered in an STD clinic might be that everyone in the clinic on one particular day would receive the intervention. The date could be randomly selected by putting all clinic dates into a hat and drawing one out. These clients would be tracked over time and compared with the other clinic population to see if there were any changes relevant to the goals and objectives of the intervention.

Convenience sampling is another method and does not involve randomness, but instead selects participants based upon the ease of contact. For example, everyone participating in a small group intervention becomes part of the study.

Matched pairs is another method of collecting data. By pairing the pre- and post-test from the same individual, the strength of the findings may be more powerful. In working with HIV, it is important to maintain awareness of confidentiality and anonymity needs. Identifiers can be made-up names, numbers, or words that the participants use on both pre- and post-test instruments.