Methods of Data Collection

 

There are many different ways to collect information that is important to the evaluation process. These include:

 

• Literature Reviews

• Focus Groups
• Community Interviews
• Key Informant Interviews
• Demonstration Sessions
• Observers
• Post-session Assessments
• Pre- and Post-tests
• Informal Surveys
• Games
• Story Telling
• Case Studies and Role Plays
• Key Informant Interviews
• Administrative Record Keeping

 

Literature Reviews - A literature review involves researching what others in your field, or a similar field have discovered. What types of programs have been piloted in other areas and what were the outcomes. By taking advantage of the experiences of others, a program planner can sidestep known pitfalls.

 

Focus Groups - Focus groups are another way to collect qualitative information. Focus groups are ideally made up of from eight to twelve people who have some characteristic in common with some interest in or connection to a project. They may be service providers, clients of a program, or members of a specific community. It is best to have a skilled facilitator run a focus group to ensure that the information is valid. Some steps to ensure the validity of the information are:

 

Confidentiality - Confidentiality of participants is assured. It is very important, especially in small communities that participants feel comfortable in speaking openly and honestly about the topic of interest. This may mean that only first names are used and participants sign a confidentiality agreement.

 

Recordings - If the session is to be taped for later transcription and analysis, it is critical to explain this process to participants and obtain their consents. Otherwise, a scribe should be available to take non-identifying notes. Ideally, a project should conduct multiple focus groups to best identify core issues, concerns, barriers, etc. A facilitator may feel confident with the findings when the conversations starts to become repetitive and s/he is hearing the same information from a variety of sources.

 

The meeting arrangements should demonstrate respect and appreciation for the assistance provided by participants. This includes providing refreshments, on-site childcare to parents (especially if the project has a maternal/child focus), transportation to and from the meeting location to those in need of this service and stipends. These typically take the form of gift certificates. When developing the evaluation plans, include these expenses in the budget.

 

When reporting on focus group findings, it is important to include background information and why the group was convened, how members were selected for participation, and non-identifying, general demographics of participants, e.g. age, gender, ethnicity.

 

Community Interviews ? Interviews can be guided, focusing on specific topic, or unguided discussions that range over many topics. They may also be a series of open-ended questions. The purpose of interviews is to identify information that is pertinent to a particular community or area of concern that may not be generally known or understood. Interviews can also provide information about how a problem or program is being perceived or understood.

 

When interviewing several people about the same topic it is important to maintain consistency across interviews. This is best accomplished through a predetermined list of open-ended questions. This does not prohibit the person from probing for additional information that may be relevant to the project when the situation arises. Often, the information collected by probing on a response yields the key to underlying issues.

 

Key Informant Interviews - Key informant interviews are discussions with community leaders or gatekeepers. Community leaders may be those formally selected by the community, employees of a health department or community organization, or others in an official capacity. They can provide information on social policies and observed areas of needs and/or gaps in services.

 

Gatekeepers can provide information on special areas of interest, community norms and perceptions, and provide suggestions for how to best work within a community, especially if this is an unfamiliar community. Gatekeepers are frequently informal community leaders or elders within a community. One method of identifying informal community leaders is to ask several different people to whom they would go for advice or information; or who do they think knows the most about how the community thinks and feels. When the same name is repeated by different sources, it is likely this person is a community gatekeeper.

 

When reporting on interviews, it is important to include background information, an explanation of how participants were selected, the circumstances of the interviews, (in homes, public settings, etc.), how the interview was conducted and non-identifying, demographic information. The qualitative report should present and focus on recurring trends that emerged across interviews.

 

Demonstration Sessions - A demonstration session is a ??practice run? or a ?pilot session? of an actual session you plan to conduct. It is the opportunity to test how HIV/STD-related presentations and seminars ?go over? with participants who represent the desired target group before they are widely or repeatedly implemented. It is usually helpful to conduct a demonstration session of any educational program, making everything about the demonstration as similar as possible to planned ongoing sessions. Select participants similar to your target audience; use the person who will actually facilitate or moderate the seminar or workshop; have ready the curriculum, agenda, and other materials you plan to use; make sure to hand out to the participants any surveys and/or forms you plan to use, such as pre- and post-tests and assessment forms; and conduct the session in the actual setting where the activity is to take place.

 

Observers - An observer is someone who is present at a presentation, seminar, or other educational activity involving community groups or individual clients for the purpose of recording comments, reactions, criticisms, or problems relevant to the activity. This person can provide useful insight into the quality of a presentation or intervention, in terms of its content, clarity, delivery, and appropriateness for the target group. Engage a participant-observer who is a member of the target group but is asked to pay particular attention to hoe the intervention is implemented, or a non-participant observer, who may be an expert on the topic but is not directly involved in the presentation or project. Observers can do process evaluation for ongoing education and prevention activities.

 

Surveys - Surveys can be written or oral, and can collect information in many types of formats such as open-ended questions, multiple choice, true/false, and scales. Self-administered surveys are completed by the person participating in the program intervention and are written. Oral surveys are administered by a trained interviewer in a one-on-one setting. For most HIV prevention work, surveys collect information on knowledge, attitudes, beliefs, and/or behaviors (KABB). Surveys are also used to assess client satisfaction. Surveys are often used before and after an intervention (pre- post-test) to measure any changes in KABB.

 

Surveys are the most quantifiable method of data collection for the typical, community based organization. The data is collected, entered into specially designed software programs and analyzed to identify among other things, statistically significant changes. The data may be collected in a number of ways, using all levels of staff, including volunteers. Data entry is very detail-oriented and needs to be handled by very conscientious individuals. The data analysis requires knowledge and expertise in statistics and is usually the task of the evaluator.

 

Types of Surveys

Post-session Assessments - Post-session assessments are usually one- to two-page survey forms that are handed out to participants at the end of a workshop or training seminar to assess the perceived quality and results of the intervention. The assessment seeks to determine whether participants learned certain information, rejected stereotypes or myths, or changed attitudes as the results of a specific intervention. Their responses provide some indication as to the effectiveness of the intervention session. Where many of the participants have limited literacy, either all or a sample of participants (every third participant, perhaps) can be interviewed orally. Or the participants can be given numbered answer sheets so they can simply circle the appropriate number of a response to a series of close-ended questions asked by the presenter (e.g., response number one, two, or three to a particular question). Entirely oral assessments are possible if the group leader requests that participants raise hands to respond to certain questions and a sufficient number of observers are present to record responses.

 

Pre- and Post-tests - Pre- and post-tests are usually short surveys designed primarily to assess what participants have learned as a results of a workshop or other education/prevention intervention. Because information is obtained before as well as after the intervention, this approach generates more specific information about how the intervention affected knowledge and attitudes than is possible with a post-test alone. Through a series of targeted questions, a pre-test measures an individual?s knowledge, attitudes, and /or behaviors about a given topic. The post-test, usually given immediately after the intervention has taken place, assesses whether an individual has gained new knowledge and perhaps changes both attitudes and/or beliefs. Pre- and post-tests enable project staff to determine whether the curriculum seems successful, and to identify types of information that should be covered more thoroughly in future sessions.

 

Information Surveys - An informal survey can be used to obtain information from prevention program participants and community residents, where formal knowledge, attitude, belief, and behavior (KABB) studies and follow-up surveys using probability samples and structured questionnaires are not feasible. Informal surveys can be used to obtain baseline data on a topic from a specific population group and to assess the effectiveness of an intervention. An informal survey is usually administered to a relatively small number of people - 30 to 50. They are generally selected through ?quota? sampling; that is, the population is divided into target groups (different security levels, ethnic groups, etc.), and a pre-determined number of respondents are chosen from each target group to ensure a representative sample of participants. Informal surveys can provide valuable information about whether interventions seem to be making a difference.

 

Games - Games or other creative techniques can be used to assess the effects of an intervention when participants in a training session or some other kind of education presentation lack basic literacy. Games are especially valuable because they require active involvement of the group, and may require participants not just to state but also to apply new knowledge. This may encourage behavior change. Games can be useful feedback tools even where literacy levels are high, since they provide an active, interactive, and often entertaining means for obtaining information about what was learned and how it is likely to be applied to real-life situations. In using games to evaluate the extent to which new knowledge, attitudes, or skills have been learned, be sure that the games are played in small groups or in a way that assures that most or all group members participate - so the learning of all can be assessed. Have a sufficient number of staff or other observers present to record the individual responses of the participants so that as little information as possible is missed. Be sure the situation - game, role-play, or case study - is carefully planned and explained to the group.

 

Story Telling ? Story telling is a traditional way of learning in Native communities and can be woven into an evaluation activity. Stories containing traditional characters such as the Trickster can be developed around the notion of protection against disease. A story can be developed to have multiple scenarios, each an example of a protective or destructive behavior. Audience members can participate by choosing a scenario and be counted as part of an evaluation activity. Audience members can be asked to create their own story and this can be used in role-play activities.

 

Case Studies and Role Play - Case studies and role-plays are other methods of obtaining active feedback. For example, if a session is designed to help men learn about athlete?s foot and ways to reduce their risk, a closing role play might put the participants in pairs or other small groups and give them a case study and role play, with staff observing and recording whether new knowledge and skills are being applied. Another approach involves presenting statements that include myths about the subject, and asking small groups to respond to them. The statements can be read to each small group by a facilitator if literacy rates are low. The purpose is to see if information presented in the session was learned and is applied appropriately to real-life situations.

 

Administrative Record-keeping - A consistently used record-keeping system is an essential component of any evaluation system. Basic record-keeping forms should be maintained for all projects. The type of information requested, whether quantitative (numerical) or qualitative (narrative), depends on how the information is to be used (e.g., will the form be used to log the number of services provided monthly, or will it be used to learn about a specific program?)