(Reliability and Validity) Improving the Reliability and Validity of Measured Variables
Now that we have considered some of the threats to the validity of measured variables, we can ask how our awareness of these potential threats can help us improve our measures. Most basically, the goal is to be aware of the potential diffi culties and to keep them in mind as we design our measures. Because the research process is a social interaction between researcher and participant, we must carefully consider how the participant perceives the research and consider how she or he may react to it. The following are some useful tips for creating valid measures:
1. Conduct a pilot test. Pilot testing involves trying out a questionnaire or other research on a small group of individuals to get an idea of how they react to it before the fi nal version of the project is created. After collecting the data from the pilot test, you can modify the measures before actually using the scale in research. Pilot testing can help ensure that participants understand the questions as you expect them to and that they cannot guess the purpose of the questionnaire. You can also use pilot testing to create self-report measures. You ask participants in the pilot study to generate thoughts about the conceptual variables of interest. Then you use these thoughts to generate ideas about the types of items that should be asked on a fi xed-format scale.
2. Use multiple measures. As we have seen, the more types of measures are used to assess a conceptual variable, the more information about the variable is gained. For instance, the more items a test has, the more reliable it will be. However, be careful not to make your scale so long that your participants lose interest in taking it! As a general guideline, twenty items are usually suffi cient to produce a highly reliable measure.
3. Ensure variability within your measures. If 95 percent of your participants answer an item with the response 7 (strongly agree) or the response 1 (strongly disagree), the item won’t be worth including because it won’t differentiate the respondents. One way to guarantee variability is to be sure that the average response of your respondents is near the middle of the scale. This means that although most people fall in the middle, some people will fall above and some below the average. Pilot testing enables you to create measures that have variability.
4. Write good items. Make sure that your questions are understandable and not ambiguous. This means the questions shouldn’t be too long or too short. Try to avoid ambiguous words. For instance, “Do you regularly feel stress?” is not as good as “How many times per week do you feel stress?” because the term regular is ambiguous. Also watch for “double-barreled” questions such as “Are you happy most of the time, or do you fi nd there to be no reason to be happy?” A person who is happy but does not find any real reason for it would not know how to answer this question. Keep your questions as simple as possible, and be specifi c. For instance, the question “Do you like your parents?” is vaguer than “Do you like your mother?” and “Do you like your father?”
5. Attempt to get your respondents to take your questions seriously. In the instructions you give to them, stress that the accuracy of their responses is important and that their responses are critical to the success of the research project. Otherwise carelessness may result.
6. Attempt to make your items nonreactive. For instance, asking people to indicate whether they agree with the item “I dislike all Japanese people” is unlikely to produce honest answers, whereas a statement such as “The Japanese are using their economic power to hurt the United States” may elicit a more honest answer because the item is more indirect. Of course, the latter item may not assess exactly what you are hoping to measure, but in some cases tradeoffs may be required. In some cases you may wish to embed items that measure something entirely irrelevant (they are called distracter items) in your scale to disguise what you are really assessing.
7. Be certain to consider face and content validity by choosing items that seem “reasonable” and that represent a broad range of questions concerning the topic of interest. If the scale is not content valid, you may be evaluating only a small piece of the total picture you are interested in.
8. When possible, use existing measures, rather than creating your own, because the reliability and validity of these measures will already be established.
