Volume 2, Issue 1
January-February 2001
Page Navigation Panel
Page 1 2
Inside this Issue
Survey Design 101 1
Performance Outcome Update 3
Inside the next issue....
Juliette Poulson, RN, MN. Interim Director,
HCA
Douglas Barton
Deputy Agency Director,
Behavioral Health Services
Behavioral Health Services
Central Quality Improvement
405 W. 5 th St., Suite 410
Santa Ana, CA. 92701
714-834-5601
Survey Design 101 -Jonathan Rich, Ph. D.
A survey is an effective and relatively inexpensive way to gather information. A well-designed survey will reveal important problems and suggest solutions. Surveys can answer questions such as: ·
- How satisfied are people (consumers, employees, managers, the public) with things as they are now? What changes would make them more satisfied?
- Do certain groups of people (age, gender, ethnicity, occupation, city of residence) have particular issues or problems?
- Are current procedures working? Are there other procedures that would work better?
While surveys can provide valuable information, a badly designed survey can be misleading and is worse than no survey at all. A good survey asks questions which are easily understood by the respondents. A good survey also asks questions in ways that do not bias the respondents. Here are some examples of good and bad survey questions:
- Avoid questions which bias the respondent by implying a socially desirable answer: Bad: Should the United States continue to be one of the few industrialized nations which imposes the death penalty on criminals? Good: Do you believe the death penalty should be imposed by the government for certain crimes?
- Avoid confusing or compound questions: Bad: Should the death penalty be imposed for certain crimes, such as murder, and life imprisonment be imposed for lessor crimes? Good: Should the death penalty be the punishment for murder? ·
- Earlier questions may bias the response to later questions. Be aware of question order. Questions which are more specific and give information about areas of concern should be placed later:
Bad:
1) Does poor ventilation in your office interfere with your productivity?
2) List any problems that interfere with your productivity. (con't. on page 2)
1
1 Page 2 3
2
Volume 2, Issue 1
January-February 2001
Survey Design 101(con't. from Page 1)
Good:
1) List any problems that interfere with your productivity.
2) Does poor ventilation in your office interfere with your productivity?
- If response options are given, they should be easily understood. The classic five-point scale, ranging from "Strongly Disagree" to "Strongly Agree" is often a good option: Bad: There are many things in my life that keep me interested and involved:
1. Possibly
2. Definitely
3. Perhaps
4. At times
5. Never
Good: There are many things in my life that keep me interested and involved:
1. Strongly disagree
2. Disagree
3. Not sure.
4. Agree
5. Strongly agree
- If response options are given, they should be all-inclusive. For instance, when asking how frequently the respondent does an activity, every possible response should fit into one of the categories: Bad: How often do you come to the clinic for a session with your psychiatrist?
1. Once a week
2. Twice a month
3. Once a month
4. Every other month
Good: How often do you come to the clinic for a session with your psychiatrist?
1. More often than once a week
2. About once a week
3. Two to three times a month
4. Once a month
5. Less than once a month
- People generally try to present themselves in the best light and downplay socially undesirable behavior. This tendency may bias your survey. While this tendency cannot be eliminated, it can be reduced by assuring respondents that their responses are confidential or anonymous, by explaining the purpose of the survey, and by asking questions in a way which reduces defensiveness.
Bad: Have you ever broken the law by using illegal drugs?
Good: Many people experiment with recreational drugs, such as marijuana, cocaine, or heroin, at some point in their life. In order to design programs to help people, we need to collect accurate, confidential information about drug use. Have you ever tried any drugs like that?
While well-written survey questions are crucial for ensuring useful results, many other factors must also be considered. How the sample of respondents is selected, how the survey is administered (i. e., mail, in-person, Internet, phone), and even the personal characteristics and training of the interviewer can impact survey accuracy. Measurement reliability and validity are important, particularly when measuring psychological constructs such as depression or intelligence. Once the data are collected, appropriate analysis methods must then be applied. These topics will be discussed in future newsletters. More information can be found about this topic in the references below. Fowler, F. J. (1995). Improving Survey Questions: Design and Evaluation. London: Sage Publications. Frankfort-Nachmisa, C. & Nachmisa, D. (1999). Research Methods in the Social Sciences. New York: Worth Publishing. www. surveysystem. com/ sdesign. htm
2
2 Page 3
3
Volume 2, Issue 1
January-February 2001
Performance Outcome Update -Jonathan Rich, Ph. D.
Since December 1999, Performance Outcome measures have been collected for nearly all children receiving services from County funded clinics. Enough data have now accumulated that we can begin to answer two crucial questions: "Are the children improving?" and "Which children are improving the most?" One of the Performance Outcome measures is the Achenbach Child Behavior Checklist (CBCL). On this instrument, parents and caretakers indicate their children's problems and competencies on a checklist. These ratings can then be compared to national averages. The ratings are grouped into four primary scales. The Externalizing scale shows the degree to which children demonstrate problems that annoy or interfere with the rights of others, including aggressive, disruptive, and destructive behavior. The Internalizing scale reflects negative emotion which is directed inward, manifesting as sadness, worry, and withdrawal. The Total Problems scale combines the Internalizing, Externalizing, and other problem scales. The Competency scale looks at the abilities and involvement that children have shown in school, at home, and in recreational activities. The first and last administration of the CBCL was compared for 39 children. The current data file contains a total of 1762 CBCL administrations. Children were selected if their first and last CBCL administrations were more than 30 days apart, and if the two administrations were during the same treatment episode (that is, there was not a discharge and readmission between the two administrations). The time between the two CBCL administrations ranged from 66 to 371 days. There were 32 boys and 7 girls. Two-thirds of the sample were Caucasian, and the remaining third were members of other ethnic groups. Here are some of the more interesting results:
- On average, these 39 children showed modest improvement. By the second administration of the CBCL, they were showing fewer problems and greater competency.
- The age and gender of the child was not related to the amount of improvement. Older children tended to improve as much as younger children, boys as much as girls.
- Fifteen of the children had a new treatment provider when the CBCL was re-administered, and 24 had the same provider. Those with a new provider fared no better or worse than the ones who continued to be seen by the same therapist.
- Caucasian children initially rated higher on the Total Problems and Externalizing scales than the other children. By the second CBCL administration, the Caucasian children showed significant improvement on these scales, whereas the non-White children did not show significant change. In other words, the Caucasian children started treatment with more reported problems than the non-White children, but after a period of treatment, they showed no more problems than the others.
The ethnic difference noted above invites speculation. The Caucasian children may be showing more problems at the first CBCL administration because they tend to be referred from different sources with different referral criteria, because of cultural differences in the decision to seek treatment, or because their caretakers are more likely to report problems. Further study is needed to better understand cultural variables.
The results do suggest that treatment is helping; as the database grows, our ability to understand the specific factors that lead to improvement will also grow. The ongoing collection of Performance Outcome data will continue to allow better understanding of factors leading to improvement, and will allow us to maximize our ability to provide effective services. 3
1 2 3
