I’ve just completed the American Community Survey (ACS) questionnaire. The ACS is a Census Bureau survey and recipients of it are required by law to complete it. That’s a good move on their part. It definitely got my attention and motivated me to complete the survey, and quickly.
[ Begin mini-rant: It is a sad fact that surveys are expensive and hard to do well, in part due to the rational temptation by many to ignore them. Because I understand the value of data for social science and the challenges of collection of voluntary data, I also support the use of administratively collected (non-survey) data for research. Administrative data don’t suffer from the response rate issues that surveys do, though they have other weaknesses. And use of administrative data does raise privacy concerns (as does survey data use). My threshold for privacy concerns is much higher than most. In general, I’d like to see greater ability for researchers to access, combine, and analyze data so long as they are de-identified whenever possible and there are strong yet reasonable penalties (with enforcement) for misuse and sound, workable remedies for those who might be harmed by such. As a country we have a long way to go in terms of sensible collection and use of data for research and penalties/remedies in the case of misuse. End mini-rant. ]
Anyway, my favorite part of completing the ACS questionnaire was thinking about the questions pertaining to the mental and physical abilities of my five year old daughter. The questions caused me to ponder:
- Do her emotions pose difficulties for concentrating, remembering, and reasoning? (It would be a rare five year old for whom emotions did not play precisely that role.)
- Is she really independent in dressing or bathing if her parents need to occasionally remind her to do them, help her into and out of a shirt or boot, and rinse the shampoo from her hair?
From my work with survey data, I do know what the ACS designer’s were getting at with their mental and physical functioning questions, so I answered “no” to questions about my daughter’s difficulty in these areas. But still, this illustrates one of many ways in which survey response errors arise. It just isn’t crystal clear what is meant by some questions.
The questions on income also reveal issues. There is no suggestion that one should pull out a pay stub, a bill, or tax return. I answered the questions from memory in very round figures which I am sure are close but not exactly right. No doubt many people have absolutely no idea what they pay per year for water and sewer services, or how much interest and dividend income they receive annually. These data have got to be very imprecise and, worse, possibly biased.
I’ve been aware for years about errors and bias in survey responses. I’ve just never been on the other side of the data. This was my first time completing the ACS or anything like it. I know it isn’t easy to design good surveys that don’t cost a fortune to implement. Now I can see quite clearly that you get what you pay for.
(Oh, and two more things. Could the designers of the ACS please send a larger return envelope? I had a hell of a time getting the survey into the return envelope without ripping it. Finally, it wasn’t necessary to send two surveys. I know why they did–to increase response rates–but I had returned my first several weeks before receiving the second. Now I wonder, did they get the first? No, I’m not filling it out twice!)
