From a procedural overview sent by a friend concerning empircal television research: "Experiments should be repeatable, as in the natural sciences; i.e., you should be able to get the same results if you follow the same procedure."
From my email: Is that really the underlying idea behind such research? I know almost nothing about this but had always figured that for the "soft" sciences there was more fuzziness involved. Experiments in the hard sciences are intended to support or falsify a hypothesis, an approach that seems rarely possible in say psychology. For instance, does watching violent TV make a person more violent: it's clear that for *individuals* there's pretty much no way to predict this so the question becomes is there an overall trend and then you have to start allowing for various factors that themselves may or may not have any effect. For instance the amount of extremely violent entertainment in Japanese culture though violent crime there is significantly lower than in the US.
There's also a sampling issue that I've always wondered about. For instance I was just looking at a CBS poll about the election and it's based on the responses of 400 people. Now this is obviously just a tiny tiny percentage of even registered voters so how reliable is it? Basically they ran four polls of 100 people and averaged the results which I guess makes it a bit more comfortable but couldn't this be completely skewed? Do pollsters try to poll in percentages that match the general population: roughly the same proportion of women/men or blacks/Hispanic/Asian or age or education? So if blacks are a bit over 10% of the US population (12.9% according to the CIA World Book) then does that mean 40 people are representing all blacks? To take an extreme example what if through luck of the draw 15 of those are gangsta rappers and 20 are lawyers/doctors then aren't the results way off? If the pollsters are actually choosing the respondents then aren't they manipulating the poll however innocent and unintentional it may be? And if not then don't they make it almost certain that the numbers are not representative? Obviously there are problems anyway with blanket categories like "the black vote" but that is one thing that polls sometimes attempt to put into numbers.
I feel sure there's some kind of theoretical underpinning and that pollsters have very strong (or at least strongly felt) arguments about it but still this bothers me. One of the longest lasting results of my debate experience was a deep suspicion of polls and studies which I don't think is cynicism so much as seeing how much the structure and process of polls create, rather than discover, the results. I guess what I'm getting at is that it seems to me that pretty much by definition you won't get the same results every time you run a poll/study, which is probably a main reason that so many of these questions have never been answered in any reliable way. Physicists argued for years about whether or not neutrinos existed but they knew what constituted evidence and what didn't; people have been arguing about the effects of art/media in some form or another since at least Aristotle.
(& yes I decided to let this run long and put it in my blog; I think that blogs may become a curse of some kind for people I send email, at least if I continue to make long-winded comments that go to both places.)