how research designs are actually carried out in practice

greenspun.com : LUSENET : History & Theory of Psychology : One Thread

It is well known that experiments are not carried out exactly in the form of the design nor in the form reported. Many changes are made along the way from conceptualization to re-interpretation. What is a good source of research on attempts to get at the "actual" process which of course can never be fully captured?

-- Gabriel Della-Piana (gdellapi@nsf.gov), June 06, 2004

Answers

I believe your statement of what is "well known" is too strong. The design reported is the one that will allow another researcher to replicate the effect. It is true that experiments are often "piloted" through a number of variations in order to demonstrate the expected effect, but that is a somewhat (and crucially) different matter from their not having ultimately been conducted as reported.

That said, I think the kind of work you're looking for can be found in the works of people such as Bruno Latour. I don't know of anyone who has done similar "anthropology of science" research with psychologists.

-- Christopher Green (christo@yorku.ca), June 06, 2004.


Hi Gabriel, I think you are correct that experiments are too often carried out differently than the original design and differently than reported. I am particularly concerned about the quality and ethics in medical research, but similar problems probably exist in psychology. Medical research is often "done on the side" by busy clinicans (and their often temporary research assistants). In addition the research team is often supported by wealthy medical companies (e.g., drug companies) that have an extremely strong vested interest in the results favoring their products. There is also extra money, perks, jobs, and promotions on the line for various research team members. This type of environment may often bring out some of the personal weaknesses in researchers. Such things as hurried weak research designs, lax execution of experiments, eliminating some subjects that do not fit expectations based on rationalizations, heavy "massaging" of the data, and nonreporting of "negative results". On the other hand, Skinner believed in experimental serendipity, and thought sometimes making a shift in experimental design during an experiment was justifed if some new more important lead emerged. In medical research, if the experimental group shows rapid and dramatic improvements, ethics dictates that the control group should be immediately given the experimental procedure (i.e., before the experiment is completed). I do not have much trust in typical high school research, where everyone in the class may be expected to do the research quickly and get typical results. However, I expect professional researchers, who I think often have strong values about truthfulness in research (and also realize getting caught in one lie about their research could ruin their careers), to be fairly honest. I think there has been some research on high school students cheating on their research, but I do not know what has been done to study the academic research scene. I do think we should learn more about what actually goes on in academic research settings and what might be done to improve the quality and honesty of research. Some say that in the long run research cheating and mistakes will be identified, when later replications do not support the poor research. However, I think that investigating and publicizing the extent of research abuses may help lead to research reform. I hope this helps. Paul

-- Paul Kleinginna (pkleinginna@georgiasouthern.edu), June 07, 2004.

Gabriel Della-Piana asked:

****"What is a good source of research on attempts to get at the 'actual' process which of course can never be fully captured?"

Hmm, that's a hard question to answer. My attempt to do so utilized the example of the area of personality research. The following link suggests that we need to recognize and assess various 'levels of maturity of research.'

"SUMMARY: This chapter introduces a methodology for evaluating the maturity of various empirical and theoretical positions in psychology. To illustrate its application, trait, situationist and interactionist theories of personality are assessed for maturity according to the kinds of questions they address. These accounts of personality are compared with A.N. Leontiev's Activity theory approach, which serves as an example of a more mature level of analysis. The methodology applies to theories of perception, memory, and learning, as well."

http://www.comnet.ca/~pballan/Personality.htm

****As for the earlier part of your posted question: "Many changes are made along the way from conceptualization to re- interpretation." There are also specific pre-supposed limitations and restrictions depending on the *research tradition* or subdiscipline which one is considering. Both Danziger and then Kusch have (in their own way) addressed that issue.

My attempt to briefly summarize their views for psychologist appears at the link below:

http://www.comnet.ca/~pballan/Socio-societal.htm

"The Danzigerian aspects of this talk had already appeared in Kurt's Constructing the Subject (1990) which outlined three old-world investigatory traditions in psychology up to the 1920s: clinical, individual differences, and Wundtian (see figures 1-3). Danziger's distinction was made on the basis of their motivation for carrying out psychological research and the resulting definition of the proper subject matter for psychology. Danziger & Ballantyne (1997) -in press at the time of the talk- expanded this analysis to the American context (up to 1945) elaborating changes in the material, social, and symbolic technologies used in psychological experiments. After describing the above elaboration, a complementary analysis by philosopher of science Martin Kusch (1995) -which focused on the power relationships within turn-of-the-20th century German (Kantian, Wundtian, and Wurzburg) settings- was introduced and then expanded upon to describe the American (Applied, Behaviorist, and Artificial Intelligence) settings of psychological research after 1945..."

Cheers,

Paul F. Ballantyne

-- Paul F. Ballantyne (pballan@comnet.ca), June 10, 2004.


Moderation questions? read the FAQ