Issues in Design

1. 10 Pitfalls in Behavioral Research (after T. X. Barber)

a) Investigators and Experimenters

(1) Little Science
(2) Big science

b) Investigator Effects

(1) Investigator Paradigm Effects
- (a) tenacity of paradigms and resistance to new discoveries
  - i) paradigms as ways of organizing data
  - ii) theories arenšt disproven, old theorists die
- (b) failing to see events
- (c) seeing non-"events"
- (d) paradigms in personality
  - i) Freudian dynamics
  - ii) trait consistency
  - iii) social-learning theory
(2) Investigator Experimental Design Effects
- (a) Confirmatory designs
- (b) Demonstrations versus tests
(3) Investigator Loose Procedures Effects
- (a) poor specification of how to conduct experiments
- (b) badly defined specification of manipulations
(4) Investigator Data Analysis Effects
- (a) "pilot studies" -- the filing cabinet syndrome
  - i) how many studies are done before "the crucial experiment"
  - ii) meaning of significance tests
- (b) stopping rules -- data "purification" -- deletion of "outliers"
- (c) post mortem ad hoc analyses
  - i) hypothesis development and "confirmation"
  - ii) over interpretation of findings
- (d) Type I error and multiple tests
  - i) meaning of significance tests
  - ii) experiment-wise error rates
(5) Investigator "Fudging" Effects -- FRAUD
- (a) Trimming, Cooking, Idealizing
  - i) Galileo, Newton, Dalton, Mendel
- (b) generation of new "confirming cases"
  - i) Burt
- (c) Outright Fraud
  - i) Summerlin, Baltimore?
  - ii) non-existent subjects
- (d) Reasons for fraud
  - i) pressures for success
  - ii) hubris
  - iii) poor supervision -- too large lab
  - iv) lack of replication -- pressure to be new

c) Experimenter Effects

(1) Experimenter Personal Attributes Effects
- (a) stable
  - i) sex, age, ethnicity, height, age, weight
- (b) modifiable
  - i) prestige, anxiety, friendliness, warmth, dress
- (c) possible interactions of experimenter characteristics and subject characteristics
  - i) sex of experimenter by sex of subject
  - ii) prestige of experimenter by anxiety of subject
  - iii) etc.
(2) Experimenter Failure to Follow Procedures Effect
- (a) impossible, difficult, and inconvient procedures
- (b) poorly specified procedures
- (c) poor training, lack of practice, lack of supervision
(3) Experimenter Misrecording Effect
- (a) mistakes at data collection
- (b) mistakes at data entry
(4) Experimenter Fudging Effects
- (a) Lack of interest in outcome
- (b) Too much interest in outcome
(5) Experimenter Unintended Expectancy Effects
- (a) Pygmalian in the Classroom (?)

d) Recommendations

(1) Be aware of one's own and alternative paradigms
- (a) Alternative procedures
- (b) Alternative explanations
(2) Tighten protocols
(3) Replicate
- (a) If it is worth doing and worth reporting, it is worth replicating
- (b) if you can't replicate your own results, who can?
(4) Delicate balance between theory testing and theory development
- (a) methodological rigour versus theoretical speculation
- (b) signifigance tests based upon prior hypotheses
- (c) speculation based upon fortuitous findings
- (d) issues of type I and type II error
  - i) statistical significance versus real significance
    - (1) point estimates and confidence ranges
    - (2) effect size
  - ii) power

2. within versus between subject designs

a) Between subjects

(1) Subject effects
- (a) Other individual differences that covary with target variable
- (b) Other individual differences that differ randomly
(2) Between subject sources of extraneous variance
- (a) intelligence
- (b) experience
- (c) motivation

b) Within subjects

(1) controls for between subject varability
(2) practice effects
(3) fatigue effects
(4) order effects
- (a) drug effects persist
- (b) memory instruction effects persist

Issues in Design

1. 10 Pitfalls in Behavioral Research (after T. X. Barber)

a) Investigators and Experimenters

a) Between subjects

3. Ethics of designs

a) subject confidentiality

b) subject discomfort/stress

c) deception