PSYCHOLOGY 635/636

Fall, 1999

S. L. Golding

ADVANCED ISSUES IN RESEARCH DESIGN AND METHODOLOGY IN CLINICAL PSYCHOLOGY

This class is designed as part of the required clinical psychology curriculum, and presumes successful completion of the introductory research design and statistics sequence, which should have included basic psychometric issues concerning the reliability and validity of dependent variables, as well as the traditional "research design" issues. If that is not true for you, you will need to pay special attention to the topics not required, but listed at the end of the syllabus. You will also need to ask me for a set of handouts on the issues of construct validation, reliability, reliability phenomena, and generalizability theory. By design, it is intended to cover topical issues, at an advanced level, that will assist students in the completion of their dissertations and in gaining competency in research design and methodological issues that are likely to occur in their post-degree research and scholarly pursuits. Obviously, there can be no claim to complete coverage of all such issues. Rather, within the confines of our schedule, I have selected, with input from faculty and graduate students, a set of topics that seem to rise to the top of the heap. There are, of course, (many) other issues that could be included. Therefore, the class will also have a set of student generated presentations and a final paper covering additional topics.

Course structure

You will receive a required readings list that assigns weekly readings, arranged by topic. A master set of the assigned readings (articles, chapters, etc.) will be available in the Clinical Office. It is your responsibility to arrange to make personal copies from this master set. There is no assigned text, although the readings required are also generally available in archival sources such as journals and books in the University Library.

It is your responsibility to read and study the materials in advance of each class. In general, I will give supplementary materials and explanations as part of a lecture during the first part of each class, but I expect all class members to be able to participate in an interactive discussion and probing of the issues. Twenty-five (25) percent of your class grade will be based upon your class participation.

Presentations and papers. There are no examinations in this course. The remainder of your course grade will be assigned on the basis of your final paper (50%) and your class presentation (25%).

In order to expand the coverage of our topics and to assist you in your own area of specialization, you should prepare a final paper, of no more than 20 double-spaced pages, exploring either a) a current methodological, psychometric, or research design issue in your area of special interest in considerable depth or b) a review of major methodological issues in a particular area of research in a survey format. The purpose of your review is to evaluate current methodological, psychometric or research design "thinking" on the selected problem. Problems chosen for option (a) should be relatively clearly defined, but not so narrow as to be esoteric. Reviews chosen for option (b) should follow the general scope of review as exemplified in the September 1995, Volume 7(3) issue of Psychological Assessment. You should choose your topic and have it approved by me no later than the end of October. I will be available by scheduled appointment to assist you in choosing a topic by discussing your interests and giving you leads. Some examples of topics are:

This paper and its presentation are, in essence, an opportunity for you to explore a design, methodological or psychometric issue that is of importance in your area of specialization, with the constraint that it be of some general interest and not too esoteric.

Presentation of papers will occur during the last class meeting.

Class schedule

Construct Validation Revisted 6

Ethical Issues 6

Review of Basic Psychometrics
8

Advanced Psychometrics 9

Introduction to multivariate issues 10

Problems with assessment and classification 10

Meta-analysis 11

Construct validational concerns with "unobtrusive" measures
12

External validity as a function of sampling
12

Statistical and clinical significance with difference, gain or change scores; covariance adjustments 12

Problems with inter-rater agreement (especially with categorical data)
13

Decision theory 14

ROC/Itemmetrics 15

Clinical judgment 16

Generalizability theory 17

Classic Research Design Issues
19

Power 20

Convergent and Discriminant Considerations
20

Construct Validation Revisted

*Cronbach, L.J., & Meehl, P.E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281-302.

^*Lykken, DT. (l968). Statistical significance in psychological research. Psychological Bulletin, 70, 151-159.

^*Meehl, P. E. (l970). Some methodological reflections on the difficulties of psychoanalytic research. In M. Radner and S. Winokur, Minnesota studies in the philosophy of science. Minneapolis: University of Minnesota Press, 4, 402-416.

*Meehl, P. (l967). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103- 115.

^*Wampold, B.E., Davis, B., & Good, R.H. (1990). Hypothesis validity of clinical research. Journal of Consulting and Clinical Psychology, 58, 360-367.

Cronbach, L. J. (1975). Beyond the two disciplines of scientific psychology. American Psychologist, 30, 116-127.

Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671-684

Feigl, H. (l959). Philosophical embarrassments of psychology. American Psychologist, 14, 115-128.

Foster, S. L., & Cone, J. D. (1995). Validity issues in clinical assessment. Psychological Assessment, 7, 248-260.

Pedhazur, E. J. & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N.J.: L. Erlbaum. Pp.52-73; 147-163; 164-179; 180-210.

Ethical Issues

^*American Psychological Association (1992). Ethical Principles of Psychologists and Code of Conduct. American Psychologist, 47, 1597-1611 Section 6.00-6.26 [Teaching, Training Supervision, Research and Publishing].

^*Bersoff, D. N. (1995). Ethical conflicts in psychology. Washington, D.C.: American Psychological Association Press, Pp. 355-392.

^*Goodyear, R. K., Crego, C. A., & Johnston, M. W. (1992). Ethical issues in the supervision of student research: A survey of critical incidents. Professional Psychology: Research and Practice, 23, 203-210.

*Hoagwood, K. (1994). The certificate of confidentiality at the National Institute of Mental Health: Discretionary considerations in its applicability in research on child and adolescent mental disorders. Special Issue: Reporting and referring child and adolescent research participants. Ethics and Behavior, 4, 123-131.

^*Sieber, J. E. (1994). Will the new code help researchers to be more ethical? Professional Psychology: Research and Practice, 25, 369- 375.

American Psychological Association. (1982). Ethical principles in the conduct of research with human participants. : Washington, D.C: Author.

Appelbaum, P., & Rosenbaum, A. (1989). Tarasoff and the researcher: Does the duty to protect apply in the research setting? American Psychologist, 44, 885-894.

Baumrind, D. (1985). Research using intentional deception: Ethical issues revisited. American Psychologist, 40, 165-174.

Ceci, S. J., Peters, D., & Plotkin, J. (1985). Human subjects review, personal values, and the regulation of social science research. American Psychologist, 40, 994-1002.

Committee on Legal Issues. (1996). Strategies for private practitioners coping with subpoenas or compelled testimony for client records or test data. Professional Psychology: Research and Practice, 27, 245-251. [Critical clinical reading, but also relevant here in terms of confidentiality of research data]

Imber, S. D., Glanz, L. M., Elkin, I., Sotsky, S. M., Boyer, J. L., & Leber, W. R. (1986). Problems in a collaborative clinical trials study. American Psychologist, 41, 137-146.

Kazdin, A. E. (1998). Research design in clinical psychology, Third Edition. Needham Heights, MA: Allyn & Bacon, Pp. 415-450.

Lehrman, N. S., & Sharav, V. H. (1997). Ethical problems in psychiatric research. Journal of Mental Health Administration., 24(2), 227-250.

Overholser, J. C. (1987). Ethical issues in prison research: A risk/benefit analysis. Behavioral Sciences and the Law, 5, 187-202.

Peterson, C. C., & Siddle, D. A. (1995). Confidentiality issues in psychological research. Australian Psychologist., 30(3), 187-190.

Rosnow, R. L. (1997). Hedgehogs, foxes, and the evolving social contract in psychological science: Ethical challenges and methodological opportunities. Psychological Methods., 2(4), 345-356.

Wolfgang, M. E. (1981). Confidentiality in criminological research and other ethical standards. Journal of Criminal Law and Criminology, 72, 345-361.

Review of Basic Psychometrics

^*Cronbach, L. J. (1990). Essentials of psychological testing (Fifth Edition). New York: Harper. Chapter 6, "How to judge tests: Reliability and other qualities," Pp. 190-222. [You probably know much of this material, but a little review can't hurt too much]]

^*Dudek, F. J. (1979). The continuing misinterpretation of the standard error of measurement. Psychological Bulletin, 86, 335-337.

^*Haynes, S. N., Nelson, K., & Blaine, D. (1999). Psychometric issues in assessment research. In P. Kendall, J. Butcher et al. (Eds.), Handbook of research methods in clinical psychology (2nd ed.) (pp. 125-154). New York: Wiley.

^*Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. American Psychologist, 50, 741-749.

^*Wilkinson, L. & The Task Force on Statistical Inference. Statistical methoids in psychology journals. American Psychologist, 54, 594-604.

Ghiselli, E. E. (l964). Theory of psychological measurement. NY: McGraw-Hill. Pp. 207-253. (Not a required reading, but the classic development of the basic equations of reliability theory)

Pedhazur, E. J. & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N.J.: L. Erlbaum. Pp. 15-29; 30-51;81-146.

Advanced Psychometrics

^*Epstein, S. (1986). Does aggregation produce spuriously high estimates of behavior stability?. Journal of Personality and Social Psychology, 50, 1199-1210.

^*Golding, S. L. (l977). Method variance, inadequate constructs, or things that go bump in the night? Multivariate Behavioral Research, 12, 89-98.

^*Meehl, P. E. (l978). Theoretical risks and tabular asterisks: Sir Karl, Sir Ronald, and the slow progress of soft psychology. Journal of Consulting and Clinical Psychology, 46, 806-834.

^*Meehl, R. E. (1991). Why summaries of research on psychological theories are often uninterpretable. In R. E. Snow & D. E. Wiley (Eds.), Improving inquiry in social science: A volume in honor of Lee J. Cronbach. Erlbaum: Hillsdale, NJ. Pp. 13-59.

^*Schmidt, F.L., & Hunter, J.E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.

*Smith, G. T., & McCarthy, D. M. (1995). Methodological considerations in the refinement of clinical instruments. Psychological Assessment, 7, 300-308.

Introduction to multivariate issues

^*Comrey, A. L. (1988). Factor-analytic methods of scale development in personality and clinical psychology. Journal of Consulting and Clinical Psychology, 56, 754-761.

^*Cronach, L. J. (1991). Methodological studies: A personal retrospective. In R. E. Snow & D. E. Wiley (Eds.), Improving inquiry in social science: A volume in honor of Lee J. Cronbach (pp. 385-400). Hillsdale, NJ: Erlbaum.

^*Floyd, F. J., & Widaman, K. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7, 286-299.

Burt, C. (1966). The early history of multivariate techniques in psychological research. Multivariate Behavioral Research, 1, 24-42.

Carpenter, W. T., Bartko, J. J., & Evans, M. (1983). Multivariate statistics and meaning in the study of psychopathology. Schizophrenia Bulletin, 9, 1-6.

Fleiss, J., & Zubin, J. (l969). On the methods and theory of clustering. Multivariate Behavioral Research, 4, 235-250.

Gleser, G. C., & Da Silva, L. C. (1991). An approach to multivariate generalizability of MMPI profiles. In R. E. Snow & D. E. Wiley (Eds.), Improving inquiry in social science: A volume in honor of Lee J. Cronbach (pp. 145-160). Hillsdale, NJ: L. Erlbaum.

Koretz, D. (l979). Validity shrinkage in psychopathology research: An example of the need for caution. Journal of Abnormal Psychology, 88, 328-329.

Problems with assessment and classification

^*Clark, L.E., Livesley, W. J., Schroeder, M.L., & Irish, S. L. (1996). Convergence of two systems for assessing specific traits of personality disorder. Psychological Assessment, 8, 294-303.

^*Risso, L.P. et al., (1994). Concordance between patients and informants on the Personality Disorder Examination. American Journal of Psychiatry, 151, 568-573.

^*Trull, T.J., Widiger, T.A., & Guthrie, P. (1990). Categorical versus dimensional status of borderline personality disorder. Journal of Abnormal Psychology, 99, 40-48.

^*Widiger, T.A., & Trull, T.J. (1991). Diagnosis and clinical assessment. In M.R. Rosenzweig and L.W. Porter (Eds). Annual Review of Psychology, 42, 109-133. Palo Alto, CA. : Annual Review Press.

^*Widiger, T.A. & Sanderson, C. J. (1995). Assessing personality disorders. In J. Butcher (Ed.), Clinical personality assessment: Practical approaches. New York: Oxford. Pp. 381-394.

^*Achenbach, T. (1995). Empirically based assessment and taxonomy: Applications to clinical research. Psychological Assessment, 7, 261-274.

Meta-analysis

^*Durlak, J. (1999). Meta-analytic research methods. In P.C. Kendall, J.N. Butcher et al. (Eds.), Handbook of research methods in clinical psychology (2nd ed.) (pp. 419-429). New York: Wiley.

*Hanson, K., & Bussiere, M. T. (1998). Predicting relapse: A meta-analysis of sexual offender recidivism studies. Journal of Consulting and Clinical Psychology, 66, 348-362.

^*Hunter, J.E., & Schmidt, F.L. (1996). Cumulative research knowledge and social policy formulation: The critical role of meta-analysis. Psychology, Public Policy and Law. 2, 324-347.

^*Rind, B., Tromovitch, P., & Bauserman, R. (1998). A meta-analytic examination of assumed properties of child sexual abuse using college samples. Psychological Bulletin, 124, 22-53.

Bonta, J., Law, M., & Hanson, K. (1998). The prediction of criminal and violent recidivism among mentally disorder offenders: A meta-analysis. Psychological Bulletin, 123, 123-142.

Construct validational concerns with "unobtrusive" measures

^*Bartell, C.H., & Holmes, D.S. (1968). High school yearbooks: A nonreactive measure of social isolation in graduates who later became schizophrenic. Journal of Abnormal Psychology, 73, 313-316.

^*Meehl, P.E. (1971). High school yearbooks: A reply to Schwarz. Journal of Abnormal Psychology, 77, 143-148.

^*Schwarz, J.C. (1970). Comment on "High school yearbooks ..." Journal of Abnormal Psychology, 75, 317-318.

External validity as a function of sampling

Goodman, S. H., Lahey, B.B., et al. (1997). Representativeness of clinical samples of youths with mental disorders: A preliminary population based study. Journal of Abnormal Psychology, 106, 3-14.

Statistical and clinical significance with difference, gain or change scores; covariance adjustments

^*Cronbach, L. J. (1990). Essentials of psychological testing (Fifth Edition). New York: Harper. Pp. 276-296.

^*Matarazzo, J. D. (1990). Psychological assessment versus psychological testing: Validation from Binet to the school, clinic, and courtroom. American Psychologist, 45, 999-1017.

^*Meehl, P. E. (l970). Nuisance variables and the ex post facto design. In M. Radner and S. Winokur (Eds.), Minnesota studies in the philosophy of science, 4, 373-402 (re-read if you haven't already from 635).

^*Pedhazur, E. J. & Schmelkin, L.P. (1991). Measurement, design and analysis. Hillsdale, N.J.: Erlbaum. Pp. 291-294; 567-582

Problems with inter-rater agreement (especially with categorical data)

^*Bartko, J. J., & Carpenter, W. T. (l976). On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163, 307-317.

^*Janes, C. L. (1979). Agreement measurement and the judgment process. Journal of Nervous and Mental Disease, 167, 343-347.

^*Uebersax, J. S. (1987). Diversity of decision-making models and the measurement of interrater agreement. Psychological Bulletin, 101, 140-146.

^*Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 374-378.

Cohen, J. (l972). Weighted Chi Square: An extension of the Kappa method. Educational and Psychological Measurement, 32, 61-74. (Not required, but one of the classic kappa articles).

Cronbach, L. J. (l949). Statistical methods applied to Rorschach scores: A review. Psychological Bulletin, 46, 393-429. (A treasure-trove of logic about analyses of such data; not required)

Fleiss, J. L.. (l966). Assessing the accuracy of multivariate observations. Journal of the American Statistical Association, 61, 403-412. (Not required; pushing kappa into multiple raters, etc.)

Fleiss, J. L., Spitzer, R. L., Endicott, J., & Cohen, J. (l972). Quantification of agreement in multiple psychiatric diagnosis. Archives of General Psychiatry, 26, 168-171. (Not required; classic kappa application)

Hanley, J. A. (1987). Standard error of the Kappa statistic. Psychological Bulletin, 102, 315-321. (Not required, but an awfully useful reference when you actually do one)

Maxwell, A. E. (l968). The effect of correlated errors on estimates of reliability coefficients. Educational and Psychological Measurement, 28, 803-811. (Not required, but a classic reference on the subject)

Maxwell, A. E, Pilliner, A. E. G,. (l968). Deriving coefficients of reliability and agreement for ratings. British Journal of Mathematical and Statistical Psychology, 21, 105-116. (Not required, but the source of the Maxwell RE coefficient discussed in Janes)

Spitzer, R. L., Cohen, J., Fleiss, J. L. Endicott, J. (l967). Quantification of agreement in psychiatric diagnosis. Archives of General Psychiatry, 17, 83-87. (Not required; another classic kappa reference; why say it once when you can say it twice in the same journal -- helps the vita and promotion)

Decision theory

^*Baldassarini, R., Finklestein, S., & Arana, G.W. (1983). The predictive power of diagnostic tests and the effect of prevalence of illness. Archives of General Psychiatry, 40, 569-573. (The classic reference in the psychiatric literature, although not the most readable -- sorry, but required nonetheless)

^*Faust, D. (1991). What if we had really listened? Present reflections on altered pasts. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol:1 Matters of public interest. Essays in honor of Paul E. Meehl. University of Minnesota Press: Minneapolis, MN. Pp. 185-216.

^*Finn, S. E. & Kampuis, J. H. (1995). What a clinician needs to know about base rates. In J. N. Butcher (Ed.), Clinical personality assessment. Oxford: New York. Pp. 224-235.

^*Kessel, J.B., & Zimmerman, M. (1993). Reporting errors in studies of diagnostic performance .... . Psychological Assessment, 5, 395-399.

^*Meehl, R F., & Rosen, A. (l955). Antecedent probability and the efficiency of psychometric signs, patterns, and cutting scores. Psychological Bulletin, 52, 194-216.

^*Wiggins, J. S. (1973). Personality and prediction : Principles of personality assessment. Reading, Mass. : Addison Wesley, Pp. 224- 274. (Perhaps the most readable of all sources on classic decision-theory; unfortunately, not presented in terms of the latest specificity, sensitivity, NPP and PPP zeigeist)

Cronbach, L. J., & Gleser, G. C. (l957). Psychological tests and personnel decisions. Urbana: University of Illinois Press. (Not required; the classic statement - fairly tough going; very useful however when you get into analytic problems)

Dawes, R. (l971). A case study of graduate admissions : Application of three principles of human decision making. American Psychologist, 26, 180-188. (Not required; early application to a situation near and dear to our hearts; origin of the dreaded GRE selection equation-since I went to Oregon and Robyn was one of my advisors, I often wonder how I made it in since my MMPI flunked me out of Minnesota!)

Hirschberg, N., & Itkin, S. (l978). Graduate student success in psychology. American Psychologist, 33, 1088- 1093. (Not required; another classic application)

Magnusson, D. (1967). Test theory. Reading, Mass.: Addison-Wesley, Pp. 198-225. (Not required; a more basic presentation of decision theory in an undergraduate text worth owning)

Motto, J. A., Heilbron, D. C., & Juster, R. P. (1985). Development of a clinical instrument to estimate suicide risk. American Journal of Psychiatry, 142, 680-686.(Not required, but very interesting application to suicide prediction)

Satz, P., Fennell, E., & Reilly, C. (l970). Predictive validity of six neurodiagnostic tests : A decision theory analysis. Journal of Consulting and Clinical Psychology, 34, 375-381. (Recommended, as an application and demonstration that different tests produce different configurations of error, but not required.)

ROC/Itemmetrics

^*Bennett, K. J., Lipman, E.L., Brown, S., et al. (1999). Predicting conduct problems:Can high risk children be identified in Kindergarten and grade 1?. Journal of Consulting and Clinical Psychology, 67, 470-480.

^*Cooke, D. J. & Michie, C. (1997). An item response theory analysis of the Hare Psychopathy Checklist-Revised. Psychological Assessment, 9, 3-14.

^*Gardner, W., et al. (1996). Clinical versus actuarial predictions of violence in patients with mental illnesses. Journal of Consulting and Clinical Psychology, 64, 602-609.

^*Rice, M.E., & Harris, G.T. (1995). Violent recidivism: Assessing predictive validity. Journal of Consulting and Clinical Psychology, 63, 737-748.

Clinical judgment

^*Garb, H.N., & Schramke, C. J. (1996). Judgment research and neuropsychological assessment: A narrative review and meta-analysis. Psychological Bulletin, U120, 140-153.

^*Goldberg, L. R. (1991).. Human mind versus regression equation: Five contrasts. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol:1 Matters of public interest. Essays in honor of Paul E. Meehl. University of Minnesota Press: Minneapolis, MN. Pp. 173-184.

^*Kleinmuntz, B. (1991). Recent developments in computerized clinical judgment. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol:1 Matters of public interest. Essays in honor of Paul E. Meehl. University of Minnesota Press: Minneapolis, MN. Pp. 217-234.

^*Kleinmuntz, B. (1990). Why we still use our heads instead of formulas: Toward an integrative approach. Psychological Bulletin, 107, 296-310.

^*Borum, R., Otto, R., & Golding, S. L. (1993). Improving clinical judgement and decision making in forensic evaluations. International Journal of Law and Psychiatry, 21, 35-76.

Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.

Einhorn, H. J., & Hogarth, R. M. (l978). Confidence in judgment:Persistence of the illusion of validity. Psychological Review, 85, 395-416.

Goldberg, L. R. (l968). Simple models or simple processes? Some research on clinical judgments. American Psychologist, 23, 483-496.

Goldberg, L. R. (l970). Man versus model of man:A rational, plus some evidence,for a method of improving clinical inferences. Psychological Bulletin, 73, 422-432. (Two classics, not required, by my old mentor and the father of linear models of clinical judgment)

Kahneman, D., & Tversky, A. (l973). On the psychology of prediction. Psychological Review, 80, 237-251. (The source for the commonly used "heuristic" analysis of social judgment processes; worth reading, but not required)

Meehl, P. E. (1973). Why I do not attend case conferences. In P. E. Meehl, Psychodiagnosis: Selected Papers of P. E. Meehl (pp. 225-304). Minneapolis, Minn. : University of Minnesota Press. (Not required, but tremendously enjoyable and thought provoking for those of us who do attend case conferences, clinical admissions meetings, and the like)

Nurius, P.S., & Nicoll, A. E. (1992). Capturing clinical expertise: An analysis of knowledge "mining" through expert system development. Clinical Psychology Review, 12, 705-717.

Sawyer, J. (l966). Measurement and prediction, clinical and statistical. Psychological Bulletin, 66, 178-200. (Another clinical judgment classic, especially if you get serious about the design of a study; not required)

Additional sections for your own re-study

Generalizability theory

Brennan, R L. (l980). Applications of generalizability theory. In R. A. Berk, Criterion-referenced measurement: State of the art. Baltimore: Johns Hopkins Press 186-232. (Not required, but good to consult for various issues; more readable than the "master.")

Cattell, R. B. (1964). Validity and reliability : A proposed more basic set of concepts. Journal of Educational Psychology, 55, 1-22. (Not a required reading; however, strongly recommended if you don't understand the concepts of generalization across domains such as time, rater, items and the like; Ray deserves his reputation of anticipating many developments in psychometrics).

Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (l963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163. (Ditto)

Cronbach, L. J., Gleser, G. C., Nanda, H. et al. (1972). The dependability of behavioral measures. New York: Wiley. (Not required; the master speaks, but he's awfully hard to understand; the classic reference and bible which you will end up reading parts of sometime in your life if you do G-theory work)

DuBois, P. H. (1970). Varieties of psychological test homogeneity. American Psychologist, 25, 532-536.

Golding, S L. (l979). Flies in the ointment : Methodological problems in the analysis of the percentage of variance due to persons and situations. Psychological Bulletin, 82, 278-288. (While written in the context of the person-situation debate, it provides a useful entry into the underlying theory and use of generalizability logic and computing techniques)

Levy, P. (l974). Generalizability studies in clinical settings. British Journal of Social and Clinical Psychology, 13, 161-172. (Not required; useful examples of application.)

McGaw, B., Wardrop, J. L., & Bunda, M. A. (l972). Classroom observation schemes : Where are the errors?. American Educational Research Journal, 9, 13-27.(Not required; application of G-theory to classroom observational data)

Shavelson, R. J., Webb, N. M., & Rowley G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932.

Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133-166. (Not required, but an early application that contains some useful examples from clinical contexts)

Shrout, P. E., & Fleiss, J. L. (l979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428. (Not required; how to compute G-coefficients without knowing you're doing it and perhaps not understanding the assumptions; the most commonly cited reference in the psychopathology literature for rater studies)

Stanley, J. C. (l971). Reliability. In R. L. Thorndike (Ed.), Educational Measurement. Washington DC: American Council on Education, Pp. 356-442. (This is not a required reading, but is well worth your reading, if you can get through it, as it pulls the entire area of classical and neo-classical reliability theories together.)

Tryon, R C. (l957). Reliability and behavior domain validity : Reformulation and historical critique. Psychological Bulletin, 54, 229-249. (Not a required reading, but the classic statement of domain theory which laid the ground work for G-theory)

Wiggins, J. (1973). Personality and prediction : Principles of personality assessment. New York: Addison-Wesley. (Not required, but out of print and a classic text if you can get a copy of it; contains a useful introduction to some G-theory concepts, albeit somewhat dated)

CLASSIC RESEARCH DESIGN ISSUES

Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Pp. 37-94.

Pedhazur, E. J. & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N.J.: L. Erlbaum. Pp. 211-233; 234-249; 250-276

Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand McNally. Pp. 95-146.

Garber, J., & Hollon, S.D. (1991). What can specificity designs say about causality in psychopathology research? Psychological Bulletin, 110, 129-136.

Pedhazur, E. J. & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N.J.: L. Erlbaum. Pp. 277-303; 304-317.

POWER

Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.

Cohen, J. (1992). A power primer. Psychological Bulletin, 112, 155-159.

Pedhazur, E. J. & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N.J.: L. Erlbaum. Pp. 336-341.

Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105, 309-316.

CONVERGENT AND DISCRIMINANT CONSIDERATIONS

Campbell, D.T., & Fiske, D.W. (1959). Convergent and discriminant validation by the multi-trait multi-method matrix. Psychological Bulletin, 56, 81-105.

Fiske, D. W., & Campbell, D.T. (1992). Citations do not solve problems. Psychological Bulletin, 112, 393-395.

Golding, S. L. (l977). Method variance, inadequate constructs, or things that go bump in the night?. Multivariate Behavioral Research, 12, 89-98.

Pedhazur, E. J. & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, N.J.: L. Erlbaum. Pp. 73-80