PSYCHOLOGY 636

Fall, 1997

S. L. Golding

ADVANCED ISSUES IN RESEARCH DESIGN AND METHODOLOGY IN CLINICAL PSYCHOLOGY

This class is designed as part of the required clinical psychology curriculum, and presumes successful completion of the introductory research design and statistics sequence, which should have included basic psychometric issues concerning the reliability and validity of dependent variables, as well as the traditional "research design" issues. If that is not true for you, you will need to pay special attention to the topics not required, but listed at the end of the syllabus. You will also need to ask me for a set of handouts on the issues of construct validation, reliability, reliability phenomena, and generalizability theory. By design, it is intended to cover topical issues, at an advanced level, that will assist students in the completion of their dissertations and in gaining competency in research design and methodological issues that are likely to occur in their post-degree research and scholarly pursuits. Obviously, there can be no claim to complete coverage of all such issues. Rather, within the confines of our schedule, I have selected, with input from faculty and graduate students, a set of topics that seem to rise to the top of the heap. There are, of course, (many) other issues that could be included. Therefore, the class will also have a set of student generated presentations and a final paper covering additional topics.

Course structure

You will receive a required readings list that assigns weekly readings, arranged by topic. A master set of the assigned readings (articles, chapters, etc.) will be available in the Clinical Office. It is your responsibility to arrange to make personal copies from this master set. The materials may not be "individually" checked out, since it is unlikely that the required readings for a particular week would be available in a timely fashion for the whole group. There is no assigned text, although the readings required are also generally available in archival sources such as journals and books in the University Library.

It is your responsibility to read and study the materials in advance of each class. In general, I will give supplementary materials and explanations as part of a lecture during the first part of each class, but I expect all class members to be able to participate in an interactive discussion and probing of the issues. Twenty-five (25) percent of your class grade will be based upon your class participation.

I will also hand out a more extensive bibliography covering some of the issues we will discuss in greater detail, and also covering a number of issues we won't discuss. This is for your current and future use. A word of caution. The bibliography is reasonably up-to -date as of 1990, and some sections have been further updated to 1995; others have not (it is a rather massive undertaking).

Presentations and papers. There is no final examination in this course. The remainder of your course grade will be assigned on the basis of your final paper (50%) and your class presentation (25%). In order to expand the coverage of our topics and to assist you in your own area of specialization, you should prepare a final paper, of no more than 20 double-spaced pages, exploring either a) a current methodological, psychometric, or research design issue in your area of special interest in considerable depth or b) a review of major methodological issues in a particular area of research in a survey format. The purpose of your review is to evaluate current methodological, psychometric or research design "thinking" on the selected problem. Problems chosen for option (a) should be relatively clearly defined, but not so narrow as to be esoteric. Reviews chosen for option (b) should follow the general scope of review as exemplified in the September 1995, Volume 7(3) issue of Psychological Assessment. Because quarters are short beasts, you should choose your topic and have it approved by me no later than 10/19/95. I will be available by scheduled appointment to assist you in choosing a topic by discussing your interests and giving you leads. Some examples of topics are:

This paper and its presentation are, in essence, an opportunity for you to explore a design, methodological or psychometric issue that is of importance in your area of specialization, with the constraint that it be of some general interest and not too esoteric.

Presentation of papers will occur during the last two class meetings.

Class schedule

Review

^*Cronbach, L. J. (1990). Essentials of psychological testing (Fifth Edition). New York: Harper. Chapter 6, "How to judge tests: Reliability and other qualities," Pp. 190-222. [You probably know much of this material, but a little review and update for those of you who don't is important. Read first]

Dudek, F. J. (1979). The continuing misinterpretation of the standard error of measurement. Psychological Bulletin, 86, 335-337.

Ghiselli, E. E. (l964). Theory of psychological measurement. NY: McGraw-Hill. Pp. 207-253. (Not a required reading, but the classic development of the basic equations of reliability theory)

Statistical and clinical significance with difference, gain or change scores

^*Cronbach, L. J. (1990). Essentials of psychological testing (Fifth Edition). New York: Harper. Pp. 276-296.

^*Matarazzo, J. D. (1990). Psychological assessment versus psychological testing: Validation from Binet to the school, clinic, and courtroom. American Psychologist, 45, 999-1017.

Meehl, P. E. (l970). Nuisance variables and the ex post facto design. In M. Radner and S. Winokur (Eds.), Minnesota studies in the philosophy of science, 4, 373-402 (re-read if you haven't already from 635).

^*Pedhazur, E. J. & Schmelkin, L.P. (1991). Measurement, design and analysis. Hillsdale, N.J.: Erlbaum. Pp. 291-294; 567-582

More Psychometrics and Problems with inter-rater agreement (especially with categorical data)

^*Bartko, J. J., & Carpenter, W. T. (l976). On the methods and theory of reliability. Journal of Nervous and Mental Disease, 163, 307-317.

*Janes, C. L. (1979). Agreement measurement and the judgment process. Journal of Nervous and Mental Disease, 167, 343-347.

^* Schmidt, F.L., & Hunter, J.E. (1996). Measurement error in psychological research: Lessons from 26 research scenarios. Psychological Methods, 1, 199-223.

^*Uebersax, J. S. (1987). Diversity of decision-making models and the measurement of interrater agreement. Psychological Bulletin, 101, 140-146.

^*Zwick, R. (1988). Another look at interrater agreement. Psychological Bulletin, 103, 374-378.

Cohen, J. (l972). Weighted Chi Square: An extension of the Kappa method. Educational and Psychological Measurement, 32, 61-74. (Not required, but one of the classic kappa articles).

Cronbach, L. J. (l949). Statistical methods applied to Rorschach scores: A review. Psychological Bulletin, 46, 393-429. (A treasure-trove of logic about analyses of such data; not required)

Fleiss, J. L.. (l966). Assessing the accuracy of multivariate observations. Journal of the American Statistical Association, 61, 403-412. (Not required; pushing kappa into multiple raters, etc.)

Fleiss, J. L., Spitzer, R. L., Endicott, J., & Cohen, J. (l972). Quantification of agreement in multiple psychiatric diagnosis. Archives of General Psychiatry, 26, 168-171. (Not required; classic kappa application)

Hanley, J. A. (1987). Standard error of the Kappa statistic. Psychological Bulletin, 102, 315-321. (Not required, but an awfully useful reference when you actually do one)

Maxwell, A. E. (l968). The effect of correlated errors on estimates of reliability coefficients. Educational and Psychological Measurement, 28, 803-811. (Not required, but a classic reference on the subject)

Maxwell, A. E, Pilliner, A. E. G,. (l968). Deriving coefficients of reliability and agreement for ratings. British Journal of Mathematical and Statistical Psychology, 21, 105-116. (Not required, but the source of the Maxwell RE coefficient discussed in Janes)

Spitzer, R. L., Cohen, J., Fleiss, J. L. Endicott, J. (l967). Quantification of agreement in psychiatric diagnosis. Archives of General Psychiatry, 17, 83-87. (Not required; another classic kappa reference; why say it once when you can say it twice in the same journal -- helps the vita and promotion)

Problems with assessment and classification

^*Clark, L.E., Livesley, W. J., Schroeder, M.L., & Irish, S. L. (1996). Convergence of two systems for assessing specific traits of personality disorder. Psychological Assessment, 8, 294-303.

^*Risso, L.P. et al., (1994). Concordance between patients and informants on the Personality Disorder Examination. American Journal of Psychiatry, 151, 568-573.

^*Trull, T.J., Widiger, T.A., & Guthrie, P. (1990). Categorical versus dimensional status of borderline personality disorder. Journal of Abnormal Psychology, 99, 40-48.

^*Widiger, T.A., & Trull, T.J. (1991). Diagnosis and clinical assessment. In M.R. Rosenzweig and L.W. Porter (Eds). Annual Review of Psychology, 42, 109-133. Palo Alto, CA. : Annual Review Press.

^*Widiger, T.A. & Sanderson, C. J. (1995). Assessing personality disorders. In J. Butcher (Ed.), Clinical personality assessment: Practical approaches. New York: Oxford. Pp. 381-394.

Decision theory

^*Baldassarini, R., Finklestein, S., & Arana, G.W. (1983). The predictive power of diagnostic tests and the effect of prevalence of illness. Archives of General Psychiatry, 40, 569-573. (The classic reference in the psychiatric literature, although not the most readable -- sorry, but required nonetheless)

*Faust, D. (1991). What if we had really listened? Present reflections on altered pasts. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol:1 Matters of public interest. Essays in honor of Paul E. Meehl. University of Minnesota Press: Minneapolis, MN. Pp. 185-216.

^*Finn, S. E. & Kampuis, J. H. (1995). What a clinician needs to know about base rates. In J. N. Butcher (Ed.), Clinical personality assessment. Oxford: New York. Pp. 224-235.

^*Kessel, J.B., & Zimmerman, M. (1993). Reporting errors in studies of diagnostic performance .... . Psychological Assessment, 5, 395-399.

*Meehl, R F., & Rosen, A. (l955). Antecedent probability and the efficiency of psychometric signs, patterns, and cutting scores. Psychological Bulletin, 52, 194-216.

^*Wiggins, J. S. (1973). Personality and prediction : Principles of personality assessment. Reading, Mass. : Addison Wesley, Pp. 224- 274. (Perhaps the most readable of all sources on classic decision-theory; unfortunately, not presented in terms of the latest specificity, sensitivity, NPP and PPP zeigeist)

Cronbach, L. J., & Gleser, G. C. (l957). Psychological tests and personnel decisions. Urbana: University of Illinois Press. (Not required; the classic statement - fairly tough going; very useful however when you get into analytic problems)

Dawes, R. (l971). A case study of graduate admissions : Application of three principles of human decision making. American Psychologist, 26, 180-188. (Not required; early application to a situation near and dear to our hearts; origin of the dreaded GRE selection equation-since I went to Oregon and Robyn was one of my advisors, I often wonder how I made it in since my MMPI flunked me out of Minnesota!)

Hirschberg, N., & Itkin, S. (l978). Graduate student success in psychology. American Psychologist, 33, 1088- 1093. (Not required; another classic application)

Magnusson, D. (1967). Test theory. Reading, Mass.: Addison-Wesley, Pp. 198-225. (Not required; a more basic presentation of decision theory in an undergraduate text worth owning)

Motto, J. A., Heilbron, D. C., & Juster, R. P. (1985). Development of a clinical instrument to estimate suicide risk. American Journal of Psychiatry, 142, 680-686.(Not required, but very interesting application to suicide prediction)

Satz, P., Fennell, E., & Reilly, C. (l970). Predictive validity of six neurodiagnostic tests : A decision theory analysis. Journal of Consulting and Clinical Psychology, 34, 375-381. (Recommended, as an application and demonstration that different tests produce different configurations of error, but not required.)

ROC/Itemmetrics

^*Cooke, D. J. & Michie, C. (1997). An item response theory analysis of the Hare Psychopathy Checklist-Revised. Psychological Assessment, 9, 3-14.

^*Gardner, W., et al. (1996). Clinical versus actuarial predictions of violence in patients with mental illnesses. Journal of Consulting and Clinical Psychology, 64, 602-609.

^*Rice, M.E., & Harris, G.T. (1995). Violent recidivism: Assessing predictive validity. Journal of Consulting and Clinical Psychology, 63, 737-748.

Clinical judgment

^*Garb, H.N., & Schramke, C. J. (1996). Judgment research and neuropsychological assessment: A narrative review and meta-analysis. Psychological Bulletin, U120, 140-153.

^*Goldberg, L. R. (1991).. Human mind versus regression equation: Five contrasts. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol:1 Matters of public interest. Essays in honor of Paul E. Meehl. University of Minnesota Press: Minneapolis, MN. Pp. 173-184.

*Kleinmuntz, B. (1991). Recent developments in computerized clinical judgment. In D. Cicchetti & W. M. Grove (Eds.), Thinking clearly about psychology: Vol:1 Matters of public interest. Essays in honor of Paul E. Meehl. University of Minnesota Press: Minneapolis, MN. Pp. 217-234.

^*Kleinmuntz, B. (1990). Why we still use our heads instead of formulas: Toward an integrative approach. Psychological Bulletin, 107, 296-310.

Borum, R., Otto, R., & Golding, S. L. (1993). Improving clinical judgement and decision making in forensic evaluations. International Journal of Law and Psychiatry, 21, 35-76.

Dawes, R. M., Faust, D., & Meehl, P. E. (1989). Clinical versus actuarial judgment. Science, 243, 1668-1674.

Einhorn, H. J., & Hogarth, R. M. (l978). Confidence in judgment:Persistence of the illusion of validity. Psychological Review, 85, 395-416.

Goldberg, L. R. (l968). Simple models or simple processes? Some research on clinical judgments. American Psychologist, 23, 483-496.

Goldberg, L. R. (l970). Man versus model of man:A rational, plus some evidence,for a method of improving clinical inferences. Psychological Bulletin, 73, 422-432. (Two classics, not required, by my old mentor and the father of linear models of clinical judgment)

Kahneman, D., & Tversky, A. (l973). On the psychology of prediction. Psychological Review, 80, 237-251. (The source for the commonly used "heuristic" analysis of social judgment processes; worth reading, but not required)

Meehl, P. E. (1973). Why I do not attend case conferences. In P. E. Meehl, Psychodiagnosis: Selected Papers of P. E. Meehl (pp. 225-304). Minneapolis, Minn. : University of Minnesota Press. (Not required, but tremendously enjoyable and thought provoking for those of us who do attend case conferences, clinical admissions meetings, and the like)

Nurius, P.S., & Nicoll, A. E. (1992). Capturing clinical expertise: An analysis of knowledge "mining" through expert system development. Clinical Psychology Review, 12, 705-717.

Sawyer, J. (l966). Measurement and prediction, clinical and statistical. Psychological Bulletin, 66, 178-200. (Another clinical judgment classic, especially if you get serious about the design of a study; not required)

Other sections for students not having 635 recently

Golding, S L. (l979). Flies in the ointment : Methodological problems in the analysis of the percentage of variance due to persons and situations. Psychological Bulletin, 82, 278-288. (While written in the context of the person-situation debate, it provides a useful entry into the underlying theory and use of generalizability logic and computing techniques)

Shavelson, R. J., Webb, N. M., & Rowley G. L. (1989). Generalizability theory. American Psychologist, 44, 922-932. (You may experience the last part as tough going (I certainly do), but the first 2/3 is another useful statement of the theory worth reading)

Brennan, R L. (l980). Applications of generalizability theory. In R. A. Berk, Criterion-referenced measurement: State of the art. Baltimore: Johns Hopkins Press 186-232. (Not required, but good to consult for various issues; more readable than the "master.")

Cattell, R. B. (1964). Validity and reliability : A proposed more basic set of concepts. Journal of Educational Psychology, 55, 1-22. (Not a required reading; however, strongly recommended if you don't understand the concepts of generalization across domains such as time, rater, items and the like; Ray deserves his reputation of anticipating many developments in psychometrics).

Cronbach, L. J., Gleser, G. C., Nanda, H. et al. (1972). The dependability of behavioral measures. New York: Wiley. (Not required; the master speaks, but he's awfully hard to understand; the classic reference and bible which you will end up reading parts of sometime in your life if you do G-theory work)

Cronbach, L. J., Rajaratnam, N., & Gleser, G. C. (l963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163. (Ditto)

Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory: 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133-166. (Not required, but an early application that contains some useful examples from clinical contexts)

Stanley, J. C. (l971). Reliability. In R. L. Thorndike (Ed.), Educational Measurement. Washington DC: American Council on Education, Pp. 356-442. (This is not a required reading, but is well worth your reading, if you can get through it, as it pulls the entire area of classical and neo-classical reliability theories together.)

Tryon, R C. (l957). Reliability and behavior domain validity : Reformulation and historical critique. Psychological Bulletin, 54, 229-249. (Not a required reading, but the classic statement of domain theory which laid the ground work for G-theory)

Wiggins, J. (1973). Personality and prediction : Principles of personality assessment. New York: Addison-Wesley. (Not required, but out of print and a classic text if you can get a copy of it; contains a useful introduction to some G-theory concepts, albeit somewhat dated)

Generalizability theory- II

DuBois, P. H. (1970). Varieties of psychological test homogeneity. American Psychologist, 25, 532-536.

Meehl, R. E. (1991). Why summaries of research on psychological theories are often uninterpretable. In R. E. Snow & D. E. Wiley (Eds.), Improving inquiry in social science: A volume in honor of Lee J. Cronbach. Erlbaum: Hillsdale, NJ. Pp. 13-59. (Required; although not strictly concerned with G-theory, it is a re-statement and re-analysis of many of the problems of construct theory intimately related to the logic of G-theory)

Levy, P. (l974). Generalizability studies in clinical settings. British Journal of Social and Clinical Psychology, 13, 161-172. (Not required; useful examples of application.)

McGaw, B., Wardrop, J. L., & Bunda, M. A. (l972). Classroom observation schemes : Where are the errors?. American Educational Research Journal, 9, 13-27.(Not required; application of G-theory to classroom observational data)

Shavelson, R. J., & Webb, N. M. (1981). Generalizability theory : 1973-1980. British Journal of Mathematical and Statistical Psychology, 34, 133-166. (Not required; more on classic applications)

Shrout, P. E., & Fleiss, J. L. (l979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86, 420-428. (Not required; how to compute G-coefficients without knowing you're doing it and perhaps not understanding the assumptions; the most commonly cited reference in the psychopathology literature for rater studies)

Reliability phenomena and problems

Dudek, F. J. (1979). The continuing misinterpretation of the standard error of measurement. Psychological Bulletin, 86, 335-337.

Epstein, S. (1986). Does aggregation produce spuriously high estimates of behavior stability?. Journal of Personality and Social Psychology, 50, 1199-1210.

Matarazzo, J. D. (1990). Psychological assessment versus psychological testing: Validation from Binet to the school, clinic, and courtroom. American Psychologist, 45, 999-1017.