Publishing Functioning JD Institutional Learning Outcomes on Legal Research: Why and How?

Abstract

This article examines the legal-research institutional learning outcomes of 196 law schools accredited by the American Bar Association and proposes improvement to best practices. The suboptimal format and content of legal-research institutional learning outcomes contribute to this deficiency, exacerbated by challenges from the upcoming new bar examination. To improve legal-research institutional learning outcomes, law schools should distinguish legal-research institutional learning outcomes from other institutional learning outcomes, include specific and effective performance criteria, and update learning outcomes periodically. To streamline this process, law schools should draw wisdom from both external benchmarks and the expertise of internal legal-research instructors.

Keywords:

• law school
• legal research
• institutional learning outcomes

Disclosure Statement

The authors report there are no competing interests to declare.

Data Availability Statement

The data that support the findings of this study are openly available in author’s Google Drive at https://docs.google.com/spreadsheets/d/1O3uopTSwzUToWXeqcaPn6ulahxfyjhaZ/edit?usp=share_link&ouid=114847982357325960396&rtpof=true&sd=true.

Notes

1 kelly terry et al., assessment of teaching and learning: a comprehensive guidebook for law schools 4 (2021).

2 roy t. stuckey, best practices for legal education: a vision and a road map 42 (2007).

3 terry et al., supra note 1, at 8–9.

4 lori shaw & victoria vanzandt, student learning outcomes and law school assessment: a practical guide to measuring institutional effectiveness 58 (2015).

5 Terry et al., supra note 1, at 9.

6 Id.

7 Id.

8 aba section of legal educ. & admissions to the bar, transition to and implementation of the new standards and rules of procedure for approval of law schools (2014), https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/governancedocuments/2014_august_transition_and_implementation_of_new_aba_standards_and_rules.pdf.

9 aba section of legal educ. & admissions to the bar, report of the outcome measures committee 4 (2008) [hereinafter measures report], https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/reports/2008_outcome_measures_committee_final_report.pdf.

10 Id.

11 stuckey, supra note 2, at 40–42, 49.

12 paul l. dressel, handbook of academic evaluation: assessing institutional effectiveness, student progress, and professional performance for decision making in higher education 303 (1976).

13 terry et al., supra note 1, at 4 Figure 1.1.

14 See infra Figure 1 (185 of 196 ABA accredited law schools have published their ILOs on their websites).

15 2021–2022 aba standards and rules of procedure for approval of law schools 17 [hereinafter 2021–2022 standards], https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/standards/2021-2022/2021-2022-aba-standards-and-rules-of-procedure.pdf.

16 stuckey, supra note 2, at 41.

17 Id.

18 Stephen M. Sheppard, The American Legal Profession in the Twenty-First Century, 62 Am. J. Comp. L. 241, 256 (2014) (including a list of the common fundamental courses offered in law schools, such as property, torts, criminal law, civil procedure, etc.).

19 E.g., Stuckey, supra note 2, at 40–42, 49.

20 Measures Report, supra note 9.

21 Id. at 6.

22 Id. at 54.

23 Id. at 57–58.

24 donald j. polden, aba standards rev. comm., statement of principles of accreditation and fundamental goals of a sound program of legal education 1 (2009), https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/standards_review/principles_and_goals_accreditation_5_6_09.pdf.

25 Id. at 4.

26 steve bahls, chair of subcommittee on student learning outcomes, key issues considered by the student learning outcomes subcommittee 1 (Dec. 15, 2009), https://researchguides.uoregon.edu/ld.php?content_id=34581676.

27 Id.

28 Id.

29 Id.

30 Id.

31 steve bahls, chair of subcommittee on student learning outcomes, key issues considered by the student learning outcomes subcommittee 2 (Apr. 17, 2010) [hereinafter key issues], https://www.albanylaw.edu/media/8335/download.

32 Id.

33 A.B.A. Sec. Of Legal Educ. And Admissions To The Bar, Explanation Of Changes 6 (2014) [hereinafter Explanation Of Changes], https://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/council_reports_and_resolutions/201408_explanation_changes.pdf.

34 Id. at 7.

35 2014–2015 ABA Standards and Rules of Procedure for Approval of Law Schools 15 [hereinafter 2014–2015 Standards], https://www.americanbar.org/content/dam/aba/publications/misc/legal_education/Standards/2014_2015_aba_standards_and_rules_of_procedure_for_approval_of_law_schools_bookmarked.pdf.

36 Explanation Of Changes, supra note 33, at 7.

37 Key Issues, supra note 31, at 4.

38 2014–2015 Standards, supra note 35, at 23.

39 Id. at 24.

40 American Bar Association, MANAGING DIRECTOR’S Guidance Memo: Standards 301, 302, 314, & 315 (June 2015) [hereinafter Managing Director’s Guidance Memo], http://www.americanbar.org/content/dam/aba/administrative/legal_education_and_admissions_to_the_bar/governancedocuments/2015_learning_outcomes_guidance.authcheckdam.pdf.

41 Id. at 3.

42 Id.

43 Id.

44 Id.

45 Key Issues, supra note 31, at 3.

46 Id.

47 managing director’s guidance memo, supra note 40, at 4.

48 Id.

49 Id.; Explanation Of Changes, supra note 33, at 6.

50 managing director’s guidance memo, supra note 40, at 4–7.

51 2021–2022 Standards, supra note 15.

52 managing director’s guidance memo, supra note 40, at 4.

53 2021–2022 Standards, supra note 15.

54 managing director’s guidance memo, supra note 40, at 4.

55 Id.

56 See id. at 3.

57 Holloran Center, Learning Outcomes Database, U. ST. THOMAS, https://www.stthomas.edu/hollorancenter/learningoutcomesandprofessionaldevelopment/learningoutcomesdatabase/ (last visited June 17, 2022).

58 laura ray & stacy etheredge, legal research learning outcomes and assessment plans: a survey of current developments and methods for law librarian engagement, presentation at the 2019 american association of law libraries annual meeting (July 2019).

59 Holloran Center, Learning Outcomes 302(b) and (d), U. ST. THOMAS, https://www.stthomas.edu/hollorancenter/learningoutcomesandprofessionaldevelopment/learningoutcomesdatabase/learningoutcomes302b/ (last visited June 17, 2022).

60 Laura Ray, Survey of Legal Research Learning Outcomes and Assessment Plans, CM Law Library Blog, https://cmlawlibraryblog.classcaster.net/2019/07/25/survey-of-legal-research-learning-outcomes-and-assessment-plans/ (last visited June 17, 2022).

61 U.S. News, 2023 Best Law Schools, https://www.usnews.com/best-graduate-schools/top-law-schools/law-rankings, (last visited June 17, 2022).

62 2021–2022 Standards, supra note 15.

63 For example, Columbia University and William and Mary Law School, as two top-30 law schools, do not publish ILOs on their websites. Some school’s ILOs are temporarily not available on their websites, which were accessible in February 2022, including Ohio State University and the University of Wisconsin.

64 E.g., Northern Illinois University, Student Learning Outcomes and Proposed Methods for Collecting Data, https://www.niu.edu/effectiveness/_files/outcomes/law/law-jd.pdf (last visited June 17, 2022.); Oklahoma City University School of Law, Learning Outcomes, https://law.okcu.edu/why-ocu-law/post-graduation/ (last visited June 17, 2022).

65 2021–2022 Standards, supra note 15.

66 See infra Figure 3.

67 This article treats an institution that explicitly accompanies a bullet point outlining the criteria for legal-research SLOs as one (1) institution. A law school is a half (0.5) institution if it mixes legal-research SLOs and corresponding criteria in a single sentence. For example, this article treats The University of Alabama Law School as one institution, https://www.law.ua.edu/academics/learning-outcomes/, while UC Hasting Law is a half institution, https://www.uchastings.edu/home/aba-required-disclosures/.

68 2021–2022 Standards, supra note 15.

69 managing director’s guidance memo, supra note 40, at 4.

70 managing director’s guidance memo, supra note 40, at 4.

71 Id.

72 managing director’s guidance memo, supra note 40, at 4–7.

73 national institute for learning outcomes assessment, more than you think, less than we need: learning outcomes assessment in american higher education 21 (2009), https://www.learningoutcomesassessment.org/wp-content/uploads/2019/02/2009NILOASurveyReport.pdf (last visited July. 17, 2022).

74 Stuckey, supra note 2, at 49; See Shaw & Vanzandt, supra note 4, at 64.

75 Shaw & Vanzandt, supra note 4, at 62, 64; Terry ET AL., supra note 1, at 68.

76 andrea susnir funk, the art of assessment: making outcomes assessment accessible, sustainable, and meaningful 45 (2017).

77 Id.

78 See generally lorin w anderson et al., a taxonomy for learning, teaching, and assessing: a revision of bloom’s taxonomy of educational objectives 63–92 (2001).

79 Shaw & Vanzandt, supra note 4, at 68.

80 Stuckey, supra note 2, at 49.

81 Terry et al., supra note 1, at 68.

82 Stuckey, supra note 2, at 49; Terry et al., supra note 1, at 44, 71.

83 Stuckey, supra note 2, at 50; Terry et al., supra note 1, at 71.

84 2021–2022 Standards, supra note 15.

85 William R. Trail & William D. Underwood, The Decline of Professional Legal Training and A Proposal for Its Revitalization in Professional Law Schools, 48 Baylor L. Rev. 201, 202 (1996).

86 See aba task force on law sch. & the profession, legal education and professional development—an educational continuum 135 (1992) [hereinafter MacCrate Report].

87 2021–2022 Standards, supra note 15, at 25.

88 Id.

89 2021–2022 Standards, supra note 15, at 25.

90 managing director’s guidance memo, supra note 40, at 4.

91 2021–2022 Standards, supra note 15, at 25.

92 mary j. allen, assessing academic programs in higher education 6 (2004).

93 Susan Hanley Duncan, They’re Back! The New Accreditation Standards Coming to a Law School Near You—A 2018 Update, Guide to Compliance, and Dean’s Role in Implementing, 67 J. Legal Educ. 462, 481 (2018).

94 Victoria L. VanZandt, Creating Assessment Plans for Introductory Legal Research and Writing Courses, 16 Legal Writing: J. Legal Writing Inst. 313, 345–46 (2010).

95 Shaw & VanZandt, supra note 4, at 104.

96 linda a. suskie, assessing student learning: a common sense guide, 290–96 (2018).

97 See id. 290; Shaw & VanZandt, supra note 4, at 93.

98 Stuckey, supra note 2, at 245.

99 See Suskie, supra note 104, at 291.

100 Stuckey, supra note 2, at 243; Shaw & VanZandt, supra note 4, at 93.

101 Andrea Susnir Funk & Kelley M. Mauerman, Starting from the Top: Using a Capstone Course to Begin Program Assessment in Legal Education, 37 Okla. City U. L. Rev. 477, 488–89 (2012).

102 See Suskie, supra note 104, at 295.

103 Terry et al., supra note 1, at 33; Shaw & VanZandt, supra note 4, at 93.

104 Suskie, supra note 104, at 291.

105 Id. at 292.

106 Id. at 291.

107 Terry et al., supra note 1, at 20–22.

108 Shaw & VanZandt, supra note 4, at 93.

109 Suskie, supra note 104, at 293–98. The value-added standards doubles the institution's workload because, in addition to assessing the competencies of graduates, the institution must also assess the competencies of incoming students so the institution can compare that evidence to assess the added value of a law school education to graduates. Value-added standards may not be accurate because it is difficult to know whether a student's improved legal-research skills are due to a good legal-research curriculum or extensive legal-research experience in an externship.

110 See id. at 295.

111 Id. at 291.

112 Terry et al., supra note 1, at 28.

113 See Suskie, supra note 104, at 297.

114 Linda A. Suskie, answering the complex question of “how good is good enough?”, 19 Assessment Update 12 (2007).

115 Suskie, supra note 104, at 297; Shaw & VanZandt, supra note 4, at 93.

116 Suskie, supra note 104, at 297.

117 See Shaw & VanZandt, supra note 4, at 111.

118 See id.

119 Id. at 28.

120 See id. at 111.

121 William m. sullivan et al., educating lawyers: preparation for the profession of law 168 (2007).

122 See infra Figure 1; See infra Figure 2; See also, infra Figure 3.

123 See infra Figure 3.

124 2021–2022 Standards, supra note 15.

125 Suskie, supra note 104, at 50.

126 For example, legal research is not merely a skill of finding information. It also requires the researcher to identify key legal issues from client’s facts, locate key legal information, and evaluate the usefulness of found resources. Another example is the relationship between problem-solving skills, and legal analysis and reasoning skills. One subcategory of problems-solving skills is that an attorney can identify a client’s problem. One lower-level skill of legal analysis and reasoning skills is for a lawyer to identify a client’s legal issue. Problem identification and legal-issues identification cannot be separated from each otherbecause problem identification also includes legal-framework identification, which covers specific legal-issues identification. See MacCrate Report, supra note 94, at 142, 151–52.

127 2021–2022 Standards, supra note 15, at 25; Managing Director’s Guidance Memo, supra note 40, at 4–7.

128 2014–2015 Standards, supra note 35, at 25.

129 E.g., Andrea A. Curcio, A better bar: why and how the existing bar exam should change, 81 Neb. L. Rev. 363, 411–12 (2002). See, e.g., Patrick J. Meyer, adding legal research to the bar exam: what would the exercise look like?, 53 Akron L. Rev. 107 (2019).

130 Nat’l Conf. Of B. Exam’rs Testing Task Force, Your Voice: Stakeholder Thoughts about the Bar Exam Phase (Aug. 2019), https://nextgenbarexam.ncbex.org/wp-content/uploads/FINAL-Listening-Session-Executive-Summary-with-Appendices-2.pdf; Nat’l Conf. Of B. Exam’rs Testing Task Force, 2019 Practice Analysis Phase 2 (Mar.2020), https://nextgenbarexam.ncbex.org/wp-content/uploads/TestingTaskForce_Phase_2_Report_031020.pdf; Nat’l Conf. Of B. Exam’rs Testing Task Force, Blueprint Development Committee And Test Design Committee Meetings Phase 3 (Nov. 2020) [hereinafter Phase 3], https://nextgenbarexam.ncbex.org/wp-content/uploads/TTF-Phase-3-Report-110420.pdf.

131 Nat’l Conf. Of Bar Exam’rs, Final Report Of The Testing Task Force 9 (Apr. 2021) [hereinafter Final Report Of The Testing Task Force], https://nextgenbarexam.ncbex.org/wp-content/uploads/TTF-Final-Report-April-2021.pdf.

132 Id. at 10.

133 Id. at 12.

134 Id. at 16.

135 Id. at 19; Phase 3, supra note 138, appendix a: bdc recommendations for tasks under skill domains.

136 Final Report of The Testing Task Force, supra note 139, at 16-17; Phase 3, supra note 138, appendix b: bdc recommendations for knowledge areas.

137 final report of the testing task force, supra note 139, at 21.

138 Nat’l Conf. Of Bar Exam’rs, NCBE Board of Trustees Votes to Approve Testing Task Force Recommendations, https://www.ncbex.org/news/ncbe-bot-vote-approves-ttf-recommendations/ (last visited Sept. 8, 2022).

139 Nat’l Conf. Of Bar Exam’rs, Implementing the Next Generation of the Bar Exam, 2021–2026, https://nextgenbarexam.ncbex.org/about/implementation-timeline/ (last visited Sept. 8, 2022).

140 See Nat’l Conf. Of Bar Exam’rs, Thanks for commenting on the Content Scope Outlines, https://nextgenbarexam.ncbex.org/thanks-for-commenting-on-the-content-scope-outlines/ (last visited Sept. 8, 2022).

141 Stephanie Farne, Is This Our Moment? Legal Research and the Next Gen Bar Exam, RIPS Law Librarian Blog (June 2, 2022), https://ripslawlibrarian.wordpress.com/2022/06/02/is-this-our-moment-legal-research-and-the-next-gen-bar-exam (stating that legal research professionals have already begun brainstorming to embrace this change, such as offering more 1L legal-research instruction or more legal-research electives).

142 Melissa Shultz, Professor, Please Help Me Pass the Bar Exam: #NEXTGENBAR2026, 69 J. Legal Educ. 34–37 (forthcoming).

143 While 2026 seems far off, law schools should make preparations for the class of 2026 as soon as they are matriculated. In other words, law schools should take action for the upcoming fall 2023 class.

144 See Suskie, supra note 104, at 291.

145 Terry et al., supra note 1, at 33; Shaw & Vanzandt, supra note 4, at 93.

146 Nat’l Conf. Of Bar Exam’rs, Next Gen Bar Exam Content Scope Outlines For Public Comment 5 (2022) [hereinafter Content Scope Outlines], https://nextgenbarexam.ncbex.org/csopc-register/.

147 Id.

148 See supra section, Lack of Legal-Research ILOs Criteria.

149 See infra Figure 1.

150 University of Minnesota Law School, Learning Outcomes, https://law.umn.edu/current-students/academic-resources/learning-outcomes#legal-thinking-460 (under “Legal Thinking & Analysis” tab) (last visited Sept. 11, 2022).

151 See infra Figure 1.

152 See id.

153 2021–2022 Standards, supra note 15.

154 See supra section, Meeting Minimum Pedagogical Requirements.

155 See laura saunders, information literacy as a student learning outcome: the perspective of institutional accreditation 185 (2011).

156 See Linda Krzykowski & Kevin Kinser, Transparency in Student Learning Assessment: Can Accreditation Standards Make a Difference?, 46 Change 67, 68 (2014).

157 See discussion, supra section, Lack of Independent Legal-Research ILOs.

158 See id.

159 University of Connecticut School of Law, Learning Outcomes, https://law.uconn.edu/academics/learning-outcomes/ (last visited Sept. 29, 2022).

160 Case Western Reserve University School of Law, Learning Outcomes, https://case.edu/law/our-school/aba-disclosures/learning-outcomes (last visited Sept. 29, 2022).

161 See infra Figure 3.

162 2021–2022 Standards, supra note 15, at 15.

163 MacCrate Report, supra note 94, at 163–71.

164 See discussion, supra section, Inadequate Content and Format.

165 Shaw & Vanzandt, supra note 4, at 62, 64; Terry et al., supra note 1, at 68.

166 Shaw & Vanzandt, supra note 4, at 64.

167 Terry et al., supra note 1, at 68.

168 Shaw & Vanzandt, supra note 4, at 62.

169 Id.

170 Id. at 118.

171 Id. at 62.

172 Id. at 118.

173 Id.

174 See id. at 143.

175 Id. at 110.

176 See id. at 119.

177 See id.

178 Saunders, supra note 163, at 187.

179 Id. at 185–87.

180 Terry et al., supra note 1, at 71.

181 See supra section, Meeting Minimum Pedagogical Requirements.

182 University of Chicago Law School, Learning Outcomes, https://www.law.uchicago.edu/learning-outcomes (last visited April. 11, 2023).

183 Shaw & Vanzandt, supra note 4, at 64.

184 Id. at 65; Suskie, supra note 104, at 47.

185 See Suskie, supra note 104, at 47.

186 Shaw & Vanzandt, supra note 4, at 66.

187 MacCrate Report, supra note 94, at 157.

188 Id. at 159.

189 Id. at 160.

190 content scope outlines, supra note 154.

191 See generally Lorin W Anderson et al., supra note 86.

192 See Suskie, supra note 104, at 49.

193 Shaw & Vanzandt, supra note 4, at 58.

194 See supra notes 146–48 and accompanying text.

195 AALL, Principles And Standards For Legal Research Competency, https://www.aallnet.org/wp-content/uploads/2020/04/AALL2020-PrinciplesStandardsLegalResearchCompetencyFull.pdf (last visited Sept. 11, 2022) [hereinafter Principles].

196 AALL, Think You Know Law Librarians? Think Again, https://www.aallnet.org/about-us/who-we-are/ (last visited Sept. 11, 2022).

197 AALL, Legal Research Competency Outreach for Law Schools, https://web.archive.org/web/20200925080255/https://www.aallnet.org/advocacy/legal-research-competency/outreach/ (last visited Sept. 11, 2022).

198 Principles, supra note 203.

199 Villanova University Charles Widger School of Law, Learning Outcomes, https://www1.villanova.edu/content/university/law/academics/learning-outcomes.html (last visited Sept. 29, 2022).

200 University of Alabama School of Law, Learning Outcomes, https://www.law.ua.edu/academics/learning-outcomes/ (last visited Sept. 29, 2022).

201 Florida A&M University, Learning Outcomes, https://law.famu.edu/students/current/learning-outcomes.php (last visited Sept. 29, 2022).

202 Shaw & Vanzandt, supra note 4, at 74.

203 American Association of Law Libraries, Academic Law Libraries, https://www.aallnet.org/careers/about-the-profession/types-of-law-libraries/ (last visited Sept. 29, 2022).

204 Id.

205 See Kristen Holmquist, Challenging Carnegie, 61 J. Legal Educ. 353, 358 (2012).

206 Shaw & Vanzandt, supra note 4, at171 tbl.6.9.

207 Texas Tech University School of Law (2022), Mission Statement & Student Learning Outcomes, https://www.depts.ttu.edu/law/about/mission-statement.php (last visited Sept. 29, 2022).

208 Shaw & Vanzandt, supra note 4, at 74.

209 Carol A. Parker, The Need for Faculty Status and Uniform Tenure Requirements for Law Librarians, 103 Law Libr. J. 7, 11 (2011) (finding that more than three quarters of nondirector legal-research instructors have faculty status).

210 Managing Director’s Guidance Memo, supra note 40.

211 Id.

212 Suskie, supra note 104, at 359.

213 Id.

214 See id.

215 Id. 356–57.

216 Texas Tech University School of Law (2020), Mission Statement & Student Learning, https://web.archive.org/web/20201203153119/https://www.depts.ttu.edu/law/about/mission-statement.php (last visited Sept. 29, 2022).

217 texas tech university school of law (2022), supra note 205.

218 Caroline L. Osborne, The State of Legal Research Education: A Survey of First-Year Legal Research Programs, or “Why Johnny and Jane Cannot Research, 108 Law Libr. J. 403, 406 (2016).

219 See supra section, The New Challenge from NextGen Bar Exam.