Assessment of the Miami Plan

Each time a faculty member proposes a course for inclusion in the Miami Plan, they must provide the following information to the Liberal Education Council:

• The language used in the syllabus to describe the Pillars and other learning outcomes for the course; and
• The assignments and other assessments in the course that advance the Pillars and other learning outcomes that are specific to the component of the MGP for which they are proposing the course.

Each course proposal is reviewed by at least two members of Liberal Education Council who use the Course Review Form to evaluate the course in terms of its outcomes and to provide the faculty with qualitative feedback to improve the course to ensure that the course advances outcomes in quality ways. The reviewers’ comments are then discussed and voted upon by the full Liberal Education Council.

Courses can be approved, approved “with revision,” rolled-back for further clarifications and review, or denied.  Faculty whose courses are not approved are provided one-on-one consultation and/or invited to join a Faculty Fellows Learning Community where they are given the opportunity and support to revise the course to advance the appropriate outcomes.

The National Survey of Student Engagement is a standardized survey tool used to assess the extent to which students engage in educational practices associated with high levels of learning and development. At Miami, NSSE is administered every other spring to first-year and senior students.

The Office of Institutional Research can provide a summary of the responses to three questions that relate directly or indirectly to liberal education outcomes. 

The components of the survey questions are mapped to the appropriate outcomes in the MP, and survey data for each outcome are integrated into the assessment report for the MP and analyzed along with other outcome-related data.

Each semester during the academic year, members of LEC will evaluate a random sampling of syllabi and assignments from one component of the plan to ensure that existing courses are continuing to:

• Explain the liberal education pillars and appropriate learning outcomes on the course syllabus
• Appropriate learning outcomes in a quality manner via assignments and other forms of assessment.

Prior to each semester, the Director of Liberal Education in consultation with the University Registrar generates a list of courses that are approved for the selected component to be reviewed for the next semester.

At least 20% of courses in the generated list of courses are randomly selected for assessment.  The faculty and department chair are contacted to request the appropriate materials (i.e., syllabus plus assignment prompts) to be submitted no later than the end of the first week of the semester. 

Course materials are evaluated using a scoring and feedback guide. 

Direct assessment of the liberal education outcomes associated with one of the Miami Plan components is conducted by collecting student artifacts produced in courses meeting that requirement and evaluating them via a rubric. 

A list of courses approved for the particular component and to be offered for the following semester or term is created each semester.  All courses on the list are eligible for participation in the assessment project.  A sample of approximately 20% of the courses from this list are randomly selected for inclusion in the assessment project. 

In consultation with the department chair, the LEC Director notifies faculty who are assigned to teach selected courses and sections via email at least one month prior to the beginning of the semester or term.  The notification will explain that their course has been randomly selected for assessment.  Faculty should do the following:

• Include suggested language in their syllabus informing students that their work may be randomly selected for assessment purposes
• Select one individual written assignment or project that aligns with the two Pillars
• Load the assignment prompt as well as a random selection of four student artifacts (e.g., the first, fourth, seventh and tenth students who are listed on the roster) related to that assignment into a specified Google folder by the end of the semester or term when they teach the course.

Once each academic year, a trained group of assessment evaluators read and score the artifacts using a normed rubric.  Assessment evaluators will undergo a norming session to encourage strong consistency in scoring prior to beginning the scoring process. 

Although the overall percentage of proposals eventually approved is strong, the percentage of proposals approved on the first submission seems low.

Liberal Education Council has noted the most common issues encountered in the proposal review, in priority order:

• Syllabus not updated
• Assignments not provided for all Miami Plan Student Learning Outcomes
• Assignments and/or syllabus is not well aligned with the Miami Plan Pillars or SLOs

LEC will review these data and develop strategies for improvement during the next reporting cycle.

The data summarized below comes from the 2019 National Survey of Student Engagement (NSSE) results which means that the data were gathered before the new Miami Plan was implemented.  We will use these data as a baseline for comparison purposes.  The means reported are for graduating seniors at Miami.

*Miami seniors estimated that they were assigned an average of 83.4 pages of writing in the current year, whereas students at comparison institutions estimated that they were assigned 74.8 pages.

These data show that in almost all categories Miami students perceive greater or similar learning as students in comparison institutions. 

Some of the strongest areas where the Miami seniors’ mean was at least .2 points higher than the mean reported by seniors at comparison institutions include:

• Connecting learning to societal issues
• Evaluating what others have concluded from numerical information
• Writing papers of varying lengths
• Working effectively with others
• Solving complex real-world problems

Areas where Miami seniors’ mean scores were lower than seniors at comparison institutions mostly relate to the lack of diversity among the student population and include:

• Having discussions with people of a different race or ethnicity
• Having discussions with people of a different socioeconomic status.

For the 2023-2024 academic year, syllabi for courses approved for the Miami Plan Social Science and Intercultural Consciousness categories were collected.  A total of twenty course syllabi in the Social Science category were collected, and a total of 40 course syllabi for Intercultural Consciousness were collected for review.

These syllabi will be reviewed using a normed rubric. Data will be summarized in the summer 2024 term and shared with Liberal Education Council in fall 2024 for interpretation and analysis. Analysis and strategies for improvement will be included in the 2024 report.

Thirty capstone courses offered during the fall 2023 semester were randomly selected for assessment purposes.  Faculty were asked to submit the assignment prompt for the capstone project and to provide a random sample of artifacts (the first, third and fifth students listed on the class roster). 

Identifying information on the assignment prompts and papers have been redacted, and projects will be scored by members of the Assessment Fellows Committee with a rubric this summer.  The Committee revised the rubric twice after undergoing several norming sessions. 

Although the overall percentage of proposals eventually approved is strong, the percentage of proposals approved on the first submission seems low.

Liberal Education Council has noted the most common issues encountered in the proposal review, in priority order:

• Syllabus not updated
• Assignments not provided for all Miami Plan Student Learning Outcomes
• Assignments and/or syllabus is not well aligned with the Miami Plan Pillars or SLOs

LEC will review these data and develop strategies for improvement during the next reporting cycle.

The National Survey of Student Engagement is a standardized survey tool used to assess the extent to which students engage in educational practices associated with high levels of learning and development. At Miami, NSSE is administered every other spring to first-year and senior students.  Data from Miami students is compared to data from students at institutions from two comparison groups: (1) institutions with the same Carnegie classification as Miami; and (2) public institutions in the Great Lakes region of the U.S.

The data summarized below comes from the 2023 National Survey of Student Engagement (NSSE) results which means that the data were gathered before the new Miami Plan was implemented.  We will use these data as a baseline for future comparison and analysis.  The means reported are for graduating seniors at Miami versus the comparison groups.

Key Findings

If we organize the outcomes by pillar, Miami students’ perceptions of their experiences are statistically higher in the following areas:

Collaboration & Innovation

• Asking another student to help you understand course material
• Explaining course material to one or more students
• Preparing for exams by discussing or working through course material with other students
• Working with other students on course projects or assignments
• Working with a faculty member on activities other than coursework (committees, student groups, etc.)
• Discussing course topics, ideas, or concepts with a faculty member outside of class
• Discussed your academic performance with a faculty member

Communication & Expression

• Giving course presentation
• Writing papers up to five pages

Critical & Integrative Thinking

• Evaluating a point of view, decision or information source
• Forming a new idea or understanding from various pieces of information
• Used numerical information to examine a real-world problem or issue
• Evaluated what others have concluded from numerical information

Areas where Miami students’ perceptions of learning were lower than students at comparison institutions include:

Civic Mindedness and Social Engagement

• Developing or clarifying a personal code of values or ethics

Global Citizenship

• Understanding people of other backgrounds (economic, racial/ethnic, political, religious, nationality, etc.)

General

• Overall level of challenge

Beginning in fall 2023 and each semester thereafter, random samples of syllabi from one component of the plan were collected and reviewed, using a rubric (see Appendix B). Members of the University Assessment Fellows Committee (a faculty-based committee with representatives from all academic divisions) conducted the review.

The syllabus assessment is designed to ensure that existing courses are continuing to:

1. Explain the liberal education pillars and appropriate learning outcomes on the course syllabus
2. Appropriate learning outcomes in a quality manner via assignments and other forms of assessment.

For the 2023-2024 academic year, syllabi for courses approved for the Miami Plan Social Science and Intercultural Consciousness categories were collected.  Syllabi for a total of 20 social science courses and 34 courses meeting the Intercultural Consciousness requirement were evaluated.

The rubric allows a score of 1 (below expectations) to 4 (exceeds expectations) for the following categories:

1. Explanation of the Miami Plan in the Syllabus
2. Promotion of Civic Mindedness & Social Engagement Pillar in the Course
3. Promotion of the Critical & Integrative Thinking Pillar in the Course
4. Promotion of the Communication & Expression Pillar in the Course
5. Promotion of the Collaboration & Innovation Pillar in the Course
6. Promotion of Requirement-Specific Outcomes in the Course. 

The tables below summarize the average scores for each component reviewed:

Key Findings

• Almost all syllabi included some mention of the Miami Plan, its Pillars or other requirements. Only two out of a total of 51 syllabi did not mention the Miami Plan.
• Of those 49 that mentioned the Miami Plan, 47 focused on the four Pillars by listing or describing them. Forty-five of those 49 that mentioned the Miami Plan also included some mention of the particular Miami Plan requirement the course meets.
• Only two syllabi gave a more detailed explanation of the purpose and value of the Miami Plan and its outcomes.
• While most syllabi provided an explanation of the Pillars and/or requirement-specific outcomes, the explanation’s connection to actual course activities and assignments was uneven.
• Almost all syllabi equated small group work in class with the Collaboration & Innovation Pillar.
• The descriptions of the ways that courses met the Civic Mindedness and Social Engagement Pillar tended to be the most vague of all of the pillar descriptions.

Twenty-nine capstone courses (out of a possible 125 capstone courses) were included in this study which accounts for 23.2%.  Faculty were asked to provide an assignment prompt that best aligned with the capstone learning outcomes as well as the artifacts from the first, fourth, seventh and tenth student listed on the course roster. 

A rubric (see Appendix) focused on the five student learning outcomes (listed below) was used to score the artifacts.

• Define real-world problem or issue.
• Explore and propose solutions or hypotheses.
• Apply knowledge gained throughout the liberal education experience.
• Employ multiple sources.
• Communicate ideas clearly and in such a way that they can be understood by a wider audience.

Key Findings

It is important to note that the new Miami Plan was implemented in fall 2023.  Thus, not all capstone courses assessed have been approved for the new plan, and consequently, some assignments were not well aligned with the student learning outcomes for the new Miami Plan capstone.

The assessment indicates that most capstone courses include a major individual or group project that focuses on a significant problem, issue or question and requires students to generate a solution or hypothesis informed by outside sources.  It also demonstrates that overall, students are able to communicate ideas clearly.

The assessment also revealed that:

• Many assignments have vague, brief and open-ended prompts. For example, some assignment prompts were only a few sentences, broadly inviting students to research something related to the course topic.  While offering students some flexibility so that they can tailor the project to their interests is beneficial, this open-ended approach also has limitations.  Not only does it invite academic dishonesty (particularly in light of Artificial Intelligence tools), but it does not ensure that the students are focusing on the capstone learning outcomes or reflecting on their liberal education experience.  Moreover, it seems to lead to inconsistency in project quality.
• Interestingly, we noted that many assignment prompts are too narrow, in the sense that they only require students to leverage sources and knowledge in the major field of study. Thus, they are more appropriate to be incorporated into the culminating course in the major rather than the major project in a liberal education capstone.
• Almost all projects require some outside sources, but students do not always provide a works cited list or cite sources appropriately. When cited, most sources come from the same discipline or field of study.
• Projects in some courses were completed by groups of students. While there are examples where this model led to thoughtful and more sophisticated projects than could have been done individually, others seemed to be too simplistic to warrant multiple authors.  A few of these group projects read more like a loosely cobbled together series of smaller projects that were not coherently woven together. 

Recommendations

• Continue to work with faculty and departments to better understand the difference between liberal education capstones and courses that are solely designed to culminate the major. Provide exemplar courses and assignments.
• Continue to offer workshops (perhaps in partnership with the Howe Center for Writing Excellence) on developing liberal education capstone assignments that are well aligned with the Miami Plan Senior Capstone student learning outcomes and that invite interdisciplinary inquiry.
• Work with the Howe Center for Writing Excellence, University Libraries and Office of Academic Integrity to offer resources on teaching students to identify and use information sources responsibly and appropriately.