Collecting Assessment Data

A rubric is a guide that describes the criteria that can be used to provide feedback and/or score or grade an assignment. It identifies the traits that are important and describes the levels of evidence of performance (e.g., unacceptable to excellent) within each of the traits.

Rubrics can:

• Clarify for student the expectations for an assignment.
• Reduce bias and improve consistency in scoring.
• Communicate to students both strengths and areas for improvement of the work.
• Assist faculty in determining which (student) skills are well-developed and which skills require improvement.

When developing a rubric:

• Determine what the rubric will encompass. Think about the following questions:
  • What is its purpose?
  • What is the assigned task, and what are the dimensions of its product?
  • What kind of feedback would benefit student learning and characterizing the quality of the produt?
  • How might levels of student performance be differentiated?
• Define the criteria. What knowledge/skills are required to complete this assignment? What evidence would demonstrate various levels of meeting the goals of the assignment. Review the learning objectives for the course (and program, if appropriate). If possible, review previous student work from the task or assignment to help inform this work. Be cognizant of the number of criteria that you have. Too many will make an unwieldy rubric. Aim to have criteria that is distinct, measurable and/or observable, and essential to the item that is being assessed.
• Design the rating scale. Determine how many levels of achievement you want to assess (typically 3-5 are used). Think about whether you will use numbers or descriptive labels for your different levels of achievement, and what those labels will be.
• For each criterion, develop descriptions for each level of the rating scale. Create statements that exemplify performance at that particular level of the criterion. Use clear, unambiguous language consistently across all levels. This provides clear expectations for your students, and other reviewers (if you will assess using multiple reviewers).
• Pilot the rubric. Ask for others to review your rubric. Ensure that the language and levels accurately reflect your intentions for the rubric. Revise as necessary.
• If you will be using the rubric with multiple reviewers, you should norm the rubric before use. When reviewers to come together before the actual assessment, they can finalize the language and composition of the rubric as well as calibrate their judements to align with the rubric and one another. Norming typically results in more consistent results across reviewers. When norming, typical steps include:
  1. The facilitator of norming (usually the person who developed the rubric), initially talks through the rubric as applied to several examples of student work.
  2. Reviewers are asked to independently score a few student examples.
  3. Raters are brought together to discuss their scoring processes and look for patterns in consistency and inconsistency.
  4. The group reconciles inconsistencies in scoring. Consensus is acceptable: not all reviewers have to agree.
  5. Repeat c-d with a new set of student examples.
  6. Consider tweaking rubric language for pervasive inconsistencies.
  7. As necessary, repeat steps c-d until score is consistent.

Surveys

Surveys, when designed and used appropriately, can provide valuable information about student learning and development.

Miami University's Office of Institutional Research and Effectiveness (OIRE) conducts multiple surveys of faculty, staff, and students each year, including national surveys that allow comparisons with benchmark institutions. This page provides information on how to using survey data for assessment purposes and the types of survey data that are already available to Miami faculty and staff.

Why Use Surveys?

Multiple types of data can be used to assess students’ learning and development. Surveys can provide an indirect measure of student learning and development (i.e., a measure that indirectly suggests a student has achieved the outcome). While direct measures are always preferred, they are not always practical.

After you identify the type of data that you need, determine whether you need to design your own survey or whether the OIRE has survey data that you can use.

Many surveys are designed to go hand-in-hand with other surveys in order to provide multiple perspectives (e.g., faculty and student perspectives). For some of these surveys, OIRE has data stretching back as far as 1970!

Online Survey Tools

Miami also subscribes to Qualtrics, a survey tool which provides additional features such as e-mailing integration and analytic tools. More information on Qualtrics is available from IT Services.

Midcourse Evaluations

The CTE has a comprehensive suite of resources and services to support instructors who wish to employ formative assessments of teaching and learning using midcourse evaluation.

End-of-Course Evaluations

It is important to consider the length, quality, and redundancy of each course evaluation, in order to reduce student fatigue and increase the quality of the information that students provide to instructors and the University about their course experiences.

Remember that full-time students will often fill out five course evaluations. It is recommended that departments and instructors each add no more than five items to prevent surveys from becoming overly lengthy.

Here are a few useful practices:

• Assess Student Perceptions of Their Learning - We recommend that departments (and instructors) consider asking course evaluations about the degree to which students perceive that this course has helped them achieve either program learning outcomes or course learning outcomes. For programs, this can serve as a supplemental indirect assessment measure that can be reported in addition to the direct assessment you do of student work.
• Evaluate Unique Forms of Pedagogy (Including Online Instruction) - Many disciplines have their own "signature pedagogy," the way that practitioners think the discipline is best taught (e.g., science labs or art studios or small discussion sections). Department or instructor evaluation questions are an excellent opportunity to ask the questions that are unique to your form of pedagogy.
• Evaluate Dimensions of Teaching Not Captured by Other Questions - You may also find, when reflecting on the University and divisional questions, that they don't capture some dimension(s) of teaching that is/are important to your department or your specific class. Or perhaps you want to know whether students found something that you changed about your class to be useful.
• Follow General Guidelines for Effective Survey Questions - As scholars, we all know that how you frame the question is fundamental to the answer you get. The same thing is true of course evaluation questions. If you are writing your own questions, you do not have to re-invent the wheel of how to construct survey questions. Examples can be found at nces.ed.gov [search for Best Practices].

For more information on the process, please visit the Provost Course Evaluation FAQ page.