Analyzing, Interpreting, Communicating, and Acting On Assessment Results

The goal of assessment is information-based decision making. To put it another way, the end of assessment is action. Assessment helps the organization determine how well it is achieving its goals and suggests effective steps for improvement.
(Walvoord, 2010, p. 4)

Reporting Assessment Results

Suskie (2009) lists five basic ways to summarize assessment results:

  • tallies
  • percentages
  • averaging results into an overall score
  • averaging results into sub-scores
  • qualitative summaries (grouped listings, themes, examples)

Look at differences

  • between groups
  • over time
  • with peers

Ensuring the Quality of Your Data and the Utility of Your Analyses

Quantitative Analyses

  • reliability
  • validity
  • comparing participants/sample/population
  • sampling error
    Sample Size Sampling Error
    196 7%
    264 6%
    384 5%
    600 4%
    1067 3%
    2401 2%
    9604 1%
  • item difficulty, item discrimination

Qualitative Analyses

  • triangulation
  • peer debriefing
  • member check
  • bracketing

Sharing Assessment Results

Suskie suggests the following guidelines for sharing your assessment results

  • Use good teaching practices to share assessment results.
  • The briefer the assessment report is, the more likely it will be used.
  • Three kinds of information are most important to share:
    • how you and your colleagues define a successful student
    • whether you are satisfied with your evidence of student success
    • what you are doing about unsatisfactory results
  • Tables, charts, and other visuals may be more effective than traditional written reports.

Honest, Balanced, Fair, and Useful Reporting of Assessment Results

Suskie recommends doing the following to ensure appropriate reporting:

  • Share only aggregated results.
  • Present results completely, fairly, and objectively.
  • Provide appropriate attribution.
  • Document the author, office, and date of assessment reports.
  • Offer to make additional information available.

What will audiences for assessment results care most about?

  • matters they can do something about
  • interesting and unanticipated findings
  • meaningful differences

Venues for Sharing Assessment Results

Suskie gives the following examples of where to share your assessment results:

  • web sites
  • emails
  • newsletters
  • alumni magazines
  • departmental memos
  • brochures
  • presentations
  • posters or banners

When People Feel Threatened by Assessment Results

If you find someone feels threatened by your assessment results, Suskie suggests that you:

  • Consult with those who may feel threatened.
  • Balance negatives with positives.
  • Be gentle and sensitive.
  • Provide corroborating information.
  • Document the quality of your assessment methods.
  • Acknowledge possible limitations in your assessment methods.
  • Help identify possible solutions.

Ways to Make Your Results Have the Most Impact

Suskie suggests that you do the following to ensure that your assessment results have the desired impact:

  • Make sure everything you include tells an important, interesting part of your story.
  • Use an engaging, meaningful title and headings.
  • Open with something intriguing.
  • Cascade from major points to details.
  • Provide a context for results.
  • Offer informed commentary.
  • Keep it short.

Using Assessment Results Effectively and Appropriately

You will want to use your assessment results effectively and appropriately. In order to do so, Suskie suggests that you:

  • Focus on important learning goals.
  • Assess processes as well as outcomes.
  • Involve those with a stake in the results in designing, carrying out, and discussing assessments.
  • Communicate findings widely and openly.
  • Discourage others from making inappropriate interpretations.
  • Don’t hold people accountable for things they cannot control.
  • Don’t let assessment results alone dictate decisions.
  • Use multiple sources of information when making decisions.
  • Keep people informed on how assessment results have affected decisions.

When Assessment Results Are Good

For those times when your assessment results are good, Suskie highly recommends that you:

  • Celebrate!
  • Reward!
  • Share!
  • Keep going!

When Assessment Results Are Bad

Suskie suggests that you look at everything carefully.

  • Do you have the right learning goals?
  • Do you have too many learning goals?
  • Take a hard look at your courses:
    • content and requirements
    • sequencing and prerequisites
    • admissions criteria
    • placement criteria
    • advising
    • tutoring
    • teaching methods
    • co-curricular activities
  • Do you need to improve your assessment methods?
  • Sometimes it really is the students’ fault.
  • Keep going.

How We Can Make Better Meaning of Our Assessment Results

Frame student learning within Astin’s (1993) Inputs-Environments-Outcomes assessment model that:

  • Is predicated on the assumption that the principal means by which assessment can be used to improve educational practice is by enlightening the educator about the comparable effectiveness of different educational policies and practices.
  • Highlights the role of student backgrounds (inputs, e.g., demographics, high school grades, test scores, values, attitudes, behaviors) and student experiences (environments, e.g., courses taken, teaching methods, employment, interactions with other students, interactions with faculty members, use of programs and services, participation in various activities) on student learning (outcomes).

Moving From Assessment Results to Action

You have completed your assessment. To ensure that you appropriately use your findings, Maki (2004) and Walvoord (2010) suggest that you:

  • Determine what is most important in the results. In addition to discussion among themselves, faculty members can consult program accreditation bodies, alumni, employers, faculty members at other institutions, librarians, writing specialists, and student affairs staff members.
  • Focus on the areas that show the greatest weaknesses.
  • Determine what is feasible now and what might be addressed in the future. Consider what changes can be made within the department and what changes involve others. Investigate resources and available assistance.
  • Keep good notes, both for your own follow-up and for reports that you might have to submit.

Case Studies of Departmental Use of Assessment Activities

The following are based upon Walvoord and the experiences of staff members in the Office of Institutional Research and Effectiveness.

  • A two-hour, annual faculty meeting is held in a department of political science where each faculty member who teaches seniors described strengths and weaknesses in research papers. Notes were kept. A vote was taken on one follow-up action to mitigate a weakness. The area identified was students’ lack of ability to construct coherent research questions. Members of the department curriculum committee followed up to investigate in which earlier courses this skill was covered. They also administered a short survey to seniors that asked which courses helped them to construct research questions and what the faculty could do to help students to improve this skill. It was decided that two junior-level courses would cover constructing research questions in greater depth. A year later a similar meeting was held, and it was determined that students’ ability to construct research questions had improved substantially. The faculty members then decided to follow up on another identified weakness.
  • The faculty member who teaches the senior capstone biological research course for biology majors developed a simple scoring system or rubric to evaluate students’ research papers. He shared the results at a department meeting. The results showed that designing experiments and controlling variables were the areas of greatest weakness. A small committee examined the curriculum and held a student focus group. They decided to institute a tutoring program that reinforces these areas.
  • Faculty members in health sciences noticed that several students were providing incorrect answers to questions in the final exam of a first-year survey course that required interpretation of written material. They began to suspect that there was a relationship between reading skills and performance in the exam. A student survey and discussions with academic advisors confirmed this relationship. As a result the faculty emphasized the importance of reading skills in their first-year survey course, they strengthened a requirement that students who scored below a certain cutoff in a diagnostic reading test enroll in the institution’s reading and study skills course, they asked experienced students to discuss with first-year students the importance of study time and effective study methods, and they provided professional development for faculty members in the department to recognize student reading problems early in their courses and to refer the students with such problems to the campus study skills center.
  • Music faculty members reviewed student portfolios and determined that there was wide variation in entering students’ knowledge of music theory. They developed a music theory diagnostic test that students took as part of their application process and created a first-semester basic music theory course for students who scored poorly on the test.
  • Faculty members offering online MBA courses noticed that some students were not turning in assignments or using the online discussion board. They emailed students a link to a short survey and also asked the institution’s IT staff to collect some statistics about students’ use of tutorials that were provided to explain how to use the features of Canvas. It turns out that some students did not understand the basics of how to navigate Canvas, and they did not complete the tutorials. A new policy was instituted that students would be de-registered from their courses unless a usage report confirmed that they had completed the tutorials and scored at least 80% on a proficiency exam.

While the level of maturity of the assessment efforts outlined in these case studies varied, the commonality is that all of the departments took some action. They did not wait years until enormous amounts of data were collected. They were better off concentrating on a few simple concerns that matter than were departments that had large amounts of information collected but did not discuss the implications and did not make any changes.

Frequently-Asked Questions

How can we ensure that results of assessment activities are used?

Faculty members, staff, and administrators who are involved in assessment planning will be more likely to use the results. Findings should be shared with department faculty members in a written report that could be used to generate discussions at faculty meetings or annual retreats. (What does the report tell you about your course or program? What possible changes are indicated? How could our department use these results to best advantage?) Additional reports or presentations can be prepared for different audiences, depending on their assessment needs.

In reporting results, do we report actual data or interpretation of the data?

This depends on the audience. Some constituents may request that the department report actual findings while others will prefer a summary that includes implications of assessment results.

Do we report everything?

The department will usually decide. It may find both expected and unexpected results to be useful. For example, an alumni survey may reveal that not only are graduates getting jobs, but the majority of graduates are securing jobs in one particular type of firm or company. This finding could be used to determine what job skills department graduates have that are making them so marketable with these companies. This information, while useful to the department, will probably not be of interest to accreditation teams or college curriculum committees.

How long should we keep assessment results?

It is advisable to keep assessment project results for at least 3 to 5 years (for ease in knowing what has been done before, for learning from past mistakes, or for doing longitudinal studies). Departments find it useful to do longitudinal studies of assessment projects to determine trends or patterns over time. Results of assessment activities may change with time along with the job market, the economy, technology, and even the department itself. Many assessment evaluations are on cycles and will necessitate the use of results from past years. (State commission or accreditation reviews may be every 5 to 10 years. Similarly, some departments do surveys annually, while others prefer to do them every 5 years.)

References

Astin, A.W. (1993). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. Phoenix, AZ: American Council on Education and the Oryx Press.

Maki, P. L. (2004). Assessment for learning: Building a sustainable commitment across the institution. Sterling, VA: American Association of Higher Education and Stylus Publishing.

Suskie, L. A. (2009). Assessing student learning: A common sense guide (2nd ed.). San Francisco: Jossey-Bass.

Walvoord, B. E. (2010). Assessment clear and simple: A practical guide for institutions, departments, and general education (2nd ed.). San Francisco: Jossey-Bass.