One Miami: Departments rally to overcome crisis

by Elizabeth Jenike, IT Services

Nothing happens in a vacuum, especially not technology infrastructure. Teamwork is necessary to ensure all of Miami’s systems are operational, and that often means that departments work together to create a successful environment.

At the end of March, IT Services, Finance and Business Services IT (FBS-IT), and Miami’s Physical Facilities Department (PFD) had a unique opportunity to experience this need firsthand.

Best-laid plans of mice and men

People standing around central command waiting to be sent to dorms

Spring break is traditionally a time that IT Services and other departments on campus conduct maintenance activities and upgrades—since few students are on campus during this time, it’s a good opportunity to test out new tools and get ducks in a row before the campus is full again. This past academic semester’s spring break brought a similar situation: There were a lot of changes and upgrades to be made, and one of those upgrades involved CS Gold, the system that manages door locks in the dorm rooms.

During the weekend of March 30, the end of spring break, this upgrade on the door lock system began. It started as a high-level update of the system itself that shouldn’t have included any upgrades to physical door locks. However, it became apparent that a firmware update setting had been missed in the implementation phase—meaning a setting on the actual locks in academic buildings and dorms was being changed, rendering the locks into a nonfunctioning state.

Example of a door lock

Once the technicians realized what was happening, they stopped the upgrade—but there was no way to know which rooms had already been impacted by the errant firmware update and which hadn’t. The result? They would need to manually check each and every door lock to ensure they were operational . . . nearly 5,500 doors.

And almost immediately, a mobilization effort got underway.

Teamwork makes the dream work

“On Saturday and Sunday, there was a small group of folks trying to work [on the issue],” said Sarah Persinger, senior technology product manager for FBS-IT. “Then we moved to all hands on deck on Monday and Tuesday.” Students were moving back in to their dorms, so door access needed to work.

“The call went out,” Persinger said. “And everyone answered.”

Staff members from PFD, IT Services, and FBS-IT convened on Monday morning to assess the situation, split the work, and head out to see to the 5,500 door locks that needed attention. They spent up to five minutes on each lock, checking the hardware and making sure via mobile application that the door was connected to the right CS Gold components. All told, around 30 people volunteered their Monday and Tuesday to check the locks not once, but twice.

Terrence Ruff stands in front of people pointing

Even Brad Grimm, the assistant vice president and chief information officer of FBS-IT, got in on the action.

“The response was immediate,” he said. “Everyone was offering help. I’ve been doing this for 20 years, and I’ve never seen a crisis responded to like this in such a positive way.”

“There were some late nights,” said Persinger, who along with Angeline Smith, administrator of campus cards for FBS-IT, manned the command center at Cook Place. “But no hard feelings. We ordered lunch and dinner in and made sure everyone was fed. The teamwork and collaboration between [PFD, IT Services, and FBS-IT] was incredible. The call went out, and we all met at Cook Place. People responded—no questions asked.”

Both Smith and Persinger were essential to the successful response to the crisis. While Persinger coordinated the deployment of over 30 people across more than 40 residence halls, Smith was responsible for training all the volunteers, as well as addressing any and all technical questions and issues that the various teams ran across. The successful partnership between Persinger and Smith was a key factor in the overall successful and smooth response to the crisis.

“They were both amazing during the entire event, and, without their partnership and expertise, I doubt our response would have been as successful,” Grimm said.

Clint Knipper, technology support coordinator in IT Services, noted that along with checking several hundred doors that week, he was able to help one student with an entirely unrelated issue—one that he was uniquely qualified to address, no less.

“She couldn’t connect to the wireless—an opportunity to provide in-person service!” he laughed.

The general feeling during the event was not the “fault” of any one party—the main issue was that students needed to get in the dorms, and it was everyone’s job to see the problem resolved.

“The general feeling was that the students were impacted,” said Dennis Schwind, senior security engineer in IT Services, “and we jumped on it because we wanted to lessen that impact. At the end of the day, that’s what it’s about—the student experience. It didn’t matter how we got there—we had an effective fix, so we did it.”