Motion Capture system for live performances in real-time environments

Project Title: Motion Capture system for live performances in real-time environments

Project Lead's Name: Matthew Board

Project Lead's Email: boardmj@MiamiOH.edu

Project Lead's Phone: 413-386-4698

Project Lead's Division: CCA

Primary Department: Art and IMS

Other Team Members and their emails:

  • Murali Pandini, paranam@MiamiOH.edu
  • Bob deSchutter, deschub@MiamiOH.edu
  • Rob Robbins, robbinrv@MiamiOH.edu
  • Eric Hodgson, hodgsoep@MiamiOH.edu
  • Phil Alexander, alexanp3@MiamiOH.edu

List Departments Benefiting or Affected by this proposal:

  • Art
  • IMS
  • Theater
  • Dance

Estimated Number of Under-Graduate students affected per year (should be number who will actually use solution, not just who is it available to): 1,800

Estimated Number of Graduate students affected per year (should be number who will actually use solution, not just who is it available to): 15

Describe the problem you are attempting to solve and your approach for solving that problem: In an effort to collaborate with the University and the College of Creative Arts in a more cohesive way, we would like to add a full-body motion capture system to our real-time performance environment. This will allow students in dance, theatre, and other movement disciplines to capture real-time performances in a real-time, virtual environment.

A recent development in motion capture is that it is not longer has to be used as pre-recorded data. It can be used at the same time as the performer is executing the movement. This is done by using a "live link" software system and a game engine. This is what Is called, "Real-Time Performance Capture."

The system being built is basically the same as what is being used at many game (Epic Games, Fortnite) and film studios (LucasArts Star Wars).

Real-time technology is not only used for making games. It is used for a variety of applications including Architectural Visualization, Film, Virtual Reality Simulation, Fine Art and Performing Arts, Automotive Visualization, government and military applications and an array of training simulation for various industries. With the enthusiastic emphasis on Virtual Reality, this is a wonderful time for 3D artists.

You can't really have a virtual environment without them. Because of this, the demand for 3D artists who can work in VR is increasing.

Competitive Analysis

Competition for the best students is a necessary component of recruitment. Our largest competitor in the state is Ohio State University. They have a facility called ACCAD, The Advanced Computing Center for Art and Design. This is the description from their website:

"The Advanced Computing Center for the Arts and Design (ACCAD) is a collaborative think space, a place to make, create, imagine and above all connect. We conduct research centered on the use and integration of emerging arts technologies. ACCAD has become internationally recognized as one of the original and leading centers of its kind, distinguished by the transdiscipllnary approach to research and teaching which is so central to its Identity. Located in the newly renovated Sullivant Hall at the heart of The Ohio State University's Arts District, ACCAD is a creative hub for scholars and practitioners of digital arts and sciences.

ACCAD functions as an applied collaboratory for time-based digital media production, both in furthering the excellence of its faculty and graduate students in residence and cultivating its own innovative research agenda clustering around animation and interactive media. Our work unfolds in a generous physical space, complemented by specialized and flexible studios for animation, motion capture, interactive design, media production and mediated performance design. ACCAD's collaborative partnerships include a wide interdisciplinary range of campus disciplines and external relationships."

ACCAD is a convergence space for the arts. There are courses taught that include dancers and musicians. Even though this is an Art and Design lab, it is actually a hybrid space for creative activity and software development. Right now, all of our resources are scattered all over campus. I believe that if we had a building where all of this activity took place in one location, a less fragmented approach, there would be greater productivity and collaboration.

How would you describe the innovation and/or the significance of your project: Recently, the presence of the game industry in Ohio and the surrounding region have shown significant growth. Higher education in the area has been evolving to meet the needs of the growing industry regionally and worldwide. Miami has one of the most established game and simulation programs in the region. We need to stay competitive with schools on the regional and national stage. The University of Cincinnati currently has a "cluster initiative" that is supported by their current president to grow interdisciplinary programs of this nature. Northern Kentucky University also has plans to hire faculty very soon that work with 30 animation, virtual reality and real-time technologies. The Greater Cincinnati area has approximately seven independent game development companies already that work with this type of technology.

Recently, I was contacted by Sam Gebhardt of Virtual Ed Systems. They are an educational simulation firm based in Columbus. They recently received funding by the city of Cincinnati to open an office downtown. They desire a partnership with us and they are interested In our students because we teach the same technology we use and they work using the same critical and theoretical methodologies we use in our courses. The following is a list of more companies that use real-time, game development tools in the region that could partner with us:

  • Marxent Labs, located in Kettering, Ohio is an ARNR firm who works with game development tools to create consumer facing software products.
  • Kaleidoscope is an industrial design and innovation firm located in Cincinnati who uses game development tools for the purposes of visualization.
  • Llghtborne is a multimedia firm known for creating the Lumenocity multimedia show in Cincinnati. They are currently working on a VR branch of their business.
  • Procter and Gamble
  • Kroger Digital
  • Kinetic Vision
  • lmmersive Wisdom
  • Gylee Games
Working with real-time tools for the purpose of virtual reality is more than just a trend, it is real and meaningful innovation. The game program is not only on the forefront of innovation, we are poised to become regional and national leaders in the field. We want to continue our pursuit of this innovation to the benefit of our students. We are asking for hardware to help students achieve their career aspirations as VR developers, artists, animators and programmers.

Basically VR is about to be in the boom stage that iPhones and the App store were in around 2007.

Another measure of the innovation of this discipline is how many other industries have found ways to use real-time toolsets. The bibliography has links to evidence of this technology being used in the industries of Automobiles, Architecture, Film and the Government for various applications.

In addition to the computers for the lab, we are asking for a facial motion capture rig. Facial motion capture is a young technology that more game development businesses are adopting every day. The facial motion capture system is a way of capturing the facial expressions of an actor or actress and using fixed data points captured from the face. That data is then used for animating the face of a virtual 3D character.

Burning Glass Technologies, a labor market analytics firm, researched the demand for 3D graphics and real-time 30 skills in the workforce. Upon mining their database of nearly a billion historical and current job postings, they found that the demand for Unreal Engine skills is growing faster than any other segment  in real-time 3D, and those jobs are paying the highest salary premiums in all of 3D graphics.

How will you assess the success of the project: Assessment of this project is based on the following:

  • The curriculum developed to support the use of the Technology.
  • Interdisciplinary Live performances used to support the technology.
  • Captures used to support the Game Major capstone.

Financial Information

Total Amount Requested: $30,000

Is this a multi-year request: No

Please address how, if at all, this project aligns with University, Divisional, Departmental or Center strategic goals: Game development, 3D animation, Live Performance capture, and real-time technologies are intrinsically innovative, STEM-related fields that carry a direct need for creatives. Performance capture is very technical, however, it requires the use of actors and dancers to capture quality data for use in other applications. Live performance capture is a new field that has only been used for production in the last three years. Because of this, there is much to discover about how the hardware and software systems can be used for various applications.