Virtual Production Meets Machine Learning
SIRT develops synthetic datasets to increase accuracy of privacy-first detection framework.
As a large generational demographic continues to age and enter retirement, it's our responsibility as a society to ensure they can enjoy their retirement years safely. This leads to the question, with such a large cohort, how can we keep up with the demand?
SPXTRM Health Inc. (the capitalized part of the name is pronounced 'spectrum') collaborated with SIRT to develop a streamlined recording system for action recognition. Actions such as falling, elder abuse, and aggressive behaviours—known as human action recognition—is a major challenge to systematically collect and label by observing real-world scenarios.
"SPXTRM AI was formed to provide secure monitoring tools for high-risk patients and elders." says Jay Couse, CEO and Co-Founder of SPXTRM AI. "When we were designing SPXTRM AI you have to give it representative datasets. We tried using public datasets. They were not robust enough. Next, we tried simulating these datasets, setting up multiple cameras in a room, using experts in the field, simulating aggressive events and falls—but we weren't getting the kind of quality we were expecting. This brought us to SIRT."
Funded through an NSERC Engage grant, this project focused on the creation of intelligent monitoring systems to improve care and respect the privacy of seniors. With virtual production tools, SIRT captured performances and generated thousands of viewing angles to increase the learning model's accuracy.
“There was an idea to use film production tools in order to not only generate, but multiply, captured actions. This way we could capture an actor falling down once, but generate 10, 20 or 100 videos of that same action from numerous vantage points within a virtual environment.” - Jason Hunter, Production Lead
Two Sheridan College Co-op students, Laura Stewart and Parker Christie, worked with Jason on this project as Junior Programmers. With clear objectives from SPXTRM, there was still room to innovate. Parker says that using virtual production tools helped the project team create two-person animations safely. "As we started developing the system we needed to act out [some violent] animations. We came up with an idea where we placed a model in the scene that I could see in front of me and then recorded the action."
While SIRT is known for research projects in the screen industries space, this project called for a new approach with motion capture and virtual production that may not have been obvious at first. Jason recalls, "It was very cool to work with SPXTRM to determine the ideal fidelity level of the deep learning system. In film we're so used to delivering the highest quality possible, but with the neural network it was a balance of making the movements realistic enough that the system would recognize the figures as human, while not being too computationally expensive."
Reflecting on the project, Jay expresses that this project led to a critical discovery. "We realized that we didn’t need [the human model's] hair, clothes, or the furniture in the room. [With this idea] SIRT came back with a model where something that took an hour [to process by the computer] now took one second. Something that took hundreds of thousands of dollars worth of equipment to set up and run, now could be done for a few thousand dollars."
“Not only did we prove the hypothesis that we could use synthetic character generation, we could actually do it at a cost and speed that would allow us to meet our objectives. We’re now at 97%+ accuracy, and a lot of it has to do with the addition of these synthetic datasets.” -
Jay Couse, CEO and Founder, SPXTRM AI
Jay is adamant that SPXTRM will be tested by their clients with further extensions of their framework. "We very much expect that we'll be back at SIRT's door to talk about new challenges as we move forward into protecting high-risk patients and elders in the healthcare setting.
