If you walked into Scotiabank Arena’s training facility on a Monday night, you’d see a room full of reporters, developers, and representatives from Amazon Web Services (AWS) and Maple Leaf Sports and Entertainment (MLSE).
What you can’t yet see are 3D renderings of mannequins representing the Toronto Raptors and Milwaukee Bucks players recreating their most recent NBA games. Unless you’re wearing an augmented reality (AR) or virtual reality (VR) headset.
With a headset, you can immerse yourself directly in the virtual action. Players come to life in front of you, giving you the chance to follow them from baseline to baseline, or just sit on the sidelines – giving you the best view of defensive meltdowns and offensive explosions, no matter how you’re feeling.
“It’s a glimpse into the future,” said Humza Teherany, MLSE’s chief technology and digital officer.
MLSE Digital Labs and AWS have partnered to launch a new research and development program, SpaceX. Its first move is to introduce augmented reality (AR) and virtual reality (VR) to change the way we consume professional sports. This is both for the fan experience and to help improve team performance, as players and coaches can relive moments that have helped them train.
As part of a Jan. 22 launch, MLSE and AWS demonstrated how AR and VR headsets can be used to display multiple variations of the data they collect. It allows those wearing the headset to immerse themselves directly in the action, or even receive real-time stats as part of their view while watching a game — like during Monday night’s game between the Toronto Maple Leafs and the New York Islanders.
“Frankly, it’s not done anywhere else in the world,” Teherany said.
“The great thing about this project is building the future of sport from Toronto to the rest of the world, allowing us to use these great things in MLSE. Hopefully it can serve as a model for other sports.”
How the technology works
The “immersive basketball experience” uses a mix of optically tracked body pose data (such as the position of each joint and limb), 3D models, and a video game development engine. The result is a life-size relive of the game, as the player is rendered in 3D around you.
For the NBA experience, MLSE and AWS are utilizing game cameras in all venues to collect the necessary data. The same experience can be used for NHL games, and with other professional sports (such as Formula 1) making progress in obtaining biomechanical data, we can expect this experience to be applicable to all sports.
For the NHL, the experience may be even more realistic, as the league uses “NHL Edge.” To collect data, the NHL uses infrared technology embedded in pucks and players’ jerseys — information that is relayed to — in this case — an AWS outpost at Scotiabank Arena. The technology allows tracking of every moving part of a game, making way for an “extended reality stat overlay” feature on AR or VR headsets.
While watching an NHL game, viewers can access overlays that display statistics for any team, such as ball possession, speed, and distance traveled. Using the laser function as part of the remote that comes with the headset, viewers can even individually select a player (such as Auston Matthews) to focus on for specific stats.
“You can enjoy the game while seeing analytics and data live in one place,” Teherany said, noting that it could be used by avid hockey fans or those who are learning the game.
Future use: Arena to everyday life
For now, the cost of several thousand dollars and the size of the headphones remain challenges in bringing them to the masses. Teherany said that when an innovation like Apple’s augmented reality glasses comes out, it will change the market, and we can expect these features to be implemented in the size of everyday glasses.
According to Eric Gales, AWS Canada country manager, the market has changed significantly in recent years before you needed a lot of devices to deliver an AR or VR experience. Now, thanks to all the information they’ve gathered in the cloud, they’re ready to deliver that experience when there’s an accessible option to make it mainstream.
An “extended reality stats overlay” is one experience the MLSE hopes to offer fans inside the arena. This will help maximize the viewing experience, especially for those sitting high up in the crowd. A similar statistical experience – without the headset element – is also displayed on a table-sized panel at the Scotiabank Arena – which could become a fixture within its luxury suite.
Outside of the arena, the question is how to license these games for AR and VR — since the broadcast rights are owned by Rogers and Bell.
For the immersive basketball experience of professional NBA games, members of MLSE and AWS wanted to extend its capabilities to people’s homes for everyday use. Given the aforementioned cost and size issues with the headset, after talking to multiple developers, a realistic timeline is 5-10 years.
Christian Magsisi, vice president of venues and digital technology at MLSE, explained that with life-sized players, you need to have bigger, better-fitting spaces. For example, if you wanted to stand next to a life-sized Kawhi Leonard during his infamous game-winning shot in 2019, you’d need a large space like a gym or playing field.
But with the capabilities of AR and VR, users will be able to display experiences on smaller surfaces, such as a tabletop, helping them see the action unfold right in front of them.
create competitive advantage
One of the benefits of the VR and AR experience is for the Professional Sports Organization section of MLSE. Magsisi says it allows players and coaches to recreate key moments in their game and learn from them.
“It allows us to create a competitive advantage,” Magsisi said, noting that members of the Raptors and Maple Leafs have already trialled the technology.
Instead of 10 real-life players recreating a specific game on an NBA court, you can use AR or VR to bring that moment to life. Players and coaches can also watch or re-watch games using the headset technology, which shows them key stats as the game progresses, giving them a chance to see exactly what is contributing to their team’s play.
Gales also pointed out that this may affect their understanding of injuries and how to avoid them to improve player safety. This type of data and innovative research has been the focus of the NFL’s partnership with AWS since 2019.
get people involved
In addition to enhancing the viewing experience and helping the team gain a competitive advantage, SpaceX is also grounded in the concept of engaging the community. They want fans to submit ideas that help guide them into the future, and they want them to try out AR and VR experiences.
“We’ve been watching sports the same way for a long time. Putting people on virtual reality goggles is a huge change. So we need to get people used to it. So even though it might come to market sooner, I think it’s going to take some The time has come for people to actually adopt.”
Those interested can start registering on spacex.io today to test the technology for themselves.
More from Yahoo Sports