The NFL on Netflix: how Netflix took on live TV and built the tech to make it work
Source: The Verge
Brandon Riegg has spent the better part of a decade trying to make live TV happen at Netflix. He joined the company in 2016, after stints at NBC, ABC, and VH1, where he’d worked on shows like Dancing with the Stars, The Voice, and America’s Got Talent. All those shows were the kind of unscripted reality fare he’d been hired to bring to Netflix, but they also incorporated things like live voting to make the whole thing feel more urgent and interactive. “I just felt like, if we’re really trying to be the preeminent entertainment service in the world,” Riegg tells me, “we should have all the tools at our disposal.”
So Riegg and Bela Bajaria, another longtime TV executive who joined Netflix around the same time and is now its chief content officer, began making the case around Netflix for why it should invest in the tech required to make live content work. Over and over, they got the same question: What do you want to do with it? And for years, Riegg says, they didn’t have a great answer. “I’d go, ‘Well, I don’t have something specific right now, but I want to be able to jump on things that require live capability if those things come up.’”
For years, that shrug of an answer didn’t work. But somewhere around two years ago, the energy shifted. “We were continuing to talk about how we wanted to have something for everyone,” he says, “and there’s a requirement of live for some programs. For us to do those things, for us to buy those things, we need to have that functionality.”
Netflix has spent the last two years slowly learning how to do live programming and live streaming. It started with a Chris Rock comedy special last March, which was a technical success and a cultural hit. A few weeks later, it did a live Love Is Blind reunion show, which was such a spectacular disaster that the reunion wound up being filmed and released later. Then there was a live feed of baby gorillas at the Cleveland Zoo, a strange golf event that teamed Formula 1 drivers up with PGA pros, the SAG Awards, a tennis exhibition, a roast of Tom Brady, and John Mulaney’s slightly unhinged late night show Everybody’s in LA.
All that was, in some ways, just practice. Because the real tests of Netflix’s live prowess came this fall. First, the Jake Paul / Mike Tyson fight in November, which the company says was watched by more than 65 million Netflix subscribers around the world — and had lots of technical difficulties and delays of its own. And next up, two NFL games on Christmas, complete with a Beyoncé halftime show. The NFL is the biggest and most valuable entertainment property in the US, and football is the most-watched thing on television by a mile. Netflix is many things, but it is also now a live TV network. And you don’t get to screw up football.
When Netflix struggled to keep up with the Paul / Tyson fight, a lot of viewers were surprised. Netflix has been streaming stuff forever… shouldn’t it be good at this? When I put that question to Elizabeth Stone, Netflix’s CTO, she says that streaming live is very different from just streaming. Maybe more different than Netflix itself originally thought.
“When we’re streaming video on demand,” Stone says, “we get the benefit of planning ahead. That content is in its final format; the video, images, audio are in nicely packaged files, and they’ve already gone through all the production steps, the encoding steps, they’re ready to be placed on servers around the world through our content delivery network and through internet service providers.” This is not trivial work, obviously, but it’s work Netflix has been doing for two decades. It has seen every problem, come up with every workaround. “So when a member clicks play,” Stone says, “we’re really ready for them to click play.”
When you’re filming and streaming live, you still have to do all that stuff and more, but you have to do it in real time. “The camera feed goes to the production truck, goes to signal ingestion, goes into the cloud to get encoded. We then have to send that through our CDN, through internet service providers, to land on your TV or your phone. And we have seconds to do that.” Streaming live, even to one person, is hard. It’s doable, of course — TV networks, streaming services, and tech companies do it every day — but it takes work.
Then there’s the whole “65 million people” thing. Stone laughs when I bring it up. Netflix builds and tests and plans as much as it can, she says, both with real events and by pummeling its infrastructure with fake traffic. “But there is no lab in which you can simulate what happens to our systems when 65 million people are watching at the same time.” Even on Netflix’s all-time busiest days, it’s not getting that kind of traffic all at once.
Stone breaks Netflix’s system into two parts. It’s a generalization, she says, but it’s close enough. “When you log into Netflix and you’re scrolling through the homepage, and you’re watching trailers and you’re deciding what to watch, that’s supported by AWS servers.” Netflix is a huge client of Amazon’s web services, which are the backbone of most of the internet at this point. It’s a huge traffic burden just to have tens of millions of people flipping through the app at the same time, but AWS scales pretty well and Stone says that part of Netflix held up even during the fight.
Once you press play, though, the system shifts to Netflix’s own Open Connect system, which is generally considered the best in the streaming business. Netflix invested heavily in its own infrastructure when it first started doing streaming, but, again: 65 million people. “I would argue that any company would have faced challenges at this type of scale,” Stone says. “We have these tight-knit connection points between our servers, Open Connect appliances, and what I’ll call the last mile that ISPs give to devices. All of that was overloaded during the fight.”
Among the things you can’t know until an event starts is who’s going to watch, where they’re going to be, and what else might be happening. The internet is a finite thing, with only so much available bandwidth in the cables that connect things; if an event is unexpectedly popular in LA, it’s going to struggle in LA even if it’s fine elsewhere. “Think of it as the difference between a truck delivering 100 bottles of water vs. having to run a live water hose to 100 people at once,” Fastly CEO Anil Dash wrote recently. “One problem is about moving some bits from one place to another, the other problem is keeping a live stream running at high volume at a massive scale. When there’s not enough water being supplied to all those hoses, everyone gets a little less.”
Stone agrees the hoses are the challenge. “All of the streamers out there,” she says, “we all face it: how much bandwidth is there? And are we going to need bandwidth at the same moment that many other streamers need bandwidth?” It’s not like Netflix can dig trenches or run more cables along your phone lines — certainly not by Christmas, anyway — so all it can do is try to optimize the system as best it can.
Since the Paul / Tyson fight, Stone says Netflix has been trying to both increase its capacity and control the flow of bandwidth more effectively. “We’ve augmented our Open Connect servers, and several of the ISPs have augmented the capacity they’re bringing to the table,” she says. They’re particularly focused on places that were overloaded during the fight, though she doesn’t specify which places those are. Internally, the team is also working on optimizing the algorithms that decide how to prioritize traffic and bandwidth.
There probably won’t be as many people watching football on Christmas as there were for the fight. It’s possible no Netflix live event will be that big ever again — there aren’t many one-off cultural moments that command an audience like that. But Stone says she’s glad to have seen the system so wildly overtaxed and stressed because now the team knows what happens. “It would have taken us a lot longer to get those learnings if we were just slightly turning the dial from some of the earlier live events,” she says. By throwing the lever all the way to the end, she thinks Netflix can now be ready for just about anything.
Though, to be clear, even Stone won’t go so far as to promise the football games will go perfectly. All she’ll say is she loves a challenge.
Even if the Christmas games go well, the Netflix team doesn’t get much of a break. On January 6th, it will stream the first episode in a new weekly series: WWE Raw, the flagship wrestling show. Netflix bought the show’s rights for $5 billion and is responsible for streaming it for the next decade. In 2027 and 2031, Netflix will also stream the FIFA Women’s World Cup. Both have big, built-in interest, and both drive big buzz around the world. They’re also recurring programs, which will keep subscribers subscribed. That stuff matters to Netflix.
It’s also just simple math. All the most popular things on TV now are live events: sports, awards shows, that sort of thing. Those are the shows that command the highest viewership and the highest ad rates, and Netflix is now rapidly trying to build its own ad business. That’s why Amazon paid for NFL rights, why Peacock went all-in on the Olympics, and why even the price of the Macy’s Thanksgiving Day Parade is going up. In an increasingly splintered entertainment landscape (which is, of course, partly Netflix’s fault), must-see live TV is more valuable than ever.
Riegg, who oversees all these content choices, is adamant that for Netflix, live and sports are not the same thing. He seems to be animated by the idea of bringing people together, of creating communal moments where everyone is watching and talking about the same thing at the same time. Netflix, of course, is maybe the company most responsible for ending that monoculture by making huge libraries of content available to everyone, everywhere, all the time. But Riegg thinks the platform should bring some of that classic live TV energy back. “Remember the Felix Baumgartner Red Bull space jump?” he asks me. “I remember everybody in the office was watching that — something where there’s still the specter that anything can happen. We’re all experiencing this at the same time.”
Netflix is interested in buying more of these events, Riegg says, but he also wants to create them. Which brings Riegg to his current big question: “What is our version of Dancing with the Stars? Or what is our version of America’s Got Talent?” That’s the stuff Netflix’s unscripted team is working on right now — taking familiar formats and adding in live elements. Because Netflix is so big, and so global, Riegg thinks it has a chance to do something genuinely new. “What if we had The Voice, and everyone around the world could opine and weigh in about who should win? That’s a different level of community viewing.”
I mention to Riegg that I was a longtime, immensely dedicated American Idol fan, and his eyes go wide. “We’ll never see another Idol,” he says, “in terms of the gap between Idol and the second-place show. But we can certainly try to say, what’s the next iteration of that?” It’s pretty clear he and the team have some ideas, though Riegg won’t tell me what they are. We’ll just all have to find out together, live.