NEW YORK -- In front of a sold-out Arthur Ashe crowd of 23,000 people on Sunday night at the US Open, world No. 1 Daniil Medvedev and Wimbledon finalist Nick Kyrgios get into a rally early in the first set.
"Let's go Kyrgios, let's go," the crowd chants in unison right before Medvedev, the defending US Open champion, begins to serve at 2-3. A few shots into the rally, Medvedev hits a backhand down the line. A voice yells "Out" -- a clean, piercing sound that travels above the din of the crowd.
The ball had just missed the line. Point, Kyrgios.
That voice, however, is not the live voice of a line judge. It's actually a recording of a former line judge. And it's automated to make a call based on where the ball lands on the court.
If that weren't enough: The loudness of the voice varies, based on how close or far from the line the ball lands. Millimeters from nicking the line? The voice is automated to go up several decibels -- almost a yell -- just to make sure the players hear it above the voices of the crowd. If the ball is about to land way out, the voice is softer, less obnoxious. The players already know, so there's no need to hit them on the head with it.
In 2020, during the COVID-19 pandemic, when the USTA had to figure out a way to minimize the number of people on the court, the best way to do it, they realized, was to ensure the players weren't surrounded by crouched humans breathing at them from all corners of the court. So they decided to use Hawk-Eye, a system that had been used as a supplement for line umpires since 2006, for all line calls.
How did it work? The USTA built a "bunker," a room within the Flushing Meadows complex that is used as a classroom for ITF camps the rest of the year. The bunker includes 17 stations -- one for every court in the stadium -- and each station has several monitors feeding live shots from cameras on each court. There are 12 cameras around each court and an additional six cameras to specifically detect foot faults during serves, Sean Cary, the managing director of the USTA's competition operations, told ESPN.
Two specialists -- a review official and a technical operator -- sit at every station to parse through the feed. All line calls, except foot faults, are automated. If a player makes a foot fault, the review official presses a button specifically designed for foot faults, and an automated voice says "foot fault" out on the court.
So, human beings are manning technology? Do mistakes happen?
It's accurate 99.9% of the time. The 0.1% -- that's human error.
"If the technical operator doesn't select the right service box for some reason -- so, you're serving to the left hand service box and they've selected the right hand box, the system is going to call it out because the system thinks you've served on the wrong side of the court," Cary said.
In those cases, the review official hits another button to override that call. The chair umpire then checks in with the review official and the call is corrected.
In order to provide a human element to this very tech-heavy process, the USTA got line umpires to record their voices to be used for calls. They got some high-pitched ones, some baritone ones, some male voices and some female voices. This comes in handy particularly where play takes place on adjacent courts: They use different voices to ensure players don't get confused by the calls on the other court.
Although this change took place in 2020, there were two courts that still employed human line umpires: Arthur Ashe and Louis Armstrong. Why? Because, at that point, Hawk-Eye was not employed throughout any one tournament. And the USTA wanted to make sure play could carry on at Ashe and Armstrong if the systems went down. But by the second week of the 2020 tournament, Cary said players began asking for Hawk-Eye on both of the marquee courts.
"We were having players say that they prefer to have the system that they know is going to be correct all the time," Cary said.
Line umpires were correct 75% of the time, compared to the machines' near-perfect average, according to data collected by the USTA for the US Open.
So in 2021, with the pandemic still ongoing and considering the players' preference for automated line calling, the US Open decided to expand Hawk-Eye to Ashe and Armstrong. This year is the third year the system is being used.
The change resulted in about 270 to 280 line umpires losing a job at the US Open. Traditionally, 400 line umpires were employed in the early rounds of the US Open, Cary said. With the change, some of them (about 120) were transitioned into other roles, such as match assistant, but the rest have not received their usual US Open contracts. (Line umpires, who are independent contractors, are still employed in about 120 pro tournaments in the U.S.)
Is the electronic line calling here to stay?
"The players have accepted it, and they expect it at the Australian Open and the US Open and it would be very hard for us to go backwards from here," Cary said.
The players might have accepted it, but it's still a process for the fans, many of whom, in the peak of their adrenaline-filled experience, yell, "That was out!" and look angrily for the line umpire who made the bad call.
Then you watch acceptance dawn on their faces as they realize, Oh, that can't be out, it's actually the Hawk-Eye calling it.
Or in some cases, they still have a hard time accepting that fact, their eyes bulging as they process the information.