QuesTec not yet showing consistency from umpires

Though it's in just its early stages, the QuesTec system has yet to make a mark on the game.

Originally Published: June 5, 2003
By Nate Silver and Keith Woolner | Special to ESPN.com

Curt Schilling is no Unfrozen Caveman Pitcher, nor, so far as we know, is he Amish. On the contrary, Schilling is one of the biggest technology geeks in all of baseball, charting every one of his pitches using a computer, preparing endlessly for every start, and spending his off days playing EverQuest against bespeckled, zit-faced teenagers everywhere, and Doug Glanville. Thus, we were a little bit surprised about the camera smashing.

Thursday, June 5
Last summer, the World Umpires Association asked Robert K. Adair, Sterling Professor Emeritus of Physics at Yale University, to study QuesTec's PitchTrax system.

And that's the last we heard of Adair and the World Umpires Association.

Why? Because the World Umpires Association didn't like what Adair told them he found.

Fortunately, Adair didn't mind sharing his findings when we spoke recently.

"I've looked at it intelligently," Adair said, "from the viewpoint of somebody who looks at data with some degree of precision, and I seriously doubt that anyone else has.

"There are problems, but it's a pretty good system. They have a limited amount of money. To get the last bugs out of a system can be very expensive, but it's not a bad system."

The umpires have complained that QuesTec's system isn't consistent. Not so, says, Adair.

"The umpire's strike zone and the QuesTec strike zone are consistent, but in different ways. The umpires' strike zone is much wider than home plate: at least a ball width on the outside corner, and half a ball on the inside. And the umpires' strike zone is smaller by a ball and a half at the bottom, and half a ball at the top."

Meanwhile, the QuesTec strike zone does closely mirror the strike zone defined in the rulebook. It's true that the zone must be adjusted up and down for each batter, but Adair says the operators generally do a good job making those adjustments. What's more, while a certain number of pitches do give incorrect readings, "Operators are given leave to kick those out, and typically they tend to throw out six or seven pitches per game."

There's nothing like a bit of objectivity to spice up a debate.

By all indications, Schilling's decapitation of a QuesTec system at Bank One Ballpark did not come in an Everettesque fit of rage, as he would have us believe; rather, it was a deliberate, premeditated act. Schill had gotten the idea, apparently, well ahead of time, from one of the umpires who was working one of his previous contests. He carried out the crime with an appropriate, carefully selected weapon (a baseball bat, of course). He spoke calmly about it afterward. It wasn't even like Schilling had suffered through a particularly bad day -- though he'd taken the loss, he struck out 11 batters against just two walks, and was harmed more by Arizona's impotent offense than anything the men in blue could have done. There's no temporary insanity defense here. Take him to the station, read him his rights, and book him.

But Schill's convinced that this QuesTec thing has done him wrong, and in a league where paper grievances are passed around more often than Pete Rose betting slips, it's hard to blame him for thinking that actions might speak louder than words. OK, so he's taken himself out of the running to be president of the Phoenix chapter of People for the Ethical Treatment of Technology (PETT). Does Schilling have a point?

According to published reports, QuesTec has been installed at 10 major-league ballparks since the start of 2002, with more parks to be added over the course of this season. Those parks are Bank One Ballpark in Phoenix, Fenway Park in Boston, Tropicana Field in St. Petersburg, Jacobs Field in Cleveland, Miller Park in Milwaukee, Edison Field in Anaheim, Network Associates Coliseum in Oakland, Minute Maid Park in Houston, and both New York ballyards.

In the face of the umpires' impending legal action, getting straight information out of anyone regarding QuesTec is only slightly less difficult than receiving accurate accounts of U.S. troop movements from Iraqi Information Minister Mohammed Saeed al-Sahhaf. But we worked the phones and confirmed the list of 10 parks with several independent sources, including Larry Gibson, an attorney for the World Umpires Association.

For several years, we've been tracking certain relevant pieces of information on a game-by-game basis, including walk rates, strikeout rates, and ball-strike counts. It's easy enough to run the numbers and see if there's any difference in their frequency between QuesTec parks, and such non-wired backwaters as Wrigley Field and Dodger Stadium. But before we do that, it's important to consider what's the fairest way to make a comparison.

If QuesTec were installed, say, only for Devil Rays, Rangers, and Brewers games, we'd expect to see a higher percentage of pitches called balls, but that would have to do with the ineptitude of the respective pitching staffs, and nothing really to do with QuesTec. We need some way to correct for this.

Our technique was to look at all games from the start of the 2002 season until May 29 of this year that fell into one of two categories:

  • Home games played by a QuesTec team against a non-QuesTec team
  • Road games played by a QuesTec team against a non-QuesTec team

    In geek speak, the latter set of games forms an effective control group, since they involve essentially the same sets of teams, only in different ballparks. In fact, this method is very close to the standard approach used to calculate park factors -- which makes sense since, when it comes right down to it, differences based on QuesTec are really park effects in another guise.

    Here's what the numbers looked like for the nearly 1,500 games in our sample:

      QuesTec Parks Non-QuesTec Parks
    Strikeouts/Batters Faced 17.44% 17.63%
    Walks/Batters Faced 8.84% 8.75%
    Percentage of Strikes Called 62.46% 62.67%
    Runs Scored (both teams) 9.29 9.27

    There is a consistent effect here, but it's negligible. Games in QuesTec parks have involved a lower percentage of pitches called strikes. Consequently, they featured a lower strikeout rate, a higher walk rate, and slightly higher run scoring.

    But let's keep things in perspective. The numbers above suggest that 0.21 percent of pitches are called differently between QuesTec games and non-QuesTec games. That works out to about one altered pitch call per 475 pitches thrown -- roughly one every other game. Using the strict standards that are favored by the scientific community, the differences above cannot be said to be statistically significant.

    But what about individual umpires? The purported goal of QuesTec, after all, isn't to lead to more balls or strikes per se, but to induce greater consistency from umpire to umpire.

    Umpires are rotated around the league on a regular basis nowadays, and there are 67 umpires who have called at least 10 games apiece in QuesTec and non-QuesTec parks since the start of the 2002 season. As it happens, there are several umpires who appear to change their game calling significantly when working the plate in QuesTec parks. First, there are some umpires who have called fewer strikes in QuesTec games:

    Strike percentage
    Umpire Other parks QuesTec parks Difference
    Ron Kulpa 64.0% 61.9% -2.1%
    Larry Poncino 62.9% 61.3% -1.6%
    Steve Rippley 63.0% 61.5% -1.5%
    Bill Hohn 63.5% 62.0% -1.5%
    Jeff Kellogg 62.6% 61.1% -1.5%

    None of these guys are likely to be on Schilling's AIM buddy list any time soon. Interestingly, however, Doug Eddings -- the home plate umpire who triggered the camera smashing -- doesn't appear here. Eddings is perhaps the best pitchers' umpire in the league, calling 64.7 percent of pitches for strikes, substantially more than the league average of 62.5 percent, and higher than any other umpire who works home plate consistently. Bet the under when he's calling balls and strikes, and you'll come out ahead. But Eddings has hardly changed his approach in QuesTec games, calling 64.4 percent of pitches for strikes in those contests. Eddings, an outlier in terms of his game calling, is one of those umpires whose behavior the QuesTec program might be designed to reform. Thus far, it hasn't worked.

    Ironically enough, there are also a number of umpires who have consistently called more strikes in QuesTec games.

    Strike percentage
    Umpire Other parks QuesTec parks Difference
    Kerwin Danley 61.6% 64.5% +2.9%
    Ed Rapuano 60.4% 62.7% +2.3%
    Mark Hirschbeck 62.5% 64.7% +2.2%
    Tim Timmons 62.1% 64.1% +2.0%
    Charles Reliford 61.5% 63.3% +1.8%
    Lance Barksdale 61.6% 63.3% +1.7%

    Except for Hirschbeck, all of these guys have been hitters' umpires in non-QuesTec parks, calling a lower percentage of strikes than the league average. They change their approach when they're being watched -- in fact, it appears that they overcompensate, calling more strikes than the league norm, and winding up further away from the middle ground than where they started from. Although we can't be certain that QuesTec is the cause of the discrepancy, the swings are statistically significant, and it's understandable that someone like Schilling, who is used to preparing for different umpires as well as different opposing hitters, is upset about it.

    Really, though, there are two types of consistency that we can talk about. QuesTec might lead to inconsistent strike zones among particular umpires -- but perhaps that's a fair price to pay for greater consistency across the entire league. Unfortunately, QuesTec has failed on that front as well. Using a metric known as standard deviation, we can measure the distribution of ball-strike counts across all the umpires in the league. A higher standard deviation indicates a wider spread -- in other words, less consistency. If the QuesTec system is having its desired effect, encouraging umpires to call a standardized, rulebook strike zone when working in parks in which the system is installed, we'd expect the standard deviation to be lower in those games.

    It isn't. In fact, the standard deviation of strike percentage between different umpires is virtually identical at QuesTec parks and at other parks. QuesTec is leading certain umpires to call their strike zones differently than they have in the past -- wider in some cases, tighter in others. It isn't leading them, as a group, to call the strike zone more consistently.

    Perhaps we've been unfair. It might be politically correct for Sandy Alderson to talk about QuesTec as providing greater consistency -- consistency is one of those things that's hard to quarrel with. But there's little doubt that the league is also striving for greater accuracy. Tracking a Pedro Martinez slider as it crosses home plate is a difficult task even without a bat in your hands; it would be surprising if the ability to do so didn't vary, even among the best umpires in the world. Like it or not, the strike zone is a more important part of baseball today than it ever has been, and it's important to get these things right.

    Alderson is correct that QuesTec has the potential to improve the game. But his talk of consistency needs to end here, at least until QuesTec is up and running at all 30 parks. It might be true, as the statistics he's cited suggest, that QuesTec doesn't make much difference on the average. The average winning percentage for the Atlanta Braves and the San Diego Padres is around .500; that doesn't tell you very much about how those teams are playing. Our numbers reveal that QuesTec has made a difference for individual umpires, each of whom has adapted to the system differently, and unpredictably.

    The one thing that pitchers and hitters can agree upon is that it's not so important what sort of strike zone is called, so long as it is called consistently. It's possible that with proper training, proper calibration, and comprehensive implementation at all ballparks, the QuesTec system will eventually be able to provide for that. But it isn't there yet, and until it is, cameras everywhere will not be safe.

    You can check out more work from the team of writers of the Baseball Prospectus at baseballprospectus.com.

  • ALSO SEE