Fantasy sports statistics offer reality check

Imagine conducting outcomes research on professional athletes without ever having to see them in person or work around their schedules or egos. Imagine doing this research without even having access to the players' official medical records, and instead using injury information publicly available on the Internet and correlating it not to clinical outcomes measures but to league statistics of player performance.

It may sound like the stuff of fantasy sports leagues, but this type of outcomes research is very much a reality. Five such studies have been presented at national society meetings and two have been published in a major professional journal. The studies all collect injury data from team releases and media reports and assess outcomes using such statistics as the number of yards gained per season in the National Football League or the number of walks and hits per innings pitched (WHIP) during a Major League Baseball game.

"We were trying to do what hadn't been done before, which was quantify the effect of injury on player performance," said Selene G. Parekh, MD, MBA, an assistant professor of orthopedic surgery at the University of North Carolina, who was a coauthor of one of the two published studies. He also is an author of a third study currently in submission that is based on research while he was a resident at the University of Pennsylvania.

The authors of these studies freely acknowledge the limitations of this type of approach, most notably working with an incomplete patient database and the inability to control for confounding clinical variables-type of surgical technique, extent and complexity of injury, rehabilitation strategy, and any ensuing complications-that could have affected outcomes. They also point out, however, that because access to the details of professional athletes' official medical information is so carefully guarded, making outcomes research contingent on those details will limit the number of studies that can be performed.

Of course, those who control the information don't think that's necessarily a bad thing.

"Doctors aren't always so open to change," said Douglas L. Cerynik, MD, a research fellow in the department of orthopaedic surgery at Hahnemann University Hospital in Philadelphia. He was the coauthor of two Internet-based outcomes studies that were presented as posters earlier this year at the annual meetings of the American Academy of Orthopaedic Surgeons and the American College of Sports Medicine.

"But given that there isn't a large sports database that everyone has access to, I think we're producing valid results with the information available, and providing information that otherwise wouldn't be out there," he said.

Even those who criticize the creative approach to data collection, however, agree that performance-specific statistics may be very useful as outcomes measures. They suggest, in effect, that fantasy stats can offer a reality check. Coaches, team administrators, and athletes themselves may be particularly interested in knowing, for example, the likelihood of a running back performing at preinjury levels after returning from an anterior cruciate ligament tear.

"I think it's a good idea to not only include variables like range of motion, joint pain, swelling, and outcomes on validated questionnaires, but to also include on-field performance," said Matthew J. Matava, MD, associate professor of orthopedic surgery and cochief of the sports medicine section at Washington University in St. Louis. He also is the head team physician for the NFL's St. Louis Rams and the assistant team physician for the National Hockey League's St. Louis Blues.

"These performance statistics are objective measures," he pointed out. "The only criticism I have is that, by themselves, they only tell part of the story."

Fantastic beginnings

The mini-trend began with a poster at the annual AAOS meeting in February 2005, in which Parekh and his colleagues at Penn analyzed 31 NFL running backs and wide receivers who had suffered ACL tears during a six-year period. Injury data were collected from NFL game summaries, play-by-play documents, weekly injury reports, and player profiles-all publicly available information.

"Basically we just culled the Internet to find as much as possible," Parekh said.

In addition to documenting whether each player returned to the NFL following his injury, the researchers also created a power rating with which to assess performance in the three seasons preceding and the three seasons following the injury; the power rating was defined as the number of total yards divided by 10, plus the number of touchdowns multiplied by six. They also compared the performance of the returning players with 146 running backs and wide receivers with no history of ACL injury.

Between 1998 and 2002, they identified 33 ACL injuries in 31 running backs and wide receivers; 21% of the injured players did not return to the NFL. In those who returned, the mean power rating per game decreased from 9.9 plus/minus 1.1 in the three preinjury seasons to 6.5 plus/minus 0.9 in the three postinjury seasons. Interestingly, the mean power rating in the control population was significantly lower than the preinjury means in the injured players, at 6.7 plus/minus 0.3, but that value essentially remained unchanged during the study period.

"You hear stories about players like Jerry Rice, and you think these elite athletes are just able to bounce back," Parekh said. "Until you see the data, you don't really comprehend the impact."

In preparing the manuscript for its eventual publication in the December issue of the American Journal of Sports Medicine, the researchers confirmed the validity and reliability of the power rating as an outcomes measure. With respect to validity, they found a statistically significant difference in power rating between 827 player-seasons not associated with Pro Bowl selection (mean power rating of 76.2 plus/minus 2.1) and 111 player-seasons associated with Pro Bowl selection (mean of 198.1 plus/minus 6.1). The measure was also found to be reliable, with an intraclass correlation coefficient of 0.835.

"Initially when we submitted the study, we didn't have a lot of the detailed statistics that we now have on how it was valid and reliable, so the majority of edits we did on that paper were just that," Parekh said.

Predictive power ratings

The following year, the Penn group had two Internet-based studies accepted as podium presentations at the annual AAOS meeting. One looked at Achilles tendon ruptures in NFL players using an approach that was very similar to that of the ACL study; however, the Achilles study also included tight ends, linebackers, cornerbacks, and defensive tackles in addition to wide receivers and running backs. Power ratings for defensive players used a formula that accounted for tackles, sacks, passes batted down, and fumbles recovered, Parekh said.

The results, which are being submitted for publication, suggest that Achilles injuries are even more devastating to NFL players than are ACL injuries: 31% of the 31 players with Achilles tendon ruptures never returned to play, and average postinjury power ratings dropped by anywhere from 64% to 95% from preinjury levels.

Perhaps even more interesting was an apparent trend in which mean power ratings for running backs and wide receivers dipped in the season prior to a player's injury. Although the numbers are too small to attain statistical significance, Parekh maintains that the finding may be clinically important.

"Is there a prodromal period? We don't know the answer to that but we'd like to look into it," he said. "If you can identify a prodromal period, there's a potential for intervention."

The other study presented at the 2006 AAOS meeting, and subsequently published in the April issue of AJSM, analyzed outcomes following ulnar collateral ligament reconstruction ("Tommy John" surgery) in 68 MLB pitchers who underwent the procedure between 1998 and 2003. As in the NFL studies, the study population was identified from team injury reports and press releases.

Because the MLB Players Association's collective bargaining agreement requires standard forms to be completed by a treating physician prior to a player being placed on the disabled list (as would certainly be the case for any player undergoing UCL reconstruction), the researchers felt that they were able to reliably identify all relevant cases, said Brett W. Gibson, MD, now a sports medicine fellow at the University of Colorado in Boulder, who participated in the study while a resident at Penn.

Performance measures in the UCL study included number of innings pitched per season, earned run average, and WHIP. Outcomes for pitchers who had undergone UCL surgery were compared with those of 112 control pitchers with no known history of UCL reconstruction.

Eighty-two percent of the pitchers returned to the major leagues. Although the study found that UCL reconstruction was more likely in starters than relievers and less likely in those with more years of major league experience and higher ERAs, it did not identify any predictors of return to competition.

Number of innings pitched per season was significantly lower in the three seasons following surgery than in the three preinjury seasons; however, the control group also experienced similar decreases. In the study population, the number of innings was significantly lower than in controls only in the first season following injury. This is likely because most pitchers did not play a full season immediately upon their return. But in the second and third seasons the number of innings did not differ significantly between the two groups. ERA and WHIP in the three postinjury seasons were not significantly different from preinjury levels for either injured players or controls. This essentially confirmed the popular contention in baseball circles that UCL reconstruction is unlikely to derail a pitcher's career.

Perhaps because of the study's relatively unremarkable findings, its unorthodox methods have generated less buzz than those of the ACL study, Gibson said.

"Because we were kind of confirming what was already felt to be true, there wasn't a lot of interest," he said. "We weren't breaking down any preexisting misconceptions."

Driving to the basket

At Hahnemann, Cerynik and colleagues conducted a similar study of isolated glenoid labral injuries in MLB pitchers. Although their study does not yet include a control group, they found that 69% of 42 pitchers who underwent glenoid repair between 1998 and 2003 returned to professional baseball and experienced no significant change in ERA or WHIP upon their return. In fact, ERA and WHIP actually saw nonsignificant increases between the three preinjury and three postinjury seasons. All pitchers, however, experienced decreases in innings pitched per season following surgery; in starters, that decrease was significant only in the first postinjury season, while in relievers, that decrease was significant for the three postinjury seasons.

The other Internet-based Hahnemann study examined outcomes of microfracture surgery for chondral defects among 24 National Basketball Association players between 1997 and 2006. Outcomes measures included number of minutes played per game and the NBA's Player Efficiency Rating, which gives players credit for points scored, rebounds, assists, steals, and blocked shots, and demerits for missed field goals, missed free throws, and turnovers.

Although some elements of the PER, such as free-throw percentage, might seem less relevant to knee function than others, Cerynik said the fact that it is a league-sanctioned measure convinced the researchers to use it.

"Nothing is going to be a perfect model," he said. "We're using the tools that are available. And this one is used by the NBA to actually classify player efficiency."

Five players never returned to play in an NBA game; 15 returned to play for two or more seasons. The average decrease in minutes played per game between the two seasons preinjury and the first season postsurgery was small but statistically significant. But by the second season postsurgery, the average number of minutes played did not differ significantly from preinjury levels. Similarly, PER decreased significantly in the first season postsurgery but had returned almost to preinjury levels by the second season.

Reality bites

Although two of the Internet-based studies have been published, the authors have heard their share of criticism. And though they freely acknowledge the limitations of their methods, they suspect other sociopolitical factors may be involved.

"This is kind of a phenomenon of generation X," Parekh said. "A lot of guys reviewing these manuscripts tend to be of an older generation who don't really appreciate this kind of data."

More experienced physicians, however, insist that their only criticisms are related to the accuracy and validity of the results.

James P. Bradley, MD, team physician for the Pittsburgh Steelers and a member of the NFL's Injury and Safety Panel, said he admired the approach used by the Penn researchers and that, in fact, the NFL is currently working to include performance-specific outcomes in future research. These will also have the advantage of being based on the league's complete injury database.

"It was a very ingenious thing they did," said Bradley, who is also a clinical professor of orthopedic surgery at the University of Pittsburgh Medical Center. "The problem is they don't have a denominator. We know there are 35 to 40 ACL injuries in the NFL every year. So there are a lot of guys who are injured that (the Penn researchers) don't know about."

All parties agree that accuracy is vital, given that team administrators and owners may be even more interested in performance-based outcomes data than physicians. Matava, for one, gave a copy of the December paper on ACL injuries to the general manager of the Rams.

"Our results in my mind may have a huge impact on workers' compensation issues, on contract negotiations, and on draft issues," Parekh said. "If you have a promising collegiate player and the year prior to the draft he has an Achilles rupture, for example, now there are data suggesting that if he ever comes back, he's not going to be as good. So it's an important challenge for us to prove these are really valid ways of studying these metrics, because they have huge implications for different aspects of the team-player relationship."

Jordana Bieze Foster is a freelance writer based in Massachusetts and a former editor of BioMechanics magazine.