All Hokie, All the Time. Period. Presented by First Bank & Trust Company

H
Joined: 02/22/2023
Posts: 344
Likes: 962
Subscriber

That’s not quite how building robust rating systems work.

I’m somewhat reluctant to even engage further, but I can’t really help myself. You do not build robust statistical models by cherry picking a few things like record and subjectively weigh them together as has been done with the RPI. You pick a loss/evaluation function to evaluate performance, then iterate on your chosen model to try to improve performance by seeing how it performs on unseen data. In essence, you want the model to learn what should be weighted from the data rather than just randomly guessing. In this case, the mean squared error (MSE) of a particular rating system’s predicted scoring differential between two teams would be the obvious evaluation function. A simple linear regression model with an indicator variable signifying if the game was played at home/away/neutral, an indicator variable for the opponent, and an indicator variable for the team themselves for each game could be the predictor variables that could be selected. The target variable would be the scoring differential between the two teams. This allows the model to learn weights to put each team’s performance in each game in context of the opponent they played. Simple linear regression utilizes a closed form solution to minimize MSE, exactly our goal here. The coefficients for each team would then give you an estimate of team strength properly controlled for location and schedule. You would then evaluate how well your model does on unseen data by holding out the 20% or so of a given season’s games and evaluating the MSE of your models’ predictions. This is the sort of methodology things like the NET, KenPom, Massey, etc. take when creating and evaluating predictive models. It sounds simple, but this approach will give a far more accurate representation of team strength when it comes to predicting which team is better than what the RPI does. ChatGPT could probably do a good job explaining this further if you were interested in learning more. Of course you are correct that selection is not being done based purely on metrics. Instead, we are allowing people to make arbitrary decisions that give them the flexility to advance their own agendas. Just take a look at the college football playoff committee. I would claim that VT (and essentially all non-SEC teams) would be better off in a system where decisions were made based of a fair, objective evaluation criterion like a rating system with publicly audible code than some committee.
Posted: 05/11/2026 at 05:56 PM ET
(In response to this post by nkhooch)

Thread Replies (17)

Full Thread →
Professional statistician and occasional poster here.
HokieEngineer22 05/11/26 11:09 AM ET
As a metric person, I would probably just say the RPI has shortcomings
nkhooch 05/11/26 03:28 PM ET
The statistical term re: the RPI would be “extremely underparameterized”
HokieEngineer22 05/11/26 04:15 PM ET
Come on now. The win/loss record "sounds" useful?
nkhooch 05/11/26 04:57 PM ET
That’s not quite how building robust rating systems work.
HokieEngineer22 05/11/26 05:56 PM ET
Best post I have seen on the subject. There is absolutely no logical
Hokie_Forester 05/11/26 02:04 PM ET
Well, how wast Angela Tincher not on the USA Olympic team, yet…
TomA 05/11/26 12:37 PM ET
Can you name the three pitchers that were selected?
MidloHokie 05/11/26 03:46 PM ET
As Ronald Coase said “If You Torture the Data Long Enough, It Will Confess”
Va AandM 05/11/26 12:32 PM ET
Yep, pick the teams you want and then select and weight the stats to
BEST2VT 05/11/26 12:58 PM ET
Ah, Finagle’s Rule: draw curve then plot your points. Works every time **
Va AandM 05/11/26 01:06 PM ET
It feels the same every time this happens. The committees
baumerj 05/11/26 11:35 AM ET
Only comeback is to win. Go Hokies. **
skautdog 05/11/26 11:18 AM ET
You're "RPI is a joke" line, although probably true, was the death ....
hokietony 05/11/26 11:14 AM ET
Then they have contradicted themselves because they didn’t follow RPI…
HokieEngineer22 05/11/26 11:20 AM ET
Committee contradicts self. Story at 11! 😉**
HokieNL 05/11/26 04:27 PM ET
In general they did, only 1 non-top 16 RPI team is a host.**
hokietony 05/11/26 11:22 AM ET
That exception is sort of what I am trying to point out.
HokieEngineer22 05/11/26 11:32 AM ET