Published: 08 December 2014
Thanks to a lot of very helpful feedback I've been able to make some good changes to the sim. Not all the problems reported were actual discrepancies, but most were. I was surprised there weren't any really bad bugs--things like after a TD return the wrong team got credit for the XP
. Instead, there were several strategic nuances that the sim wasn't handling as well as I'd like. For example, the offense was spiking the ball too often. I'll detail the updates below.
The demo on the sim will continue, and I appreciate any more feedback you have. You can leave comments here or on the original demo post from last week
. As with any software fix, the solution usually creates some other unintended problem, so please give it another whirl.
It's also looking more likely that I'll be able to host some sort of limited version of the sim that will allow multiple replications. In other words, you'd be able to run a bunch of games from the same starting point to get an estimate of simulated win probability. The real value of a sim is in its "excursions," that is, alternate specifications of the model that can be compared. For example, an excursion would be setting up the sim to start the 2-minute offense for one team at 5 min to play rather than 2 min to play. We could compare how often that team wins in each specification, and then perhaps gain some insight into why the results come out as they do.
There are endless examples of what we can investigate. What if a team took a fair catch on every punt, preventing penalties but forgoing a chance at a return? Would they win more or less often? What if you do that when ahead? When behind? What if the Giants, who were up 21-3 at halftime, had knelt-out the entire second half against the Jaguars last week instead of giving up two big turnovers for scores?
Unfortunately, the endless variety of things we can study with the WOPR means that it needs and endless variety of plugs and hooks in the software to allow users to tell one team to always kneel
or to fair catch when ahead
. It's not hard for me to do this on my end inside the code, but it's not easy to anticipate what controls and inputs are needed for every possible research question. Ultimately, only the guy with the code would be able to fully exploit the sim.
Before I get to the list of fixes, here's a fun (but pretty meaningless) exercise. If I program the Giants to always kneel in the second half (except on 4th down or when they are inside their own 3-yard line), they end up winning about 71% of the time in the WOPR. Given that the Jaguars offense is far from average, the Giants may have had an even better chance than that. That stunned me at first, but New York was up by 18, and scoring 18 points in a half is not easy to do even when the other team isn't moving the ball. Plus, in some of the simulated games the Giants got some easy points off turnovers or special teams.
Playing conventionally (not kneeling) would have given the Giants a better than 95% chance of winning, so even though 71% from the kneel strategy is pretty high it's not something I would recommend. Still, it's interesting to think that somewhere between 30 min to play and the end of the game it might make sense to start kneeling a lot earlier than your final possession.
You can play with the sim demo at this link
or via the Tools menu. Here's a list of fixes for the WOPR over the weekend.
1. Trailing teams will now onside when down by more than one score, but not when down by more than 15 points in the endgame.
2. Trailing defenses were not using timeouts when the two-minute warning was approaching before the next play. In other words, they often wasted large amounts of time knowing the two-minute warning would stop the clock before the next snap, but that could still be up to 40 seconds. Now defenses use timeouts if the offense can burn a significant amount of time prior to the warning.
3. The trailing offense was not using timeouts often enough between 2 and 1 minute remaining. Now they do.
4. A team trailing by 3 or less would forego an extra-long FG (outside the 35 yl) and go for it on 4th down in the endgame. Now they will accept a fg attempt as far as a 60-yd attempt (the 42 yl) in desperation.
5. Offenses were calling timeouts or spiking in the 2-minute offense after an incomplete pass. Fixed.
6. Changed the loss of yardage when kneeling to 2 yards from 3, with a slight amount of noise built in.
7. Teams would use timeouts after a kickoff. (I couldn't duplicate this, and think it might be a misunderstanding of the sim output...A timeout in the note field means it was called following the play, not prior). But in any case, I added additional logic to prevent this.
8. Spikes after plays were not burning enough time. Corrected.
9. Play situations with very small samples of actual data now fall back on a more general distribution of outcomes. For example, very rarely but on occasion a team will run on 3rd and long in the 2-minute offense in the redzone. There may only be a handful of such plays since '08. In this example, the sim falls back on 3rd and long runs when not in the 2-min offense to determine the gain on the play. (The sim was always designed to do this, but it wasn't working exactly right.)
10. Offenses are too conservative on 4th down, even for actual coaches. Currently, the sim always kicks or punts unless a team is behind in the endgame. This was by design so it can be compared with other 4th down doctrines. I haven't changed anything yet, but I can see how this approach might bias the results against teams trailing by a couple scores who could be willing to go for it in favorable situations.