How Baseball Sim Engine Uses ChatGPT to Help Create More Accurate Baseball Environments

October 9, 2026 by Patrick Toner

Baseball Sim Engine tries to make simulated baseball behave like real baseball.

At first, it seems like this should be simple.

We already know what a real MLB environment looks like. We know the league walk rate, strikeout rate, batting average, on-base percentage, slugging percentage, runs per game, stolen-base rate, and so on.

So why not just feed those numbers into the simulator and be done?

Because those numbers are connected.

If we increase walk rate, we do not just increase walks. We also increase on-base percentage, OPS, the number of baserunners, and usually the number of runs scored.

If those other numbers were already close to their real MLB targets, fixing walks can knock them out of tune.

The same thing happens all over the simulator. Changing contact affects hits and strikeouts. Changing extra-base-hit frequency affects slugging and runs. Changing baserunning affects scoring without necessarily changing the batting line at all.

So creating an accurate league environment is less like setting a list of independent values and more like tuning a musical instrument.

You tighten one string, test it, and then make small adjustments elsewhere until the whole thing sounds right together.

That is what the tuning process is doing.

For each tuning candidate, Baseball Sim Engine creates two teams made up of league-average players and has them play each other 750 times: five separate samples of 150 games.

Then we combine the results and compare the simulated environment with the real MLB environment.

A tuning result looks like this:

Metric Sim MLB
Runs/Game 5.08 4.71
AVG .269 .264
OBP .336 .331
SLG .425 .417
BABIP .296 .303
BB% 8.3% 8.7%
SO% 15.0% 17.5%
HR% 2.8% 2.6%
SB Attempts/Game .84 .75
SB/Game .73 .58

That is the basic tuning problem:

Adjust a few tuning knobs, run another 750 games, and see whether the entire environment moves closer to the real one without knocking something else out of tune.

What We Send to the Model

Most of the straightforward tuning is handled by normal code first.

For example, if walk rate is clearly too high or stolen-base attempts are too low, we can make targeted adjustments directly.

The model is used later for cases where several parts of the environment interact and there is not an obvious one-to-one adjustment.

The API call uses a system message, a user message containing the current environment and simulation results, and a strict JSON schema defining exactly what may come back.

The beginning of the actual system message looks like this:

You are a deterministic baseball simulation tuning engine.
Return only one JSON object.
The JSON object must exactly match the PitchEnvironmentTuning TypeScript interface.
Do not include markdown.
Do not include explanations.
Do not include fields outside the PitchEnvironmentTuning interface.
BB%, SO%, BABIP/AVG, steal attempts, and doubles have already been tuned deterministically.

From there, the prompt gets more specific about which tuning knobs should remain locked, which ones may move, how large those changes should be, and what previous attempts have already shown us.

The user message includes things like:

  • the current tuning values
  • the latest simulation results
  • the real MLB targets
  • the difference between the two
  • which metrics are already within tolerance
  • recent tuning attempts and their results
  • valid ranges for the available tuning knobs

The full implementation is available in the repository:

View the pitch environment tuning source on GitHub

The important distinction is that the large PitchEnvironmentTarget is context. It describes the baseball environment we are trying to reproduce.

What we want back is a new PitchEnvironmentTuning: the knobs that can actually change the behavior of the simulator.

What Comes Back

The response is required to match a strict JSON schema.

It cannot return an explanation like:

Try increasing baserunning a little.

It has to return a complete tuning object that our code can use directly.

A response looks roughly like this:

{
  "_id": "...",
  "tuning": {
    "contactQuality": {
      "evScale": 0,
      "laScale": 0,
      "distanceScale": 0,
      "outOutcomeScale": 0.025,
      "doubleOutcomeScale": 0,
      "tripleOutcomeScale": 0,
      "homeRunOutcomeScale": 0
    },
    "swing": {
      "pitchQualityZoneSwingEffect": 0,
      "pitchQualityChaseSwingEffect": 0,
      "disciplineZoneSwingEffect": 0,
      "disciplineChaseSwingEffect": 0,
      "walkRateScale": -0.015
    },
    "contact": {
      "pitchQualityContactEffect": 0.24,
      "contactSkillEffect": 0.24
    },
    "running": {
      "stealAttemptAggressionScale": 0.531,
      "advancementAggressionScale": 0.05
    },
    "meta": {
      "fullPitchQualityBonus": 0,
      "fullTeamDefenseBonus": 0,
      "fullFielderDefenseBonus": 0
    }
  }
}

Almost everything in that object may remain unchanged.

The meaningful proposal might simply be:

Try changing advancementAggressionScale from 0 to 0.05.

But that is still only a proposal.

Then We Verify It

A recent 2009 tuning run shows why that matters.

The process started with deterministic adjustments for things we can tune directly. The model was then given the remaining problem and produced several additional candidates.

Its first proposal performed best during the initial 750-game search.

At that point, we had a candidate that looked like the winner.

But that first test is not the final decision.

We keep the strongest candidates from the search and run each of them again over a larger sample: five sets of 300 games, or 1,500 games per candidate.

That second test changed the result.

The model proposal that had ranked first during the initial search did not hold up as well over the larger sample.

Meanwhile, the much simpler lock-BB candidate — which only changed the deterministic walk-rate tuning and had ranked fourth during the initial search — performed best in verification.

So the result was simple:

Initial 750-game search: the model proposal looked best.

Larger 1,500-game verification: the deterministic walk-rate tuning performed best.

The deterministic candidate was selected.

That is fairly representative of how this system works. The model can help explore combinations we may not want to hard-code, but it rarely produces the final selected environment.

More importantly, it never gets to decide that for itself.

We give it a narrow problem, require a machine-readable proposal, and then run that proposal through the same simulation and verification process as everything else.

The model proposes a tuning. The simulator decides whether it actually works.

The complete output from that 2009 run is below for anyone who wants to follow the process in detail:

npm run generate:env 2009

========================================
GENERATING ENV: 2009
========================================
Starting season 2008 synchronization
Loaded 2463 scheduled games for season 2008 in 127ms
Loaded 2462 completed game PKs for season 2008 in 39ms
Downloaded 2430/2463 (gamePk: 235968, date: 2008-09-29) in 1.30s
Season 2008 synchronization timing:
  Schedule load: 127ms
  Completed-PK query: 39ms
  Cached PK checks: 0ms
  Game downloads and writes: 1.30s
  Throttling: 0ms
  Full game loop: 1.31s
  Other overhead: 6ms
  Total: 1.48s
Finished processing season 2008: 1 downloaded, 2462 already complete, 0 skipped, 1.48s
[IMPORTER] PITCH ENVIRONMENT WINDOW targetSeason=2009 startDate=2008-04-05 endDateExclusive=2009-04-05 regularSeasonGames=2368
[IMPORTER] HOME FIELD ADVANTAGE targetSeason=2009 startDate=2008-04-05 endDateExclusive=2009-04-05 value=0.058326289095519845
[IMPORTER] TUNING START mode=deterministic-locks-then-chatgpt-shape games=150 verifyGames=300 workers=25
[IMPORTER] BATCH START baseline games=150 samples=5 workers=25
[IMPORTER] BATCH DONE baseline results=5
[IMPORTER] EVAL SAMPLE baseline samples=5 gamesEach=150 totalGames=750
[IMPORTER] baseline i=0 G=150 R=5.29/4.71 AVG=0.271/0.264 OBP=0.348/0.331 SLG=0.431/0.417 OPS=0.778/0.747 BABIP=0.305/0.303 BB=0.096/0.087 SO=0.162/0.175 1B=0.161/0.156 2B=0.049/0.048 3B=0.005/0.005 HR=0.028/0.026 SBA=0.535/0.75 SB=0.464/0.58 T[O=0 2b=0 3b=0 hr=0 bb=0 so=0 sb=0 br=0 td=0 fd=0]
[IMPORTER] BATCH START lock-BB games=150 samples=5 workers=25
[IMPORTER] BATCH DONE lock-BB results=5
[IMPORTER] EVAL SAMPLE lock-BB samples=5 gamesEach=150 totalGames=750
[IMPORTER] DETERMINISTIC lock-BB currentPenalty=2006324.734 nextPenalty=1634513.674
[IMPORTER] lock-BB i=2 G=150 R=5.117/4.71 AVG=0.273/0.264 OBP=0.342/0.331 SLG=0.432/0.417 OPS=0.773/0.747 BABIP=0.308/0.303 BB=0.085/0.087 SO=0.165/0.175 1B=0.164/0.156 2B=0.05/0.048 3B=0.005/0.005 HR=0.028/0.026 SBA=0.487/0.75 SB=0.417/0.58 T[O=0 2b=0 3b=0 hr=0 bb=-0.015 so=0 sb=0 br=0 td=0 fd=0]
[IMPORTER] BATCH START lock-SO games=150 samples=5 workers=25
[IMPORTER] BATCH DONE lock-SO results=5
[IMPORTER] EVAL SAMPLE lock-SO samples=5 gamesEach=150 totalGames=750
[IMPORTER] DETERMINISTIC lock-SO currentPenalty=1634513.674 nextPenalty=3700769.752
[IMPORTER] lock-SO i=3 G=150 R=5.395/4.71 AVG=0.279/0.264 OBP=0.346/0.331 SLG=0.444/0.417 OPS=0.79/0.747 BABIP=0.306/0.303 BB=0.083/0.087 SO=0.146/0.175 1B=0.168/0.156 2B=0.051/0.048 3B=0.005/0.005 HR=0.029/0.026 SBA=0.488/0.75 SB=0.422/0.58 T[O=0 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0 br=0 td=0 fd=0]
[IMPORTER] BATCH START lock-OBP games=150 samples=5 workers=25
[IMPORTER] BATCH DONE lock-OBP results=5
[IMPORTER] EVAL SAMPLE lock-OBP samples=5 gamesEach=150 totalGames=750
[IMPORTER] DETERMINISTIC lock-OBP currentPenalty=3700769.752 nextPenalty=2201923.023
[IMPORTER] lock-OBP i=4 G=150 R=5.118/4.71 AVG=0.272/0.264 OBP=0.341/0.331 SLG=0.433/0.417 OPS=0.774/0.747 BABIP=0.299/0.303 BB=0.086/0.087 SO=0.146/0.175 1B=0.164/0.156 2B=0.049/0.048 3B=0.006/0.005 HR=0.028/0.026 SBA=0.49/0.75 SB=0.415/0.58 T[O=0.025 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0 br=0 td=0 fd=0]
[IMPORTER] SKIP lock-BABIP reason=babip-locked actual=0.299 target=0.303
[IMPORTER] BATCH START lock-SBA-1 games=150 samples=5 workers=25
[IMPORTER] BATCH DONE lock-SBA-1 results=5
[IMPORTER] EVAL SAMPLE lock-SBA-1 samples=5 gamesEach=150 totalGames=750
[IMPORTER] DETERMINISTIC lock-SBA-1 currentPenalty=2201923.023 nextPenalty=1325150.984
[IMPORTER] lock-SBA-1 i=5 G=150 R=5.085/4.71 AVG=0.27/0.264 OBP=0.339/0.331 SLG=0.43/0.417 OPS=0.769/0.747 BABIP=0.295/0.303 BB=0.086/0.087 SO=0.145/0.175 1B=0.162/0.156 2B=0.05/0.048 3B=0.005/0.005 HR=0.028/0.026 SBA=0.823/0.75 SB=0.702/0.58 T[O=0.025 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0.531 br=0 td=0 fd=0]
[IMPORTER] SKIP lock-2B-1 reason=doublePercent-locked actual=0.05 target=0.048
[IMPORTER] SKIP lock-runs-advancement reason=runs-not-low actual=5.085 target=4.71
[IMPORTER] BATCH START chatgpt-shape-1 games=150 samples=5 workers=25
[IMPORTER] BATCH DONE chatgpt-shape-1 results=5
[IMPORTER] EVAL SAMPLE chatgpt-shape-1 samples=5 gamesEach=150 totalGames=750
[IMPORTER] CHATGPT PROPOSAL pass=1 currentPenalty=1325150.984 bestCandidatePenalty=1325150.984 proposalPenalty=1191841.339
[IMPORTER] chatgpt-shape-1 i=1 G=150 R=5.08/4.71 AVG=0.269/0.264 OBP=0.336/0.331 SLG=0.425/0.417 OPS=0.761/0.747 BABIP=0.296/0.303 BB=0.083/0.087 SO=0.15/0.175 1B=0.162/0.156 2B=0.05/0.048 3B=0.004/0.005 HR=0.028/0.026 SBA=0.843/0.75 SB=0.727/0.58 T[O=0.025 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0.531 br=0.05 td=0 fd=0]
[IMPORTER] BATCH START chatgpt-shape-2 games=150 samples=5 workers=25
[IMPORTER] BATCH DONE chatgpt-shape-2 results=5
[IMPORTER] EVAL SAMPLE chatgpt-shape-2 samples=5 gamesEach=150 totalGames=750
[IMPORTER] CHATGPT PROPOSAL pass=2 currentPenalty=1191841.339 bestCandidatePenalty=1191841.339 proposalPenalty=1612778.586
[IMPORTER] BATCH START chatgpt-shape-3 games=150 samples=5 workers=25
[IMPORTER] BATCH DONE chatgpt-shape-3 results=5
[IMPORTER] EVAL SAMPLE chatgpt-shape-3 samples=5 gamesEach=150 totalGames=750
[IMPORTER] CHATGPT PROPOSAL pass=3 currentPenalty=1191841.339 bestCandidatePenalty=1191841.339 proposalPenalty=2850686.075
[IMPORTER] BATCH START verify-1-chatgpt-shape-1 games=300 samples=5 workers=25
[IMPORTER] BATCH DONE verify-1-chatgpt-shape-1 results=5
[IMPORTER] EVAL SAMPLE verify-1-chatgpt-shape-1 samples=5 gamesEach=300 totalGames=1500
[IMPORTER] VERIFY CANDIDATE rank=1 label=chatgpt-shape-1 searchPenalty=1191841.339 verifyPenalty=1873865.77
[IMPORTER] verify-1-chatgpt-shape-1 i=0 G=300 R=5.239/4.71 AVG=0.272/0.264 OBP=0.341/0.331 SLG=0.431/0.417 OPS=0.772/0.747 BABIP=0.298/0.303 BB=0.086/0.087 SO=0.145/0.175 1B=0.164/0.156 2B=0.048/0.048 3B=0.005/0.005 HR=0.029/0.026 SBA=0.821/0.75 SB=0.709/0.58 T[O=0.025 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0.531 br=0.05 td=0 fd=0]
[IMPORTER] BATCH START verify-2-lock-SBA-1 games=300 samples=5 workers=25
[IMPORTER] BATCH DONE verify-2-lock-SBA-1 results=5
[IMPORTER] EVAL SAMPLE verify-2-lock-SBA-1 samples=5 gamesEach=300 totalGames=1500
[IMPORTER] VERIFY CANDIDATE rank=2 label=lock-SBA-1 searchPenalty=1325150.984 verifyPenalty=2187886.108
[IMPORTER] verify-2-lock-SBA-1 i=0 G=300 R=5.274/4.71 AVG=0.274/0.264 OBP=0.341/0.331 SLG=0.438/0.417 OPS=0.779/0.747 BABIP=0.299/0.303 BB=0.084/0.087 SO=0.145/0.175 1B=0.164/0.156 2B=0.05/0.048 3B=0.005/0.005 HR=0.029/0.026 SBA=0.84/0.75 SB=0.721/0.58 T[O=0.025 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0.531 br=0 td=0 fd=0]
[IMPORTER] BATCH START verify-3-chatgpt-shape-2 games=300 samples=5 workers=25
[IMPORTER] BATCH DONE verify-3-chatgpt-shape-2 results=5
[IMPORTER] EVAL SAMPLE verify-3-chatgpt-shape-2 samples=5 gamesEach=300 totalGames=1500
[IMPORTER] VERIFY CANDIDATE rank=3 label=chatgpt-shape-2 searchPenalty=1612778.586 verifyPenalty=1837147.267
[IMPORTER] verify-3-chatgpt-shape-2 i=0 G=300 R=5.229/4.71 AVG=0.272/0.264 OBP=0.339/0.331 SLG=0.432/0.417 OPS=0.771/0.747 BABIP=0.297/0.303 BB=0.083/0.087 SO=0.145/0.175 1B=0.163/0.156 2B=0.05/0.048 3B=0.005/0.005 HR=0.028/0.026 SBA=0.815/0.75 SB=0.702/0.58 T[O=0.025 2b=0 3b=0 hr=0 bb=-0.015 so=0.24 sb=0.531 br=0.04 td=0 fd=0]
[IMPORTER] BATCH START verify-4-lock-BB games=300 samples=5 workers=25
[IMPORTER] BATCH DONE verify-4-lock-BB results=5
[IMPORTER] EVAL SAMPLE verify-4-lock-BB samples=5 gamesEach=300 totalGames=1500
[IMPORTER] VERIFY CANDIDATE rank=4 label=lock-BB searchPenalty=1634513.674 verifyPenalty=1756231.026
[IMPORTER] verify-4-lock-BB i=0 G=300 R=5.204/4.71 AVG=0.272/0.264 OBP=0.341/0.331 SLG=0.435/0.417 OPS=0.776/0.747 BABIP=0.306/0.303 BB=0.085/0.087 SO=0.167/0.175 1B=0.163/0.156 2B=0.05/0.048 3B=0.005/0.005 HR=0.029/0.026 SBA=0.509/0.75 SB=0.438/0.58 T[O=0 2b=0 3b=0 hr=0 bb=-0.015 so=0 sb=0 br=0 td=0 fd=0]
[IMPORTER] BATCH START verify-5-baseline games=300 samples=5 workers=25
[IMPORTER] BATCH DONE verify-5-baseline results=5
[IMPORTER] EVAL SAMPLE verify-5-baseline samples=5 gamesEach=300 totalGames=1500
[IMPORTER] VERIFY CANDIDATE rank=5 label=baseline searchPenalty=2006324.734 verifyPenalty=1790016.373
[IMPORTER] verify-5-baseline i=0 G=300 R=5.243/4.71 AVG=0.27/0.264 OBP=0.345/0.331 SLG=0.428/0.417 OPS=0.773/0.747 BABIP=0.304/0.303 BB=0.095/0.087 SO=0.164/0.175 1B=0.16/0.156 2B=0.049/0.048 3B=0.005/0.005 HR=0.028/0.026 SBA=0.523/0.75 SB=0.458/0.58 T[O=0 2b=0 3b=0 hr=0 bb=0 so=0 sb=0 br=0 td=0 fd=0]
[IMPORTER] VERIFY SELECTED label=lock-BB penalty=1756231.026
[IMPORTER] verify i=0 G=300 R=5.204/4.71 AVG=0.272/0.264 OBP=0.341/0.331 SLG=0.435/0.417 OPS=0.776/0.747 BABIP=0.306/0.303 BB=0.085/0.087 SO=0.167/0.175 1B=0.163/0.156 2B=0.05/0.048 3B=0.005/0.005 HR=0.029/0.026 SBA=0.509/0.75 SB=0.438/0.58 T[O=0 2b=0 3b=0 hr=0 bb=-0.015 so=0 sb=0 br=0 td=0 fd=0]
[IMPORTER] VERIFY WARNING [
  {
    "key": "teamRunsPerGame",
    "actual": 5.204333333333333,
    "target": 4.71,
    "tolerance": 0.18,
    "diff": 0.4943333333333326,
    "ok": false
  },
  {
    "key": "teamSBAttemptsPerGame",
    "actual": 0.5093333333333334,
    "target": 0.75,
    "tolerance": 0.12,
    "diff": -0.24066666666666658,
    "ok": false
  },
  {
    "key": "teamSBPerGame",
    "actual": 0.438,
    "target": 0.58,
    "tolerance": 0.12,
    "diff": -0.14199999999999996,
    "ok": false
  }
]

========================================
GENERATE ENV COMPLETE
SEASONS: 2009-2009
========================================
{
  "firstSeason": 2009,
  "endSeason": 2009,
  "seasonsGenerated": 1
}

Waiting for the debugger to disconnect...
Waiting for the debugger to disconnect...