built this for my chess app and figured this sub would appreciate the details, so it's now MIT on github + npm.
the pipeline: board detection is gradient peak analysis (ported chessboard_finder.py from the old tensorflow_chessbot to typescript, credited in the repo). then each of the 64 tiles goes through a small CNN, ~330k params, 1.3MB onnx, running on onnxruntime wasm fully client side. no server, works offline once loaded.
the fun part: i never labeled a single screenshot. the training data is 100% synthetic. render known positions across ~72 piece sets and ~55 board themes (lichess assets, procedural flat boards, hatched book-diagram styles) and every tile is labeled by construction. then you beat the renders up with real world damage: jpeg artifacts, blur, dimming overlays for the reddit lightbox case, resize round trips, corner jitter. and the detail that matters most: training tiles go through the exact same grayscale/crop/resize code that runs in the browser, so there's zero train/serve skew.
second trick i'm happy with: book diagrams fool the corner detection by about a quarter tile because the boards are edge-rich. instead of chasing perfect detection, i classify two alignment candidates (detected corners + a checkerboard grid-snap) and keep whichever one the model is more confident about. let the classifier arbitrate the geometry.
orientation gets resolved from pawn direction (white pawns sit lower on average in any natural position, a black-POV screenshot inverts that).
results: the legacy tensorflow_chessbot model misreads up to 34 tiles per board on my eval set and mixes up K/Q on some themes. this one ships at zero wrong tiles on all positive cases, and board-free images still get rejected. everything returns per-tile confidence, low reads get flagged as unreliable instead of returned silently.
the whole training pipeline (corpus generator, train script, eval gate) is in the repo too, so the model is reproducible, not just downloadable.
demo (just paste a screenshot): https://scoriiu.github.io/fenshot/
repo: https://github.com/scoriiu/fenshot
npm: https://www.npmjs.com/package/@scoriiu/fenshot
limitations: 2d screenshots and diagrams only, no 3d/perspective sets yet. happy to answer anything about the synthetic corpus approach, it worked way better than i expected and i suspect it generalizes to any domain where you can render your labels.