Experiment 1

Summary

Use 32 PSQT buckets and layer stacks for SmallNet.

Description

Setup

nnue-pytorch

Base commit

70781454bb9fbd5fbaf3d47281daa22a82565984

Try to catch one source of errors (#367)

Changes
diff --git a/model/lightning_module.py b/model/lightning_module.py
index 0000ba3..c249650 100644
--- a/model/lightning_module.py
+++ b/model/lightning_module.py
@@ -36,8 +36,8 @@ class NNUE(L.LightningModule):
         gamma=0.992,
         lr=8.75e-4,
         param_index=0,
-        num_psqt_buckets=8,
-        num_ls_buckets=8,
+        num_psqt_buckets=32,
+        num_ls_buckets=32,
         loss_params=LossParams(),
     ):
         super().__init__()
diff --git a/model/model.py b/model/model.py
index 359db58..1f1d019 100644
--- a/model/model.py
+++ b/model/model.py
@@ -161,8 +161,8 @@ class NNUEModel(nn.Module):
         feature_set: FeatureSet,
         config: ModelConfig,
         quantize_config: QuantizationConfig,
-        num_psqt_buckets: int = 8,
-        num_ls_buckets: int = 8,
+        num_psqt_buckets: int = 32,
+        num_ls_buckets: int = 32,
     ):
         super().__init__()

diff --git a/training_data_loader.cpp b/training_data_loader.cpp
index d335f67..a467e64 100644
--- a/training_data_loader.cpp
+++ b/training_data_loader.cpp
@@ -435,7 +435,8 @@ private:
         is_white[i] = static_cast<float>(e.pos.sideToMove() == Color::White);
         outcome[i] = (e.result + 1.0f) / 2.0f;
         score[i] = e.score;
-        psqt_indices[i] = (e.pos.piecesBB().count() - 1) / 4;
+        // psqt_indices[i] = (e.pos.piecesBB().count() - 1) / 4;
+        psqt_indices[i] = e.pos.piecesBB().count() - 1;
         layer_stack_indices[i] = psqt_indices[i];
         fill_features(FeatureSet<Ts...>{}, i, e);
     }

Training Script

train.sh
#!/bin/sh

case "$1" in
  *.ckpt)
    echo "Resuming from the checkpoint: $1"
    resume_option="--resume-from-checkpoint=$1"
    ;;
  *.pt)
    echo "Resuming from the model: $1"
    resume_option="--resume-from-model=$1"
    ;;
esac

python train.py \
  /data/linrock/dual-nnue/hse-v1/dfrc99-16tb7p-eval-filt-v2.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/leela96-filt-v2.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test60-novdec2021-12tb7p-filter-v6-dd.min-mar2023.unmin.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test77-nov2021-2tb7p.no-db.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test77-dec2021-16tb7p.no-db.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test77-jan2022-2tb7p.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test78-jantomay2022-16tb7p-filter-v6-dd.min-mar2023.unmin.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test78-juntosep2022-16tb7p-filter-v6-dd.min-mar2023.unmin.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test79-apr2022-16tb7p.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test79-may2022-16tb7p-filter-v6-dd.min-mar2023.unmin.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test80-apr2022-16tb7p.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test80-may2022-16tb7p.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test80-jun2022-16tb7p-filter-v6-dd.min-mar2023.unmin.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test80-jul2022-16tb7p.v6-dd.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test80-sep2022-16tb7p-filter-v6-dd.min-mar2023.unmin.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/dual-nnue/hse-v1/test80-nov2022-16tb7p-v6-dd.min.high-simple-eval-1k.min-v2.binpack \
  /data/linrock/test80-2022/test80-2022-08-aug-16tb7p.v6-dd.min.binpack \
  /data/linrock/test80-2022/test80-2022-10-oct-16tb7p.v6-dd.binpack \
  /data/linrock/test80-2022/test80-2022-12-dec-16tb7p.min.binpack \
  /data/linrock/test80-2023/test80-2023-01-jan-16tb7p.v6-sk20.min.binpack \
  /data/linrock/test80-2023/test80-2023-02-feb-16tb7p.v6-dd.min.binpack \
  /data/linrock/test80-2023/test80-2023-03-mar-2tb7p.v6-sk16.min.binpack \
  /data/linrock/test80-2023/test80-2023-04-apr-2tb7p.v6-sk16.min.binpack \
  /data/linrock/test80-2023/test80-2023-05-may-2tb7p.v6.min.binpack \
  /data/linrock/test80-2023/test80-2023-06-jun-2tb7p.min-v2.binpack \
  /data/linrock/test80-2023/test80-2023-07-jul-2tb7p.min-v2.binpack \
  /data/linrock/test80-2023/test80-2023-08-aug-2tb7p.v6.min.binpack \
  /data/linrock/test80-2023/test80-2023-09-sep-2tb7p.binpack \
  /data/linrock/test80-2023/test80-2023-10-oct-2tb7p.binpack \
  /data/linrock/test80-2023/test80-2023-11-nov-2tb7p.min-v2.binpack \
  /data/linrock/test80-2023/test80-2023-12-dec-2tb7p.min-v2.binpack \
  /data/linrock/test80-2024/test80-2024-01-jan-2tb7p.min-v2.v6.binpack \
  /data/linrock/test80-2024/test80-2024-02-feb-2tb7p.min-v2.v6.binpack \
  /data/linrock/test80-2024/test80-2024-03-mar-2tb7p.min-v2.v6.binpack \
  /data/linrock/test80-2024/test80-2024-04-apr-2tb7p.min-v2.v6.binpack \
  --threads=4                           \
  --gpus=0,                             \
  --max_epochs=800                      \
  --num-workers=64                      \
  --batch-size=16384                    \
  --features=HalfKAv2_hm^               \
  --l1=128                              \
  --no-wld-fen-skipping                 \
  --start-lambda=1.0                    \
  --end-lambda=0.6992605384421289       \
  --gamma=0.9942746303116422            \
  --lr=0.0012181558724738395            \
  --in-scaling=317.54982869522763       \
  --out-scaling=379.8040378799748       \
  --in-offset=269.50654301142134        \
  --out-offset=253.51483568025657       \
  --pow-exp=2.4829519732155125          \
  --random-fen-skipping=3               \
  --simple-eval-skipping=931            \
  --network-save-period=10              \
  --compile-backend=cudagraphs          \
  $resume_option

Local Testing

  • 10,000 fixed games

  • Openings: UHO_Lichess_4852_v1

  • Time control: 10+0.1, nodestime=69 (simulated 1+0.01)

  • Engine options: Threads=1, Hash=16

Epoch 200

Epoch 250

Epoch 300

Epoch 350

Epoch 400

Stopped after epoch 400 as no noticeable loss / Elo improvement is observed.

Note

  • Bucket 0, 1, 2 are never used. For bucket 31, it is theoretically possible to handcraft positions by playing without capturing and making pawns promote, but such positions are not present in datasets and will never appear in normal matches.

Last updated