GSM8K-TR - Turkish Math Reasoning Benchmark

How to Participate

1. What this benchmark measures

GSM8K-TR evaluates a language model's ability to solve Turkish grade-school math word problems. Each item is a Turkish problem statement; the model must return a final numeric answer.

• Items: 100 held-out problems sampled from malhajar/gsm8k_tr-v0.2 (test split) using seed 42.
• Metric: Exact-match accuracy on the final number.
• Scorer: Looks for #### <number> first, otherwise takes the last number in the response.

2. Build your submission

Create a model.py containing class GSM8KModel with a solve(prompt) -> str method. See the Model Template page for a runnable example.

my_submission.zip
└── model.py

3. Submit

1. Register for the competition.
2. Upload my_submission.zip on the Submissions tab.
3. Wait for ingestion + scoring to finish — final scores appear on the leaderboard.

4. Tips

• Always end your response with #### <number> for safer parsing.
• Return integers without thousands separators (1500, not 1,500).
• Prefer temperature=0 for reproducibility.
• A 512-token chain-of-thought is usually enough for GSM8K-style problems.