From engine to leaderboard.

Your first run

Check the whole workflow with a working engine before optimizing anything.

1. Sign in with GitHub and create a team or join one. Ask a team owner or admin for the six-character invite code; entering it joins immediately. You can leave a team from the Team page.
2. Open the starter repository, select Use this template → Create a new repository, and clone your new repository. The included engine works as-is; the baseline score is 100.
3. Open Submissions, have a team owner or admin select Connect a repository, and set the engine folder to engine.
4. Push a new commit to the default branch, or open Repositories and select Run now to use the existing commit. Dryft submits the engine and starts a public run. Follow the results and logs on the submission page.
5. Improve engine/engine.py without changing its two methods. Give AGENTS.md to your coding agent; use agent/ if you want an automated research loop.

Public runs give feedback, not a leaderboard score. Start an official run to rank; the baseline scores 100.

Submit without GitHub

Any team member can open Submissions, select Download starter archive, and upload that file unchanged under Submit an archive. Open the new submission in the history and select Start public run. Uploading alone does not start a run. No CLI, repository connection, or local GPU is needed.

For an edited engine, install the CLI from the starter and package the contents of engine/:

./bin/dryft validate engine
./bin/dryft package engine --output submission.tar.gz

Upload submission.tar.gz. Its root must contain engine.py, without a containing folder or ./ path prefixes. Limits: 2 MiB compressed, 16 MiB expanded, and 200 files. ZIP archives are not supported.

Allowed file extensions: .py .pyi .yaml .yml .json .toml .txt .md .cfg .ini. The engine itself must be Python or Triton source.

Teams and permissions

You can compete alone by creating a team. To invite someone, an owner or admin copies the invite code from Team. A code grants immediate membership, so share it only with intended teammates.

Every member

Download the starter, upload engines, run connected repositories, start public or official runs, cancel runs, and read the team’s submissions, results, and logs.

Team owners and admins

Connect or disconnect repositories, change their engine folders and automatic-run settings, inspect webhook delivery, create or revoke team API tokens, and remove members. Only owners change member roles.

Repository connection also requires permission to install or configure the GitHub App for that repository. Organization repositories may need a GitHub organization owner’s approval. Choose “Only select repositories” and grant access to your own copy of the starter.

If you belong to multiple teams, use the account menu at the bottom of the sidebar to switch teams. API tokens belong to the team that created them. A token’s full value appears only once; ask its creator to share it securely or create a replacement if it was lost.

Submissions, code, and logs stay private to your team. Ranked team pages show members and best-score history. Leaving a team does not delete its work; the leave dialog explains ownership transfer when applicable.

What to submit

Submit a folder with engine.py and any source files it imports. engine.py must export class Engine. Dryft names each submission from its repository and team slot, such as fast-qwen #7.

In the starter repository, only engine/ is submitted. Keep your agent, notes, credentials, and local tools outside that folder.

engine.py exports a class that owns the whole generation loop over the fixed weights:

class Engine:
    def __init__(self, model_path: str) -> None:
        """Load the pinned checkpoint from model_path. Untimed, budgeted."""

    def generate(self, input_ids: list[list[int]], max_new_tokens: int):
        """Yield one list of token ids per step, one id per sequence,
        exactly max_new_tokens times. Greedy; do not stop at end-of-sequence."""

Every token must match the baseline’s greedy choice. You may change how the work is done—KV-cache layout, CUDA graphs, fused kernels, prefill, or exact speculative decoding—but not the answer. Quantized or approximate output is not allowed.

The checkpoint is Qwen/Qwen3-4B-Instruct-2507, revision cdbee75f17c01a7cc42f958dc650907174af0554, with BF16 weights on one NVIDIA H100. The platform provides the weights through model_path; do not download them in your engine.

The runtime uses Python 3.11, CUDA 12.4, PyTorch 2.5.1, Triton 3.1.0, Transformers 4.51.3, safetensors 0.5.3, and tokenizers 0.21.1. The starter’s requirements.txt is for optional local GPU development. See its engine contract for the complete interface.

Ship Python or Triton source only. Do not include model weights, credentials, compiled binaries, or your research agent. The sandbox has PyTorch, Triton, and Transformers but no network access.

Scoring and correctness

Every run checks output correctness and resource limits. Official runs also enforce latency and timing stability. These are the published values for the Qwen3 4B decode benchmark; sign in to see the live definition.

The organizer also sets a whole-run time limit, visible on signed-in run pages. A run over that limit is canceled with an explanation. This is separate from the per-sample budget; a changed limit applies to runs already in progress.

Use the CLI

The starter installs the right CLI for macOS, Linux, or Windows and checks the download before installing it. You do not need Python or access to the platform source.

./install-dryft.sh                 # macOS or Linux
# .\install-dryft.ps1              # Windows PowerShell

export DRYFT_TOKEN="dryft_pat_REPLACE_ME"
./bin/dryft doctor

A team owner or admin creates a token under API tokens. Keep it out of git and your submission. The CLI already knows the event server.

On Windows PowerShell, use:

.\install-dryft.ps1
$env:DRYFT_TOKEN = "dryft_pat_REPLACE_ME"
.\bin\dryft.exe doctor
.\bin\dryft.exe validate engine
.\bin\dryft.exe submit engine
.\bin\dryft.exe run SUBMISSION_ID --mode public --wait 3000

Published CLI binaries support macOS Intel and Apple Silicon, Linux x86_64, and Windows x86_64. Linux ARM64 and Windows ARM64 users can submit archives in the browser instead. The optional Python loop in the starter’s agent/ folder uses the standard library and documents its own setup.

On macOS or Linux, validate, submit, and run your engine:

./bin/dryft validate engine
./bin/dryft submit engine
./bin/dryft run SUBMISSION_ID --mode public --wait 3000

For engines uploaded in the browser, copy the Submission ID from the submission’s Details panel. Copy the Run ID beside its run heading for log, result, and cancel commands. The number in a title such as “Submission #7” is not an API ID.

submit prints the submission ID used by run. Validation happens on your computer; the run happens on Dryft.

./bin/dryft submissions
./bin/dryft runs
./bin/dryft logs RUN_ID --follow
./bin/dryft result RUN_ID --wait --timeout 3000
./bin/dryft rank RUN_ID
./bin/dryft cancel RUN_ID --reason "superseded"
./bin/dryft --help

Run on push

In Repositories, an owner or admin can connect your GitHub repository and set the engine folder to the folder containing engine.py, usually engine. Give the GitHub App access only to that repository.

Connecting alone does not run the existing commit. Select Run now for the first run, or push a new commit to the default branch. Pushes to other branches do not submit. The next push to the default branch starts a public run. Change Evaluation on push if you want official runs, or turn Auto-run on push off for no automatic run. Run now submits the latest commit once without changing that setting.

Use the HTTP API

The API is on the same origin as this website. Send Authorization: Bearer YOUR_TOKEN with a team API token. The token selects its team. Browser uploads need no separate token.

Use a new idempotency key for each intended run and reuse it when retrying an uncertain request. The upload response contains submission.id; run creation and inspection contain run.id. Logs return items, nextAfter, and complete; pass nextAfter as the next after.

Results and leaderboard

Official scores are normalized points, not tokens per second: 100 is the baseline, and 150 means 1.5× the baseline speed. Per-workload TPS in sample results is actual throughput. TTFT means time to first token; TPOT means time per output token.

Public runs

Each public run measures the public workloads once, for quick feedback. Use the output check, timings, and logs to iterate. Public runs never change the leaderboard.

Official runs

The runs that count, marked Official wherever they appear; public runs are marked Public. You are asked to confirm before one starts. Hidden workloads determine your score, five samples each. Runs must pass the output rule, latency, memory, and timing-stability checks to rank. A score of 100 is the baseline; higher is faster. Measured baseline runs can vary slightly.

Your team’s best

The Leaderboard refreshes every 10 seconds and keeps your best eligible score. A slower run stays in history without replacing your best. When the benchmark’s rules change, earlier results are invalidated and every team runs again.

Team pages

Select a team on the leaderboard to see its members and every result that raised its best score. A team’s page is visible to everyone while it holds a place on the board; runs, submissions and code stay private to the team.

Troubleshooting

My repository is missing

An owner or admin should open Connect a repository, choose the repository on GitHub, and then refresh the list. Check that the GitHub App has access to the correct account or organization. If the App is unavailable, use a browser archive upload.

My push did nothing

Confirm the default branch, engine folder, and Auto-run on push setting in Repositories. An owner or admin can use Check delivery. Run now submits the latest commit without needing another push.

Submissions or evaluations are closed

The organizer controls availability separately from the displayed round clock. You can prepare code and read existing results while closed. Teams are created and joined only while submissions are open. Check the announcement for reopening; a countdown alone does not enable submissions.

The CLI refuses my token

Check that DRYFT_TOKEN is set in the same terminal, the token was copied fully, and its creator is still a member of the team. A revoked or expired token needs a replacement from an owner or admin. Run dryft doctor to check connectivity. Do not put a token in git, an engine archive, or a bug report.

I need help with a failed run

Open its logs and sample results. For incorrect_output, candidate_error, timeout, latency_limit, memory_limit, or unstable_timing, check the engine and benchmark limits. For infra_error or harness_error, retry once; if it persists, contact the event organizer with the submission link, full Run ID, error message, and time. Cancel unwanted queued or active work with Cancel run.