Commercial AI gateways sit between you and the foundational models. They can quietly downgrade models, inject hidden instructions, or buffer connections. Here is how to test and verify them objectively using reproducible evidence. Gateway reseller AI menjadi perantara antara aplikasi kita dan model asli. Tanpa transparansi, reseller bisa saja mendowngrade model diam-diam, menyisipkan instruksi rahasia, atau mengalami jeda buffering. Panduan ini menjelaskan cara mengujinya secara objektif berbasis data empiris.
When an application connects to an LLM reseller, requests do not go directly to the model creator (like OpenAI or Anthropic). Instead, they traverse an intermediary reverse proxy that can alter headers, adjust system prompts, or silently change the upstream routing target based on current load. Saat aplikasi kita memanggil API reseller LLM, request tersebut tidak langsung menuju ke penyedia resmi (seperti OpenAI atau Anthropic). Permintaan tersebut melewati reverse proxy perantara yang memiliki kemampuan mengubah header, menyisipkan instruksi sistem, atau mengalihkan rute server secara diam-diam berdasarkan beban trafik saat itu.
Before spending tokens or running benchmarks, execute these three steps to understand what the gateway advertises and how its API responds. Sebelum buang-buang token atau menjalankan pengujian besar, lakukan 3 langkah awal ini untuk memetakan katalog model dan respons dasar gateway.
Never hardcode API keys in scripts. Keep them in shell variables: Simpan API key di variabel environment terminal, jangan ditulis langsung di kode:
export AUDIT_KEY="sk-..."
export GATEWAY_URL="https://api.reseller.com/v1"Query the catalog to see advertised model IDs, permissions, and pricing: Ambil daftar model untuk melihat ID yang terdaftar dan perizinannya:
curl -s "$GATEWAY_URL/models" \
-H "Authorization: Bearer $AUDIT_KEY" | jqSend a 1-token prompt to check HTTP status, latency, and response ID format: Kirim prompt singkat untuk mencatat status HTTP, latensi, dan format ID:
curl -s "$GATEWAY_URL/chat/completions" \
-H "Authorization: Bearer $AUDIT_KEY" \
-d '{"model":"target","max_tokens":15,...}'An LLM cannot be verified by asking simple questions like "What model are you?" because system prompts can easily instruct any model to claim an identity. Similarly, individual word choices (like picking a specific animal) are strong clues, but a single answer is never proof. Models are probabilistic engines. Model LLM tidak bisa diverifikasi hanya dengan bertanya "Kamu model apa?" karena system prompt bisa dengan mudah menyuruh model apapun mengaku identitas tertentu. Pilihan kata tertentu (seperti hewan favorit) adalah petunjuk yang baik, tapi satu jawaban saja tidak pernah bisa jadi bukti mutlak. Model AI bekerja secara probabilitas.
We test models across 20 open-ended tasks in 4 languages (English, Russian, Chinese, Arabic) at sampling temperature 1.0. We discard uninformative prompts (like asking for a number from 1 to 10, which almost all modern models answer with "7").
Kita menguji model di 20 pertanyaan terbuka dalam 4 bahasa (Inggris, Rusia, Mandarin, Arab) pada suhu sampling 1.0. Kita membuang prompt yang tidak informatif (seperti meminta angka 1 sampai 10, di mana hampir semua model menjawab "7").
JSD measures how close two probability curves are on a mathematical scale from 0.0 (identical probability distributions) to 1.0 (completely disjoint answers).
JSD mengukur seberapa dekat dua kurva probabilitas dalam skala matematika dari 0.0 (distribusi probabilitas identik) sampai 1.0 (jawaban sama sekali tidak beririsan).
JSD(P || Q) = 0.5 * KL(P || M) + 0.5 * KL(Q || M)
where M = 0.5 * (P + Q)
To avoid subjective bias, verdicts are determined strictly by mathematical thresholds:
• Looks right. [GENUINE]: Distance to claimed reference is within 2× internal noise floor, and at least 2.0× closer than the nearest plausible decoy.
• Does not match. [SUBSTITUTED]: Distance ratio < 1.0× (meaning the model is mathematically closer to a cheaper decoy than to the model you paid for).
• Cannot tell yet. [INCONCLUSIVE]: Separation between 1.0× and 2.0×, uncalibrated noise floor, or fewer than 12 shared valid cells.
Untuk menghindari bias subjektif, vonis diputuskan murni oleh ambang batas matematika:
• Asli [GENUINE]: Jarak ke model referensi berada di dalam 2× noise floor internal, dan setidaknya 2.0× lebih dekat dibanding model tiruan (decoy) terdekat.
• Palsu [SUBSTITUTED]: Rasio jarak < 1.0× (artinya model terbukti lebih dekat ke model murah daripada model yang dibeli).
• Belum Jelas [INCONCLUSIVE]: Separasi antara 1.0× dan 2.0×, noise floor belum terkalibrasi, atau kurang dari 12 cell beririsan.
These five patterns represent the most common architectural anomalies observed when auditing commercial resellers. Use them to diagnose unexpected model responses. Lima pola ini mewakili anomali arsitektur yang paling sering ditemukan saat menguji reseller komersial. Gunakan pola ini untuk mendiagnosis kejanggalan respons model.
| Scenario | Observed Metrics | Underlying Mechanism | Verdict |
|---|---|---|---|
| Scenario 01 Silent Distillation Downgrade |
D(Claimed Flagship): 0.5836 D(Flash Decoy): 0.3102 Ratio: 0.53× |
The gateway charges flagship prices, but internally routes traffic to a 10× cheaper distilled model. The statistical distance collapsed directly into the Flash decoy baseline. Gateway mematok tarif flagship, tetapi di backend mengalihkan trafik ke model distilasi Flash yang 10× lebih murah. Jarak statistiknya runtuh mendekati baseline Flash. | SUBSTITUTED |
| Scenario 02 Collusive Multi-Model Wrapper |
D(Model A vs B): 0.1291 Output Overlap: 42.5% Identical Hidden CoT: 169–515 tokens |
Two supposedly competing models from different creators produce 42.5% identical text under identical seeds. Both labels route to a single underlying model burning hidden reasoning tokens. Dua model dari lab berbeda menghasilkan output 42.5% sama persis di seed yang sama. Keduanya ternyata diarahkan ke satu model yang sama dan membakar token reasoning tersembunyi. | SUBSTITUTED |
| Scenario 03 Prompt-Injected Identity Wrapper |
D(Claimed Model): 0.6206 Ratio vs Decoy: 0.81× Thinking Sig: "" (empty) |
An open-weights model is wrapped with a system prompt saying "You are [Brand], made by [Creator]...". Multilingual queries cause the model to break character and leak its injected instructions. Model open-weights dibungkus system prompt: "Kamu adalah [Brand], buatan [Lab]...". Pertanyaan multibahasa membuat model keluar karakter dan membocorkan instruksi rahasianya. | SUBSTITUTED |
| Scenario 04 Unannounced Outage Fallback |
Outage JSD to Decoy: 0.1130 ID Shift: msg_ → chatcmpl-Restored JSD: 0.0838 |
When upstream enterprise credentials expire, the proxy automatically redirects requests to an alternative cloud model without notifying callers. Once fixed, JSD returns to normal. Saat akun upstream kedaluwarsa, proxy otomatis mengalihkan request ke model alternatif tanpa pemberitahuan. Begitu diperbaiki, JSD kembali normal ke batas noise floor. | FALLBACK |
| Scenario 05 Multi-Decoy Falsification of Rumors |
D(Claimed): 0.0472 (Floor: 0.065) D(Decoy 1): 0.3962 • D(Decoy 2): 0.6144 Ratio: 8.39× |
Community rumors claimed a newly released flagship was secretly an older deployment. Tested across 1,000 paired draws, the model sat at 0.0472 JSD (within noise floor) and >8× further from both decoys, disproving the rumor. Rumor komunitas menduga model baru hanyalah rebrand dari model lama. Diuji lewat 1.000 draw berimbang, model berada di 0.0472 JSD (di dalam noise floor) dan >8× lebih jauh dari kedua decoy, mematahkan rumor tersebut. | LOOKS RIGHT |
Frontier coding models cannot be validated solely by short text completions. DeepSWE tests an agent's ability to navigate large real-world software repositories, modify code files, and pass unit tests inside isolated Docker sandboxes. Model coding frontier tidak bisa divalidasi hanya dengan prompt teks singkat. DeepSWE menguji kemampuan agen AI menjelajahi repositori perangkat lunak dunia nyata, mengedit berkas kode, dan lulus pengujian unit di dalam container Docker terisolasi.
Pier (v0.3.1+) executes tasks inside Docker sandboxes and connects to custom gateways via agent configuration YAML: Pier (v0.3.1+) menjalankan task di dalam Docker dan terhubung ke gateway kustom lewat file konfigurasi agen YAML:
# Install Pier runner
uv tool install datacurve-pier
# Run task with custom gateway YAML
pier run -p deep-swe/tasks/katex-multicolumn-array-spans \
--agent mini-swe-agent \
--model custom/target-model \
--ak config_file=custom_agent.yaml \
--max-retries 1max_tokens: 16384:
Coding agents emit large multi-file edits. Setting output ceilings below 16k causes output truncation and crashes with RepeatedFormatError.
Agen coding butuh menulis perubahan file yang panjang. Membatasi token di bawah 16k menyebabkan output terpotong dan memicu error RepeatedFormatError.
--max-retries 1:
Caps trial-level retries so persistent failures are properly recorded rather than hidden by endless loops.
Membatasi retry trial agar kegagalan model yang persisten dicatat secara objektif, bukan diulang-ulang tanpa henti.
Advanced reasoning models deliberate internally before transmitting their first token. In our audits, standard models respond in 1.5s to 2.5s, while heavy reasoning models can exhibit up to 44.1 seconds of dead air where zero bytes are sent across the network.
Model reasoning canggih berpikir secara internal sebelum mengeluarkan token teks pertama. Dalam pengujian, model standar merespons dalam 1.5s sampai 2.5s, sementara model reasoning berat bisa mengalami jeda hening (dead air) hingga 44.1 detik tanpa ada data yang terkirim di jaringan.
90 seconds to prevent prematurely terminating connections while the model is thinking.
Pastikan library HTTP client lo punya batas timeout baca minimal 90 detik agar koneksi tidak diputus saat model sedang memproses penalaran.
HTTP 429 means you sent too many requests. Commercial gateways enforce rate limits across your entire account, not per key. Creating 10 new API keys will not help because they all draw from the exact same account bucket.
HTTP 429 berarti request terlalu cepat. Gateway komersial menerapkan pembatasan trafik di level akun, bukan per key. Membuat 10 API key baru tidak akan berpengaruh karena semuanya berbagi kuota yang sama.
probe_lock) to ensure only one request is in flight at a time.