How to Reduce Model Costs with an AI Router, and Failover Design for External AI API Outages

Applicability of This Document

Ready-to-Run Commands

# 対象: 自組織のAI Routerのルーティングポリシー定義
# 権限: ポリシー定義の編集権限
# 変更作業: なし（定義の記述例）
# Production 実行: 該当なし（設計サンプル）

# 「小 / 中 / 大」は自組織で採用しているモデルの相対的な区分を指す。
# 具体的なモデル名と価格は変動するため、ここでは書かずに定義側で解決する。

タスク種別            | 既定の割り当て | 出力上限 | 品質ゲート                   | 不合格時の扱い
----------------------|----------------|----------|------------------------------|----------------------------
分類・ラベル付け      | 小             | 短       | 許可ラベル集合に含まれるか   | 中へ1回だけ再試行
構造化抽出（JSON）    | 小             | 中       | JSONスキーマ検証             | 中へ1回だけ再試行
要約（短文）          | 小             | 中       | 入力に無い固有名詞が無いか   | 中へ1回だけ再試行
要約（長文・横断）    | 中             | 長       | 参照元の提示があるか         | 大へ1回だけ再試行
コード生成（定型）    | 中             | 中       | 構文解析とテスト実行         | 大へ1回だけ再試行
設計判断・長文推論    | 大             | 長       | 人間のレビュー               | 再試行しない（人へ回す）
自由入力の対話        | 中             | 中       | 出力フィルタ                 | 中で再試行（上位へ上げない）

共通の制御:
  timeout:            タスク種別ごとに設定。長文推論だけ長くする
  max_retries:        2 まで。ジッタ付き指数バックオフ
  prompt_cache:       システムプロンプトと共通コンテキストは前方固定にして再利用する
  result_cache:       入力のハッシュをキーにする。ただし個人情報を含む入力は対象外
  batching:           即時性が不要なタスクのみ。締切のあるタスクは対象外
  budget_guard:       日次・月次の上限に対する消費割合を監視し、しきい値で通知する

計測の前提:
  - 振り分けの妥当性は、自組織の利用ログ（タスク種別ごとの件数・入出力量・
    品質ゲートの合格率・上位モデルへの再試行率）で判断する
  - 単価と提供モデルは変わるため、各プロバイダの現行の価格ページを定期的に確認する

# 対象: 自組織のAI Router（複数プロバイダを束ねる中継層）
# 権限: 各プロバイダのAPIキー参照権限とRouterホストへのシェルアクセス
# 変更作業: なし（疎通確認と状態ファイルの読み書きのみ）
# Production 実行: 可能（本番トラフィックとは別経路のヘルスチェック）

STATE_DIR=/var/lib/ai-router/breaker
FAIL_THRESHOLD=5          # 連続失敗がこの回数に達したら OPEN にする
mkdir -p "$STATE_DIR"

for p in provider-a provider-b; do
  # 軽量なエンドポイントを使う。本番の推論リクエストで死活を測らない
  code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 5 "https://$p.example.com/v1/models")
  fails=$(cat "$STATE_DIR/$p.fails" 2>/dev/null || echo 0)

  case "$code" in
    200)
      fails=0
      ;;
    429)
      # レート制限は「プロバイダは生きている」。切り離さずバックオフとキューイングで受ける
      echo "$p: 429 rate limited -> backoff, do not open breaker"
      ;;
    5*|000)
      # 5xx と接続不能は障害として数える（000 は curl が到達できなかった場合）
      fails=$((fails + 1))
      ;;
    4*)
      # 429 以外の 4xx はこちらのリクエストの問題。切り離しても直らない
      echo "$p: client error $code -> check request, do not open breaker"
      ;;
  esac

  echo "$fails" > "$STATE_DIR/$p.fails"

  if [ "$fails" -ge "$FAIL_THRESHOLD" ]; then
    echo "$p: OPEN -> 新規ルーティングを停止し、次候補へ退避。一定時間後に HALF_OPEN で試行"
  else
    echo "$p: CLOSED (http=$code consecutive_fails=$fails)"
  fi
done

How to Read the Results

When to Use This

• AI usage costs are rising more than expected, but it's unclear which tasks are driving it
• All processing is routed through the same large model
• When the external AI API becomes unstable, the entire app stops responding
• A 429 response is being treated the same as an outage, cutting off the path
• Switching providers changed the output format and broke downstream processing
• Frequent retries are increasing both wait time and cost

Possible Causes (Most Likely First)

1. 01

   The same model is used regardless of task difficulty

   Routing routine work like classification or extraction through the same path as reasoning like design judgment means the routine work also pays for the expensive path. Separating task types first is the prerequisite.

2. 02

   There is no cap on output length

   Without an output token limit, unnecessarily long output can be generated. Set the cap based on the length that downstream processing actually uses.

3. 03

   There is no retry policy

   Unlimited retries on every failure increase cost and wait time. Decide on a retry-count cap, backoff, and conditions under which retrying stops.

4. 04

   Provider-specific calls are scattered across the entire app

   Without an abstraction layer, switching during an outage requires a code change. Consolidating calls into one place means switching is just a configuration change.

5. 05

   Rate limiting is not distinguished from outages

   A 429 indicates the provider is operating. Treating it as an outage and cutting off the path can divert requests that would have succeeded with waiting onto another path, worsening the situation.

6. 06

   What to stop during degradation is undecided

   Trying to keep all functions running slows everything down. Deciding priorities in advance lets you keep only the core functions running.

Diagnostic Steps

1. 1

   Aggregate usage by task type

   Read-only

   Aggregate counts, input volume, output volume, and failure rate by task type from your usage logs. This is the starting point for routing design.

2. 2

   Check the distribution of output tokens

   Read-only

   Look at actual output length as a distribution to judge where to set the cap. An average alone cannot determine the cap.

3. 3

   Separate retry rate by cause

   Read-only

   Aggregate separately whether retries are due to timeouts, 429s, 5xxs, or quality-gate failures. The response differs for each.

4. 4

   Check each provider's current pricing and available models

   Read-only

   Reference each provider's pricing page and apply it to your own aggregated figures. Since pricing changes, review it periodically.

5. 5

   Verify operation with one provider disabled

   Medium

   Disable one path in a test environment and confirm the app keeps working without any code changes.

Solutions

Differences by Version and Environment

If This Doesn't Resolve It

• Identify which task types are driving up cost

  Look separately at high-volume tasks and tasks with large per-call input/output volume. The response differs for each.

• Review quality-gate failure rates by task type

  For types with high failure rates, raising the default allocation may end up being cheaper overall.

• Check cache hit rates

  If hits are low, the cause lies in key design or variance in input.

• Actually trigger a failover

  Bring down one provider in a test environment and confirm the switchover and degraded mode behave as expected.

• Decide when to check each provider's current pricing page

  Pricing and available models change. Include a review cadence in your operating procedures.

Basis and Limitations of This Document

FAQ

Questions This Document Covers

• How do you design failover for external AI API outages?
• Where should you start to reduce LLM usage costs?
• How do you build a setup that can switch between multiple AI providers?

What the Risk Labels Mean

• Read-onlyDoes not change data or settings.
• LowImpact is limited, but permissions and load should be checked.
• MediumMay affect performance, locking, or cost.
• HighMay cause an outage, data loss, or require recovery work.
• Expert Review RequiredRequires a separate review before applying to production.

GIIP's Scope of Support

Everything above can be implemented without a specific product. GIIP uses a configuration that switches among multiple AI models depending on the use case, and includes path switching and degraded mode in its operating procedures so that processing doesn't stop when an external API misbehaves. Which model is actually applied to which category depends on the type of task and quality requirements, so it must be decided based on your own usage logs. If you'd like help organizing how to aggregate that decision data, you can reach out for a consultation.

Related Services

• クラウドコスト最適化
• AIプロダクション運用
• AIインフラ運用