What to Configure When Monitoring Servers and Databases 24/7

この文書の適用条件

そのまま実行できるコマンド

# 対象: Amazon CloudWatch（AWS CLI v2）
# 権限: cloudwatch:PutMetricAlarm
# 変更作業: あり（同名アラームが存在する場合は上書きされる）
# Production 実行: 可能。通知先とランブックを用意してから作成すること
#
# 通知先の SNS トピック ARN は環境変数で渡す（この文書には実 ARN を記載しない）
# export SNS_TOPIC_ARN="<自環境の SNS トピック ARN>"

aws cloudwatch put-metric-alarm \
  --alarm-name "rds-db-sample-instance-cpu-sustained" \
  --alarm-description "CPU使用率が継続して高い状態を検知する" \
  --namespace AWS/RDS \
  --metric-name CPUUtilization \
  --dimensions Name=DBInstanceIdentifier,Value=db-sample-instance \
  --statistic Average \
  --period 300 \
  --evaluation-periods 3 \
  --datapoints-to-alarm 3 \
  --threshold 80 \
  --comparison-operator GreaterThanThreshold \
  --treat-missing-data missing \
  --alarm-actions "$SNS_TOPIC_ARN" \
  --ok-actions "$SNS_TOPIC_ARN"

-- 対象: MySQL 5.7 以降 / Amazon Aurora MySQL
-- 権限: グローバル状態変数の参照権限（PROCESS 権限相当）
-- 変更作業: なし（参照のみ）
-- Production 実行: 可能

-- 現在の接続数
SHOW STATUS LIKE 'Threads_connected';

-- 起動以降の最大同時接続数（ピークの把握に使う）
SHOW STATUS LIKE 'Max_used_connections';

-- 接続上限
SHOW VARIABLES LIKE 'max_connections';

-- 上限に達して拒否された接続の累計
SHOW STATUS LIKE 'Aborted_connects';

-- 対象: SQL Server 2012 以降 / Amazon RDS for SQL Server
-- 権限: VIEW SERVER STATE
-- 変更作業: なし（参照のみ）
-- Production 実行: 可能
SELECT
    (SELECT COUNT(*) FROM sys.dm_exec_sessions
      WHERE is_user_process = 1)                 AS user_sessions,
    (SELECT COUNT(*) FROM sys.dm_exec_requests
      WHERE session_id > 50)                     AS active_requests,
    (SELECT COUNT(*) FROM sys.dm_exec_requests
      WHERE blocking_session_id <> 0)            AS blocked_requests,
    @@MAX_CONNECTIONS                            AS max_connections_configured;

# 対象: Amazon CloudWatch（AWS CLI v2）
# 権限: cloudwatch:DescribeAlarms
# 変更作業: なし（参照のみ）
# Production 実行: 可能
aws cloudwatch describe-alarms \
  --query "MetricAlarms[].{Name:AlarmName,Metric:MetricName,\
Threshold:Threshold,Periods:EvaluationPeriods,Missing:TreatMissingData,\
State:StateValue,Actions:AlarmActions[0],Enabled:ActionsEnabled}" \
  --output table

結果の読み方

こういう状況で使います

• An incident is sometimes learned about from a customer report
• There are too many alerts, and important ones get buried
• Who responds overnight and on holidays hasn't been decided
• Alerts arrive, but the recipient doesn't know what to do
• Monitoring is in place, but there's no list of what is actually being monitored
• Only easy-to-understand metrics like CPU utilization are watched, while lock waits and replication lag are not

考えられる原因（可能性の高い順）

1. 01

   The three types of monitoring are not distinguished

   Liveness monitoring (does the process or endpoint respond), performance monitoring (does it respond but has it slowed down), and log monitoring (are errors occurring) differ in purpose and in what they can detect. Relying on just one misses whatever the other two would have caught.

2. 02

   Only internal metrics are being watched

   Even with CPU, memory, and connection count all normal, a problem with DNS, the load balancer, a certificate, an application exception, or an external dependency can leave the service unusable from the user's side. No amount of internal metrics can monitor that path.

3. 03

   Thresholds are designed as a single fixed value

   A single-shot condition like "notify if CPU exceeds 80%" fires even on a normal spike. Alerts that fire too often get ignored, and real alerts end up missed as a result. Duration (how many times in a row it was exceeded) and rate of change (the deviation from normal) need to be built into the condition.

4. 04

   Alerts have no response procedure attached

   Without deciding what the recipient should check, how far they can handle it themselves, and under what condition to escalate, a notification stops at "noticed" and never turns into a response.

5. 05

   No structure is designed for nights and holidays

   24/7 monitoring is not "the monitoring tool runs 24/7" — it is "it reaches someone who can respond, 24/7." Without an on-call rotation, a next contact when someone cannot be reached, and a way to spread the load among responders, a notification at 2am just sits there until the next morning.

6. 06

   The monitoring configuration has never been inventoried

   A notification destination still pointing at a former employee, an action left disabled, or a target resource that has already been deleted — these all become "monitoring that exists but does not work." Correct at the time it was configured, it degrades over time.

確認手順

1. 1

   Build a list of what is monitored

   参照のみ

   List out what is currently monitored at each of the host, database, and application layers. Make the current state visible before discussing gaps.

2. 2

   Check whether synthetic monitoring exists

   参照のみ

   Check whether anything monitors, from outside, the path users actually take (DNS → LB → app → DB). If not, this is the top-priority addition.

3. 3

   Inventory alarms

   参照のみ

   Use `describe-alarms` to find ones that are disabled, have an empty notification destination, or have been `INSUFFICIENT_DATA` for a long time.

4. 4

   Count how many alerts fired in the past

   参照のみ

   Count how many fired in the last month and how many of those actually required a response. A high no-response-needed ratio points to threshold design as the cause.

5. 5

   Review how past incidents were actually detected

   参照のみ

   For recent incidents, check whether monitoring caught it first or a user report came first. Any instance of the latter pinpoints a monitoring gap.

6. 6

   Write out the escalation path

   参照のみ

   Document the first responder, the second responder, a fallback when someone cannot be reached, and who is responsible when a judgment call is needed.

7. 7

   Add alarms

   低

   Create alarms for any gaps with `put-metric-alarm`. Prepare the notification destination and runbook before creating one.

対応方法

バージョン・環境による違い

これで解決しない場合に確認すること

• Test whether notifications actually arrive

  Send a test notification and confirm it reaches the recipient, including overnight hours. Even when the configuration is correct, it can be dropped by a filter on the receiving end.

• Check for resources that fell through the cracks of monitoring

  Periodically check whether newly created instances are included in the monitoring setup. Automating alarm creation at instance-creation time is effective.

• Feed past incidents back into the monitoring items

  For each incident that occurred, verify whether a given monitoring item would have caught it sooner, and add whatever was missing.

• Periodically review how many alerts have fired

  If many alerts require no action, review the thresholds or the monitoring items themselves. Use the trend in the count as a metric in its own right.

• Check whether the runbook content still matches reality

  If the runbook isn't updated after a configuration change, the procedure drifts from reality. Assign ownership for periodic updates.

この文書の根拠と限界

よくある質問

この文書がカバーする質問

• How to prevent customers from discovering an incident first
• Want to know what metrics to watch for database monitoring
• Too many alerts to handle
• How to build a structure for overnight/holiday incident response

リスク表示の意味

• 参照のみデータと設定を変更しません。
• 低影響は限定的ですが、権限と負荷の確認が必要です。
• 中性能・ロック・コストに影響する可能性があります。
• 高障害・データ損失・復旧作業が発生する可能性があります。
• 専門家レビュー必須本番適用前に別途レビューが必須です。

GIIPの対応範囲

Building a list of monitoring items is itself just a design task that finishes. What's hard about 24/7 monitoring is two things: having someone awake overnight and on holidays, and being able to judge what to do once an alert arrives. At GIIP, an AI agent handles the first-pass triage (cross-checking metrics, reviewing recent change history, identifying the scope of impact) across several databases on AWS and Azure and roughly 30 web services, while a human specialist makes irreversible decisions such as a restart or a failover. The idea is not to assume a human is glued to a screen 24 hours a day, but to keep at least the initial response always running.

関連サービス

• 24時間監視の対象を確認する
• 障害対応の初動を任せる
• AIインフラ運用