Amazon Linuxで残りディスク容量アラートを設定

メトリクス送信設定

CloudWatchのメトリクスをLinuxから送信して、ディスク容量が減ったらアラートメールを送信するようにする。

Amazon Linux

http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html

http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/mon-scripts.html

https://aws.amazon.com/code/8720044071969977

設定対象のLinuxで作業

Metrics通知ツール本体の前に、使っているライブラリ?をインストール。

sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https

本体をダウンロード。

mkdir ~/cloudwatch/
cd ~/cloudwatch/
curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O

解凍して、aws-scripts-mon/ を /usr/local/cloudwatch/ の下に移動。

sudo mkdir /usr/local/cloudwatch
sudo mv aws-scripts-mon/ /usr/local/cloudwatch/

IAMユーザーを作成して、アクセスキーを設定。

cd /usr/local/cloudwatch/aws-scripts-mon
cp awscreds.template awscreds.conf
vi awscreds.conf

IAM

IAMユーザーにアタッチしたポリシー。

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudwatch:PutMetricData",
                "cloudwatch:GetMetricStatistics",
                "cloudwatch:ListMetrics",
                "ec2:DescribeTags"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

↓だとダメだった。

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "cloudwatch:PutMetricData",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "ec2:DescribeTags"
    ],
      "Resource": [
        "arn:aws:logs:*:*:*",
        "arn:aws:cloudwatch:*:*:*"        
    ]
  }
 ]
}

cronに設定

cronに設定前にコマンドラインから試す。

/usr/local/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail

crontab -e

*/5 * * * * /usr/local/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail --from-cron

以上でMetrics送信の設定は完了。
動いているかの確認は、AWS Management Console の CloudWatchで

Metrics > Linux System > Filesystem, InstanceId, MountPath

残りディスク容量が減った時のアラートメールは、AWS Consoleでやる。

Alarms mail

AWS Management Console の CloudWatchに行く(インスタンスが動いているリージョン)

Alarms > Create Alarm
Search Metrics

DiskSpaceUtilization

で検索。
チェックボックスをチェックして、Name, Description, >= 80
で作成。
メール送信の機能は AWS SNS の機能を使っているらしい。

メトリクスではなくてCloudWatchにログを出力(メモ)

CloudWatchにログを出力することもできるらしい。
この手順はMetrics送信の設定の前にやったけど、これじゃなかった。

この記事の通りにやった。
http://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html

2017-02-06 02:06:45,273 - cwlogs.threads - ERROR - 12442 - Thread-23 - Exception caught in <EventBatchPublisher(Thread-23, started daemon 139840766506752)>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run
    self._run()
  File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1391, in _run
    self._publish_event_batch()
  File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1198, in _publish_event_batch
    self.sequence_token = self._put_log_events(self.event_batch)
  File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1236, in _put_log_events
    response = self.logs_service.put_log_events(**params)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 159, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 483, in _make_api_call
    operation_model, request_dict)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 117, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 142, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 126, in create_request
    operation_name=operation_model.name)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
    return self._emit(event_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
    response = handler(**kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 147, in sign
    auth.add_auth(request)
  File "/usr/lib/python2.7/dist-packages/botocore/auth.py", line 314, in add_auth
    raise NoCredentialsError
NoCredentialsError: Unable to locate credentials

sudo vi /var/log/awslogs.log
を書き換えてリスタート
sudo service awslogs restart

2017-02-06 02:10:23,622 - cwlogs.push.reader - INFO - 12842 - Thread-4 - Start reading file from 0.
2017-02-06 02:10:25,109 - cwlogs.push.publisher - WARNING - 12842 - Thread-2 - Caught exception: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::9915xxxxxxxxxx:user/mmr2host-awslogs is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:us-east-1:9915xxxxxxxxxx:log-group:/var/log/messages:log-stream:i-5ca0f181
2017-02-06 02:10:25,110 - cwlogs.threads - ERROR - 12842 - Thread-2 - Exception caught in <EventBatchPublisher(Thread-2, started daemon 139899077621504)>
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run
    self._run()
  File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1391, in _run
    self._publish_event_batch()
  File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1198, in _publish_event_batch
    self.sequence_token = self._put_log_events(self.event_batch)
  File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1236, in _put_log_events
    response = self.logs_service.put_log_events(**params)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 159, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 494, in _make_api_call
    raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::9915xxxxxxxxxx:user/mmr2host-awslogs is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:us-east-1:9915xxxxxxxxxx:log-group:/var/log/messages:log-stream:i-5ca0f181
2017-02-06 02:21:13,763 - cwlogs.push.reader - INFO - 12842 - Thread-24 - Replay events end at 14798.
2017-02-06 02:21:13,763 - cwlogs.push.reader - INFO - 12842 - Thread-24 - Start reading file from 0. 
2017-02-06 02:21:14,890 - cwlogs.push.publisher - WARNING - 12842 - Thread-23 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist. 
2017-02-06 02:21:14,890 - cwlogs.push.batch - INFO - 12842 - Thread-23 - Creating log group /var/log/messages.
2017-02-06 02:21:15,081 - cwlogs.push.batch - INFO - 12842 - Thread-23 - Creating log stream i-5ca0f181.
2017-02-06 02:21:15,316 - cwlogs.push.publisher - INFO - 12842 - Thread-23 - Log group: /var/log/messages, log stream: i-5ca0f181, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1486264861000, 'start_position': 0L, 'end_position': 151L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1486345988000, 'start_position': 14706L, 'end_position': 14798L}, 'source_id': '9eb733a4a488d353c22b89549724bc1e', 'num_of_events': 159, 'batch_size_in_bytes': 18773}
2017-02-06 02:21:19,437 - cwlogs.push.publisher - INFO - 12842 - Thread-23 - Log group: /var/log/messages, log stream: i-5ca0f181, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1486347485000, 'start_position': 14798L, 'end_position': 14900L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1486347485000, 'start_position': 14984L, 'end_position': 15076L}, 'source_id': '9eb733a4a488d353c22b89549724bc1e', 'num_of_events': 3, 'batch_size_in_bytes': 353}

ログの出力場所

正しく設定できているはずなのに、CloudWatchに出てこないと思ったら、
us-east-1 (US East (N. Virginia)) に出来ていた。
監視対象は us-west-2 (US West (Oregon))にあるのに。
yumで入れた設定ファイルの初期値で region = us-east-1 と書いてある。

[plugins]
cwlogs = cwlogs
[default]
region = us-east-1