Amazon Linuxで残りディスク容量アラートを設定
メトリクス送信設定
CloudWatchのメトリクスをLinuxから送信して、ディスク容量が減ったらアラートメールを送信するようにする。
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html
http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/mon-scripts.html
https://aws.amazon.com/code/8720044071969977
設定対象のLinuxで作業
Metrics通知ツール本体の前に、使っているライブラリ？をインストール。
sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https
本体をダウンロード。
mkdir ~/cloudwatch/ cd ~/cloudwatch/ curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O
解凍して、aws-scripts-mon/ を /usr/local/cloudwatch/ の下に移動。
sudo mkdir /usr/local/cloudwatch sudo mv aws-scripts-mon/ /usr/local/cloudwatch/
IAMユーザーを作成して、アクセスキーを設定。
cd /usr/local/cloudwatch/aws-scripts-mon cp awscreds.template awscreds.conf vi awscreds.conf
IAM
IAMユーザーにアタッチしたポリシー。
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics",
"ec2:DescribeTags"
],
"Resource": [
"*"
]
}
]
}
↓だとダメだった。
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData",
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics",
"ec2:DescribeTags"
],
"Resource": [
"arn:aws:logs:*:*:*",
"arn:aws:cloudwatch:*:*:*"
]
}
]
}
cronに設定
cronに設定前にコマンドラインから試す。
/usr/local/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail
crontab -e
*/5 * * * * /usr/local/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail --from-cron
以上でMetrics送信の設定は完了。
動いているかの確認は、AWS Management Console の CloudWatchで
Metrics > Linux System > Filesystem, InstanceId, MountPath
残りディスク容量が減った時のアラートメールは、AWS Consoleでやる。
Alarms mail
AWS Management Console の CloudWatchに行く（インスタンスが動いているリージョン）
Alarms > Create Alarm
Search Metrics
DiskSpaceUtilization
で検索。
チェックボックスをチェックして、Name, Description, >= 80
で作成。
メール送信の機能は AWS SNS の機能を使っているらしい。
メトリクスではなくてCloudWatchにログを出力（メモ）
CloudWatchにログを出力することもできるらしい。
この手順はMetrics送信の設定の前にやったけど、これじゃなかった。
この記事の通りにやった。
http://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
2017-02-06 02:06:45,273 - cwlogs.threads - ERROR - 12442 - Thread-23 - Exception caught in <EventBatchPublisher(Thread-23, started daemon 139840766506752)>
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run
self._run()
File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1391, in _run
self._publish_event_batch()
File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1198, in _publish_event_batch
self.sequence_token = self._put_log_events(self.event_batch)
File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1236, in _put_log_events
response = self.logs_service.put_log_events(**params)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 159, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 483, in _make_api_call
operation_model, request_dict)
File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 117, in make_request
return self._send_request(request_dict, operation_model)
File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 142, in _send_request
request = self.create_request(request_dict, operation_model)
File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 126, in create_request
operation_name=operation_model.name)
File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit
return self._emit(event_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit
response = handler(**kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler
return self.sign(operation_name, request)
File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 147, in sign
auth.add_auth(request)
File "/usr/lib/python2.7/dist-packages/botocore/auth.py", line 314, in add_auth
raise NoCredentialsError
NoCredentialsError: Unable to locate credentials
sudo vi /var/log/awslogs.log
を書き換えてリスタート
sudo service awslogs restart
2017-02-06 02:10:23,622 - cwlogs.push.reader - INFO - 12842 - Thread-4 - Start reading file from 0.
2017-02-06 02:10:25,109 - cwlogs.push.publisher - WARNING - 12842 - Thread-2 - Caught exception: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::9915xxxxxxxxxx:user/mmr2host-awslogs is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:us-east-1:9915xxxxxxxxxx:log-group:/var/log/messages:log-stream:i-5ca0f181
2017-02-06 02:10:25,110 - cwlogs.threads - ERROR - 12842 - Thread-2 - Exception caught in <EventBatchPublisher(Thread-2, started daemon 139899077621504)>
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run
self._run()
File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1391, in _run
self._publish_event_batch()
File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1198, in _publish_event_batch
self.sequence_token = self._put_log_events(self.event_batch)
File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1236, in _put_log_events
response = self.logs_service.put_log_events(**params)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 159, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 494, in _make_api_call
raise ClientError(parsed_response, operation_name)
ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::9915xxxxxxxxxx:user/mmr2host-awslogs is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:us-east-1:9915xxxxxxxxxx:log-group:/var/log/messages:log-stream:i-5ca0f181
2017-02-06 02:21:13,763 - cwlogs.push.reader - INFO - 12842 - Thread-24 - Replay events end at 14798.
2017-02-06 02:21:13,763 - cwlogs.push.reader - INFO - 12842 - Thread-24 - Start reading file from 0.
2017-02-06 02:21:14,890 - cwlogs.push.publisher - WARNING - 12842 - Thread-23 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist.
2017-02-06 02:21:14,890 - cwlogs.push.batch - INFO - 12842 - Thread-23 - Creating log group /var/log/messages.
2017-02-06 02:21:15,081 - cwlogs.push.batch - INFO - 12842 - Thread-23 - Creating log stream i-5ca0f181.
2017-02-06 02:21:15,316 - cwlogs.push.publisher - INFO - 12842 - Thread-23 - Log group: /var/log/messages, log stream: i-5ca0f181, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1486264861000, 'start_position': 0L, 'end_position': 151L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1486345988000, 'start_position': 14706L, 'end_position': 14798L}, 'source_id': '9eb733a4a488d353c22b89549724bc1e', 'num_of_events': 159, 'batch_size_in_bytes': 18773}
2017-02-06 02:21:19,437 - cwlogs.push.publisher - INFO - 12842 - Thread-23 - Log group: /var/log/messages, log stream: i-5ca0f181, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1486347485000, 'start_position': 14798L, 'end_position': 14900L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1486347485000, 'start_position': 14984L, 'end_position': 15076L}, 'source_id': '9eb733a4a488d353c22b89549724bc1e', 'num_of_events': 3, 'batch_size_in_bytes': 353}
ログの出力場所
正しく設定できているはずなのに、CloudWatchに出てこないと思ったら、
us-east-1 (US East (N. Virginia)) に出来ていた。
監視対象は us-west-2 (US West (Oregon))にあるのに。
yumで入れた設定ファイルの初期値で region = us-east-1 と書いてある。
[plugins] cwlogs = cwlogs [default] region = us-east-1
