Ansible --start-at-task
Ansible には –start-at-task という、
指定したタスクから開始するオプションがあるけどうまく動かない。
https://github.com/ansible/ansible/issues/11692
deploy% touch ansible.cfg
deploy% vi ansible.cfg
[defaults] task_includes_static = True handler_includes_static = True
% ansible-playbook -i inventory/hosts/localvm playbook/deploy.yml --start-at-task="Display all variables/facts known for a host"
で途中から実行できた。
TASK [carlosbuenosvinos.ansistrano-deploy : Display all variables/facts known for a host] *** ok: [192.168.3.43] => { "hostvars[inventory_hostname]": { "ansible_all_ipv4_addresses": [ "192.168.3.43" ...
メモ
https://github.com/ansible/ansible/issues/4561
roleを指定しているとダメだというチケットがあったので、
| で区切って指定したけど動かなかった。
–start-at-task=“carlosbuenosvinos.ansistrano-deploy : Display all variables/facts known for a host”
–start-at-task=“carlosbuenosvinos.ansistrano-deploy | Display all variables/facts known for a host”
ansibleを2.1.0 から ansible-2.2.1.0 にしてもansible.cfgを書かないと–start-at-taskは動かなかった。
ansible をアップグレードしようとしたらエラー
pip install –upgrade ansible でエラー。
pip install --upgrade ansible Collecting ansible Downloading ansible-2.2.1.0.tar.gz (2.5MB) 100% |################################| 2.5MB 404kB/s build/temp.macosx-10.10-x86_64-2.7/_openssl.c:1107:27: error: unknown type name 'EVP_PKEY_CTX' int EVP_PKEY_encrypt_init(EVP_PKEY_CTX *); ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. error: command 'clang' failed with exit status 1 ---------------------------------------- Failed building wheel for cryptography Running setup.py clean for cryptography creating build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7 clang -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kubotad/.pyenv/versions/2.7.9/include/python2.7 -c build/temp.macosx-10.10-x86_64-2.7/_openssl.c -o build/temp.macosx-10.10-x86_64-2.7/build/temp.macosx-10.10-x86_64-2.7/_openssl.o build/temp.macosx-10.10-x86_64-2.7/_openssl.c:680:30: error: expected function body after function declarator typedef LHASH_OF(CONF_VALUE) Cryptography_LHASH_OF_CONF_VALUE; ^ build/temp.macosx-10.10-x86_64-2.7/_openssl.c:886:5: error: expected ')' DH *DHparams_dup(DH *); ^ /usr/include/openssl/dh.h:164:25: note: expanded from macro 'DHparams_dup' #define DHparams_dup(x) ASN1_dup_of_const(DH,i2d_DHparams,d2i_DHparams,x) ^ Command "/Users/kubotad/.pyenv/versions/app1/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip-build-UehdPg/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip-bUFtd6-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/kubotad/.pyenv/versions/app1/include/site/python2.7/cryptography" failed with error code 1 in /private/var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip-build-UehdPg/cryptography/
↑このコマンドは、後で気づいたけど
/Users/kubotad/PycharmProjects/app1/.python-version
というファイルがある場所で実行した。
なので、pyenvの中のpythonバージョンで実行されている。
sudoでやってみても失敗。
sudo pip install --upgrade ansible Password: The directory '/Users/kubotad/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. The directory '/Users/kubotad/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. ^ build/temp.macosx-10.10-x86_64-2.7/_openssl.c:1107:27: error: unknown type name 'EVP_PKEY_CTX' int EVP_PKEY_encrypt_init(EVP_PKEY_CTX *); ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. error: command 'clang' failed with exit status 1 ---------------------------------------- Command "/Users/kubotad/.pyenv/versions/app1/bin/python2.7 -u -c "import setuptools, tokenize;__file__='/private/tmp/pip-build-E79wWq/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-wAEgKs-record/install-record.txt --single-version-externally-managed --compile --install-headers /Users/kubotad/.pyenv/versions/app1/include/site/python2.7/cryptography" failed with error code 1 in /private/tmp/pip-build-E79wWq/cryptography/
↓このコマンドは多分ホームディレクトリで実行した。
% pip install --upgrade ansible Downloading/unpacking ansible Downloading ansible-2.2.1.0.tar.gz (2.5MB): 2.5MB downloaded Running setup.py (path:/private/var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip_build_kubotad/ansible/setup.py) egg_info for package ansible changing mode of /Users/kubotad/.pyenv/versions/2.7.9/bin/ansible-pull to 755 changing mode of /Users/kubotad/.pyenv/versions/2.7.9/bin/ansible-vault to 755 Running setup.py install for PyYAML checking if libyaml is compilable clang -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/kubotad/.pyenv/versions/2.7.9/include/python2.7 -c build/temp.macosx-10.10-x86_64-2.7/check_libyaml.c -o build/temp.macosx-10.10-x86_64-2.7/check_libyaml.o build/temp.macosx-10.10-x86_64-2.7/check_libyaml.c:2:10: fatal error: 'yaml.h' file not found #include <yaml.h> ^ 1 error generated. libyaml is not found or a compiler error: forcing --without-libyaml (if libyaml is installed correctly, you may need to specify the option --include-dirs or uncomment and modify the parameter include_dirs in setup.cfg) Found existing installation: setuptools 7.0 Uninstalling setuptools: Successfully uninstalled setuptools Running setup.py install for pycrypto Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/setuptools/__init__.py", line 10, in <module> from six.moves import filter, map ImportError: No module named six.moves Complete output from command /Users/kubotad/.pyenv/versions/2.7.9/bin/python2.7 -c "import setuptools, tokenize;__file__='/private/var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip_build_kubotad/pycrypto/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip-ijhgF2-record/install-record.txt --single-version-externally-managed --compile: Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/setuptools/__init__.py", line 10, in <module> from six.moves import filter, map ImportError: No module named six.moves ---------------------------------------- Cleaning up... Command /Users/kubotad/.pyenv/versions/2.7.9/bin/python2.7 -c "import setuptools, tokenize;__file__='/private/var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip_build_kubotad/pycrypto/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip-ijhgF2-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /private/var/folders/1s/14vkt60n019bjctmy0rvg6cm0000gn/T/pip_build_kubotad/pycrypto Storing debug log for failure in /Users/kubotad/.pip/pip.log
~% pip install six Downloading/unpacking six Downloading six-1.10.0-py2.py3-none-any.whl Installing collected packages: six Successfully installed six Cleaning up...
~% pip install setuptools Requirement already satisfied (use --upgrade to upgrade): setuptools in ./.pyenv/versions/2.7.9/lib/python2.7/site-packages Cleaning up... [kubotad@Kubotads-iMac](19:34:07)~%
Ansibleの修復
関連パッケージを変えたので試しに、ansible のバージョンを確認したらエラー。壊れた。
~% ansible --version ERROR! Unexpected Exception: No module named markupsafe the full traceback was: Traceback (most recent call last): File "/Users/kubotad/.pyenv/versions/2.7.9/bin/ansible", line 80, in <module> from ansible.cli.adhoc import AdHocCLI as mycli File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/ansible/cli/adhoc.py", line 28, in <module> from ansible.executor.task_queue_manager import TaskQueueManager File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/ansible/executor/task_queue_manager.py", line 29, in <module> from ansible.executor.play_iterator import PlayIterator File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/ansible/executor/play_iterator.py", line 30, in <module> from ansible.playbook.block import Block File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/ansible/playbook/__init__.py", line 27, in <module> from ansible.playbook.play import Play File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/ansible/playbook/play.py", line 27, in <module> from ansible.playbook.base import Base File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/ansible/playbook/base.py", line 29, in <module> from jinja2.exceptions import UndefinedError File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/jinja2/__init__.py", line 33, in <module> from jinja2.environment import Environment, Template File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/jinja2/environment.py", line 13, in <module> from jinja2 import nodes File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/jinja2/nodes.py", line 19, in <module> from jinja2.utils import Markup File "/Users/kubotad/.pyenv/versions/2.7.9/lib/python2.7/site-packages/jinja2/utils.py", line 531, in <module> from markupsafe import Markup, escape, soft_unicode ImportError: No module named markupsafe [kubotad@Kubotads-iMac](19:34:36)~%
% pip install markupsafe Downloading/unpacking markupsafe Downloading MarkupSafe-0.23.tar.gz Cleaning up... setuptools must be installed to install from a source distribution Storing debug log for failure in /Users/kubotad/.pip/pip.log
% pip install -U pip Downloading/unpacking pip from https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144 Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB): 1.3MB downloaded Installing collected packages: pip Found existing installation: pip 1.5.6 Uninstalling pip: Successfully uninstalled pip Successfully installed pip Cleaning up... [kubotad@Kubotads-iMac](19:39:23)~% [kubotad@Kubotads-iMac](19:39:24)~% [kubotad@Kubotads-iMac](19:39:24)~% pip install setuptools Requirement already satisfied: setuptools in ./.pyenv/versions/2.7.9/lib/python2.7/site-packages Requirement already satisfied: six>=1.6.0 in ./.pyenv/versions/2.7.9/lib/python2.7/site-packages (from setuptools) Collecting appdirs>=1.4.0 (from setuptools) Using cached appdirs-1.4.0-py2.py3-none-any.whl Collecting packaging>=16.8 (from setuptools) Using cached packaging-16.8-py2.py3-none-any.whl Collecting pyparsing (from packaging>=16.8->setuptools) Using cached pyparsing-2.1.10-py2.py3-none-any.whl Installing collected packages: appdirs, pyparsing, packaging Successfully installed appdirs-1.4.0 packaging-16.8 pyparsing-2.1.10
markupsafe がインストールできた。
~% pip install markupsafe Collecting markupsafe Installing collected packages: markupsafe Successfully installed markupsafe-0.23
直った。
% ansible --version ansible 2.2.1.0 config file = configured module search path = Default w/o overrides
app1 env
pyenv に入っているpythonでは、まだAnsibleが2.1.0.0のままだった。
pip install --upgrade cryptography
を実行するとエラーになる。
% pip install -U pip Requirement already up-to-date: pip in /Users/kubotad/.pyenv/versions/app1/lib/python2.7/site-packages
ansible==2.2.1.0 の環境
% pip freeze ansible==2.2.1.0 appdirs==1.4.0 Jinja2==2.8.1 MarkupSafe==0.23 packaging==16.8 paramiko==2.1.1 pyparsing==2.1.10 PyYAML==3.12 six==1.10.0 virtualenv==15.0.2
ansible==2.1.0.0 の環境
% pip freeze amqp==1.4.9 ansible==2.1.0.0 anyjson==0.3.3 appdirs==1.4.0 billiard==3.3.0.23 boto==2.41.0 bpmappers==0.8.2 celery==3.1.23 cffi==1.9.1 Django==1.8.13 django-celery==3.1.17 django-kombu==0.9.4 ecdsa==0.13 enum34==1.1.6 Fabric==1.11.1 idna==2.2 ipaddress==1.0.18 Jinja2==2.8.1 kombu==3.0.35 MarkupSafe==0.23 packaging==16.8 paramiko==1.17.1 psycopg2==2.6.2 pyasn1==0.2.2 pycparser==2.17 pycrypto==2.6.1 pyparsing==2.1.10 pytz==2016.4 PyYAML==3.12 redis==2.10.5 six==1.10.0
openssl
openssl version
でバージョンを確認したら、0.9.8 だった。
upgradeしようとしてもすでにインストール済みというメッセージがでる。
http://stackoverflow.com/questions/15185661/update-openssl-on-os-x-with-homebrew
% brew link openssl --force Linking /usr/local/Cellar/openssl/1.0.2h_1... 1601 symlinks created
opensslをアップグレードしたら、cryptography がインストールできるようになった。
pip install --upgrade cryptography Successfully built cryptography Installing collected packages: cryptography Successfully installed cryptography-1.7.2
ansible
% pip install --upgrade ansible Collecting ansible Installing collected packages: paramiko, ansible Found existing installation: paramiko 1.17.1 Uninstalling paramiko-1.17.1: Successfully uninstalled paramiko-1.17.1 Found existing installation: ansible 2.1.0.0 Uninstalling ansible-2.1.0.0: Successfully uninstalled ansible-2.1.0.0 Successfully installed ansible-2.2.1.0 paramiko-2.1.1
うまくいった。
Amazon Linuxで残りディスク容量アラートを設定
メトリクス送信設定
CloudWatchのメトリクスをLinuxから送信して、ディスク容量が減ったらアラートメールを送信するようにする。
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/mon-scripts.html
http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/mon-scripts.html
https://aws.amazon.com/code/8720044071969977
設定対象のLinuxで作業
Metrics通知ツール本体の前に、使っているライブラリ?をインストール。
sudo yum install perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https
本体をダウンロード。
mkdir ~/cloudwatch/ cd ~/cloudwatch/ curl http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip -O
解凍して、aws-scripts-mon/ を /usr/local/cloudwatch/ の下に移動。
sudo mkdir /usr/local/cloudwatch sudo mv aws-scripts-mon/ /usr/local/cloudwatch/
IAMユーザーを作成して、アクセスキーを設定。
cd /usr/local/cloudwatch/aws-scripts-mon cp awscreds.template awscreds.conf vi awscreds.conf
IAM
IAMユーザーにアタッチしたポリシー。
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "ec2:DescribeTags" ], "Resource": [ "*" ] } ] }
↓だとダメだった。
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "cloudwatch:PutMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "ec2:DescribeTags" ], "Resource": [ "arn:aws:logs:*:*:*", "arn:aws:cloudwatch:*:*:*" ] } ] }
cronに設定
cronに設定前にコマンドラインから試す。
/usr/local/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail
crontab -e
*/5 * * * * /usr/local/cloudwatch/aws-scripts-mon/mon-put-instance-data.pl --disk-path=/ --disk-space-util --disk-space-used --disk-space-avail --from-cron
以上でMetrics送信の設定は完了。
動いているかの確認は、AWS Management Console の CloudWatchで
Metrics > Linux System > Filesystem, InstanceId, MountPath
残りディスク容量が減った時のアラートメールは、AWS Consoleでやる。
Alarms mail
AWS Management Console の CloudWatchに行く(インスタンスが動いているリージョン)
Alarms > Create Alarm
Search Metrics
DiskSpaceUtilization
で検索。
チェックボックスをチェックして、Name, Description, >= 80
で作成。
メール送信の機能は AWS SNS の機能を使っているらしい。
メトリクスではなくてCloudWatchにログを出力(メモ)
CloudWatchにログを出力することもできるらしい。
この手順はMetrics送信の設定の前にやったけど、これじゃなかった。
この記事の通りにやった。
http://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/logs/QuickStartEC2Instance.html
2017-02-06 02:06:45,273 - cwlogs.threads - ERROR - 12442 - Thread-23 - Exception caught in <EventBatchPublisher(Thread-23, started daemon 139840766506752)> Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run self._run() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1391, in _run self._publish_event_batch() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1198, in _publish_event_batch self.sequence_token = self._put_log_events(self.event_batch) File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1236, in _put_log_events response = self.logs_service.put_log_events(**params) File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 159, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 483, in _make_api_call operation_model, request_dict) File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 117, in make_request return self._send_request(request_dict, operation_model) File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 142, in _send_request request = self.create_request(request_dict, operation_model) File "/usr/lib/python2.7/dist-packages/botocore/endpoint.py", line 126, in create_request operation_name=operation_model.name) File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 227, in emit return self._emit(event_name, kwargs) File "/usr/lib/python2.7/dist-packages/botocore/hooks.py", line 210, in _emit response = handler(**kwargs) File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 90, in handler return self.sign(operation_name, request) File "/usr/lib/python2.7/dist-packages/botocore/signers.py", line 147, in sign auth.add_auth(request) File "/usr/lib/python2.7/dist-packages/botocore/auth.py", line 314, in add_auth raise NoCredentialsError NoCredentialsError: Unable to locate credentials
sudo vi /var/log/awslogs.log
を書き換えてリスタート
sudo service awslogs restart
2017-02-06 02:10:23,622 - cwlogs.push.reader - INFO - 12842 - Thread-4 - Start reading file from 0. 2017-02-06 02:10:25,109 - cwlogs.push.publisher - WARNING - 12842 - Thread-2 - Caught exception: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::9915xxxxxxxxxx:user/mmr2host-awslogs is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:us-east-1:9915xxxxxxxxxx:log-group:/var/log/messages:log-stream:i-5ca0f181 2017-02-06 02:10:25,110 - cwlogs.threads - ERROR - 12842 - Thread-2 - Exception caught in <EventBatchPublisher(Thread-2, started daemon 139899077621504)> Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cwlogs/threads.py", line 58, in run self._run() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1391, in _run self._publish_event_batch() File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1198, in _publish_event_batch self.sequence_token = self._put_log_events(self.event_batch) File "/usr/lib/python2.7/dist-packages/cwlogs/push.py", line 1236, in _put_log_events response = self.logs_service.put_log_events(**params) File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 159, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/lib/python2.7/dist-packages/botocore/client.py", line 494, in _make_api_call raise ClientError(parsed_response, operation_name) ClientError: An error occurred (AccessDeniedException) when calling the PutLogEvents operation: User: arn:aws:iam::9915xxxxxxxxxx:user/mmr2host-awslogs is not authorized to perform: logs:PutLogEvents on resource: arn:aws:logs:us-east-1:9915xxxxxxxxxx:log-group:/var/log/messages:log-stream:i-5ca0f181
2017-02-06 02:21:13,763 - cwlogs.push.reader - INFO - 12842 - Thread-24 - Replay events end at 14798. 2017-02-06 02:21:13,763 - cwlogs.push.reader - INFO - 12842 - Thread-24 - Start reading file from 0. 2017-02-06 02:21:14,890 - cwlogs.push.publisher - WARNING - 12842 - Thread-23 - Caught exception: An error occurred (ResourceNotFoundException) when calling the PutLogEvents operation: The specified log group does not exist. 2017-02-06 02:21:14,890 - cwlogs.push.batch - INFO - 12842 - Thread-23 - Creating log group /var/log/messages. 2017-02-06 02:21:15,081 - cwlogs.push.batch - INFO - 12842 - Thread-23 - Creating log stream i-5ca0f181. 2017-02-06 02:21:15,316 - cwlogs.push.publisher - INFO - 12842 - Thread-23 - Log group: /var/log/messages, log stream: i-5ca0f181, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1486264861000, 'start_position': 0L, 'end_position': 151L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1486345988000, 'start_position': 14706L, 'end_position': 14798L}, 'source_id': '9eb733a4a488d353c22b89549724bc1e', 'num_of_events': 159, 'batch_size_in_bytes': 18773} 2017-02-06 02:21:19,437 - cwlogs.push.publisher - INFO - 12842 - Thread-23 - Log group: /var/log/messages, log stream: i-5ca0f181, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1486347485000, 'start_position': 14798L, 'end_position': 14900L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1486347485000, 'start_position': 14984L, 'end_position': 15076L}, 'source_id': '9eb733a4a488d353c22b89549724bc1e', 'num_of_events': 3, 'batch_size_in_bytes': 353}
ログの出力場所
正しく設定できているはずなのに、CloudWatchに出てこないと思ったら、
us-east-1 (US East (N. Virginia)) に出来ていた。
監視対象は us-west-2 (US West (Oregon))にあるのに。
yumで入れた設定ファイルの初期値で region = us-east-1 と書いてある。
[plugins] cwlogs = cwlogs [default] region = us-east-1
PostgreSQLのリプリケーションをしてみるテスト
準備
postgresql94
AWS上にt1.smallでPostgresqlが動いているAmazon Linuxを2台構築。
Amazon Linux
$ cat /etc/os-release NAME="Amazon Linux AMI" VERSION="2016.09" ID="amzn" ID_LIKE="rhel fedora" VERSION_ID="2016.09" PRETTY_NAME="Amazon Linux AMI 2016.09" ANSI_COLOR="0;33" CPE_NAME="cpe:/o:amazon:linux:2016.09:ga" HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
参考にしたウェブサイト
PostgreSQL レプリケーション構成を組んでみる - tom__bo’s Blog
PostgreSQL9.4でストリーミングレプリケーションする - マコーの日記
エラー1
-bash-4.2$ pg_basebackup -h 172.40.24.192 -p 5432 -D /var/lib/pgsql94/data --xlog --progress --verbose pg_basebackup: could not connect to server: could not connect to server: Connection timed out Is the server running on host "172.40.24.192" and accepting TCP/IP connections on port 5432?
AWS の security group に自分自身のsgを許可(5432)
エラー2
-bash-4.2$ pg_basebackup -h 172.40.24.192 -p 5432 -D /var/lib/pgsql94/data --xlog --progress --verbose pg_basebackup: could not connect to server: FATAL: no pg_hba.conf entry for replication connection from host "172.40.135.122", user "postgres", SSL off
Postgresql doesn't accept replication connection - Database Administrators Stack Exchange
-bash-4.2$ diff pg_hba.conf pg_hba.conf.2017203 85,86d84 < host all all 172.40.135.122/32 trust < host replication postgres 172.40.135.122/32 trust
pg_basebackup が動いた。
-bash-4.2$ pg_basebackup -h 172.40.24.192 -p 5432 -D /var/lib/pgsql94/data --xlog --progress --verbose transaction log start point: 0/E000028 on timeline 1 51646/51646 kB (100%), 1/1 tablespace transaction log end point: 0/E000128 pg_basebackup: base backup completed
メモ
Amazon Linux の場合、
recovery.conf.sample
は
/usr/share/pgsql94/recovery.conf.sample
にあった。
エラー3
INSERT文が止まる。
アプリも動かない(nginx 504 error: Gateway Timeout)
app1=> select application_name, client_addr, backend_start, state, sync_state from pg_stat_replication; application_name | client_addr | backend_start | state | sync_state ------------------+-------------+---------------+-------+------------ slave01 | | | | (1 row)
参考にしたウェブサイトと違って client_addr や state などが空になっていた。
ただ、この状態はエラーの原因ではなかった。
psql から INSERTして応答がなかったので、controll + c した時のメッセージ。
Cancel request sent WARNING: canceling wait for synchronous replication due to user request DETAIL: The transaction has already committed locally, but might not have been replicated to the standby. INSERT 0 1
このエラーは、
master db の postgresql.conf に書く、synchronous_standby_names の値と
slave db の recovery.conf に書く、primary_conninfo の中の値が違っていると発生するっぽい。
レコード自体は作られていた。
postgresql.conf in master db. synchronous_standby_names = 'slave01' # standby servers that provide sync rep recovery.conf in slave db. primary_conninfo = 'host=172.40.24.192 port=5432 application_name=slave01'
cronで作成日時から一定期間過ぎたファイルを削除
メモ:
cronで作成日時から一定期間過ぎたファイルを削除
MAILTO="" # delete files older than 3 days # 1 3,15 * * * find /var/www/app1/shared/docs/ -maxdepth 2 -type f -ctime +3 -delete 1 3,15 * * * find /var/www/app1/shared/docs/input/ -maxdepth 1 -type f -ctime +3 -delete 5 3,15 * * * find /var/www/app1/shared/docs/output/ -maxdepth 1 -type f -cmin +720 -delete
ansistranoのafter_symlinkでsudo(become)を実行する
ansible の YAML
ansible ansistrano の after_symlink hook で
/etc/init.d/
にシンボリックリンクを作る。
- name: initd symlink become: true file: state: link src: /var/www/app1/current/deploy/initd/app1_uwsgi #dest: /home/user1/app1_uwsgi dest: /etc/init.d/app1_uwsgi mode: "u=rwx,g=rx,o=rx"
sudoをするにはbecomeを使うと書いてあったけど、
指定の方法が分からなかった。
“配列の中にハッシュをネスト” という方法で書かないといけないっぽい。
http://docs.ansible.com/ansible/become.html
http://www.task-notes.com/entry/20150922/1442890800
ansible-playbook -i inventory/hosts/localvm playbook/deploy.yml -K
ansible-playbook コマンドの最後に -K を付けると最初にsudo passwordを聞かれる。
Ansible, sudoパスワード要求を忘れただけでめんどくなる - Goldstine研究所
エラーメッセージ
その1
fatal: [192.168.3.43]: FAILED! => {"failed": true, "reason": "no action detected in task. This often indicates a misspelled module name, or incorrect module path.\n\nThe error appears to have been in '/Users/aaa/PycharmProjects/app1/deploy/playbook/ansistrano/after_symlink.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- tasks:\n ^ here\n\n\nThe error appears to have been in '/Users/aaa/PycharmProjects/app1/deploy/playbook/ansistrano/after_symlink.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- tasks:\n ^ here\n"}
その2
fatal: [192.168.3.43]: FAILED! => {"failed": true, "reason": "Syntax Error while loading YAML.\n\n\nThe error appears to have been in '/Users/aaa/PycharmProjects/app1/deploy/playbook/ansistrano/after_symlink.yml': line 4, column 18, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n- name: create django static files\n django_manage:\n ^ here\n"}
完成品
PycharmProjects/app1/deploy/playbook/ansistrano/after_symlink.yml
- name: create django static files django_manage: command: collectstatic app_path: "{{ ansistrano_deploy_to }}/current/" virtualenv: "/home/user1/.pyenv/versions/app1/" - name: init.d app1_uwsgi symlink become: true file: state: link src: /var/www/app1/current/deploy/initd/app1_uwsgi #dest: /home/user1/app1_uwsgi dest: /etc/init.d/app1_uwsgi mode: "u=rwx,g=rx,o=rx" - name: init.d app1_celeryd symlink become: true file: state: link src: /var/www/app1/current/deploy/initd/app1_celeryd dest: /etc/init.d/app1_celeryd mode: "u=rwx,g=rx,o=rx" - name: restart app1_uwsgi become: true service: name: app1_uwsgi state: restarted - name: restart app1_celeryd become: true service: name: app1_celeryd state: restarted
lsのソースコードを読む 2017
lsのソースコードを読んでみたくなったのでメモ。
lsコマンドをハックしてみよう - Yahoo! JAPAN Tech Blog
https://svnweb.freebsd.org/base/head/bin/ls/
当初の目標
- Visual Studio Codeでステップ実行
- ソースを書き換えてファイル名がabc.txtのファイルは表示しないようにする
色々試してMacOS Sierra 10.12.2 とxcodeでビルドとステップ実行ができたので、
Visual Studio Codeじゃなくてxcodeにした。
ファイルの取得
MacOSのlsコマンドのソースコードを取得する。
自分のMac(Sierra 10.12.2)
file_cmds-242.tar.gz
をダウンロード。
https://opensource.apple.com/tarballs/file_cmds/file_cmds-264.30.2.tar.gz
後で、
libutil-47.30.1
と
Libinfo-503.30.1
も必要になった。
xcode
AppStoreから無料ダウンロード。
ls ビルド設定 (Build Settings)
Project navigator トップ(root)の file_cmds をクリック。
2段目のfile_cmdsをクリックしてlsを選択。
Debug構成
Project navigator トップ(root)の file_cmds をクリック。
2段目でfile_cmdsを選択。
Info > Configurations
で + をクリックして、Duplicate
Debugという構成を作成。
optimization
エラー
何回かやっていたら以下のエラーが出た。
The run destination My Mac is not valid for Running
Build Settings
Base SDKをmacOS 10.12
に変更。
ClipStudioPluginSDKのビルドが出来ない[mac] - Qiita
membershipPriv.h not found
ビルドしたらエラーが出た。
/Users/kubotad/codes/ls/file_cmds-264.30.2/ls/print.c:56:10: 'membershipPriv.h' file not found
EI Capitanでsudo付けているOperation not permittedが出た時の対処法 - いつかエンジニアになりたい
https://opensource.apple.com/release/macos-10122.html
の
libutil-47.30.1
と
Libinfo-503.30.1
をダウンロード。
解凍
http://www.ict-fractal.com/blog/2012/07/xcode.html
Search Pathsに追記
2.Build Settingsの一覧が表示されますので、Search PathsカテゴリのHeader Search Pathsという項目を見つけて下さい。 3. Header Search Pathsをダブルクリックすると項目の編集画面が開きます。 編集画面で+ボタンをクリックすると項目が追加されますので、そこにパスを入力して下さい。 ※パスはダブルクォーテーションで括りましょう。
Header Search Path
"../Libinfo-503.30.1/membership.subproj"
"../libutil-47.30.1"
How do I use a relative path in Xcode project settings? - Stack Overflow
実行ファイル出力先
Project navigator
に Products というディレクトリ的なものが表示されているけど実物はない。
詳細の Full Pathに以下のPathと書いてあった。
~/Library/Developer/Xcode/DerivedData/file_cmds-bocccevvdifphpdyuxszwxcbgtxo/Build/Products/Release
実行時引数
http://stackoverflow.com/questions/1698830/giving-command-line-arguments-in-xcode-in-c-program
Keyboard shortcut is command shift <
メニューの
Product > Scheme > Edit Scheme...
ステップ実行のためのoptimization変更
ls was compiled with optimization - stepping may behave oddly; variables may not be available.
Optimization Level
BasicじゃなくてAllにしないと表示されない。
メニューの
Product > Scheme > Edit Scheme...
で
Run > Info > Build Configuration
をDebugに変更。
ls was compiled with optimization
のメッセージが消えた。
ソースコード書き換え
Linked lists - Learn C - Free Interactive C Tutorial
// kubotti customize FTSENT *cur_temp = NULL; for (cur = list; cur; cur = cur->fts_link) { if (!strcmp(cur->fts_name, "kubotti.txt")){ if(cur_temp == NULL){ list = cur->fts_link; //free(cur); }else{ cur_temp->fts_link = cur->fts_link; } } cur_temp = cur; }
code modification. not display if the name is kubotti.txt · kubotti/ls_cmd_mod@816e9a6 · GitHub
動作確認
~/Library/Developer/Xcode/DerivedData/file_cmds-boccc ....
にできたlsの実行ファイルを適当なディレクトリにコピー。
./ls /tmp/dir1
で実行。
kubotti.txt というファイルは表示しないlsコマンドができた。
以上、終了。
(メモ)Visual Studio Code
最初にVisual Studio Codeを試した時のメモ。
.tar.gzを解凍して、Visual Studio Codeで開いた。
C/C++ extensionをインストールすることを推奨 と出たので、インストール。
ms-vscode.cpptools publisher:"Microsoft"
launch.json
"program": "enter program name, for example ${workspaceRoot}/a.out",
ソースの取得
https://opensource.apple.com/release/os-x-10105.html
file_cmds-242.tar.gz (Yosemite 10.10.5)
を解凍したけど、
にMakefileがない。
https://opensource.apple.com/source/file_cmds/file_cmds-188/ls/Makefile にはあるけど。
https://opensource.apple.com/source/file_cmds/file_cmds-212/ がMakefileがある中で一番数字が大きい。
ls/mac212% make Makefile:16: /Library/Developer/CommandLineTools/Makefiles/CoreOS/ReleaseControl/BSDCommon.make: No such file or directory make: *** No rule to make target `/Library/Developer/CommandLineTools/Makefiles/CoreOS/ReleaseControl/BSDCommon.make'. Stop.