AWS Lambda エラーメモ1
作成したLambda + API Gatewayにアクセスしたら以下のエラー。
{"message":"Missing Authentication Token"}
Lambda管理画面のTriggers で、
既存のものを削除して作り直したら、エラーが出ずに正常に実行された。
エラー2
Lambda > Functions > funcName
Triggers
Add trigger
API Gatewayを選んで作成しようとしたら、以下のエラー。
There was an error creating the trigger: An integration is already present on this method
API nameは、プルダウンから選べるけど、そのままSubmitするとこのエラーが出るっぽい。
新しい名前で作ると、API Gatewayの管理画面で一覧に表示される。
とりあえずserverlessでpythonを試してみる
npmのインストール
なんかインストール方法が複数あるっぽいけど、
nodebrewというのを入れるのがよさげだったのでそれにしてみる。
（会社のmacにはanyenvというのが入っていた）
http://www.hirooooo-lab.com/entry/development/install-node
brew install nodebrew nodebrew ls-remote nodebrew install-binary v6.9.4
インストールに失敗した。
% nodebrew install-binary v6.9.4 Fetching: https://nodejs.org/dist/v6.9.4/node-v6.9.4-darwin-x64.tar.gz Warning: Failed to create the file Warning: /Users/kubotad/.nodebrew/src/v6.9.4/node-v6.9.4-darwin-x64.tar.gz: No Warning: such file or directory curl: (23) Failed writing body (0 != 941) download failed: https://nodejs.org/dist/v6.9.4/node-v6.9.4-darwin-x64.tar.gz
ディレクトリを作ったらうまく行った。おかしい。
mkdir -p ~/.nodebrew/src % nodebrew install-binary v6.9.4 Fetching: https://nodejs.org/dist/v6.9.4/node-v6.9.4-darwin-x64.tar.gz ######################################################################## 100.0% Installed successfully
パスを通す。
echo 'export PATH=$PATH:~/.nodebrew/current/bin' >> ~/.zshrc % source ~/.zshrc % nodebrew use v6.9.4 use v6.9.4
バージョン確認
% node -v v6.9.4 % npm -v 3.10.10
https://nodejs.org/
から.pkgをダウンロードしてインストールでもいいような気がする。
anyenvのndenvの場合
インストール可能バージョンの一覧
ndenv install -l
# install ndenv install v6.9.4 # installed list ndenv versions # switch ndenv global v6.9.4
serverlessのインストール
# Install serverless globally npm install serverless -g
% npm install serverless -g /Users/kubotad/.nodebrew/node/v6.9.4/bin/serverless -> /Users/kubotad/.nodebrew/node/v6.9.4/lib/node_modules/serverless/bin/serverless /Users/kubotad/.nodebrew/node/v6.9.4/bin/sls -> /Users/kubotad/.nodebrew/node/v6.9.4/lib/node_modules/serverless/bin/serverless /Users/kubotad/.nodebrew/node/v6.9.4/bin/slss -> /Users/kubotad/.nodebrew/node/v6.9.4/lib/node_modules/serverless/bin/serverless > serverless@1.5.0 postinstall /Users/kubotad/.nodebrew/node/v6.9.4/lib/node_modules/serverless > node ./scripts/postinstall.js /Users/kubotad/.nodebrew/node/v6.9.4/lib └─┬ serverless@1.5.0 ├─┬ agent-base@2.0.1 │ └── semver@5.0.3 ├── ansi-regex@2.0.0 ├── ansi-styles@2.2.1 ├─┬ archiver@1.2.0 │ └── async@2.1.4 ├── archiver-utils@1.3.0 ├── argparse@1.0.9 ├── async@1.5.2 ├─┬ aws-sdk@2.7.13 │ └── uuid@3.0.0 ├── balanced-match@0.4.2 ├── base64-js@1.2.0 ├─┬ bl@1.1.2 │ └── readable-stream@2.0.6 ├── bluebird@3.4.6 ├── brace-expansion@1.1.6 ├── buffer@4.9.1 ├── buffer-crc32@0.2.13 ├── buffer-shims@1.0.0 ├── capture-stack-trace@1.0.0 ├── caw@2.0.0 ├── chalk@1.1.3 ├── combined-stream@1.0.5 ├── commander@2.8.1 ├── component-emitter@1.2.1 ├── compress-commons@1.1.0 ├── concat-map@0.0.1 ├── cookiejar@2.0.6 ├── core-util-is@1.0.2 ├── crc32-stream@1.0.0 ├── create-error-class@3.0.2 ├── crypto-browserify@1.0.9 ├── debug@2.3.3 ├── decompress@4.0.0 ├── decompress-tar@4.1.0 ├── decompress-tarbz2@4.1.0 ├── decompress-targz@4.0.0 ├── decompress-unzip@4.0.1 ├── deep-extend@0.4.1 ├── delayed-stream@1.0.0 ├── download@5.0.2 ├── duplexer3@0.1.4 ├── encoding@0.1.12 ├─┬ end-of-stream@1.1.0 │ └── once@1.3.3 ├── escape-string-regexp@1.0.5 ├── esprima@2.7.3 ├── extend@3.0.0 ├── fd-slicer@1.0.1 ├── file-type@3.9.0 ├── filename-reserved-regex@1.0.0 ├── filenamify@1.2.1 ├── filesize@3.3.0 ├── form-data@1.0.0-rc3 ├── formidable@1.0.17 ├── fs-extra@0.26.7 ├── fs.realpath@1.0.0 ├── get-proxy@1.1.0 ├── get-stdin@5.0.1 ├── get-stream@2.3.1 ├── glob@7.1.1 ├── glob-all@3.1.0 ├── got@6.6.3 ├── graceful-fs@4.1.11 ├── graceful-readlink@1.0.1 ├── has-ansi@2.0.0 ├── https-proxy-agent@1.0.0 ├── iconv-lite@0.4.15 ├── ieee754@1.1.8 ├── inflight@1.0.6 ├── inherits@2.0.3 ├── ini@1.3.4 ├── is-absolute@0.1.7 ├── is-natural-number@2.1.1 ├── is-redirect@1.0.0 ├── is-relative@0.1.3 ├── is-retry-allowed@1.1.0 ├── is-stream@1.1.0 ├── isarray@1.0.0 ├── jmespath@0.15.0 ├── js-yaml@3.7.0 ├─┬ json-refs@2.1.6 │ └── commander@2.9.0 ├── jsonfile@2.4.0 ├── klaw@1.3.1 ├── lazystream@1.0.0 ├── lodash@4.17.2 ├── lowercase-keys@1.0.0 ├── methods@1.1.2 ├── mime@1.3.4 ├── mime-db@1.25.0 ├── mime-types@2.1.13 ├── minimatch@3.0.3 ├── minimist@1.2.0 ├─┬ mkdirp@0.5.1 │ └── minimist@0.0.8 ├── moment@2.17.0 ├── ms@0.7.2 ├── native-promise-only@0.8.1 ├── node-fetch@1.6.3 ├── node-status-codes@2.0.1 ├── normalize-path@2.0.1 ├── object-assign@4.1.0 ├── once@1.4.0 ├── path-is-absolute@1.0.1 ├── path-loader@1.0.1 ├── pend@1.2.0 ├── pify@2.3.0 ├── pinkie@2.0.4 ├── pinkie-promise@2.0.1 ├── prepend-http@1.0.4 ├── process-nextick-args@1.0.7 ├── punycode@1.3.2 ├── qs@2.3.3 ├── querystring@0.2.0 ├── rc@1.1.6 ├── readable-stream@2.2.2 ├── reduce-component@1.0.1 ├── replaceall@0.1.6 ├── rimraf@2.5.4 ├── sax@1.1.5 ├── seek-bzip@1.0.5 ├── semver@5.3.0 ├── semver-regex@1.0.0 ├── shelljs@0.6.1 ├── slash@1.0.0 ├── sprintf-js@1.0.3 ├── string_decoder@0.10.31 ├── strip-ansi@3.0.1 ├─┬ strip-dirs@1.1.1 │ └── get-stdin@4.0.1 ├── strip-json-comments@1.0.4 ├── strip-outer@1.0.0 ├── sum-up@1.0.3 ├─┬ superagent@1.8.4 │ ├── isarray@0.0.1 │ └── readable-stream@1.0.27-1 ├── supports-color@2.0.0 ├── tar-stream@1.5.2 ├── through@2.3.8 ├── timed-out@3.0.0 ├── traverse@0.6.6 ├── trim-repeated@1.0.0 ├── tunnel-agent@0.4.3 ├─┬ unbzip2-stream@1.0.10 │ ├── base64-js@0.0.8 │ └── buffer@3.6.0 ├── unzip-response@2.0.1 ├── uri-js@2.1.1 ├── url@0.10.3 ├── url-parse-lax@1.0.0 ├── util-deprecate@1.0.2 ├── uuid@2.0.3 ├── wrappy@1.0.2 ├── xml2js@0.4.15 ├─┬ xmlbuilder@2.6.2 │ └── lodash@3.5.0 ├── xtend@4.0.1 ├─┬ yargs@1.2.6 │ └── minimist@0.1.0 ├── yauzl@2.7.0 └── zip-stream@1.1.0
https://serverless.com/
のトップページに載っていたコマンドを入力してみる。
serverless create --template aws-nodejs % ll total 24 drwxr-xr-x 5 kubotad staff 170 1 12 14:47 ./ drwx------@ 10 kubotad staff 340 1 12 01:10 ../ -rw-r--r-- 1 kubotad staff 86 1 12 01:10 .npmignore -rw-r--r-- 1 kubotad staff 466 1 12 01:10 handler.js -rw-r--r-- 1 kubotad staff 2410 1 12 01:10 serverless.yml
とりあえずインストールできた。
デプロイ
AWS IAMでアクセスキーを作成。
https://serverless.com/framework/docs/providers/aws/guide/credentials/
serverless config credentials --provider aws --key AKIAIOSFODNN7EXAMPLE --secret wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
% serverless deploy
Error --------------------------------------------------
ServerlessError: ServerlessError: User: arn:aws:iam
is not authorized to perform: cloudformation:DescribeStackResources
https://github.com/serverless/serverless/issues/588
Serverless Error ---------------------------------------
An error occurred while provisioning your stack: ServerlessDeploymentBucket
- API: s3:CreateBucket Access Denied.
kubotad_lambda_inline_policy - Inline policy
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudformation:Describe*",
"cloudformation:List*",
"cloudformation:Get*",
"cloudformation:PreviewStackUpdate"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"cloudformation:CreateStack",
"cloudformation:UpdateStack",
"cloudformation:DeleteStack"
],
"Resource": "arn:aws:cloudformation:*:*:*"
},
{
"Effect": "Allow",
"Action": [
"lambda:Get*",
"lambda:List*",
"lambda:CreateFunction"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"lambda:AddPermission",
"lambda:CreateAlias",
"lambda:DeleteFunction",
"lambda:InvokeFunction",
"lambda:PublishVersion",
"lambda:RemovePermission",
"lambda:Update*"
],
"Resource": "arn:aws:lambda:*:*:function:${project}*"
},
{
"Effect": "Allow",
"Action": [
"iam:PassRole"
],
"Resource": "arn:aws:iam::*:role/${project}*"
},
{
"Effect": "Allow",
"Action": [
"apigateway:GET"
],
"Resource": "arn:aws:apigateway:*::/restapis"
},
{
"Effect": "Allow",
"Action": [
"apigateway:*"
],
"Resource": "arn:aws:apigateway:*::/restapis/GATEWAY_ID/*"
}
]
}
Serverless Error ---------------------------------------
An error occurred while provisioning your stack: IamRoleLambdaExecution
is not authorized to perform: iam:GetRole on resource:
IAM権限の設定でエラーが出て直らなかったので、
AdministratorAccess
にして
% serverless deploy
を実行したら、エラーが出ずにデプロイできた。
↓のIAM設定にしたらデプロイできた。
AWSLambdaFullAccess - AWS Managed policy IAMFullAccess - AWS Managed policy AmazonS3FullAccess - AWS Managed policy AmazonAPIGatewayInvokeFullAccess - AWS Managed policy AmazonAPIGatewayAdministrator - AWS Managed policy AWSLambdaExecute - AWS Managed policy AWSCloudFormationReadOnlyAccess - AWS Managed policy kubotad_lambda_inline_policy - Inline policy
Python
chalice
chaliceというのを試してみる。
http://dev.classmethod.jp/cloud/aws/preview-the-python-serverless-microframework-for-aws/
GitHub - awslabs/chalice: Python Serverless Microframework for AWS
pyenv install 2.7.11 pyenv local 2.7.11 pyenv virtualenv 2.7.11 chalice_dev pyenv version pyenv local chalice_dev pyenv version pip install chalice chalice new-project helloworld && cd helloworld cat app.py chalice deploy
is not authorized to perform: lambda:AddPermission on resource: arn:aws:lambda:us-west-2:
IAM設定が分からないので、Admin権限のを作成した。
% chalice deploy Updating IAM policy. Updating lambda function... Regen deployment package... Sending changes to lambda. Lambda deploy done. API Gateway rest API already found. Deleting root resource id Done deleting existing resources. Deploying to: dev https://xxxxxxxxxx.execute-api.us-west-2.amazonaws.com/dev/
動いた。
Docker TensorFlow
『pythonによるスクレイピング&機械学習 開発テクニック』
の５章 (5-2)だけ試してみた。
docker run -it b.gcr.io/tensorflow/tensorflow:latest-devel
本の通りにpython3で試したらエラー。
root@ac3b891d06fd:~# python3 Python 3.4.3 (default, Oct 14 2015, 20:28:29) [GCC 4.8.4] on linux Type "help", "copyright", "credits" or "license" for more information. >>> >>> import tensorflow as tf Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'tensorflow'
python 2系だといけた。
root@ac3b891d06fd:~# python
Python 2.7.6 (default, Jun 22 2015, 17:58:13)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>>
>>> import tensorflow as tf
>>> sess = tf.Session()
>>> hello = tf.constant('Hello')
>>> sess.run(hello)
'Hello'
>>>
>>>
>>> a = tf.constant(10)
>>> b = tf.constant(32)
>>> sess.run(a+b)
42
rmする時、ワイルドカードでマッチするファイルの内、一つだけ除外したい場合
~/Documents/ に
FileZilla_ で始まる消したいファイルがあったけど、
最新の１個だけ残しておきたかった時のメモ。
rm FileZilla_*
で全部消せたけど、マッチするファイルの内１個だけ残したい。
同じディレクトリに消したくないファイルが他にある。
find . -type f -name 'FileZilla*' -not -name 'FileZilla_3.23.0.2_macosx-x86.app.tar.bz2' -print0 | xargs -0 rm
bash - rm all files except some - Stack Overflow
コマンドが長すぎてめんどくさい。
macOSにpipが入っていない
macOS Sierra
Version 10.12.2
% brew install pip Updating Homebrew... ==> Auto-updated Homebrew! Updated 1 tap (homebrew/core). ==> Updated Formulae antlr youtube-dl Error: No available formula with the name "pip" Homebrew provides pip via: `brew install python`. However you will then have two Pythons installed on your Mac, so alternatively you can install pip via the instructions at: https://pip.readthedocs.io/en/stable/installing/
% curl -kL https://bootstrap.pypa.io/get-pip.py | python
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1558k 100 1558k 0 0 3886k 0 --:--:-- --:--:-- --:--:-- 3895k
Collecting pip
Downloading pip-9.0.1-py2.py3-none-any.whl (1.3MB)
100% |████████████████████████████████| 1.3MB 912kB/s
Collecting wheel
Downloading wheel-0.29.0-py2.py3-none-any.whl (66kB)
100% |████████████████████████████████| 71kB 4.8MB/s
Installing collected packages: pip, wheel
Exception:
Traceback (most recent call last):
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/req/req_set.py", line 784, in install
**kwargs
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/req/req_install.py", line 851, in install
self.move_wheel_files(self.source_dir, root=root, prefix=prefix)
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/req/req_install.py", line 1064, in move_wheel_files
isolated=self.isolated,
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/wheel.py", line 345, in move_wheel_files
clobber(source, lib_dir, True)
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/wheel.py", line 316, in clobber
ensure_dir(destdir)
File "/var/folders/t_/1c8g5tlj7y70t3s4f3pvsqkm0000gn/T/tmpdpNht1/pip.zip/pip/utils/__init__.py", line 83, in ensure_dir
os.makedirs(path)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py", line 157, in makedirs
mkdir(name, mode)
OSError: [Errno 13] Permission denied: '/Library/Python/2.7/site-packages/pip'
sudo を付けるらしい
curl -kL https://bootstrap.pypa.io/get-pip.py | sudo python
インストールできた。
Visual Studio CodeでPython
MacBookProを買ったので環境設定。
Visual Studio CodeでPythonの開発環境構築を構築してみた。 ｜ Developers.IO
Python - Visual Studio Marketplace
Visual Studio Codeで
Command + P
ext install python
donjayamanne.python
If using a custom Python Version or a Virtual Environment, use the command Select Workspace Interpreter)
pyenvとvirtualenvの設定
pyenvのメモ - kubotti’s memo
pyenv virtualenv anaconda3-4.2.0 pydev3
Visual Studio Codeで
Command + P
shell
Homebrewのインストール
MacbookPro 2016 13インチを買った。
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
