SlideShare a Scribd company logo

Node.js 與 google cloud storage

•
•
O
onlinemad

The document discusses Node.js and Google Cloud Storage. It covers topics like using OAuth2 to authenticate with JSON Web Tokens and service accounts, uploading files via simple, multipart, and resumable upload methods, and managing file metadata, access control lists, versions, and directories without a true folder structure in Cloud Storage. The author reflects on lessons learned like ensuring proper permissions when accessing buckets and the value of sharing knowledge gained from experimenting with Google services.

Related slideshows

Firebase_not_really_yohoho
Firebase_not_really_yohohoFirebase_not_really_yohoho
Firebase_not_really_yohoho
 
Getting started with DataStax .NET Driver for Cassandra
Getting started with DataStax .NET Driver for CassandraGetting started with DataStax .NET Driver for Cassandra
Getting started with DataStax .NET Driver for Cassandra
 
Ecom2
Ecom2Ecom2
Ecom2
 
1 of 26
Download to read offline
Node.js 與 Google Cloud Storage 多麼痛的領悟
關於我 • Ian Wu • 瘋⼈人院院⻑⾧長 • 頑⽪皮⼯工坊 Backend Engineer • http://blog.ianwu.tw/about-me/
Why google cloud storage • 內建 CDN • Google Cloud Storage behaves essentially like a Content Delivery Network (CDN) with no work on your part because publicly readable objects are, by default, cached in the Google Cloud Storage network. • try try see • try 到死 • 有 USD 500 的 credit
OAuth2 • JWT (JSON Web Token) • Google Cloud console • credential: service account • covert p12 > pem Authentication - Google Cloud Storage — Google Cloud Platform https://cloud.google.com/storage/docs/authentication#service_accounts
OAuth2 • Get token • payload { iss: '460520686343-k6tfn73sentmh0ss5nu67kniorbcta8n@developer.gserviceaccount.com', scope: 'https://www.googleapis.com/auth/devstorage.full_control', aud: 'https://accounts.google.com/o/oauth2/token', exp: 1418280623, iat: 1418280563 } • jwt sign // sign with RSA SHA256 var cert = fs.readFileSync('google_cloud_key.pem'); // get private key var claim = jwt.sign(payload, cert, { Get Google JWT token. https://gist.github.com/onlinemad/28341a343ecde186a410 algorithm: 'RS256' });
OAuth2 • token { access_token: 'ya29.2QA9sZg_YtCTGJf1d6Vzxr_4ypioiaIdHJBmgxq6b1HsJuAPODCHnCvt', token_type: 'Bearer', expires_in: 3600 } • 使⽤用 token headers: { Authorization: 'Bearer ' + token.access_token }
Upload URI • Upload URI, for media upload requests • upload/storage/v1/b/bucket/o • Metadata URI, for metadata-only requests: • storage/v1/b/bucket/o • APIs Explorer currently supports metadata requests only.
Upload method • simple • 就 post 上傳檔案 • multipart（推薦使⽤用） • 可以連 metadata ⼀一起上傳 • request 某⼀一個版本以上才有⽀支援 • resumable • 沒⽤用過 • node-youtube-resumable-upload https://github.com/grayleonard/node-youtube-resumable-upload
multipart • request var url = 'https://www.googleapis.com/upload/storage/v1/b/yourbucket/o?' + qs.stringify(querystring); request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [ { 'Content-Type': 'application/json', body: JSON.stringify(metadata) }, { body: __newFile } ], headers: { Authorization: 'Bearer ' + token.access_token } });
multipart • body { cacheControl: 'public, max-age=604800', acl: [{ entity: 'allUsers', role: 'READER' }, { entity: 'project-owners-692227494718', role: 'OWNER' }] } • query string • 不能跟 Request body ⼀一起⽤用
Directory structure • ⼀一切都是平的 • 跟 s3 ⼀一樣 • 所以沒有建⽴立 folder 這件事情 • name = foo/bar.jpg;
Directory structure • simple • /o?name=foo%2Fbar.jpg • multipart • body.name = foo/bar.jpg
Access URL • Standard（推薦） • storage.googleapis.com/<bucket>/<object> • <bucket>.storage.googleapis.com/<object> • CNAME • travel-maps.example.com CNAME c.storage.googleapis.com • no ssl • Cookie-based Authentication • 沒⽤用過
Versioning • 預設是關掉的 ➜ ~ gsutil versioning get gs://onlinemad-versioning gs://onlinemad-versioning: Suspended ➜ ~ gsutil versioning set on gs://onlinemad-versioning Enabling versioning for gs://onlinemad-versioning/... ➜ ~ • qs + generation { "kind": "storage#object", "id": "onlinemad-dev/uploaded.jpg/1418291876469000", "selfLink": "https://www.googleapis.com/storage/v1/b/onlinemad-dev/o/uploaded.jpg", "name": "uploaded.jpg", "bucket": "onlinemad-dev", "generation": "1418291876469000", "metageneration": "1", "contentType": "image/jpeg", "updated": “2014-12-11T09:57:56.468Z”, }
ACL [ { "entity": "project-owners-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "owners" }, "role": "OWNER" }, { "entity": "project-editors-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "editors" }, "role": "OWNER" }, { "entity": "project-viewers-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "viewers" }, "role": "READER" }, { "entity": "user-00b4903a9745459d3abf193213c0f30d5dea50ee7e3e318007a7edfaecb646e5", "entityId": "00b4903a9745459d3abf193213c0f30d5dea50ee7e3e318007a7edfaecb646e5", "role": "OWNER" } ]
ACL • 我需要 public read • 所以request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [{ 'Content-Type': 'application/json', body: JSON.stringify({ name: 'acl_multipart_upload_public_read.jpg', acl: [{ entity: 'allUsers', role: 'READER' }] }) }, { body: data }], headers: { Authorization: 'Bearer ' + token.access_token } })
ACL ➜ ~ gsutil acl get gs://onlinemad-dev/ acl_simple_upload_public_read.jpg AccessDeniedException: Access denied. Please ensure you have OWNER permission on gs://onlinemad-dev/ acl_simple_upload_public_read.jpg.
Node.js 與 google cloud storage
這是 feature 不是 bug 這是 feature 不是 bug 這是 feature 不是 bug
ACL
Node.js 與 google cloud storage
ACL request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [{ 'Content-Type': 'application/json', body: JSON.stringify({ name: 'acl_multipart_upload_public_read_add_owner.jpg', acl: [{ entity: 'allUsers', role: 'READER' }, { entity: 'project-owners-460520686343', role: 'OWNER' }] }) }, { body: data }], headers: { Authorization: 'Bearer ' + token.access_token } })
我的領悟
「還沒有⼈人分享 Google Service 時， 請勿輕易嘗試」 – Ian Wu
「當你試了 Google Service 時， 請���分享」 – Ian Wu
謝謝⼤大家

More Related Content

Node.js 與 google cloud storage

  • 1. Node.js 與 Google Cloud Storage 多麼痛的領悟
  • 2. 關於我 • Ian Wu • 瘋⼈人院院⻑⾧長 • 頑⽪皮⼯工坊 Backend Engineer • http://blog.ianwu.tw/about-me/
  • 3. Why google cloud storage • 內建 CDN • Google Cloud Storage behaves essentially like a Content Delivery Network (CDN) with no work on your part because publicly readable objects are, by default, cached in the Google Cloud Storage network. • try try see • try 到死 • 有 USD 500 的 credit
  • 4. OAuth2 • JWT (JSON Web Token) • Google Cloud console • credential: service account • covert p12 > pem Authentication - Google Cloud Storage — Google Cloud Platform https://cloud.google.com/storage/docs/authentication#service_accounts
  • 5. OAuth2 • Get token • payload { iss: '460520686343-k6tfn73sentmh0ss5nu67kniorbcta8n@developer.gserviceaccount.com', scope: 'https://www.googleapis.com/auth/devstorage.full_control', aud: 'https://accounts.google.com/o/oauth2/token', exp: 1418280623, iat: 1418280563 } • jwt sign // sign with RSA SHA256 var cert = fs.readFileSync('google_cloud_key.pem'); // get private key var claim = jwt.sign(payload, cert, { Get Google JWT token. https://gist.github.com/onlinemad/28341a343ecde186a410 algorithm: 'RS256' });
  • 6. OAuth2 • token { access_token: 'ya29.2QA9sZg_YtCTGJf1d6Vzxr_4ypioiaIdHJBmgxq6b1HsJuAPODCHnCvt', token_type: 'Bearer', expires_in: 3600 } • 使⽤用 token headers: { Authorization: 'Bearer ' + token.access_token }
  • 7. Upload URI • Upload URI, for media upload requests • upload/storage/v1/b/bucket/o • Metadata URI, for metadata-only requests: • storage/v1/b/bucket/o • APIs Explorer currently supports metadata requests only.
  • 8. Upload method • simple • 就 post 上傳檔案 • multipart（推薦使⽤用） • 可以連 metadata ⼀一起上傳 • request 某⼀一個版本以上才有⽀支援 • resumable • 沒⽤用過 • node-youtube-resumable-upload https://github.com/grayleonard/node-youtube-resumable-upload
  • 9. multipart • request var url = 'https://www.googleapis.com/upload/storage/v1/b/yourbucket/o?' + qs.stringify(querystring); request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [ { 'Content-Type': 'application/json', body: JSON.stringify(metadata) }, { body: __newFile } ], headers: { Authorization: 'Bearer ' + token.access_token } });
  • 10. multipart • body { cacheControl: 'public, max-age=604800', acl: [{ entity: 'allUsers', role: 'READER' }, { entity: 'project-owners-692227494718', role: 'OWNER' }] } • query string • 不能跟 Request body ⼀一起⽤用
  • 11. Directory structure • ⼀一切都是平的 • 跟 s3 ⼀一樣 • 所以沒有建⽴立 folder 這件事情 • name = foo/bar.jpg;
  • 12. Directory structure • simple • /o?name=foo%2Fbar.jpg • multipart • body.name = foo/bar.jpg
  • 13. Access URL • Standard（推薦） • storage.googleapis.com/<bucket>/<object> • <bucket>.storage.googleapis.com/<object> • CNAME • travel-maps.example.com CNAME c.storage.googleapis.com • no ssl • Cookie-based Authentication • 沒⽤用過
  • 14. Versioning • 預設是關掉的 ➜ ~ gsutil versioning get gs://onlinemad-versioning gs://onlinemad-versioning: Suspended ➜ ~ gsutil versioning set on gs://onlinemad-versioning Enabling versioning for gs://onlinemad-versioning/... ➜ ~ • qs + generation { "kind": "storage#object", "id": "onlinemad-dev/uploaded.jpg/1418291876469000", "selfLink": "https://www.googleapis.com/storage/v1/b/onlinemad-dev/o/uploaded.jpg", "name": "uploaded.jpg", "bucket": "onlinemad-dev", "generation": "1418291876469000", "metageneration": "1", "contentType": "image/jpeg", "updated": “2014-12-11T09:57:56.468Z”, }
  • 15. ACL [ { "entity": "project-owners-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "owners" }, "role": "OWNER" }, { "entity": "project-editors-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "editors" }, "role": "OWNER" }, { "entity": "project-viewers-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "viewers" }, "role": "READER" }, { "entity": "user-00b4903a9745459d3abf193213c0f30d5dea50ee7e3e318007a7edfaecb646e5", "entityId": "00b4903a9745459d3abf193213c0f30d5dea50ee7e3e318007a7edfaecb646e5", "role": "OWNER" } ]
  • 16. ACL • 我需要 public read • 所以request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [{ 'Content-Type': 'application/json', body: JSON.stringify({ name: 'acl_multipart_upload_public_read.jpg', acl: [{ entity: 'allUsers', role: 'READER' }] }) }, { body: data }], headers: { Authorization: 'Bearer ' + token.access_token } })
  • 17. ACL ➜ ~ gsutil acl get gs://onlinemad-dev/ acl_simple_upload_public_read.jpg AccessDeniedException: Access denied. Please ensure you have OWNER permission on gs://onlinemad-dev/ acl_simple_upload_public_read.jpg.
  • 19. 這是 feature 不是 bug 這是 feature 不是 bug 這是 feature 不是 bug
  • 20. ACL
  • 22. ACL request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [{ 'Content-Type': 'application/json', body: JSON.stringify({ name: 'acl_multipart_upload_public_read_add_owner.jpg', acl: [{ entity: 'allUsers', role: 'READER' }, { entity: 'project-owners-460520686343', role: 'OWNER' }] }) }, { body: data }], headers: { Authorization: 'Bearer ' + token.access_token } })
  • 24. 「還沒有⼈人分享 Google Service 時， 請勿輕易嘗試」 – Ian Wu
  • 25. 「當你試了 Google Service 時， 請來分享」 – Ian Wu