1

I'm checking if there's a better way to run a query on Google BigQuery and export the result to AWS S3. Reading a bit about it I saw there's a way by following these steps:

  • Export query result to an external table
  • Move created table to Google Cloud Storage
  • Move files from Google Cloud Storage into AWS S3

But my question is, is there a way to avoid the external table creation and just move the query result to GCS? is there another clever/proper way to do the same?

Thanks for your help.

Improve this question

1 Answer 1

Reset to default
2

Export of query result is not (yet?) supported in BigQuery - so for you there is only way to materialize result first (it is what you refer as export query result to an external table).

One more potential way is use BQ API and read table using tabledata.list - API ref - so if say result of your query is small (MBs) it may be more easy way of moving data. But even in this case you need to materialize result in new table - but no need to export to GCS.

Improve this answer
4
  • Mmmm the issue here is that the query result is some Gbytes, so I don't think that would help. In terms of a "query export" utility from Bigquery, I'm not aware of anything like that, but that would be amazing.
    – Pepeluis
    Commented May 30, 2018 at 19:20
  • 1
    Agree with you - but for now if your result is in GB - only one way is to store query result in table - export table to GCS - copy from GCS to AWS :(
    – Alexey Maloletkin
    Commented May 30, 2018 at 19:22
  • I was expecting to find out there something to avoid that many steps, apart from the cost associated with every step.
    – Pepeluis
    Commented May 31, 2018 at 9:54
  • A feature request has been created on your behalf. Please star it so that you could receive updates about this feature request and do not hesitate to add additional comments to provide details of the desired implementation. You can track the feature request by following this link - issuetracker.google.com/issues/111339864
    – Philipp Sh
    Commented Jul 11, 2018 at 13:39

Not the answer you're looking for? Browse other questions tagged or ask your own question.