Questions tagged [google-bigquery]
BigQuery is Google's fully managed, scalable, serverless data warehouse. With BigQuery, you get a columnar and ANSI SQL database that can analyze petabytes of data in seconds.
11
questions
3
votes
1
answer
64
views
Trigger Cloud Function with Eventarc and then extract BigQuery table as csv files
Problem:
Whenever new data are inserted, extract BigQuery tables as csv files and
store them in Cloud Storage.
My plan:
Set up an Eventarc trigger based on Event method
...
- 33
0
votes
1
answer
48
views
Efficiency of Apache Beam + BigQuery pipeline
This is my first Apache pipeline. It takes a JSON file and saves the correctly formatted rows in one table, and the misformed rows in another.
My biggest worry is the efficiency. I have tested the ...
- 635
0
votes
1
answer
196
views
Execute several queries and save them to files in Bash
I wrote the following bash script to run queries saved in files and save them to csv files. The queries have to be run against a BigQuery database. I use the bq-cli ...
- 319
2
votes
2
answers
142
views
Calculates start time and end time of jobs in a dataproc cluster
I have the below function get_status_time which calculates the start time and the end time of the spark job which has already completed its run (status could be ...
- 55
2
votes
1
answer
57
views
building a shell command inside python function based on different conditions
I have written the below python function which is working fine but there are lot of code which looks redundant to me and hence i want to make it better by using best coding guidelines.
Basically I am ...
3
votes
1
answer
49
views
How to fine tune the complex python function for creating merge command to run in bigquery
I created a python code to run the merge command in Google BigQuery, which is used to do UPDATE, INSERT, and DELETE in a single statement. I'd appreciate it if someone could assist me in fine-tuning ...
- 55
1
vote
1
answer
132
views
Creating SQL query for relative timeframes and multiple tables
What I want to achieve:
I have several tables of the following schema:
+------+-------+
| date | val |
+------+-------+
| DATE | INT64 |
+------+-------+
I want ...
- 319
3
votes
2
answers
159
views
Keeping only maximum date rows in a group
The following SQL code keeps only the MAX(date) rows with the same id and question values. I would like to know if there is a simpler/ shorter syntax returning the same result.
...
- 133
2
votes
0
answers
45
views
SQL query that extracts a topic keyword from article titles
I have a query like this to select the only row with title contains specific programming topics.
...
- 359
3
votes
1
answer
1k
views
Insert an SQL Server Table rows into a Bigquery table in one bloc
I want to insert all rows of an SQL server Table into a BigQuery Table having the same schema.
The streaming insert row by row is very slow: to insert 1000 rows the execution of the code below took ...
- 151
2
votes
0
answers
379
views
Insert all SQL Server table rows in a Google BigQuery Table
I want to know the best way (best performance and reaction time) to insert all rows of an SQL table into a Google BigQuery table. Actually, I am looping over a query result and insert the rows one by ...
- 151