Google Datastream errors on larger MySQL tables

Question

I have set up a Datastream service, in order to replicate data from Cloud SQL (MySQL) to BigQuery.

Everything is set up correctly, connection works. But the weird thing is that only tables < 10mb size are replicated without issues.

The larger tables (100+ MB) all fail. When checking the error status, it only says "Timed-out while waiting for the query to complete."

I have not found anything useful regarding this error. What approaches can I try? Backfilling a specific table give the same error.

Source (MySQL) database connection flags are set to the recommended values:

net_read_timeout: 3600 seconds (1 hour)
net_write_timeout: 3600 seconds (1 hour)
wait_timeout: 86400 seconds (24 hours)

Answer 1

I was able to resolve the issue.

What needed to be adjusted was the MySQL flag max_execution_time

Once that was incremented, Datastream could run and finish successfully.