How do you debug GRPC error when there is no details?

Question

I have already found a discussion regarding better debugging for GRPC: How to debug grpc call?

but I am running into a scenario where even after adding those flags GRPC_VERBOSITY and GRPC_TRACE, I couldn't know the actual error because details are empty!

I am trying to create data transfer from S3 to Big Query but using a programmatic way, so this is what I wrote by first going through some stuff here: https://github.com/googleapis/googleapis/blob/master/google/cloud/bigquery/datatransfer/v1/transfer.proto and a little bit of AI help.

import * as bigQueryDataTransfer from '@google-cloud/bigquery-data-transfer';
const client = new bigQueryDataTransfer.DataTransferServiceClient();
client
 .createTransferConfig({
 parent: `projects/project-name`,
 transferConfig: {
 destinationDatasetId: 'dataset-id',
 displayName:'Test Transfer',
 dataSourceId: 'amazon_s3',
 schedule: 'every 3 hours',
 params: {
 fields: {
 bucket_name: {
 stringValue: 'bucket-name',
 },
 object_path: {
 stringValue: 'path/to/your/data/*',
 },
 aws_access_key_id: {
 stringValue: 'access-key',
 },
 aws_secret_access_key: {
 stringValue: 'secret-key',
 },
 region_name: {
 stringValue: 'us-west-1',
 },
 },
 },
 },
 })

After running above code I am running into this kind of error

Error: 3 INVALID_ARGUMENT: 
...
...
 at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
 code: 3,
 details: '',
 metadata: Metadata { internalRepr: Map(0) {}, options: {} }
}

Now the problem is that the details field is empty and I can't figure out where the actual error is happening, even in the verbose output it doesn't reveal anything details is still empty

D 2024年12月26日T13:37:51.639Z | v1.12.5 26468 | subchannel_call | [3] received status code 3 from server
D 2024年12月26日T13:37:51.639Z | v1.12.5 26468 | subchannel_call | [3] ended with status: code=3 details=""
D 2024年12月26日T13:37:51.640Z | v1.12.5 26468 | load_balancing_call | [2] Received status
D 2024年12月26日T13:37:51.640Z | v1.12.5 26468 | load_balancing_call | [2] ended with status: code=3 details="" start time=2024年12月26日T13:37:50.437Z
D 2024年12月26日T13:37:51.640Z | v1.12.5 26468 | retrying_call | [1] Received status from child [2]

So I want to know how you folks debug this kind of error. There is not much documentation regarding how to perform S3 transfer using npm SDK that's why I have to do this try-and-error play, also couldn't get much information from .profo files from Github.

Don't get me wrong sometimes it does provide useful errors at one point I entered the dataset ID wrong then it gave me a detailed message like dataset can not be found but sometimes it provides just an empty message with Error code 3: INVALID_ARGUMENT, but which argument?

• Visrut: can help for Big Query, not js but Python has some good stuff: cloud.google.com/python/docs/reference/bigquerydatatransfer/…
• Visrut: good stuff if you are creating data transfer in Python: stackoverflow.com/questions/62033526/…

Answer 1

Ok, this is not an official answer on how to debug when the detailed message is empty, but I have to say the Python library has some really good stuff for sure and I wrote my working node code after writing in Python first.

So When I specified a field called unknown_stuff it just actually gave me a good error message that 400 Data source definition doesn't define this parameter Id: unknown_stuff

from google.cloud import bigquery_datatransfer
from google.protobuf import struct_pb2
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file(
 "your-credentials.json"
)
client = bigquery_datatransfer.DataTransferServiceClient(credentials=credentials)
parent = f"projects/your-google-project-name"
params = struct_pb2.Struct()
params.update({
 "destination_table_name_template": "destination-table-name",
 "data_path": "s3://path/to/files/*",
 "access_key_id": 'access-key',
 "secret_access_key": 'secret-key',
 "file_format": "JSON",
 "ignore_unknown_values": True,
 "write_disposition": "WRITE_APPEND",
 "unknown_stuff": "What is this" ### <--- This is not valid and got the error nice
})
transfer_config = bigquery_datatransfer.TransferConfig(
 destination_dataset_id="your-dataset-id",
 display_name=f"Data transfer for XYZ stuff",
 data_source_id="amazon_s3",
 params=params,
 schedule="every 3 hours"
)
try:
 response = client.create_transfer_config(
 request={
 "parent": parent,
 "transfer_config": transfer_config
 }
 )
 print(f"Created transfer config: {response.name}")
except Exception as e:
 print(f"Error creating transfer config: {str(e)}")

In the case of node, I didn't get this kind of friendly error message, but I should be getting an error called object_path is not valid. Now Eventually I wrote working code in node but all the stuff I had to derive was from Python so there is a plus point there.

One more thing I liked is the documentation reference in the Python client is good, for example, compare the below 2 links, I really liked the Python docs for sure.

Node.JS: https://cloud.google.com/nodejs/docs/reference/bigquery-data-transfer/latest/bigquery-data-transfer/protos.google.cloud.bigquery.datatransfer.v1.itransferconfig

Python: https://cloud.google.com/python/docs/reference/bigquerydatatransfer/latest/google.cloud.bigquery_datatransfer_v1.types.TransferConfig

For those who are wondering about the code I wrote in node afterward check this:

import * as bigQueryDataTransfer from '@google-cloud/bigquery-data-transfer';
process.env.GOOGLE_APPLICATION_CREDENTIALS = './google-service-file.json';
const client = new bigQueryDataTransfer.DataTransferServiceClient();
const transferConfig = {
 destinationDatasetId: 'dataset-id',
 displayName: 'Data transfer for Table Name',
 dataSourceId: 'amazon_s3',
 params: {
 destination_table_name_template: 'table-name',
 data_path: 's3://path/to/files/*',
 access_key_id: 'access-key',
 secret_access_key: 'secret-key',
 file_format: 'JSON',
 ignore_unknown_values: true,
 write_disposition: 'WRITE_APPEND',
 },
 schedule: 'every 3 hours',
};
const parent = 'projects/project-name';
const [response] = await client.createTransferConfig({
 parent: parent,
 transferConfig: {
 displayName: transferConfig.displayName,
 dataSourceId: transferConfig.dataSourceId,
 destinationDatasetId: transferConfig.destinationDatasetId,
 schedule: transferConfig.schedule,
 params: {
 fields: {
 destination_table_name_template: {
 stringValue: transferConfig.params.destination_table_name_template,
 },
 data_path: {
 stringValue: transferConfig.params.data_path,
 },
 access_key_id: {
 stringValue: transferConfig.params.access_key_id,
 },
 secret_access_key: {
 stringValue: transferConfig.params.secret_access_key,
 },
 file_format: {
 stringValue: transferConfig.params.file_format,
 },
 ignore_unknown_values: {
 boolValue: transferConfig.params.ignore_unknown_values,
 },
 write_disposition: {
 stringValue: transferConfig.params.write_disposition,
 },
 },
 },
 },
});