ProcessDocument API Errors - No remaining quota for ParseDocument

Question

As part of our workflow we invoke DocumentAI ProcessDocument API (v1) API from our back end and the code has been in place for over 6 months and running without any errors. In the past one week we have been seeing random errors. But the main one being different variation of the error message "No remaining quota for ParseDocument". When we check the 'systems and quotas' page we do not see anything which has even neared the quota limits and lost how to debug this further or resolve it. We make use of both OCR and Form Parser processors. Also, note that not all requests fail, only some of them fail.

Multiple occurrences of the below error message with only the page number at the end: No remaining quota for ParseDocument client:documentai-core-frontend pages:6 No remaining quota for ParseDocument client:documentai-core-frontend pages:3 etc

In addition to the above, we are seeing errors like: Error occurred during processing pages. Detail: FormParsing failed for page: 1. Detailed: CalculatorGraph::Run() failed: Calculator::Process() for node "UcaipFormPairDetectorCalculator" failed: Harpoon reply has empty body. Fetch status: URL_UNREACHABLE-UNREACHABLE_NO_RESPONSE

Error occurred during processing pages. Detail: FormParsing failed for page: 1. Detailed: CalculatorGraph::Run() failed: Calculator::Process() for node "UcaipFormKeyDetectorCalculator" failed: Took too long to respond

Error observed only last Wednesday: Service Control: Quota check error RESOURCE_EXHAUSTED (8) dvs_import_page Service Control: Quota check error RESOURCE_EXHAUSTED (8) dvs_import_page,dvs_import_gocr_page

As the error occurs randomly, we have not been able to pinpoint anything specific. Our code which invokes the API has not changed for over six months and these errors started only since last Wednesday and hence we are suspecting it to be something on GCP side. We could not find anything specific when we search online. Any pointers to debug this would be helpful.

• DazWilkin: It's possible that Google has added quotas to the services you're using but has not yet published these. It's also possible that the services you're using are themselves exceeding quota in upstream services. Either way, your best approach is likely to open a support ticket with Google Cloud and have them diagnose.
• Charles: Thanks @DanWilkin, that's what we would end up doing if we are unable to figure it out ourself or through the community. Paying Google additional money (over 100ドル) every month for a support subscription for the one service we use and for this one technical issue seems a bit too much, but when it comes down to it, we would just have to do it. Thanks once again for your inputs.