Propogating db queries from a host that cannot initiate communication

Question

We have jobs running on-premises, many of which directly query our (on-prem) database. At the start of these jobs a new record is created in the database, and the record id is returned so that it may be referenced in future queries. We’ve recently started running other jobs in the cloud, but we have yet to run jobs that require a database connection. Due to security constraints, communication between sites is restricted and we are only able to initiate pushes/pulls from on-prem.

What can we do to enable these jobs that need to propagate data to (and receive a response from) our on-prem database? Is a message queue an appropriate solution here (ie instead of querying the database directly, submit queries to a message queue in the cloud, and periodically poll the queue from on-prem, popping off messages as they're received)? I don't have much experience with databases and I have zero knowledge of message queues so when I looked into this I'm having trouble filtering for what's relevant to this issue.

• Doc Brown: Polling from the on-prem site sounds fine. Depends, however, if you can find a reasonable polling interval (too often - too much traffic, too seldom - too much latency).

Answer 1

That sounds like your policy is there to prevent data leakage, and your proposal is a way of making a run around the security policy? If a job requires data from the on-prem database, and you're not allowed to leak data from on-prem into the cloud, then you cannot run that job in the cloud.

• byte_zero: I don't think this is an issue - the policy is there to secure our internal network. We don't have a problem with moving data in/out of the cloud. In any case we are working with IT/SecOps on both sides to ensure whatever we decide to do is kosher.