-
Notifications
You must be signed in to change notification settings - Fork 738
Support not using a container in Azure Batch #5536
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support not using a container in Azure Batch #5536
Conversation
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
...Batch Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
It's not a good idea, but it can unblock users if they need to use a bare VM.
Given this, would it be worth keeping the strict validation on by default and instead providing an override config option to turn it off (like azure.requireContainer=false or something) then updating the validation error message to explain that the option exists but you should only use it if you know what you're doing?
Given this, would it be worth keeping the strict validation on by default and instead providing an override config option to turn it off (like
azure.requireContainer=falseor something) then updating the validation error message to explain that the option exists but you should only use it if you know what you're doing?
That's a great idea!
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
b327beb to
ad07555
Compare
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
@tom-seqera implemented now as azure.batch.requireContainer.
I'm still not sure this is a wise thing to do, but it unblocks some very specific use cases and it's a relatively buried feature.
My understanding is that some tasks my require a container, some other may not, therefore a global setting is sub-optimal.
Also, could not be Dragen accessed by the container anyway? this is for example how it works with AWS Batch (the dragen path in the host is made accessible in the container context)
Also, could not be Dragen accessed by the container anyway? this is for example how it works with AWS Batch (the dragen path in the host is made accessible in the container context)
Our implementation only works on AWS using a custom AMI. The AMI provided by Illumina uses CentOS 7, which is EOL. And obviously, it's AWS only.
The official AMI provided by Illumina for Azure is not compatible with Docker and they do not want to update it. So we end up either using a Docker compatible image which does not work on Azure Batch, or a specific Dragen based image which does not use Docker. We spent a long time playing around with this and this was the most parsimonious solution we could make work.
The official AMI provided by Illumina for Azure is not compatible with Docker
What this means exactly?
Illumina provide a machine image for Dragen, it’s not compatible with Docker.
1f834a0 to
6454605
Compare
5a93547 to
27345a6
Compare
Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com>
2a9153e to
1587653
Compare
f6a3696 to
49b58d2
Compare
b4b321e to
069653d
Compare
b7b4221 to
c1114bc
Compare
There are rare cases where a user needs to run a process on a bare VM on Azure Batch. Most notably, when using the Dragen which requires you to use a specific machine image that does not support a Docker container.
Previously, Nextflow prevented this, but this PR removes that safeguard and replaces it with a warning. It's not a good idea, but it can unblock users if they need to use a bare VM.
If nothing else, this PR functions as proof it can be done if anyone else needs to use it.