Terraform init and modules hosted in Azure Devops failed

Question

So, we are using Azure Devops to store our Terraform config and all the self created module code. We also want to use a DevOps Pipeline to apply the configuration.

As we are not allowed to use ssh for accessing the repos from our developer workstations (traffic has to go trough the https-proxy), so we have to use https for the repository integration so that our source for the module looks like

source = "git::https://<<ADO_ORG>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>"

Locally running terraform init works completely fine. When running it in the pipeline we see following

Error: Failed to download module
Could not download module "xyz" (main.tf:3)
source code from
"git::https://<<ADO_ORG>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>"
error downloading
'https://<<ADO_ORG>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>'
/usr/bin/git exited with 128: Cloning into
'.terraform/modules/xyz'...
fatal: could not read Password for 'https://<<ADO_ORG>>@dev.azure.com':
terminal prompts disabled

We have tested many things right now and only with changeing source to

source = "git::https://<<PAT>>@dev.azure.com/<ADO_ORG>>/<<ADO_PROJECT>>>/_git/<<ADO_REPO>>"

we were able to run terraform init but checking in PAT to git sounds not very right as variables can't be used at that parameter.

We have also tested added the repos as resource to the pipeline and using

git config --global http.https://<ADO_ORG>>@dev.azure.com.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"

But as repo ressources are limited to max 20 with a pipeline, this will not work.

Anyone an idea on that? Regards Joerg

Answer 1

After many tests, I was able to solve the issue. As I can see, the solution has two parts.

1) Script in the pipeline yml

- script: |
 git config --global url."https://[email protected]".insteadOf "https://<<ADO-ORG>>@dev.azure.com"
 displayName: 'set extra header'
 env:
 SYSTEM_ACCESSTOKEN: $(System.AccessToken)

2) Deactivating the setting "Protect access to repositories in YAML pipelines"

enter image description here

With both parts together, everything seams to work now.

Answer 2

Set the build service to have permissions to the repo in question, then (it should, by default):

steps:
- checkout: self
 persisteCredentials: true

Add that before wherever you perform your terraform init. At the end, clean it all up:

steps:
- checkout: self
 clean: true

https://learn.microsoft.com/en-us/azure/devops/pipelines/scripts/git-commands

If you can manually run a git clone as a step in the pipeline (to test), then Terraform should work because it just piggybacks everything that is already working:

Terraform installs modules from Git repositories by running git clone, and so it will respect any local Git configuration set on your system, including credentials. To access a non-public Git repository, configure Git with suitable credentials for that repository...

If using the HTTP/HTTPS protocol, or any other protocol that uses username/password credentials, configure Git Credentials Storage to select a suitable source of credentials for your environment.

https://www.terraform.io/language/modules/sources#generic-git-repository

• Jörg Lang: I have tryed to use ``` steps: - checkout: self persisteCredentials: true ``` without any success. The permissions looks okay so far. I will look into some more stuff related to git credential storeand try to see, how we can integrate that into an Azure Pipeline running on Linux.