Azure IMDS publicKeys contain \r\n which prevents ssh access to vms using cloud-generated ssh keys.

Bug Description

== Begin SRU Template ==
[Impact]
The previous version of cloud-init used OpenSSL to process the SSH keys provided by the Azure platform. cloud-init 20.4 replaced that code with a more efficient implementation, but one which did not use OpenSSL: this meant that users passing certificates to instances, or users generating SSH keys in Azure's web UI (which inserts \r\n sequences into the public key content), were regressed: their certificates and malformed SSH keys were no longer handled, so they could fail to gain access to newly-launched instances.

This release is only a single functional cherry-pick which solely affects Azure platform. It is a critical bug we wish to release as soon as possible

* Azure: cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys
(#760) (LP: #1910835)

The functional changeset here introduces a raise KeyError exception which forces cloud-init to revert to previous released logic of the previous cloud-init public release 20.3.

[Test Case]

As this is a single commit backport, the cloud-init SRU exception need not apply. An upstream integration test has been written for this issue (https://github.com/canonical/cloud-init/blob/master/tests/integration_tests/bugs/test_lp1910835.py).

A full run of the upstream test suite on Azure will therefore regression test the update generally and test this issue specifically: a log of a test run for each suite will be attached.

[Regression Potential]

The proposed change only modifies code paths used on Azure, specifically to revert to a previous behaviour: users unaffected by the bug should see no change (their keys will get to their instance via a different route), and users affected by the bug would have been unable to access their instances before (so cannot be relying on this behaviour in a way which we could break by fixing it).

[Discussion]
This should only affect public Azure VM launched which use Azure to --generate-ssh-keys either from the dashboard or from the `az cli`

Any other cloud-platform is not affected by this change.

== End SRU Template ==

* cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys
(#760) (LP: #1910835)

== Original Description ==

cloud-init 20.4 or later will incorrectly add Azure publicKeys to .ssh/authorized_keys preventing ssh access for cloud-generated keys.

To reproduce: launch an ubuntu VM from the portal.azure.com choosing to generate new ssh key.

When the instance is launched you can see that the ssh-rsa content provided in the metadata publicKeys value contains CRLF characters (\r\n) thus splitting the content of the pubkey onto multiple lines when it is rendered into .ssh/authorized_keys.

the solution is either for IMDS to stop adding the CRLF characters or cloud-init to strip them out.

Here is the IMDS value provided to cloud-init

cloud-init query --format '{{ds.meta_data.imds.compute.publicKeys}}'

[{'keyData': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d\r\nk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R\r\n9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW\r\nlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq\r\n4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7\r\n6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu\r\niKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht\r\n6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9\r\nS2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure\r\n', 'path': '/home/ubuntu/.ssh/authorized_keys'}]

cloud-init renders this directly to .ssh/authorized_keys without processing the string, resulting in an invalid keyline:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1d k/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R^M
9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUW^M
lkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq^M
4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh7^M
6xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhu^M
iKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht^M
6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9^M
S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU= generated-by-azure

this prevents ssh from actually reading the right key from azure:

$ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys

If we strip the CRLF (^M) characters and reparse with ssh-keygenm we see the proper key registered:

$ ssh-keygen -lf /home/ubuntu/.ssh/authorized_keys
3072 SHA256:PQ9EKxTKONJKFC2N56UpL6+Oc/cujfA9HpsF5VW2QDI generated-by-azure (RSA)

If cloud-init (or IMDS) were to strip those \r\n characters from each line ssh

To work around this issue in cloud-init 20.4 on azure, one can launch a VM providing cloud-config userdata which contains the unsplit pubkey content.

userdata can be added in the Advanced tab in Azure portal dashboard or on the az cli via the cmdline parameters --custom-data mycloudconfig.yaml

For the auto-generated key listed in the above bug description the following would be the "right" #cloud-config to provide at VM launch time

#cloud-config
ssh_authorized_keys:
- ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCllNnyHXFWlMb9EKD9LZrOxt1dk/QxYwQ0HYEP8n6TUWoUsN3mv/Qk/qWH76Pa6f33hefzTFRiom7Ls/tJMcr/ki8R9FqyYOu0xxHmpXTUWFoZQCZtGRMtvDl/s76Wr1sCsE/ez+EcAPeGGm/B7jHtDAUWlkINfuPVBDfRtSfmnlCKS+sIf1XOqvRASGWi05zAW921T4OkiattyXyhaOimJOwq4jAXmydwtNCN2iGGKWS8YeXbtgveReqZVVKtcDKevgWdNyqZa69uq9tRujobjCh76xxCkQcdCLospgqX79GBbdRys6mVxVgc349RIWjQwglRQpJwNzkeOG5Q+La2MEhuiKqKJMvYVhil3khzMuZwzmTrGbRx0E8AS+Cm064RBgbcdjCW8dDYGLuk2eQ2v9Ht6eERfxMBNg3udv1jmiKpjjHIg99HDU4VqhL3aHmg+TSrxByd0cAgFBV+H0CiUVC9S2mLJ6Peu/HDwd88E8Wqiv3eAsjcaCRH3QiQVaU=

For those who have access to the pem file auto-generated on their local system you can get your public key from the pem file with the following command:

ssh-keygen -f ~/Downloads/my-generated-azure_key.pem -y

The Azure auto-generated public key can also be pasted directly into the portal dashboard at instance launch time by setting "Use existing public key" in the "SSH Public Key Source" option and copying the output from `ssh-keygen -f ~/Downloads/my-generated-azure_key.pem -y` into the textarea.

Hello Chad, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~16.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Chad, or anyone else affected,

Accepted cloud-init into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~18.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Chad, or anyone else affected,

Accepted cloud-init into groovy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~20.10.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-groovy to verification-done-groovy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-groovy. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Hello Chad, or anyone else affected,

Accepted cloud-init into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/20.4-0ubuntu1~20.04.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

This bug was fixed in the package cloud-init - 20.4-0ubuntu1~20.10.2

---------------
cloud-init (20.4-0ubuntu1~20.10.2) groovy; urgency=medium

* cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys
(#760) (LP: #1910835)

-- Daniel Watkins <email address hidden> 2021年1月11日 17:10:13 -0500

The verification of the Stable Release Update for cloud-init has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

This bug was fixed in the package cloud-init - 20.4-0ubuntu1~20.04.2

---------------
cloud-init (20.4-0ubuntu1~20.04.2) focal; urgency=medium

* cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys
(#760) (LP: #1910835)

-- Chad Smith <email address hidden> 2021年1月11日 15:25:31 -0700

This bug was fixed in the package cloud-init - 20.4-0ubuntu1~18.04.2

---------------
cloud-init (20.4-0ubuntu1~18.04.2) bionic; urgency=medium

* cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys
(#760) (LP: #1910835)

-- Daniel Watkins <email address hidden> 2021年1月11日 17:31:19 -0500

This bug was fixed in the package cloud-init - 20.4-0ubuntu1~16.04.2

---------------
cloud-init (20.4-0ubuntu1~16.04.2) xenial; urgency=medium

* cherry-pick 4f62ae8d: Fix regression with handling of IMDS ssh keys
(#760) (LP: #1910835)

-- Daniel Watkins <email address hidden> 2021年1月11日 17:41:09 -0500

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3825