lunar riscv64 preinstalled image shows login prompt before login is possible

Bug Description

Testing the lunar unmatched daily image in qemu, I was given a login prompt on console and tried to log in, and it repeatedly failed.

This is because cloud-init had not finished running, so the user that was provisioned via nocloud data was not actually available.

There should not be a race. The login prompt should not be shown until after the user exists - either by changing the ordering of the systemd units, or by not using cloud-init to configure the initial user. (I don't see any obvious reason that a hard-coded initial user/password should be done via cloud-init instead of being prepopulated in /etc/passwd, but feel free to enlighten me.)

This bug has been reported on the Ubuntu ISO testing tracker.

A list of all reports related to this bug can be found here:
https://iso.qa.ubuntu.com/qatracker/reports/bugs/2013403

This sounds like possibly something should be waiting on cloud-init and isn't?

I reproduced this error with the following:

qemu-system-riscv64 \
-machine virt \
-cpu rv64 \
-m 1G \
-device virtio-blk-device,drive=hd \
-drive file=lunar-preinstalled-server-riscv64+unmatched.img,if=none,id=hd \
-device virtio-net-device,netdev=net \
-netdev user,id=net,hostfwd=tcp::2222-:22 \
-bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_jump.elf \
-kernel /usr/lib/u-boot/qemu-riscv64_smode/uboot.elf \
-object rng-random,filename=/dev/urandom,id=rng \
-device virtio-rng-device,rng=rng \
-append "root=LABEL=rootfs console=ttyS0" \
-nographic

Log:

[ OK ] Created slice system-getty.slice - Slice /system/getty.
[ OK ] Started <email address hidden> - Getty on tty1.
[ OK ] Reached target getty.target - Login Prompts.
[ OK ] Started systemd-logind.service - User Login Management.
[ OK ] Started unattended-upgrade...0m - Unattended Upgrades Shutdown.

Ubuntu Lunar Lobster (development branch) ubuntu ttyS0

ubuntu login: ubuntu
Password:

Login incorrect
ubuntu login: [ 148.735449] Dev loop3: unable to read RDB block 8
[ 166.631180] cloud-init[715]: Cloud-init v. 23.1.1-0ubuntu2 running 'modules:config' at 2023年3月31日 18:48:02 +0000. Up 165.88 seconds.
[ 167.335569] cloud-init[715]: 2023年03月31日 18:48:03,547 - cc_set_passwords.py[WARNING]: DEPRECATION: key 'lists' is now deprecated. Use 'users'.
[ 177.934571] cloud-init[751]: Cloud-init v. 23.1.1-0ubuntu2 running 'modules:final' at 2023年3月31日 18:48:13 +0000. Up 177.29 seconds.
ci-info: no authorized SSH keys fingerprints found for user ubuntu.
<14>Mar 31 18:48:15 cloud-init: #############################################################
<14>Mar 31 18:48:15 cloud-init: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Mar 31 18:48:15 cloud-init: 1024 SHA256:szpuJaLNqHJR/i5rtmAW0QBDc4rOiZkbzg0oJr+5x2g root@ubuntu (DSA)
<14>Mar 31 18:48:15 cloud-init: 256 SHA256:c7UTAof0IjqZWPy43dgZmqxdo4ajOcWCsAwt7W/EoOY root@ubuntu (ECDSA)
<14>Mar 31 18:48:16 cloud-init: 256 SHA256:ZCyPk622f7s3RgJdgB5ps5X00BMA5/8wUfM8L3sgZDY root@ubuntu (ED25519)
<14>Mar 31 18:48:16 cloud-init: 3072 SHA256:MrkVMol0wlpj6LM2LQvW5wsZbOrP4BXv5V2m8p5+u1A root@ubuntu (RSA)
<14>Mar 31 18:48:16 cloud-init: -----END SSH HOST KEY FINGERPRINTS-----
<14>Mar 31 18:48:16 cloud-init: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIuN1I/bvbih1NnNZ7kNBBEbZBDjQ/yLwB+S+iLJIbR3n70G8VHsPNP7B9DQmA/DGaC6kTL78i1pLp17EggzjNI= root@ubuntu
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDjt0gd0c3qEw7HrZfWFFpWlh0Folwu2Wh1vR9xmkxEh root@ubuntu
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCJcFHCuEnviGbmy/Q33WgKlC71YaNM88of/mJh7FY/edNMjjF6va1oB0QLRuZQ3OTWFcujOCej3mdqwcF6R/SjKRnzXsmvPRJv+KE4dgMU0PVKWeHM5fu4QChHskVMF6j8FHq+RbPV5iytKQCQ3BA6I1WkV4Y6MVXlGN+jMoeKPENEBmhA8t1yPZBH6yeq2Aq9UuuLYjZhiO3mm05HaXqzwKDP8TRU1MBMsyOhoVmcgiQ7uH3wY00mu11/xGgKo7lSPVV3C1v7gME6ROX2zRLklMOWDt7wehvSEae7NJ8CcihJxczrUhw30/mbNoPS7BHw8Lbq4kffRHwznX4j8YKZrke0QIOFqTguIw90x2GaDmaY5iw/TZEyav/gvl0SicA...

The "something" is the core tty units, which are part of systemd upstream and should not have dependencies on cloud-init. Cloud-init needs to handle it from its side.

this is actually true of pretty much all cloud-images, at least that i'm aware. I've seen this functionally on every cloud image, all arches, since i joined CPC.

I think it's generally not considered a "bug" for cloud-images, as the standard user interaction is via SSH. for that use case, a user is not able to SSH into the image until cloud-init has finished provisioning SSH keys, and so most users will not have an issue.

I'm with vorlon on this generally -- if we view reaching the login prompt before user finishes configuration as a bug, it's a bug in cloud-init / systemd (with some sort of blocking mechanism therein) to disallow a terminal console from showing a login prompt. as far as a cloud-image, it's "operating as expected" though less than optimally.

Thanks for the feedback Steve and John. Agreed, the fix belongs in cloud-init.

I just proposed a change[1] that I think should resolve the issue for console access. It addresses the ssh use case too (though I only tested the tty case).

[1] https://github.com/canonical/cloud-init/pull/2111/files

and here i was thinking it'd be more than a systemd target change :) I had the hope, but didn't want to get excited.

I'll give a general +1 there, with the caveat of wanting to test in a bunch of scenarios (of course)

Status changed to 'Confirmed' because the bug affects multiple users.

Performance testing[1] reveals no regression. The fix has landed in upstream cloud-init.

[1] https://github.com/canonical/cloud-init/pull/2111#issuecomment-1616634930

This bug was fixed in the package cloud-init - 23.3~2gcf474da7-0ubuntu1

---------------
cloud-init (23.3~2gcf474da7-0ubuntu1) mantic; urgency=medium

* Upstream snapshot based on upstream/main at cf474da7.
- Bugs fixed in this snapshot: (LP: #2013403)

-- Alberto Contreras <email address hidden> 2023年7月11日 10:22:04 +0200

This change resulted in regression bug report bug 2039441. Whether we should treat it as a real regression or not isn't yet clear to me. Did we change the outcome of a race condition, or introduce a new one? Anyway, discussion in that other bug please.