cc_final_message will create /var/lib/cloud/instance as a directory if the symlink does not exist

Bug Description

cc_final_message calls util.write_file, which calls ensure_dir; if nothing has caused the symlink to be created, then this means that /var/lib/cloud/instance will be a directory, and future boots will fail because they fail to remove it (because it's a directory, so can't just be `rm`d.)

(We're seeing this in bug 1883661; I don't think it's the root cause, hence this separate bug.)

Dan,
Thanks!

https://bugs.launchpad.net/cloud-init/+bug/1531880
https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1796875

I've tried to reproduce this before and never could.

I think the above are basically duplicates.

A PR for this is open at https://github.com/canonical/cloud-init/pull/445

Thanks for the fix.

After applying the fix I'm getting one more error with the same test script I've mentioned in:
https://bugs.launchpad.net/cloud-init/+bug/1883661

```
Jun 23 11:15:16 phdev cloud-init[1786]: Traceback (most recent call last):
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/bin/cloud-init", line 11, in <module>
Jun 23 11:15:16 phdev cloud-init[1786]: load_entry_point('cloud-init==20.2', 'console_scripts', 'cloud-init')()
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/lib/python3.7/site-packages/cloudinit/cmd/main.py", line 893, in main
Jun 23 11:15:16 phdev cloud-init[1786]: get_uptime=True, func=functor, args=(name, args))
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/lib/python3.7/site-packages/cloudinit/util.py", line 2618, in log_time
Jun 23 11:15:16 phdev cloud-init[1786]: ret = func(*args, **kwargs)
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/lib/python3.7/site-packages/cloudinit/cmd/main.py", line 600, in status_wrapper
Jun 23 11:15:16 phdev cloud-init[1786]: util.ensure_dirs((data_d, link_d,))
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/lib/python3.7/site-packages/cloudinit/util.py", line 1645, in ensure_dirs
Jun 23 11:15:16 phdev cloud-init[1786]: ensure_dir(d, mode)
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/lib/python3.7/site-packages/cloudinit/util.py", line 1698, in ensure_dir
Jun 23 11:15:16 phdev cloud-init[1786]: os.makedirs(path)
Jun 23 11:15:16 phdev cloud-init[1786]: File "/usr/lib/python3.7/os.py", line 221, in makedirs
Jun 23 11:15:16 phdev cloud-init[1786]: mkdir(name, mode)
Jun 23 11:15:16 phdev cloud-init[1786]: FileExistsError: [Errno 17] File exists: '/var/lib/cloud/data'
Jun 23 11:15:16 phdev systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE
Jun 23 11:15:16 phdev systemd[1]: cloud-init.service: Failed with result 'exit-code'.
```

Is it okay to replicate the behavior of 'mkdir -p' for this?

A fix for this landed in https://github.com/canonical/cloud-init/commit/0755cff078d5931e1d8e151bdcb84afb92bc0f02

Hi Shreenidhi, can you file a separate bug for that, including the tarball produced by `cloud-init collect-logs` on an affected instance, and reproduction instructions? Thanks!

Hi Dan Watkins,

Sure I'll log a separate bug with log tarball. Thanks for the response.

Hi Dan Watkins,

I created a new bug: https://bugs.launchpad.net/cloud-init/+bug/1884819. Please let me know if you need any more information.

This bug is believed to be fixed in cloud-init in version 20.3. If this is still a problem for you, please make a comment and set the state back to New

Thank you.

Tracked in Github Issues as https://github.com/canonical/cloud-init/issues/3710