cloud-init

cloud-init logs leak hashed passwords

Bug #1978422 reported by Mike Stroyan on 2022年06月13日

256

This bug affects 1 person

Affects		Status	Importance	Assigned to	Milestone
cloud-init	Fix Released	Critical	Unassigned

Bug Description

The recent update of cloud-init to Version: 22.2-0ubuntu1~20.04.1 on ubuntu 20.04 LTS
has started logging a warning that includes hashed passwords into at least three files readable by all users-

/var/log/cloud-init.log

2022年06月12日 21:23:48,866 - util.py[DEBUG]: Read 100004 bytes from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
2022年06月12日 21:23:48,963 - schema.py[WARNING]: Invalid cloud-config provided:
users.0: {'gecos': 'Mike Stroyan', 'groups': ['adm', 'cdrom', 'dip', 'plugdev', 'lxd', 'sudo'], 'lock-passwd': False, 'name': 'mike', 'passwd': 'HASHED_PASSWORD!', 'shell': '/bin/bash'} is not valid under any of the given schemas
2022年06月12日 21:23:48,964 - util.py[DEBUG]: Reading from /var/lib/cloud/instance/cloud-config.txt (quiet=False)

/var/log/cloud-init-output.log

2022年06月12日 21:23:48,963 - schema.py[WARNING]: Invalid cloud-config provided:
users.0: {'gecos': 'Mike Stroyan', 'groups': ['adm', 'cdrom', 'dip', 'plugdev', 'lxd', 'sudo'], 'lock-passwd': False, 'name': 'mike', 'passwd': 'HASHED_PASSWORD!', 'shell': '/bin/bash'} is not valid under any of the given schemas

/var/log/syslog

Jun 12 15:23:49 b2 cloud-init[800]: 2022年06月12日 21:23:48,963 - schema.py[WARNING]: Invalid cloud-config provided:
Jun 12 15:23:49 b2 cloud-init[800]: users.0: {'gecos': 'Mike Stroyan', 'groups': ['adm', 'cdrom', 'dip', 'plugdev', 'lxd', 'sudo'], 'lock-passwd': False, 'name': 'mike', 'passwd': 'HASHED_PASSWORD!', 'shell': '/bin/bash'} is not valid under any of the given schemas
Jun 12 15:23:49 b2 systemd[1]: Finished Initial cloud-init job (metadata service crawler).

It looks like the warning about not being compliant with schemas comes from both the use of a "lock-passwd" key and by representation of users groups as an array of strings instead of a single string containing a comma separated list of groups.

/var/lib/cloud/seed/nocloud-net/user-data is written with "lock-passwd" by original subiquity in 20.04 server release.
That was later changed to "lock_passwd" in this pull merge-
https://github.com/canonical/subiquity/pull/784
But installations done with the original 20.04 release will still have "lock-passwd".
That propagates to several files in /var/lib/cloud/instance/.

The treatment of the "groups" key as an array of strings continues in subiquity.

Both "lock-passwd" and "groups" conflict with /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json.

That would be more minor issue if the warning put into multiple log files didn't contain the password hash that is otherwise only readable by root.

CVE References

2022-2084

Revision history for this message

James Falcon (falcojr) wrote on 2022年06月13日: #1

Thanks Mike. We'll get this fixed ASAP.

Changed in cloud-init:

status: New → Triaged

importance: Undecided → Critical

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2022年06月13日: #2

Thanks Mike,

James, will these errors halt the install process entirely? Or will it continue on without expected users?

Will fixing this (adding aliases?) cause unexpected *success* in cloud-config use in deployments? If we add aliases, what happens if someone supplies both lock_passwd and lock-passwd keys in their user data?

I'm a bit surprised this is the first we're hearing a consequence of a change from two years ago.

Thanks

Revision history for this message

James Falcon (falcojr) wrote on 2022年06月13日: #3

The cloud-config from this bug contains 'lock-passwd' which is still supported but now deprecated. We logged the bit of the cloud-config that doesn't pass validation, but in doing so we have exposed sensitive user data.

To specifically answer you questions:
will these errors halt the install process entirely?
No.

Or will it continue on without expected users?
Neither. It will do the right thing. It is complaining about the supplied keys not matching the schema.

Will fixing this (adding aliases?) cause unexpected *success* in cloud-config use in deployments?
We won't be changing aliases for the fix. The fix is to write a more generic warning to the log telling the user to run a command as root to see why their config is invalid.

If we add aliases, what happens if someone supplies both lock_passwd and lock-passwd keys in their user data?
This is currently already valid. We're trying to deprecate 'lock-passwd'

"I'm a bit surprised this is the first we're hearing a consequence of a change from two years ago."
The change is in 22.2 and was SRUed last week.

A little more background might be helpful here. Nothing has changed in the code handling of users or groups. The thing that changed is that we have added schema validation to our cloud-init cloud-config. In the users/groups module being used here, cloud-init has supported keys with dashes '-' and keys with underscores '_', though the documentation for the past 4 years or so have only shown underscores. When we introduced the schema to this module, we effectively deprecated the dashes, and in the future we'll only be supporting underscores. If a module fails schema validation, we currently warn but do not raise an exception.
The cloud-config from this bug contains 'lock-passwd' which is still supported but now deprecated. We logged the bit of the cloud-config that doesn't pass validation, but in doing so we have exposed sensitive user data.
To specifically answer you questions:
will these errors halt the install process entirely? 
No.
Or will it continue on without expected users?
Neither. It will do the right thing. It is complaining about the supplied keys not matching the schema.
Will fixing this (adding aliases?) cause unexpected *success* in cloud-config use in deployments? 
We won't be changing aliases for the fix. The fix is to write a more generic warning to the log telling the user to run a command as root to see why their config is invalid.
If we add aliases, what happens if someone supplies both lock_passwd and lock-passwd keys in their user data?
This is currently already valid. We're trying to deprecate 'lock-passwd'
"I'm a bit surprised this is the first we're hearing a consequence of a change from two years ago."
The change is in 22.2 and was SRUed last week.

Revision history for this message

Mike Stroyan (stroyan) wrote on 2022年06月13日: #4

diff --git a/cloudinit/config/schemas/schema-cloud-config-v1.json b/cloudinit/config/schemas/schema-cloud-config-v1.json
index a5be310a..4e07c0c9 100644
--- a/cloudinit/config/schemas/schema-cloud-config-v1.json
+++ b/cloudinit/config/schemas/schema-cloud-config-v1.json
@@ -35,7 +35,8 @@
},
"groups": {
"description": "Optional comma-separated string of groups to add the user to.",
- "type": "string"
+ "type": ["array", "string"],
+ "items": {"type": "string"}
},
"homedir": {
"description": "Optional home dir for user. Default: ``/home/<username>``",
@@ -51,6 +52,11 @@
"description": "Disable password login. Default: ``true``",
"type": "boolean"
},
+ "lock-passwd": {
+ "default": true,
+ "description": "Disable password login. Default: ``true``",
+ "type": "boolean"
+ },
"no_create_home": {
"default": false,
"description": "Do not create home directory. Default: ``false``",

Note that the warning message from the schema check is triggered by both the "lock-passwd" key and the representation of users_groups.user groups key as an array of strings rather than a single string with comma separated group names. I need to change both of those in the json file to make it accept the way that Ubuntu subiquity installer wrote the cloud config.
diff --git a/cloudinit/config/schemas/schema-cloud-config-v1.json b/cloudinit/config/schemas/schema-cloud-config-v1.json
index a5be310a..4e07c0c9 100644
--- a/cloudinit/config/schemas/schema-cloud-config-v1.json
+++ b/cloudinit/config/schemas/schema-cloud-config-v1.json
@@ -35,7 +35,8 @@
 },
 "groups": {
 "description": "Optional comma-separated string of groups to add the user to.",
- "type": "string"
+ "type": ["array", "string"],
+ "items": {"type": "string"}
 },
 "homedir": {
 "description": "Optional home dir for user. Default: ``/home/<username>``",
@@ -51,6 +52,11 @@
 "description": "Disable password login. Default: ``true``",
 "type": "boolean"
 },
+ "lock-passwd": {
+ "default": true,
+ "description": "Disable password login. Default: ``true``",
+ "type": "boolean"
+ },
 "no_create_home": {
 "default": false,
 "description": "Do not create home directory. Default: ``false``",

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2022年06月14日: #5

Thanks for the explanations James,

Please use CVE-2022-2084 for this issue.

Thanks

Revision history for this message

Chad Smith (chad.smith) wrote on 2022年06月14日: #6

Thanks Mike for the context here and spending time understanding the symptoms.

In our deprecation of this key, we should have also added a DEPRECATED description to this in our schema. Looks like we covered that approach in other keys like `grub-dpkg` -> grub_dpkg[1] and `remove-defaults` -> remove_defaults[2] but not in this case. We have PRs in flight to fix this and will make sure subiquity is also onboard with our deprecation schedule for these keys too so we don't re-introduce this problem when the hyphenated keys are officially dropped in upstream.

References:
[1] https://github.com/canonical/cloud-init/blob/main/cloudinit/config/schemas/schema-cloud-config-v1.json#L760
[2] https://github.com/canonical/cloud-init/blob/main/cloudinit/config/schemas/schema-cloud-config-v1.json#L167

Revision history for this message

James Falcon (falcojr) wrote on 2022年06月15日: #7

Hey Seth, what do you think about the severity for this bug?

Revision history for this message

Seth Arnold (seth-arnold) wrote on 2022年06月16日: #8

Chad, James, I've added you both to https://launchpad.net/~ubuntu-security/+archive/ubuntu/ubuntu-security-collab -- I believe 'dput' should work to this ppa.

Thanks

Revision history for this message

Chad Smith (chad.smith) wrote on 2022年06月28日: #9

Thanks Seth.
Email sent out on this Friday with the suggested fix, the packages have been uploaded to this private PPA and are available for security review and upload into the appropriate -security pockets for Bionic, Focal, Impish, and Jammy.

Revision history for this message

Chad Smith (chad.smith) wrote on 2022年06月28日: #10

Download full text (8.8 KiB)

Proposed fix, generalize schema validation warnings messages to avoid reporting potentially sensitive user-data values in /var/log/cloud-init.log.

cloud-init.postinst fixes to redact historic sensitive logs.

diff --git a/debian/changelog b/debian/changelog
index 135671ae..baec8fd1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+cloud-init (22.2-0ubuntu1~22.04.3) jammy; urgency=medium
+
+ * d/cloud-init.postinst: redact previously leaked schema errors from logs
+ * Remove schema errors from log (LP: #1978422) (CVE-2022-2084)
+
+ -- James Falcon <email address hidden> 2022年6月14日 06:31:00 -0500
+
cloud-init (22.2-0ubuntu1~22.04.2) jammy; urgency=medium

* cherry-pick a2e62738: Fix cc_phone_home requiring 'tries' (#1500)
diff --git a/debian/cloud-init.postinst b/debian/cloud-init.postinst
index 683ba86d..85788a98 100644
--- a/debian/cloud-init.postinst
+++ b/debian/cloud-init.postinst
@@ -125,6 +125,27 @@ handle_preseed_local_cloud_config() {
db_unregister "${debconf_name}" || :
}

+fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema() {
+ local oldver="1ドル" last_bad_ver="22.2-0ubuntu1~22.04.2"
+ dpkg --compare-versions "$oldver" le "$last_bad_ver" || return 0
+
+ MSG="Redacting sensitive logs due to invalid cloud-config user-data from"
+ INVALID_USERDATA_LOG="Invalid cloud-config provided:"
+ if grep -q "${INVALID_USERDATA_LOG}" /var/log/cloud-init.log; then
+ echo "${MSG} /var/log/cloud-init.log"
+ # Redact all schema warnings between
+ # 'Invalid cloud-config provided:' and the next timestamped log 2022-
+ sed -i '/Invalid cloud-config provided:/,/2022-/{/^[^2202-]/d};s/Invalid cloud-config provided:.*/Invalid cloud-config provided. To see errors, run: sudo cloud-init schema --system/' /var/log/cloud-init.log
+ fi
+ if grep -q "${INVALID_USERDATA_LOG}" /var/log/cloud-init-output.log; then
+ echo "${MSG} /var/log/cloud-init-output.log"
+ # Redact all schema warnings between
+ # 'Invalid cloud-config provided:' and the public/private key gen at
+ # 'Generating public/private rsa key pair' OR 'Cloud'
+ sed -i '/Invalid cloud-config provided:/,/Generating\|Cloud/{/Cloud/b; /^[^Generating]/d};s/Invalid cloud-config provided:.*/Invalid cloud-config provided. To see errors, run: sudo cloud-init schema --system\nGenerating public\/private rsa key pair./' /var/log/cloud-init-output.log
+ fi
+}
+
fix_1336855() {
### Begin fix for LP: 1336855
# fix issue where cloud-init misidentifies the location of grub and
@@ -375,6 +396,9 @@ EOF
cleanup_ureadahead "2ドル"
fix_lp1889555 "2ドル"
change_cloud_init_output_log_permissions "2ドル"
+
+ # Redact schema sensitive warning logs on invalid user-data
+ fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema "2ドル"
fi

#DEBHELPER#
diff --git a/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log b/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log
new file mode 100644
index 00000000..01e886d7
--- /dev/null
+++ b/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log
@@ -0,0 +1,148 @@
+From b0534cbf05221b141ebd2edb5a71e94742b...

Proposed fix, generalize schema validation warnings messages to avoid reporting potentially sensitive user-data values in /var/log/cloud-init.log.
cloud-init.postinst fixes to redact historic sensitive logs.
diff --git a/debian/changelog b/debian/changelog
index 135671ae..baec8fd1 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,10 @@
+cloud-init (22.2-0ubuntu1~22.04.3) jammy; urgency=medium
+
+ * d/cloud-init.postinst: redact previously leaked schema errors from logs
+ * Remove schema errors from log (LP: #1978422) (CVE-2022-2084)
+
+ -- James Falcon <james.falcon@canonical.com> 2022年6月14日 06:31:00 -0500
+
 cloud-init (22.2-0ubuntu1~22.04.2) jammy; urgency=medium
 
 * cherry-pick a2e62738: Fix cc_phone_home requiring 'tries' (#1500)
diff --git a/debian/cloud-init.postinst b/debian/cloud-init.postinst
index 683ba86d..85788a98 100644
--- a/debian/cloud-init.postinst
+++ b/debian/cloud-init.postinst
@@ -125,6 +125,27 @@ handle_preseed_local_cloud_config() {
 db_unregister "${debconf_name}" || :
 }
 
+fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema() {
+ local oldver="1ドル" last_bad_ver="22.2-0ubuntu1~22.04.2"
+ dpkg --compare-versions "$oldver" le "$last_bad_ver" || return 0
+
+ MSG="Redacting sensitive logs due to invalid cloud-config user-data from"
+ INVALID_USERDATA_LOG="Invalid cloud-config provided:"
+ if grep -q "${INVALID_USERDATA_LOG}" /var/log/cloud-init.log; then
+ echo "${MSG} /var/log/cloud-init.log"
+ # Redact all schema warnings between
+ # 'Invalid cloud-config provided:' and the next timestamped log 2022-
+ sed -i '/Invalid cloud-config provided:/,/2022-/{/^[^2202-]/d};s/Invalid cloud-config provided:.*/Invalid cloud-config provided. To see errors, run: sudo cloud-init schema --system/' /var/log/cloud-init.log
+ fi
+ if grep -q "${INVALID_USERDATA_LOG}" /var/log/cloud-init-output.log; then
+ echo "${MSG} /var/log/cloud-init-output.log"
+ # Redact all schema warnings between
+ # 'Invalid cloud-config provided:' and the public/private key gen at
+ # 'Generating public/private rsa key pair' OR 'Cloud'
+ sed -i '/Invalid cloud-config provided:/,/Generating\|Cloud/{/Cloud/b; /^[^Generating]/d};s/Invalid cloud-config provided:.*/Invalid cloud-config provided. To see errors, run: sudo cloud-init schema --system\nGenerating public\/private rsa key pair./' /var/log/cloud-init-output.log
+ fi
+}
+
 fix_1336855() {
 ### Begin fix for LP: 1336855
 # fix issue where cloud-init misidentifies the location of grub and
@@ -375,6 +396,9 @@ EOF
 cleanup_ureadahead "2ドル"
 fix_lp1889555 "2ドル"
 change_cloud_init_output_log_permissions "2ドル"
+
+ # Redact schema sensitive warning logs on invalid user-data
+ fix_1978422_redact_sensitive_logs_on_invalid_userdata_schema "2ドル"
 fi
 
 #DEBHELPER#
diff --git a/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log b/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log
new file mode 100644
index 00000000..01e886d7
--- /dev/null
+++ b/debian/patches/cpick-b0534cbf-Remove-schema-errors-from-log
@@ -0,0 +1,148 @@
+From b0534cbf05221b141ebd2edb5a71e94742b369a3 Mon Sep 17 00:00:00 2001
+From: James Falcon <james.falcon@canonical.com>
+Date: 2022年6月14日 06:24:40 -0500
+Subject: [PATCH] Remove schema errors from log
+
+When schema errors are encountered, the section of userdata in question
+gets printed to the cloud-init log. As this could contain sensitive
+data, so log a generic warning instead and redirect user to run
+`cloud-init schema --system` as root.
+
+LP: #1978422
+---
+ cloudinit/cmd/main.py | 4 +++-
+ cloudinit/config/schema.py | 15 +++++++++++---
+ tests/integration_tests/modules/test_cli.py | 20 ++++++++++++------
+ tests/unittests/config/test_schema.py | 23 ++++++++++++++++++++-
+ 4 files changed, 51 insertions(+), 11 deletions(-)
+
+--- a/cloudinit/cmd/main.py
++++ b/cloudinit/cmd/main.py
+@@ -454,7 +454,9 @@ def main_init(name, args):
+ 
+ # Validate user-data adheres to schema definition
+ if os.path.exists(init.paths.get_ipath_cur("userdata_raw")):
+- validate_cloudconfig_schema(config=init.cfg, strict=False)
++ validate_cloudconfig_schema(
++ config=init.cfg, strict=False, log_details=False
++ )
+ else:
+ LOG.debug("Skipping user-data validation. No user-data found.")
+ 
+--- a/cloudinit/config/schema.py
++++ b/cloudinit/config/schema.py
+@@ -196,6 +196,7 @@ def validate_cloudconfig_schema(
+ schema: dict = None,
+ strict: bool = False,
+ strict_metaschema: bool = False,
++ log_details: bool = True,
+ ):
+ """Validate provided config meets the schema definition.
+ 
+@@ -208,6 +209,9 @@ def validate_cloudconfig_schema(
+ logging warnings.
+ @param strict_metaschema: Boolean, when True validates schema using strict
+ metaschema definition at runtime (currently unused)
++ @param log_details: Boolean, when True logs details of validation errors.
++ If there are concerns about logging sensitive userdata, this should
++ be set to False.
+ 
+ @raises: SchemaValidationError when provided config does not validate
+ against the provided schema.
+@@ -232,12 +236,17 @@ def validate_cloudconfig_schema(
+ errors += ((path, error.message),)
+ if errors:
+ if strict:
++ # This could output/log sensitive data
+ raise SchemaValidationError(errors)
+- else:
++ if log_details:
+ messages = ["{0}: {1}".format(k, msg) for k, msg in errors]
+- LOG.warning(
+- "Invalid cloud-config provided:\n%s", "\n".join(messages)
++ details = "\n" + "\n".join(messages)
++ else:
++ details = (
++ "Please run 'sudo cloud-init schema --system' to "
++ "see the schema errors."
+ )
++ LOG.warning("Invalid cloud-config provided: %s", details)
+ 
+ 
+ def annotated_cloudconfig_file(
+--- a/tests/integration_tests/modules/test_cli.py
++++ b/tests/integration_tests/modules/test_cli.py
+@@ -18,11 +18,18 @@ runcmd:
+ - echo 'hi' > /var/tmp/test
+ """
+ 
++# The '-' in 'hashed-password' fails schema validation
+ INVALID_USER_DATA_SCHEMA = """\
+ #cloud-config
+-updates:
+- notnetwork: -1
+-apt_pipelining: bogus
++users:
++ - default
++ - name: newsuper
++ gecos: Big Stuff
++ groups: users, admin
++ sudo: ALL=(ALL) NOPASSWD:ALL
++ hashed-password: asdfasdf
++ shell: /bin/bash
++ lock_passwd: true
+ """
+ 
+ 
+@@ -69,11 +76,12 @@ def test_invalid_userdata_schema(client:
+ assert result.ok
+ log = client.read_from_file("/var/log/cloud-init.log")
+ warning = (
+- "[WARNING]: Invalid cloud-config provided:\napt_pipelining: 'bogus'"
+- " is not valid under any of the given schemas\nupdates: Additional"
+- " properties are not allowed ('notnetwork' was unexpected)"
++ "[WARNING]: Invalid cloud-config provided: Please run "
++ "'sudo cloud-init schema --system' to see the schema errors."
+ )
+ assert warning in log
++ assert "asdfasdf" not in log
++
+ result = client.execute("cloud-init status --long")
+ if not result.ok:
+ raise AssertionError(
+--- a/tests/unittests/config/test_schema.py
++++ b/tests/unittests/config/test_schema.py
+@@ -304,11 +304,32 @@ class TestValidateCloudConfigSchema:
+ assert "cloudinit.config.schema" == module
+ assert logging.WARNING == log_level
+ assert (
+- "Invalid cloud-config provided:\np1: -1 is not of type 'string'"
++ "Invalid cloud-config provided: \np1: -1 is not of type 'string'"
+ == log_msg
+ )
+ 
+ @skipUnlessJsonSchema()
++ def test_validateconfig_schema_sensitive(self, caplog):
++ """When log_details=False, ensure details are omitted"""
++ schema = {
++ "properties": {"hashed_password": {"type": "string"}},
++ "additionalProperties": False,
++ }
++ validate_cloudconfig_schema(
++ {"hashed-password": "secret"},
++ schema,
++ strict=False,
++ log_details=False,
++ )
++ [(module, log_level, log_msg)] = caplog.record_tuples
++ assert "cloudinit.config.schema" == module
++ assert logging.WARNING == log_level
++ assert (
++ "Invalid cloud-config provided: Please run 'sudo cloud-init "
++ "schema --system' to see the schema errors." == log_msg
++ )
++
++ @skipUnlessJsonSchema()
+ def test_validateconfig_schema_emits_warning_on_missing_jsonschema(
+ self, caplog
+ ):
diff --git a/debian/patches/series b/debian/patches/series
index 9e8c77f7..d44f86ae 100644
--- a/debian/patches/series
+++ b/debian/patches/series
@@ -1 +1,2 @@
 cpick-a2e62738-Fix-cc_phone_home-requiring-tries-1500
+cpick-b0534cbf-Remove-schema-errors-from-log

Revision history for this message

Chad Smith (chad.smith) wrote on 2022年06月29日: #18