Config for httpd as an nfs-client while also handling Kerberos auth of its clients #39

It would be necessary, at the very least to know what httpd and nfs configurations you are starting with.

We changed them some over time, so a pointer to the one you have installed on your system would be good.

Secondarily a description of exactly what you want to achieve with HTTPD as an NFS client is also needed. Does not need to delve into proprietary details, just what you expect to happen in general when Http is in the mix.

TLDR: I may have this working. See last paragraph.

I'm building my web server atop Fedora 35 and have joined the host to my FreeIPA domain.

One goal is to serve HTML content from remote storage that has been mounted with sec=krb5. If I mounted with sec=sys, there was no issue. With sec=krb5 however, httpd logs indicated that the apache user lacked read access even though the file/dir modes would otherwise be fine (as they were with sec=sys).

gssproxy-0.8.4-4.fc35.x86_64 to the rescue! It provides /etc/gssproxy/99-nfs-client.conf:

[service/nfs-client]
 mechs = krb5
 cred_store = keytab:/etc/krb5.keytab
 cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U
 cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab
 cred_usage = initiate
 allow_any_uid = yes
 trusted = yes
 euid = 0

That matches exactly (nice!) what I find at https://github.com/gssapi/gssproxy/blob/main/docs/NFS.md#keytab-based-client-initiation and all makes reasonably good sense to me, so I left this alone. I restarted httpd, gssproxy and sssd thinking that should cover everything and found it to work nicely.

The other goal I want to achieve is Kerberos authentication of the user accessing the same httpd service, say for something private. I took https://github.com/gssapi/gssproxy/blob/main/docs/Apache.md to be perfect for that. So I now have the following config for httpd:

<Location /private>
 AuthType GSSAPI
 AuthName "GSSAPI Login"
 Require valid-user
</Location>

and I couple that with the other steps for adding the service principal, getting the keytab, starting httpd with GSS_USE_PROXY=1. My rpm doesn't include the mentioned 80-httpd.conf, live, example, or otherwise (bummer). So I created my own /etc/gssproxy/80-httpd.conf from the git repo here, so it looks like:

[service/HTTP]
 mechs = krb5
 cred_store = keytab:/etc/gssproxy/http.keytab
 cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
 euid = apache

I restart the same services again, and it all works perfectly. That was a surprise because that's not how it went the first time and I think my attempts at correcting it only made things worse or me more confused. I've rebooted the server and it still works as expected. Maybe much of my problems were only in how I tested. I eventually did gain a new distinct impression that when getting a new keytab from the KDC I must also do a new kinit to renew myself as a client before testing access to services secured with those keytabs. If that is indeed true, perhaps that's where I messed up.

I would appreciate a review of my keytabs to ensure I didn't do something stupid & insecure. These have been sanitized of specifics. www is a CNAME for webby. I especially question the host principals for the 2nd and 3rd keytabs as I don't think I included those in my earliest (troubled) testing.

# klist -k /etc/krb5.keytab
Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
 1 host/webby.example.com@EXAMPLE.COM
 1 host/webby.example.com@EXAMPLE.COM
# klist -k /etc/gssproxy/http.keytab
Keytab name: FILE:/etc/gssproxy/http.keytab
KVNO Principal
---- --------------------------------------------------------------------------
 2 HTTP/webby.example.com@EXAMPLE.COM
 2 HTTP/webby.example.com@EXAMPLE.COM
 1 HTTP/www.example.com@EXAMPLE.COM
 1 HTTP/www.example.com@EXAMPLE.COM
 2 HTTP/webby.example.com@EXAMPLE.COM
 2 HTTP/webby.example.com@EXAMPLE.COM
 1 HTTP/www.example.com@EXAMPLE.COM
 1 HTTP/www.example.com@EXAMPLE.COM
 1 host/webby.example.com@EXAMPLE.COM
 1 host/webby.example.com@EXAMPLE.COM
# klist -k /var/lib/gssproxy/clients/48.keytab
Keytab name: FILE:/var/lib/gssproxy/clients/48.keytab
KVNO Principal
---- --------------------------------------------------------------------------
 1 apache@EXAMPLE.COM
 1 apache@EXAMPLE.COM
 1 apache@EXAMPLE.COM
 1 apache@EXAMPLE.COM
 1 host/webby.example.com@EXAMPLE.COM
 1 host/webby.example.com@EXAMPLE.COM

So you elected to create a principal named apache@ for file access, that is an acceptable solution to allow the httpd process access to the NFS share.
However I do not fully understand why you also added the host keys to that keytab. From a security pov only the gss-proxy should be given access to those keys, so there is no immediate issue of leaking them to the apache process. However an intruder that may gain privileges as the apache user might be able (in certain configurations) to obtain tickets to itself (the "host" principal) as any user, which could then be used to ssh into the host as any user.

Similarly for the http.keytab.

the host/fqdn principal can be considred the "machine key", and it is used by the SSH daemons to authenticate incoming users. It should not be randomly distributed to other services. And access permissions to keytabs containing those keys should be strictly reseved to system components (ssh, gssproxy, root user).

HTH.

Ah yes, the user principle for the file access was another piece I forgot to mention.
I'm quite certain my thinking for adding host keys to the keytab was because I was getting desperate as to why I struggled for an entire day with only one of the two goals achieved or errors. Sort of a nuke option. Since it's all working and given your gracious and excellently detailed security stance on it, I did the only reasonable thing and busted out kt_util to delete the host/ entries. Those are now gone from everywhere except /etc/krb5.tab and everything is still working.

I honestly can't say why I had so much trouble now, but things are always easier once you've actually seen the thing work. I am most appreciative of the security implications as that really helps my understanding of Kerberos in practice. Thank you so much!

I'm afraid my reported success has stopped. gssproxy behaves well to authorize me as Kerberos user to httpd but I am no longer to access files served by httpd when they are mounted with sec=krb5. httpd logs:

==> /var/log/httpd/ssl_error_log <==
[Wed Jan 19 15:57:57.597523 2022] [core:error] [pid 1266:tid 1294] (13)Permission denied: [client 192.168.10.12:51968] AH00035: access to /repos/index.html denied (filesystem path '/srv/www/repos/index.html') because search permissions are missing on a component of the path
[Wed Jan 19 15:57:57.815278 2022] [core:error] [pid 1266:tid 1294] (13)Permission denied: [client 192.168.10.12:51968] AH00035: access to /repos/index.php denied (filesystem path '/srv/www/repos/index.php') because search permissions are missing on a component of the path

repos/ is just a directory I'm serving with Options Indexes MultiViews FollowSymLinks. I'm not sure why the latter is attempted, but I have MediaWiki on the same server and with it PHP, of course. Anyway, neither file exists but a normal directory listing would be served if this directory was mounted instead with sec=sys. In any event, here are the relevant permissions:

# namei -l /srv/www/repos/
f: /srv/www/repos/
dr-xr-xr-x root root /
drwxr-xr-x root root srv
drwxr-xr-x root root www
drwxr-xr-x apache apache repos

I think I caught the problem in the gssproxy logs, however:

Jan 19 16:17:24 webby.example.com gssproxy[1964]: [2022年01月19日 21:17:24]: Client [2022年01月19日 21:17:24]: (/usr/sbin/rpc.gssd) [2022年01月19日 21:17:24]: connected (fd = 12)[2022年01月19日 21:17:24]: (pid = 759) (uid = 48) (gid = 48)[2022年01月19日 21:17:24]: (context = system_u:system_r:gssd_t:s0)[2022年01月19日 21:17:24]:
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022年01月19日 21:17:24]: Connection matched service HTTP
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022年01月19日 21:17:24]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48,socket: (null)
Jan 19 16:17:24 webby.example.com gssproxy[1964]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "HTTP/webby.example.com@EXAMPLE.COM" [ { "HTTP/webby.example.com@EXAMPLE.COM" { 1 2 840 113554 1 2 2 } ACCEPT 0 4294967295 } ] [ u..U..bm....MHR.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 )
Jan 19 16:17:24 webby.example.com gssproxy[1964]: GSSX_RES_ACQUIRE_CRED( status: { 720896 { 1 2 840 113554 1 2 2 } 100001 "The referenced credential has expired" "Success" [ ] } output_cred_handle: { "" [ ] [ ] 0 } )
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022年01月19日 21:17:24]: Connection matched service HTTP
Jan 19 16:17:24 webby.example.com gssproxy[1964]: [CID 12][2022年01月19日 21:17:24]: gp_rpc_execute: executing 6 (GSSX_ACQUIRE_CRED) for service "HTTP", euid: 48,socket: (null)
Jan 19 16:17:24 webby.example.com gssproxy[1964]: GSSX_ARG_ACQUIRE_CRED( call_ctx: { "" [ ] } input_cred_handle: { "HTTP/webby.example.com@EXAMPLE.COM" [ { "HTTP/webby.example.com@EXAMPLE.COM" { 1 2 840 113554 1 2 2 } ACCEPT 0 4294967295 } ] [ u..U..bm....MHR.... ] 0 } add_cred: 0 desired_name: <Null> time_req: 4294967295 desired_mechs: { { 1 2 840 113554 1 2 2 } } cred_usage: INITIATE initiator_time_req: 0 acceptor_time_req: 0 )
Jan 19 16:17:24 webby.example.com gssproxy[1964]: GSSX_RES_ACQUIRE_CRED( status: { 720896 { 1 2 840 113554 1 2 2 } 100001 "The referenced credential has expired" "Success" [ ] } output_cred_handle: { "" [ ] [ ] 0 } )

The part that catches my eye is that it's matching on the service HTTP when I would expect nfs-client here instead. If I'm right, that means httpd would be using the keys for the service principles (HTTP/www.example.com, etc.) instead of the apache user principle key. On that hunch, I thought I might be able to better constrain the service by adding program = /usr/sbin/httpd to 80-httpd.conf. After I did that, I once again restarted gssproxy, httpd and sssd and that looks to have resolved the problem.

However.... I can take that line back out, restart the services and still access the repos/ directory when I would have expected it to fail once again. I believe I want that line, but I don't understand my process to confirm that I do. Short of experts here, that is.

Ok, I can imagine how that might create issues. I've revised as you suggested and so far so good. Would it make sense to update the example documentation to steer others away from my mistake?