Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Sign up
Appearance settings

Allow certificates to contain subjects with allowed external names #630

Open

Description

Overview

Currently, secret-operator can only sign/request certificates with IP/DNS names based on these scopes:

  • listener-volume (The external-name that is reported back by the Load Balancer controller1 ).
  • node (eg: 10.0.0.1 or node-1.local.domain. If I understand correctly, this becomes obsolete since listener-volume scope)
  • pod (STS) (eg: pod-0.app-service.app-namespace.svc.cluster.local)
  • service (FQDN of the named service, eg: app-service.app-namespace.svc.cluster.local

Customers often want to use an external2 DNS name, and in many cases use external-dns to automatically create DNS records for Services/Ingress/Gateway resources.

Common environment setups

In many environments (both Cloud and On-Prem):

  • The kubelet clusterDomain is not usable outside of the cluster (especially
    when it is the default of cluster.local), and
  • Customers often use names under a suffix separate from the kubelet
    clusterDomain. Often this is a shorter name (eg: app.team.example.com
    instead of app.app-service.app-namespace.cluster.domain).
  • DNS is often auto-configured using External DNS, which is configured in these
    ways:
    • For Services, an annotation is place to map the DNS name to the Service
      (NodePort, or LoadBalancer) IP.
    • For Ingress/Gateway, there are dedicated fields for specifying the DNS name.
  • Load balancers do TLS termination
    • In the case of internet-facing load balancers, this can be used to hide the
      subject names on the internal certificate which leak information such as the
      Kubernetes namespace and clusterDomain.

In some cases, they can work around this by using a Load Balancer that does L5 (TLS termination) or L7 (HTTP proxying), however with SNI checks becoming more prevalent these methods are becoming less effective when the external name used is not in the certificate used by the application.

This issue is about allowing external names to be added to certificates and a suggested approach follows...

Suggested approach

  1. Extend the SecretClass to allow a list of suffixes (and minDepth/maxDepth number of dots) for additional names that can be added to the certificate.
  2. Extend Listener with an externalName (or externalNames?) field.
    • If the listener-volume scope is used, and the suffix is permitted (within the minDepth/maxDepth bounds), then secret-operator will add that as a subject to the certificate request.
  3. Extend Listener with a serviceAnnotations field.
    • We could use this opportunity to consistently name the fields in Listener and ListenerClass (either all fields to be passed to Service are prefixed with service, or we put everything under ServiceOverrides.

Important

Question: If the externalName set on the listener is not permitted in the certificate (or minDepth/maxDepth are out of bounds), where should an error event go? It could go on the Listener, but the listener doesn't concern itself too much with TLS details.

Example config:

Have a SecretClass which allows signing names under certain additional suffixes.

apiVersion: secrets.stackable.tech/v1alpha1
kind: SecretClass
metadata:
 name: org-pki
spec:
 backend:
 # Using autoTls as an example, but same applies for any supported backend
 autoTls:
 ca:
 secret:
 name: secret-provisioner-tls-ca
 namespace: default
 autoGenerate: true
 maxCertificateLifetime: 15d
 # 👇 New
 allowedSuffixes: # maybe should be additionalSuffixes, since it is on top of existing names
 - suffix: internal.example.com
 maxDepth: 0 # allow anything under internal.example.com, eg: a.b.c.d.internal.example.com
 # or
 - suffix: internal.example.com
 maxDepth: 1 # default, only allow one dot between the name and the suffix, eg: a.internal.example.com
 # or
 - suffix: internal.example.com
 # Example: the organisation allows signing certs for $app.$team.internal.example.com
 # by requiring two dots between the name and the suffix
 # allow a.b.internal.example.com
 # disallow a.internal.example.com
 # disallow a.b.c.internal.example.com
 minDepth: 2
 maxDepth: 2
 # alternatively we could define depth and make it mutually exclusive to minDepth/maxDepth
 - suffix: internal.example.net

Have a ListenerClass ready for making public facing AWS NLBs with TLS termination

apiVersion: listeners.stackable.tech/v1alpha1
kind: ListenerClass
metadata:
 name: aws-ec2-nlb-public
spec:
 # I think it would make more sense to make a serviceOverrides key instead of prefixing some with "service" and missing it in others (eg: loadBalancerClass)
 serviceType: LoadBalancer
 # https://docs.aws.amazon.com/eks/latest/userguide/auto-configure-nlb.html#_sample_service
 loadBalancerClass: eks.amazonaws.com/nlb
 loadBalancerAllocateNodePorts: false
 preferredAddressType: HostnameConservative
 serviceExternalTrafficPolicy: Local
 serviceAnotations:
 service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
 service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip # Forward directly through node to Pod IP instead of an L3 hop/NAT through the node.
 service.beta.kubernetes.io/aws-load-balancer-attributes: >-
 proxy_protocol_v2.enabled=true

Have a NifiCluster, specifying an externalName (to be used in the resultant Listener) and service annotations to enable TLS temination on the Load Balancer, and a hostname to be configured by External DNS.

apiVersion: nifi.stackable.tech/v1alpha1
kind: NifiCluster
metadata:
 name: simple-nifi
spec:
 clusterConfig:
 tls:
 serverSecretClass: org-pki
 nodes:
 roleConfig:
 # I think this should be moved under listenerOverrides as className...
 listenerClass: aws-nlb-tls
 # 👇 New
 listenerOverrides:
 className: aws-nlb-tls # moved from roleConfig.listenerClass
 externalName: app.internal.example.com
 serviceAnnotations:
 external-dns.alpha.kubernetes.io/hostname: app.internal.example.com
 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:123456789012:certificate/4e12c4fe-eed9-48db-98d8-820b6b50ace4
 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "8443"

Based on the configurations above, the objects below should be created by the NiFi Operator:

# This is the Listener produced by NifiCluster
kind: Listener
metadata:
 name: the-nifi-listener
spec:
 className: aws-nlb-tls-public
 # 👇 New
 externalName: app.internal.example.com # this came from the NifiCluster overrides
 # 👇 Not yet available, see (see: https://github.com/stackabletech/listener-operator/issues/331)
 serviceAnnotations:
 external-dns.alpha.kubernetes.io/hostname: app.internal.example.com
 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:123456789012:certificate/4e12c4fe-eed9-48db-98d8-820b6b50ace4
 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "8443"
# This is the Pod produced by the Deployment produced by the NifiCluster
apiVersion: v1
kind: Pod
metadata:
 name: nifi-0
spec:
 volumes:
 - name: tls
 ephemeral:
 volumeClaimTemplate:
 metadata:
 annotations:
 secrets.stackable.tech/class: org-pki
 secrets.stackable.tech/scope: pod,service=nifi,listener-volume=the-nifi-listener

Based on the configurations above, the object below should be created by the Listener Operator:

apiVersion: v1
kind: Service
metadata:
 name: nifi-listener
 annotations:
 external-dns.alpha.kubernetes.io/hostname: app.internal.example.com
 service.beta.kubernetes.io/aws-load-balancer-attributes: proxy_protocol_v2.enabled=true
 service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
 service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
 service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:eu-central-1:123456789012:certificate/4e12c4fe-eed9-48db-98d8-820b6b50ace4
 service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "8443"
spec:
 type: LoadBalancer
 loadBalancerClass: eks.amazonaws.com/nlb
 loadBalancerAllocateNodePorts: false
 externalTrafficPolicy: Local
 ports: ...

Footnotes

  1. Often, this is a dynamic (and not user-friendly) name and is not intended to be used directly, but rather be the value of the DNS CNAME record of the intended external name. ↩

  2. Whether that is on the internet, or just within their organization - it is external to the Kubernetes cluster. ↩

Metadata

Metadata

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      AltStyle によって変換されたページ (->オリジナル) /