Microservices blindly trusting anything due to network segmentation?

Question

So I just fell in a project where microservices are inside private subnets and therefore aren't reacheable through the internet.

There is a balancer that can reach this microservices and this balancer is restricted through ACLs and Security Groups (AWS) to be accessible to very limited set of IPs (the IPs from the API Gateway).

At a very high level, this is the architecture:

architecture

As you see, the only entrypoint to everything is the API Gateway.

They have multiple front-end applications and automated backend systems that access the microservices through the API Gateway.

Until here, extremely common.

Currently, human users (using some websites, apps etc..) authenticate via a password flow (through the API Gatewat) that involves a backend service issuing a JWT. This JWT is wrapped in an opaque token by the API Gateway and passed to the front-end. The API Gateway validates the opaque token on each request, extracts the JWT, and forwards it to the microservices, which validate the JWT using a shared secret. In this authentication, the API Gateway routes the request to an authentication operation from the backend (this is only possible in their Gateway when grant_type is password).

However, automated applications authenticate directly with the API Gateway using grant_type=client_credentials. The Gateway returns an opaque token but does not issue a JWT, so the microservices cannot validate or authorize these automated requests.

When a request from these automated applications arrive, the API Gateway authenticates them based on the opaque token (just like it does with the front-end apps) but doesn't pass anything to the microservices.

The microservices simply blindly allows everything without a JWT to enter and accept every request. Basically it assumes these requests are safe because it already passed through all the network restrictions and client authentication on the API Gateway.

I once asked another person on the team about it and they told me that they must use the API Gateway to generate tokens, because all the consumer APPs for their APIs are registered within the API Gateway and then the API Gateway becomes a "fail-fast" mechanism for non-existing and non-authenticated entities. Also, they told me it is safe because of all the network segmentations.

Okay, understood, but it it hurts the Zero-Trust principle by allowing everything from within the boundaries of the network to reach the microservice without any validation whatsoever.

Their API Gateway restrictions are the following:

1. Their API Gateway can't issue an JWT;
2. Their API Gateway can't connect to an external authorization server (it has one buil-in);
3. Their API Gateway doesn't even expose an introspection endpoint to allow verification of the opaque token.

I'm thinking about some options on this, and the only one I'm seeing is using Client Certificates from the API Gateway to the microservices. But even so, the microservices are unable to authorize who is calling and therefore are unable to restrict which operations it can perform. If there was a JWT with scopes in it, the microservices could look and authorize based on that.

Morewise, this customer is extremely hard to deal with and will not help us by just telling us what we ask for and nothing more.

Given their API Gateway constraints and their current state, what would you suggest to enable security (not only network, as it is already secured) in machine-to-machine communication in this case?

• Philip Kendall: What you write is correct, but what problem are you actually trying to solve under what constraints? Asking for "thoughts and suggestions" is probably going to get closed on any Stack Exchange site as "bad subjective".
• Matheus: @PhilipKendall. . . I'm sorry, I was somewhat vague, I agree. I just edited my question to be more explicit on what I need help on.
• JimmyJames: "The Gateway returns an opaque token but does not issue a JWT" Is there are particular reason for this?
• JimmyJames: @Ewan It's normal in the sense that it's how things have been done in the past but I think the general consensus with security experts is that this is an outdated approach.
• JimmyJames: @Ewan An attacker can be on your network without direct access to a given host. I call this style the 'crustacean' security model. Once the shell is cracked, the guts just ooze out.

Answer 1

If you are worried that the services could be compromised, and want to convince your pairs or the stakeholders that this is a real possibility, your better option would be to do a demo where you do access those services without passing through the gateway (by using IP spoofing or tampering with the network traffic or any other means). "Hey, see by yourself how easy it is to hack our system" is much more convincing than any meetings filled with theoretical discussions about the evaluation of risks and mitigation strategies.

Note that while preparing switch a demo, you might encounter something you didn't know. Such as a network mechanism that would prevent you from performing IP spoofing in the first place.

Once the people who decide are convinced that there is a problem, be ready to tell them how to fix it. HTTPS connection with a client-side certificate is one thing. Another possible solution is to have a VPN between the gateway and the service.

---

As noted by Flater:

1. "Depending on context, hacking that system may be a crime in and of itself, and even if it is not, it may still land you in hot water at certain companies."

2. The answer applies only if you actually have the skills to do such demo.

• Greg Burghardt: I ran into a similar thing with our SSO solution. No amount of links to articles on the web or in-depth e-mails explaining the vulnerability works quite like a quick Invoke-WebRequest ... in PowerShell, open the resulting HTML file, and asking a manager, "Is this your home page? I didn't know your password." The resulting look on their face is all the argument you need.
• Flater: Two points of concern w.r.t. this answer: (a) Depending on context, hacking that system may be a crime in and of itself, and even if it is not, it may still land you in hot water at certain companies (b) I don't personally know how to hack a database but I don't think my advice on not storing plaintext passwords in our database should be ignored purely because I didn't have the skill to hack the database myself.
• Arseni Mourzenko: @Flater: those are two very important points, indeed. I integrated them in my answer, in case someone reads only the answer, but not the comments.
• JimmyJames: While it would be an effective demonstration, it's a logical fallacy to think that if you and your colleagues can't think of a way to hack something, no one can. This kind of hubris is a large part of the reason that the state of system and software security is such a travesty.

Answer 2

With this design you have chosen NOT to have MS -> MS security. This is fine.

Let's say you have a new MS and you want to make sure only Admin users can call its methods.

You can restrict this on the API gateway. The gateway can validate the token, check claims or client id and allow or deny access to specific microservices.

Ahh! but I want to make sure that other MS's don't call the Admin MS and forward the data!

Well don't write those microservices! If you did have full JWTs with claims passed around and you needed this type of functionality, you would just be adding the Admin claim to some JWTs used by MSs anyway. Its fully within your control to allow or not allow that functionality not writing the code.

---

Given your comments below that the API gateway can't restrict the user to a sub set of microservices or endpoints, I think my recommendation would be to replace the API gateway with an off the shelf product.

this keeps the internal architecture and the auth protocols the same, but allows fine grained control over user access

• Matheus: . . . "You can restrict this on the API gateway. The gateway can validate the token, check claims or client id and allow or deny access to specific microservices." . . . Actually, this is not possible. The API Gateway doesn't know the roles of the user that is authenticated and using that opaque token, because its authentication occurs on another backend service. Basically, when the grant_type=password, their API Gateway routes the token issuing request to a identity API that is configured to go to one microservice. The backend returns 200 OK instructing the API Gateway to generate the token.
• Ewan: ahh well, that is a bit unusual. Im not sure I follow your "API Gateway routes the token issuing request to a identity API that is configured to go to one microservice" But if the security is limited to all or nothing access that is a flaw
• Matheus: Yes, I find hard to grasp this "API Gateway routes the token issuing request to a identity API that is configured to go to one microservice". Basically, when the grant_type is password, the Gateway token issuing endpoint recognizes this and invokes a so called "Identity API" and this API calls the backend that validates usename and password and returns the HTTP 200 with JWT so that the Gateway can put in a memory grid. And I'm 100% on you: this is a flaw + I've never seen such architecture.
• Ewan: it seems normal up until you said "memory grid"
• Matheus: I think their API Gateway uses Infinispan as in-memory storage. That is what I meant by "memory grid"