This is our story about building a multi-tenant Kubernetes environment that facilitates various DevOps teams (tenants) with their own Kubernetes namespace and private container registry.
A dance troupe has to know the choreography, so together they make a great show. Similarly, micro-services as dancers on a stage (Kubernetes) are instructed by their choreographers (DevOps engineers) about how to interact with each other.
This is my story is about building a multi-tenant Kubernetes environment that facilitates various DevOps teams (tenants) with their own Kubernetes namespace and private container registry (Harbor v2.1.0) with Single-Sign-On On (Keycloak v10.0.0) and service mesh (Istio 1.6.14) included.
Harbor provides a container image registry, vulnerability scanning, container image signature and validation, OIDC based authentication and authorization. The fully-featured version is composed of ten micro-services. It is also CNCF graduated OSS.
In Harbor, a project represents a container image registry, exposed under a unique URL, For example, “harbor.otomi.io/team-demo/”, where team-demo is a project name.
By creating many projects you can achieve a multi-tenant container image repository for workloads in your Kubernetes cluster.
In Harbor, you can also define project membership, first by defining OIDC groups (Figure 2) and then assigning them with a given project (Figure 3). For example, the OIDC group team-demo is a member of a team-demo project.
From the figure, the team-demo group has a Developer role that limits user’s permissions in this project.
Next, we want pods from a given Kubernetes namespace to pull container images from a private registry. The Harbor robot accounts are made for that purpose. I recommend creating two robot accounts in each Harbor project (see: Figure 4). The first one for using Kubernetes as a PullSecret at a given namespace and the second one for CI/CD pipeline.
The Keycloak application can be used as an identity provider. In Harbor, a user can log in by performing an OIDC authentication code flow with Keycloak. Upon successful authentication, a user is redirected back to Harbor with a JSON Web Token (JWT) signed by the identity Provider (Keycloak). The JWT contains a ID token.
Harbor can verify JWT signature and automatically assign a user to a role and a project, based on groups claim from the ID token.
The following code snippet present an ID token with a groups claim:
There is a “Joe Doe” user that belongs to team-dev and team-demo groups, which in Harbor can be matched to predefined OIDC groups. The ID token is issued by the Keycloak (iss property) that is running in the same Kubernetes cluster. Harbor can be configured to leverage ID tokens by specifying a set of authentication parameters that are presented in Figure 5.
There is an OIDC endpoint URL, which is matched against iss property from the ID token. Next, OIDC Client ID with OIDC client secret is used by Harbor to authenticate with a client at Keycloak. The group claim name is crucial for enabling Harbor OIDC group matching.
If you want to perform an automatic user onboarding process you should provide the following OIDC scopes: OpenID (iss and sub-properties) and email scope (email and email_verified properties).
Do not forget about Keycloak, which requires an additional configuration of the OIDC client.
As you can see in “Figure 6”, the client id is Otomi and the client secret is defined in the credentials tab. There are also valid redirect URLs and Web-origins that have to be set so a user can be redirected from and to the Harbor dashboard upon successful login.
The Istio ensures service interconnectivity, encrypted traffic (mTLS), and routing (VirtualService + Gateways). Integrating Harbor with Istio is mostly about setting up proper URI routing.
Harbor is composed of ten microservices:
The Virtual Services redirect URI paths into Harbor core service:
All other URI paths are redirected to the Harbor portal service (dashboard).
The destination hosts from harbor VirtualService is a Fully Qualified Domain Name (FQDN) that indicates the Kubernetes namespace of the Harbor services. It makes it possible for the Istio Ingress gateway to route the incoming traffic.
You might have noticed that traffic is routed to port 80 (HTTP) instead of 433 (HTTPS). It is because I disabled Harbor internal TLS in favor of the Istio proxy sidecar that enforces mTLS for each Harbor service.
With Otomi Container Platform (or simply Otomi), we strive to integrate best of breed Open-Source projects and provide multi-tenancy awareness out-of-the-box.
Multi-tenancy is challenging and requires configuration automation to ensure scalability.
Part of Otomi’s automation is to configure applications, so they are aware of each other. We do it either by using a declarative approach when that is possible, or else by interacting with their (REST) APIs directly.
We have generated REST API clients based on the open API specification for Harbor and Keycloak. You are welcome to use our factory
Next, we implemented idempotent tasks that leverage these REST API clients and automate service configuration. These are run as Kubernetes jobs whenever a configuration changes.For Harbor, we have automated the creation of projects, OIDC groups, project membership, and OIDC settings.
For Keycloak, we have automated configuration of the external identity provider, group names normalization, deriving Client ID, Client Secret and more. Are you inspired about both? Then take a look at our open-source code.
Each OSS project has its own goals and milestones, thus it may be challenging to integrate various projects to work together. Here, I share just a few Issues that I stumbled upon.
The container image registry, provided by Harbor, and Docker CLI do not support the OIDC protocol. Instead, it uses a username/password-based authentication. It means that whenever you perform Docker Login/push/pull commands, the HTTPS traffic from a docker client to the container registry does not carry JWT. Make sure to exclude /v1/, /v2/ and /service/ Harbor URI paths from the JWT verification. Otherwise, you won’t be able to use the registry.
Next, OIDC users may experience issues with their Docker credentials (CLI secrets) that suddenly are invalidated. The CLI secret depends on the validity of the ID token, which has nothing in common with the container registry. This hybrid security solution is something that a regular docker user does not expect and can be a source of many misunderstandings.
Follow this thread to get more insights.
The good news is that if you are an automation freak like me, you don’t actually need CLI secrets. Instead, you can use Harbor robot accounts that do not depend on OIDC authentication.
If your organization decides to migrate users to another identity provider you may experience a duplicated user error: “Conflict, the user with the same username or email has been onboarded”.
It is because sub and/or iss scopes from ID token may change, so the same user trying to login to the harbor dashboard will be treated as a new one. The onboarding process starts but fails because Harbor requires each user to have a unique email address. I ended up removing existing OIDC users from Harbor and allowing them to onboard once again. Interestingly the community of Harbor users is having a broad debate about using OIDC protocol and could not agree on a final solution so far. I encourage you to take a look at a very insightful conversation about it.
While making Harbor services part of the Istio service mesh, it is very important that Kubernetes services are using port names that follow the Istio convention. For example, the Harbor registry services should have an HTTP-registry port name, instead of a registry. See example:
If the service port name does not follow the Istio convention, Harbor core service is not able to communicate with the Harbor registry service in Istio service mesh. Attempting to login into the Docker registry will end up with an “authentication required” error.
I hope that this article provides you a good insight into more advanced Harbor integration in the Kubernetes cluster.
Interested in Otomi Container Platform? Request a free demo or try out our open source. Part of Otomi’s automation is to configure applications, so they are aware of each other. We do it either by using a declarative approach when that is possible, or else by interacting with their (REST) APIs directly. We have generated REST API clients based on the open API specification for Harbor and Keycloak. You are welcome to use our factory for building and publishing open API clients.
This article was originally posted by Jehoszafat Zimnowoda on medium.com.
Interested in Kubernetes, cloud-native and platforms to scale up your business? Get exclusive tech insights delivered straight to your inbox.