Getting Started with OAuth for Confluent Cloud Using Azure AD DS

Released in December 2022, OAuth support on Confluent Cloud allows Confluent Cloud users to integrate their own third-party identity provider (IdP) with Confluent Cloud, centralizing account management across all of their cloud services. This article explains how to configure Azure Active Directory DS (Azure AD DS) and Confluent Cloud so that the Azure Directory can be used to authenticate and authorize applications to use Confluent Cloud clusters. 

A detailed description of the OAuth authentication process can be found in the Confluent Documentation, but put simply the authentication flow is:

1. A Confluent Cloud organization administrator configures Confluent Cloud to trust the third-party IdP.

2. A Confluent Cloud organization administrator creates and configures identity pools in Confluent Cloud that will allow authenticated applications to access resources.

3. When a client wants to connect to Confluent Cloud:

   1. It first connects to the IdP over HTTPS, authenticates itself with the IdP, and requests a token using the "Client Credentials" grant.

   2. The IdP creates a JSON Web Token (JWT) for the client. This token contains a number of claims, such as the identity of the user or application.

   3. The client connects to Confluent Cloud using the SASL mechanism "OAUTHBEARER" and presents the JWT to the Confluent Cloud broker. The client specifies the identity pool it wants to use.

   4. The server validates that the JWT is valid and has been signed by a trusted IdP. If so, the server accepts the claims in the JWT.

   5. Confluent Cloud applies role-based access control (RBAC) authorization based on the client's identity pool and whether the rules for that pool allow clients with the claims in the JWT.

   6. The JWT will expire after some period. The client will periodically renew their JWT with the IdP.

4. Identity pools are used to grant access to resources based on the claims within the JWT. 

The OAuth authentication mechanism, introduced to Apache Kafka® in KIP-768, requires the client to request a "Client Credentials" grant from their IdP. This type of grant is intended for use by applications that are authenticating to other applications, so in that respect it is equivalent to creating a service account in Confluent Cloud and granting that account an API key.

As with many other Kafka features, it's possible to replace the Java classes that implement the default OAuth bearer authentication logic with your own classes if, for example, you need to use a different grant type. The important thing is that the application client has obtained a JWT from an IdP that Confluent Cloud had been configured to trust.

For more information, refer to the OAuth website or the OAuth on Confluent Cloud documentation.

Configuration

Each component of the OAuth mechanism requires different configuration settings.

Configuring the IdP to authenticate clients

The configuration settings required to configure the OAuth identity provider will depend on that system. This document provides examples for Azure Active Directory Domain Services, but other systems exist that can provide these services. 

When configured, the IdP will provide an Issuer URI and a JWKS URI.

In the IdP, you need to create at least:

1. A client application, identified by a client ID.

2. A secret for that client application.

3. A scope for that client application. This will depend on the IdP. Strictly speaking, requests are not required to provide a scope, but servers may reject a request that does not provide one.

Configuring Confluent Cloud to use an IdP

To configure Confluent Cloud to use an IdP you will need the following:

1. The Issuer URI: This URI uniquely identifies the IdP.

2. The JWKS URI: This URI allows the service (Confluent Cloud) to download the IdP's public keys. These public keys are used by Confluent Cloud to verify the signature on the OAuth JWT tokens presented by clients.

These URIs are obtained from the IdP configuration. Note that in many cases the IdP will provide a "Discovery URL" in the format "https://{the IdP's domain}/[IdP-specific resource/].well-known/openid-configuration". The exact format depends on the IdP. A http GET from that URL will return a JSON document containing the jwks_uri and issuer fields. Confluent Cloud can extract those fields from the Discovery URL automatically.

Configuring the client to use OAuth

Apache Kafka Client 3.2.1 or later is required for Java clients. Confluent's librdkafka library v1.9.2 or later is required for clients using Confluent's C++, Python, Go, or .NET clients. 

Client settings are listed in the documentation, but briefly, the Kafka client uses a JAAS callback handler to authenticate to the IdP before passing the JWT to Kafka. The callback handler needs to be configured with the URI of the IdP and a JAAS configuration containing the user's credentials for the IdP and some additional data about which cluster and resource pool in Confluent Cloud will be used. The configuration items are:

You will need to obtain your Kafka cluster ID and identity pool ID from your Confluent Cloud administrator. 

If you are the Confluent Cloud administrator, a cluster's ID can be found in the Cluster Settings page for the cluster in the Confluent Cloud console, via the confluent kafka cluster list command in the Confluent CLI, or by querying the Confluent Cloud REST API.

Identity pool IDs are listed under the identity provider's details in the Confluent Cloud console and can be listed using the confluent iam pool list command in the Confluent CLI or by querying the Confluent Cloud REST API.

Setting up Azure AD as an IdP

Microsoft's explanation of this flow can be found on their website.

This example shows how to configure an Azure AD tenant to allow the OAuth login to Confluent Cloud. A tenant "Acme Examples Inc" was created for this example, allowing AD examples to be created.

Note the tenant ID and primary domain from the directory.

In this example, the tenant ID is "67e9d472-7fbe-48f4-ade6-03e3f3c0dbe6" and the primary domain is "acmeexamples.onmicrosoft.com".

For this example, we will register two applications in Azure AD. One application represents Confluent Cloud and will allow us to configure the roles that client applications can claim in Confluent Cloud. 

The second is a client application. This application will have a secret to allow it to authenticate to Active Directory and will be assigned one or more roles defined by the Confluent Cloud application we registered.

Let's start with the Confluent Cloud registration.

1. In the directory management screen, choose Add -> App registration to add a new application registration.

2. Give the application a name that reflects the fact that it will grant permissions for Confluent Cloud. The application does not need a redirect URI.

3. Once registered, the application will be assigned an ID. Click through the Add an Application ID URI link to add a URI for this service.

4. Click the Set link.

5. Set the application URI, or keep the default. This string will be used by the client applications to identify this application.

6. Once the Application URI is set, go to App roles.

7. Create the first App role by selecting Create app role.

8. Create a role for Confluent Cloud. The fields to fill in are:

   1. Display name: a meaningful name for Azure to display.

   2. Allowed member types: Select Applications.

   3. Value: This is the string that will appear in the claims section of the JWT.

   4. Description: A useful description of what this role is supposed to allow.

9. Create as many roles as are appropriate for your authentication needs.

10. Do not skip this step! It is easy to miss but it is necessary. We then need to update the application's manifest to ensure that Azure AD actually returns version 2 tokens on its version 2 endpoint. Go to the Manifest in the left-hand menu. Find the key for accessTokenAcceptedVersion and update the value from null to 2, so that it looks like the values below.

11. Make sure you have not skipped step 10. We've seen a lot of our customers hit this problem.

12. Save the manifest.

13. Return to the Overview and view the Endpoints.

14. Here you can see that the Discovery URL is https://login.microsoftonline.com/67e9d472-7fbe-48f4-ade6-03e3f3c0dbe6/v2.0/.well-known/openid-configuration. This allows the OAuth client to discover the other important URLs.

15. The OAuthv2.0 token endpoint is https://login.microsoftonline.com/67e9d472-7fbe-48f4-ade6-03e3f3c0dbe6/oauth2/v2.0/token.

16. Return to the app registrations page and add a new application. This will register a client application of Confluent Cloud.

17. Name the client application. Again, no redirect URI is necessary.

18. Go into Certificates & secrets on the left-hand menu and create a new client secret.

19. Like many other secrets, this secret will soon be hidden, so copy it now.

20. Now go to the API permissions tab and select Add a permission.

21. In the Request API permissions blade, select the My APIs option.

22. Select your Confluent Cloud registration.

23. Add the roles this client application will need according to your Confluent Cloud registration.

24. After adding the permissions, click the Grant admin consent for … button and confirm the grant. In this case, we can grant the admin consent because we are a domain admin. If you are not a domain admin, you may need to be granted admin rights to the Confluent Cloud application in order to grant this consent to the client application.

25. Confirm that the admin consent has been granted.

This completes the Azure AD configuration for a client application. 

We can now generate a JWT with a curl command that POSTs a form payload to the OAuth 2.0 token endpoint: 

curl -X POST -H "Content-Type: application/x-www-form-urlencoded" -d 'client_id=75f2f796-363d-4de2-b299-713920db01a7&scope=api://be587d08-a466-45e9-a02a-dfdda6b4c765/.default&client_secret=yOZ8QPiSK_Zh9IA5HzicuY.lKnlkaVIkpuxc35&grant_type=client_credentials' 'https://login.microsoftonline.com/67e9d472-7fbe-48f4-ade6-03e3f3c0dbe6/oauth2/v2.0/token'

Note that in this command we're providing the client id, the client secret, the scope, and the grant type of client_credentials. The request URL contains the tenant ID.

The response should be a JSON Web Token like this:

{
  "token_type": "Bearer",
  "expires_in": 3599,
  "ext_expires_in": 3599,
  "access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiGERYUE54gpZCI6Ii1LSTNROW5OUjdiUm9meG1lWm9YdldyJ9
.
eyJhdQiOiJiZTU4N2QwOC1hNDY2LTQ1ZTktYTAyYS1kZmRkYTZiNGM3NjUiLCJpc3MiOiJodHRwczovL2xvZ2luLm1pY3Jvc29mdG9ubGluZS5jb20vNjdlOWQ0NzItN2ZiZS00OGY0LWFkZTYtMDNlM2YzYzBkYmU2L3YYXQiOjE2Nzk1NDY0MDAsIm5iZiI6MTY3OTU0NjQwMCwiZXhwIjoxNjc5NTUwMzAwLCJhaW8iOiJFMlpnWUtoaXRVL1N1TE0zTDcrQ3cvWmJaZTFXQUE9PSIsImF6cCI6IjNkNmFjMzRlLWFjZmMtNDVkYi04Y2MwLTJkMjc4MTMtNGQxMS04MjJhLTI1M2VmMjgzNWI0YyIsInRpZCI6IjY3ZTlkNDcyLTdmYmUtNDhmNC1hZGU2LTAzZTNmM2MwZGJlNiIsInV0aSI6IlBRZTlWYWF3aEVPZmdrQlhKRE0tQUEiLCJ2ZXIiOiIyLjAifQ
.
soofLrQD8dShrGoKH71X_7K4pFZU9ne7xxrJxNOe-KbsmUEAi2setYY0b0QzOSLnb_zz-bq_mZtMyJkZkVswoe6QQeKvKdR52YYd7n2JveqLDwXieS9dXDIbjAoKF4VoGlAKiVzBZ7f7u_OdA_a9WgfD6QWct6AP8hGvIqesp0CgGvWkqO3mwAzr8CyospN2iXHny2KWDDG8oSivdgTVqFGQO-DwZy9qdLJy6TSGtGwszwn2vre9Ec3Ka41ieKMOKLAgUNZaDBSHJhnFMpwByagLe-nOfY6yTJlQ"
}

This example has been formatted to highlight that the token is split into three parts, separated by "." characters. The middle part is the payload. This token can be decoded using a tool such as https://jwt.io.

The payload for the above token is:

{
  "aud": "be587d08-a466-45e9-a02a-dfdda6b4c765",
  "iss": "https://login.microsoftonline.com/67e9d472-7fbe-48f4-03e38e10dbe6/v2.0",
  "iat": 1679546400,
  "nbf": 1679546400,
  "exp": 1679550300,
  "aio": "E2ZgYKhitU/SuLM3L7+Cw/ZbZe1WAA==",
  "azp": "3d6ac34e-acfc-45db-8cc0-2d2781097e33",
  "azpacr": "1",
  "oid": "b1841e28-d943-4d11-822a-253ef2835b4c",
  "rh": "0.AUIActTpZ75_9Eit5gPj88Db5gh9WL5mpOlFoCrf3aa0x2VCAAA.",
  "roles": [
    "dev.environmentAdmin"
  ],
  "sub": "b1841e28-d943-4d11-822a-253ef2835b4c",
  "tid": "67e9d472-7fbe-48f4-ade6-03e3f3c0dbe6",
  "uti": "PQe9VaawhEOfgkBXJDM-AA",
  "ver": "2.0"
}

The aud value is the Application ID of the client application we created.

The sub value is the application's Object ID in Azure AD.

The roles value contains the role the client application has been assigned. Note that this is an array. The application may be assigned multiple roles.

Configuring Confluent Cloud for AD IdP

To configure Confluent Cloud we have to first register the IdP.

Go to Accounts & access from the menu in the top right, and select the Identity providers tab.

Select the Add Identity Providers button and choose Azure AD, then press Next.

In the next screen, enter a name for this provider, a description of the provider, and the tenant ID from Azure. Pressing the Import from Tenant ID button will populate the Issuer and JWKS URI fields.

This creates the provider, but we still have to configure identity pools. Do this by clicking the identity provider link.

From this screen, we can create a new identity pool by pressing the Add identity pool button.

The client pool needs a name, an identity claim, and a filter.

The identity claim is the field of the token that we consider to be the principal of the application logging in, for the purposes of audit. It defaults to claims.sub (subject) but in this example, we have set it to claims.azp (authorized party).

The filter specifies which identities will be included in this pool by inspecting the contents of the token. If the application registrations have been set up as described earlier, a rule such as 'dev.environmentAdmin' in claims.roles will be used to include clients in this identity pool if they have been assigned the Dev Admin role for the Confluent Cloud application in Azure AD. Note that we are using the Advanced tab, since claims.roles may contain multiple roles. 

The next step is to assign permission to the identity pool using RBAC. Import permissions will apply the RBAC permissions of an existing service account. Add new permissions will allow new role bindings to be applied to this identity pool.

After assigning RBAC permissions, you'll be prompted to validate and save the identity pool.

If all goes well, you will be returned to the provider's screen:

Note the pool ID, clients will need to supply it when they authenticate.

Using identity pools

In Confluent Cloud, identity pools are used to authorize actions by OAuth-authenticated clients. In combination with matching roles, scopes, or profiles defined for Confluent Cloud within the IdP, this allows security administration for Confluent Cloud to be externalized to the IdP.

Each identity pool is associated with a set of RBAC role bindings that a client requires in order to perform its role. Clients can join the pool and inherit the pool's role bindings if they have a claim that matches the pool's filter.

Exactly which claim should be matched depends on the IdP used and what information that IdP puts into the JWTs it generates. Given that different client applications might have different authorizations within Confluent Cloud, the IdP needs some mechanism to distinguish which Confluent Cloud resources the IdP has authorized the client application to use. 

The Azure IdP uses a single default scope for the application API that represents Confluent Cloud. The Azure IdP then allows "App roles" to be defined within that scope and client applications can be assigned those app roles. Any app roles assigned to a client are included in the JWT.

If we needed a role in our Confluent Cloud organization that allowed applications to have the CloudClusterAdmin role binding for a development cluster we could create the Dev.ClusterAdmin app role and assign client applications that role.

In your organization, you should plan your identity pools according to the needs of your applications. Remember that an identity pool can be used by different applications just by setting the right filters, so if you have a resource that is used by lots of different applications they can easily all use the same identity pool when they authenticate. 

This helps to reduce the number of service accounts and API keys needed in Confluent Cloud for resources that you intend to make widely available.

In contrast, an application that has complex authorization requirements might be the only application that uses a particular identity pool. In this case, the identity pools act a lot like a service account in terms of authorization, but without having to manage a service account and API keys in your Confluent Cloud account.

Also remember that an application is required to specify an identity pool when it authenticates to Confluent Cloud, so a single identity could be used to authenticate to Confluent Cloud, but different identity pools could be used to grant access to development or production clusters. Maybe you have one identity pool for all applications in your Dev environment that grants every application full access, but in Prod you have an identity pool defined for each application that only authorizes access to the resources that application needs. In this case, you could promote an application from Dev to Prod in Confluent Cloud by changing just the bootstrap server and the identity pool. 

Conclusion

We used Azure AD DS to demonstrate how applications can authenticate to Confluent Cloud using OAuth, but the process works similarly with other identity providers, including Okta, Ping Identity, and Amazon Cognito.