Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ


Choose your language

InfoQ Homepage Articles Don't SCIM over your Data Model

Don't SCIM over your Data Model

With the advent of Cloud Computing, "Identity Management in the Cloud" has become a hot topic, and many standards have emerged to deal with the challenges of Identity and Access Management (IAM) at Web scale. The older XML-based trio of SAML for authentication, XACML for authorisation and SPML for provisioning have achieved only mixed success, with only SAML having gained some traction in enterprise systems. But in the new world of the cloud, more lightweight alternatives are in demand. The "OAuth2 family" of protocol standards promises to deliver what the industry is asking for.

The OAuth2 protocol itself, as a means of enabling constrained delegated authorisation of access, is an excellent base technology to protect other IAM APIs (protocols)1, - specifically, such APIs as OpenID Connect for Authentication and SCIM for User Provisioning.

[SCIM used to stand for "Simple Cloud Identity Management" but now stands for "System for Cross-domain Identity Management", which is more generic.]

However, OAuth2 by itself is only a necessary and not a sufficient enabler. The design of these latter specifications needs equal attention. For example, a well-designed protocol requires a good data model underpinning it, and both need to be based on sound principles.

This article looks at some aspects of the SCIM protocol as currently published. There are three major areas where SCIM’s underlying data model is somewhat inadequate:

  1. Tight coupling of identifiers
  2. Inelegant handling of multi-valued attributes
  3. Clumsy PATCH command syntax

The Real External ID

The SCIM protocol is based on REST, and provides a set of RESTful methods to create, update and delete resources in the cloud. It is assumed that a cloud provider will implement these RESTful interfaces and an enterprise client will invoke them. In all these RESTful calls, the resource is identified by a URI (a template populated with a distinct "Resource ID") generated by the cloud provider when the resource is created. All subsequent manipulation of the resource by the enterprise client is expected to reference this URI.

This is classic REST. According to the SCIM Draft API, a resource is created through a POST to a collection URI:

Fig. 1 - A resource creation request in SCIM. Note the exposure of the client’s internal identifier in the field "externalId".

A successful response looks like this:

(Click on the image to enlarge it)

Fig. 2 - The SCIM response to the above resource creation request, revealing the cloud provider’s identifier for the newly created resource and confirming the enterprise client’s own identifier for it.

Here, the URI of the newly created resource is "", and the Resource ID itself is "2819c223-7f76-453a-919d-413861904646".

In other words, the Resource ID is generated by the cloud provider and returned to the enterprise client. The client is then expected to refer to this resource using the URI with this Resource ID embedded.

So far, so good. SCIM goes a bit further here by recognising that the enterprise client may already have an identifier for the resource internally. For convenience, the client should be able to associate its own internal identifier with the one generated by the cloud provider and be able to refer to the resource using this friendlier and more meaningful value.

From the perspective of the cloud provider, this latter identifier is how the "external" system (i.e., the enterprise client) refers to the resource, so it is called the "External ID". A glance at the request message will show that a field called "externalId" is being populated with the value "bjensen".

However, in spite of the good intentions of this feature, there are certain drawbacks to it:

  1. The rest of the protocol does not meaningfully use the enterprise client’s identifier, the "external ID" at all, even though it was ostensibly introduced to make things friendlier for the client. All subsequent API calls are seen to use the URI supplied by the cloud provider, which contains the cloud-generated Identifier, so it is not clear what purpose was achieved by revealing the client’s internal identifier to the cloud.
  2. From an encapsulation perspective, the nomenclature has it backwards - the cloud provider's ID and the enterprise client's ID are both "Internal IDs" with respect to their domains. There needs to be an entirely new and separate ID shared by the two, and that is the one that should by rights be called the "External ID", because that is the identifier that each exposes to the other. Each party should map this to their respective Internal IDs.
  3. The External ID should be the only (and mandatory) identifier used in the API. Internal IDs, whether belonging to the cloud provider or the enterprise client, should not be exposed in messages at all, because that creates tight coupling. (As any DBA will attest, database primary keys should not be revealed to the outside world, only candidate keys created for that specific purpose.)

This is what the underlying data model for cross-domain identity management should really look like:

(Click on the image to enlarge it)

Fig. 3 - How domains should exchange resource identifiers (i.e., Keep Internal IDs private, and map them to a shared External ID)

If "Domain A" in the diagram is the enterprise client and "Domain B" is the cloud provider, then the Domain-private Identifier ‘x’ would correspond to ‘bjensen’ (the enterprise client’s Internal ID). The Domain-private Identifier ‘y’ would be some arbitrary value generated by the cloud provider that is hidden from everyone else, and the only identifier we would see in the API, the External Identifier ‘q’, would correspond to the string "2819c223-7f76-453a-919d-413861904646". Note that we do not know what the resource is actually called within the cloud, nor do we care. The cloud provider does not know what the enterprise client calls the resource, nor does it care. All they share is a meaningless identifier that is loosely coupled to their own internal ones, and in that way, they interoperate with a minimum of dependency.

Also from figure 3, the enterprise client can expose other External IDs such as ‘p’ to third parties, and the cloud can similarly expose External IDs such as ‘r’.

With this suggestion incorporated, the SCIM POST request now looks like this (there is no mention of the enterprise client’s Internal ID ‘bjensen’ (misleadingly called ‘externalId’):

Fig. 4 - A resource creation request with no indication of the resource’s Internal ID within the enterprise client

The cloud provider’s response to the resource creation request now looks like this:

(Click on the image to enlarge it)

Fig. 5 - The response to the resource creation request - Note that the exposed identifier is by definition the External ID, since Internal IDs are never exposed.

Notice that what the response now calls "externalId" is the meaningless string shared between the two domains, not "bjensen" or the Internal ID of the resource within the cloud. This External ID is the string that is also seen in the URI that must be used in subsequent API calls.

With this suggestion, we believe SCIM can encourage a more loosely coupled relationship between enterprise client and cloud provider.

The Challenge of Multi-valued Attributes

The manipulation of resources containing multi-valued attributes (elements) within them seems a bit clumsy. Look at the spec (or even the latest version) for examples.

There are three types of attributes for a resource:

Fig. 6 - Simple attribute

Fig. 7 - Composite attribute

Fig. 8 - Multi-valued (repeating) attribute (a list or array)

If you look closely, you will see that multi-valued attributes (and not simple or composite attributes) are the real problem when attempting to manipulate sub-entities of an entity. SCIM does not seem to recognise this problem, hence doesn’t have a simple means of managing multi-valued attributes. Here is the model we propose instead.

The secret is that every value needs a key, and multi-valued attributes lack that. So our solution is quite simple - turn every list or array (of values) into a dictionary (of key-value pairs) by providing each value with a unique and meaning-free identifier.

(Click on the image to enlarge it)

Fig. 9 - A generic model for entities. Note that repeating attributes need to be assigned a separate identifier for each value.

Our recommended way to represent multi-valued attributes in a JSON format would be as follows:

(Click on the image to enlarge it)

Fig. 10 - Multi-valued attributes with unique and stable identifiers (a dictionary)

In other words, lists or arrays are converted to dictionaries, and every single value inside a resource is uniquely addressable, including those found in lists or arrays.

[If the sequence of items in the array is important and you don't want to lose that when turning it into a dictionary, generate keys that are in sequence, only taking care to see that there are enough "gaps" between keys so that new items can be inserted between them later, if required. Something like a B-Tree algorithm will be useful.]

Note that we do not consider positional indexes to be adequate identifiers. They need to be stable as well. After all, if the value "email-addrs[1]" is ever deleted, then "email-addrs[2]" moves up to become the new "email-addrs[1]", and any concurrent update to "email-addrs[1]" by another actor would probably be applied to the wrong element. Hence positional indexes will not do as identifiers for our purposes.

We recommend that the SCIM protocol should return a unique key per value on successful creation of any multi-valued attribute, and refer to it in all resource representations, as illustrated in Fig. 11.

This diagram illustrates a client’s resource creation request, and the server’s creation response. It isn’t strictly necessary that the key be a UUID, but UUIDs are cheap and virtually inexhaustible, so that’s what we would recommend.

(Click on the image to enlarge it)

Fig. 11 - The recommended transformation of a resource to enable easier manipulation in future

Patching PATCH

It’s good to see that SCIM is one of the first RESTful protocols to start using the HTTP verb PATCH. For those who don’t know, PATCH is a relatively new HTTP verb that has slightly different semantics from PUT. PUT implies a wholesale replacement of a resource with a fresh representation of itself. PATCH means a partial update of just some elements of the resource.

The nice thing about PATCH is that it allows elements of a resource to be added, modified and deleted, which is very flexible indeed. The not-so-nice thing about it is that best practice around the use of PATCH has not yet evolved.

When the SCIM protocol uses PATCH, there are areas where it seems a bit clumsy.

The first source of clumsiness is of course because of multi-valued attributes, more specifically the lack of stable and unique identifiers for individual values. We believe we have already solved that part of the puzzle by specifying the need to assign such identifiers as part of the protocol.

We now exploit the assumption that every attribute value is uniquely addressable to build up the next layer of the protocol.

The second area where SCIM’s use of PATCH could be improved is to specify, in a simple and unambiguous manner, how to add, modify and delete attribute values inside a top-level PATCH verb. In other words, how best to nest POST, PUT/PATCH and DELETE verbs inside an outer PATCH.

In order to avoid confusion between nested operations and the regular HTTP verbs (and also to scratch a longstanding itch of ours to clean up the HTTP verbs themselves!), let us postulate new "reserved words" for the following operations inside a PATCH command:

Nested operation inside PATCH

HTTP Equivalent




Add the given attribute value to a collection and return the unique key generated for it.



Add the given attribute key-value pair at the location implied by the key. (If there’s already an attribute with this key, return an error status.)



Replace the current value of the given key with this new value. (If there’s no such key, return an error status.)



This means PLACE or REPLACE. (At the end of this operation, we want the specified key to refer to the given value whether the key already existed or not.)



Delete, deactivate or otherwise render inaccessible the attribute with the given key.



(This verb is just listed for completeness. We probably don’t need a nested PATCH since PATCH cascades to every level of the tree.)

Table 1 - Reserved names for nested operations within a PATCH command

A third quibble with SCIM’s PATCH-based API is the use of the HTTP status code "200 OK" for all responses, even though the WebDAV status code of "207 Multi-Status" would be more appropriate.

With these three changes in mind, let’s see how an enterprise client could use SCIM to update part of a resource without having to replace it wholesale.

Let’s say the resource representation is the following, with unique and stable identifiers assigned to all multi-valued attributes:

(Click on the image to enlarge it)

Fig. 12 - Resource before update

The SCIM command to update the resource would look like this:

(Click on the image to enlarge it)

Fig. 13 - PATCH command to update resource

At the end of the operation (if all goes well), the resulting resource representation would look like this:

(Click on the image to enlarge it)

Fig. 14 - Resource after update

(This is the resource representation that will be provided as the response to a regular GET too, so the keys of all values of a multi-valued attribute can always be discovered.)

The SCIM response (if all goes well) would then look like this:

(Click on the image to enlarge it)

Fig. 15 - PATCH response on successful update of resource

Note that when a new resource is created, as with the addition of a new email address, the server returns a "201 Created" and a UUID key for the new value.

The following figure shows how a PATCH command and its nested operations will act on the elements of a resource and transform it.

(Click on the image to enlarge it)

Fig. 16 - PATCH and its nested operations transforming a resource

[How do the inserted values fit as neatly as shown (e.g., middle-name between first-name and last-name)? Well, they don’t necessarily fit so neatly, but it doesn’t matter, because the order of values isn’t guaranteed, just their position in the hierarchy.]

Some of the nested operations could result in errors, such as "409 Conflict" if an update would put the resource in an inconsistent state and "404 Not Found" if a non-existent element was being updated or deleted, etc. These status codes should be checked for, since "207 Multi-Status" just says to look inside.

Summary and Conclusions

In this article, we suggest a few changes to the underlying data model to improve the SCIM protocol.

The first suggestion is to hide internal identifiers for all resources and instead share an explicit External ID between enterprise client and cloud provider. This enables loose coupling at the data level, which somehow always seems to be neglected in distributed computing.

The second suggestion is the assignment of identifiers that are stable and unique to each value of a multi-valued attribute, at any level of a resource. The principle is that every value of an attribute must have its own unique identifier.

The third suggestion is a change in the syntax of the PATCH command to manipulate resources in three respects. The exploitation of unique identifiers for every attribute of a resource is the first improvement that cleans up the API drastically. New and specialised reserved words to denote specific operations are a second innovation. The third change is the use of the "207 Multi-Status" code with nested status codes corresponding to the results of specific operations.

The same PATCH syntax may also be used to manipulate collections of entities, such as users within groups. A similar nesting structure could possibly also be considered for bulk operations using POST, where a large set of operations on resources are batched together and submitted as one.

There are other, non-data aspects of SCIM which may require review, such as its synchronous request-response interaction model, which is a form of tight coupling and could prove to be a source of brittleness. However, this article focuses purely on data-related aspects of SCIM.

These suggestions have also been posted on the SCIM mailing list. We hope they will be considered for incorporation into the SCIM protocol to make it simpler, more elegant and more powerful.

About the Author

Ganesh Prasad is a 25-year IT veteran who has spent the last decade as an architect for Enterprise Shared Services including Identity and Access Management, across the banking, insurance and telecom industries. His experiences in this discipline combined with the general lessons learnt over a decade of architecture practice and consulting, have prompted him to document his learnings for the benefit of other professionals in the field. He has co-authored a how-to manual on designing a corporate IAM solution ("Identity Management on a Shoestring"), now available as an InfoQ eBook. An upcoming article of his will elaborate on 50 basic principles underlying the IAM data model.



1. Recently, there has been some strong criticism of OAuth 2.0 from no less than one of the lead authors of the specification, Eran Hammer. The criticism is that OAuth 2.0 is too flexible and extensible to be a good specification, with the result that (1) it is possible for a less sophisticated developer to produce a non-secure implementation of it, and (2) it is possible that two implementations of the specification will fail to interoperate. On analysis, this does not seem to be a fatal flaw since Hammer acknowledges the existence of secure implementations, and one of these profiles can in future form the basis of an interoperability standard along the lines of the WS-I Basic Profile for SOAP-based Web Services.

Rate this Article