BT

Your opinion matters! Please fill in the InfoQ Survey!

Metadata-Driven Design: Building Web APIs for Dynamic Mobile Apps

| Posted by Aaron Kendall Follow 0 Followers on Sep 01, 2015. Estimated reading time: 15 minutes |

A note to our readers: As per your request we have developed a set of features that allow you to reduce the noise, while not losing sight of anything that is important. Get email and web notifications by choosing the topics you are interested in.

 

Going back a few years (when I had more hair on my head), Google launched Gmail on the date of April 1st, causing many to question whether the whole campaign was actually an elaborate joke. And who could blame them? After all, the Internet wasn’t yet fully mature and had just entered its adolescence; even Web API servers were still a nascent technology. In that moment, it wasn’t uncommon for remote processes on heterogeneous machines to still communicate via “handshaking” methods with generated procedure stubs (like CORBA).

Around the same time, software architect Kevin Perera recognized a pattern to the communication layers of his ongoing projects, and in an effort to simplify such architectures and implementation builds, he proposed a solution1 in the way of a new design method. He called it “metadata-driven design and development”, and it offered a way to integrate design and development, as well as unifying both sides in a client-server paradigm of software development. In his example, he explained how a service and its exposed methods could be described through a few lines of metadata, and existing on a more abstract level than an IDL, this same metadata could then be used to drive the entire construction, interaction, and navigation of the distributed system. If one wanted to later add a method to the service, the required effort would include a few more lines of metadata and a fraction of the needed code changes from beforehand; by employing a few simple functions and data structures, the server’s interface wouldn’t require any subsequent alteration since the metadata would effectively serve as the contract for interaction. As an early advocate of metadata-driven design, Mr. Perera had helped the greater software community by drawing attention to a simple yet novel idea: by only abstracting software design a bit more, we could create solutions where design enhancements could be planned and implemented in the same moment. Written nearly a decade ago, this proposal still holds merit today.

So, what exactly is metadata-driven design? For the sake of brevity, it can be summarized as an approach to software design and implementation where metadata can constitute and integrate both phases of development. In my last article2, I wrote how metadata-driven design (i.e., MDD) could be applied in order to design an engine that ingests data from an API server. Now, we’ll extract the same core ideas from that scenario and demonstrate how MDD can assist us with forging a contemporary architecture for an API server, specifically one that’s used by an iOS mobile application.

Even though I’m not an expert on the subject, I have dabbled in the realm of creating mobile apps. While building these apps on iOS and Android, I took note of the additional time that was inherent to their development on a native level, especially when compared to normal desktop applications. Besides the unquantifiable test of an app’s user interactivity, a significant amount of time was required to organize the application’s flow of navigation when using a more complex framework (like Cocoa). Of course, there was also the time needed to submit the app for approval and then the subsequent effort to modify and/or tailor any aspects considered undesirable by the app store’s vendor and/or the app’s users. (For one, I learned that you can’t call a feature in your app by the name “Magic 8-Ball”, since Mattel might consider it as a trademark infringement. Who knew?) When all of this time and effort is taken into account, it becomes obvious that a hefty investment is required to simply maintain and update such an application. After only a few hours of development, I realized that it would be beneficial if there were a way to update my app’s data and behavior dynamically. If I could configure the app to be driven by metadata, I could attenuate the investment needed in order to make any changes. In fact, this code could then reach out to an API server for this metadata, and by implementing the changes in this provided metadata sent back, I could bypass any need to resubmit my project to its relevant app store. Better yet, it would serve me well to implement this API server using MDD as well, so that the client and server could be more tightly integrated.

Of course, some professionals might debate a proposal that would create such a close coupling between the client and server. After all, it’s generally desirable to have an untangled separation between the two of them, and they would have a number of valid arguments to support their position. For one, by having a clear demarcation, both the client and server teams can develop with a significant amount of autonomy and flexibility; the close coupling negates such freedom. Also, for many, it implies a stateful interaction, which obviously counters the stateless sessions sought by so many who implement and utilize RESTful APIs. Most importantly, this close coupling adds more work to the scenario where we might want to alter the architecture, like switching our server platform. So, in numerous cases, it is true that this method would not befit the goals of the project’s designers.

For example, any popular mobile application involves the simultaneous effort of multiple teams and the frantic addition of enhancements that are needed in a competitive market (like games). Such a volatile undertaking requires extreme flexibility in order to accommodate the fervent pace and constant surprises that accompany such a field. Obviously, in this type of scenario, this method would not be a viable candidate for such a project. So, with all of these negatives, why might we want to commit to such a decision? We then have to remember the applications in the world whose expectations are drastically different. For those projects that have a reduced scope and/or where consistency far exceeds the low probability of any new features, this method can be very extremely beneficial. For example, the enterprise mobile app of a small, focused team could evince the strengths inherent in such a tactic, especially if the app were only available to a limited set of users. Enterprise projects have less cause for concern since they are generally not exposed to the same set of unstable factors in commercial mobile applications. In these types of cases, the positives of this method have a strong chance to outweigh the negatives. It’s also important to note that even though this method advocates a closer coupling between the client and server, it does not necessitate stateful sessions between the tiers of our mentioned enterprise project.

So, how do we go about accomplishing such a feat? Let’s say that our goal is to create an API server that will serve two purposes. One, it will provide a simple interface that will never require alteration, even when we intend to eventually provide incremental functionality to clients. Two, its secondary purpose will be to provide both data and direction to a simple iOS app, which will be implemented by utilizing a variation of the HATEOAS (a.k.a., Hypermedia as the Engine of Application State) method. Much like the proposal within Roy Fielding’s doctoral dissertation 3, it is our intention for the app “to be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations”. (Hence, our client will act by choosing those options supplied by the metadata via the server’s responses.) In general, the app will only have a few modest goals: to allow a user to navigate through a few screens; to request a calculation be performed on the server; to start a particular batch job on the server (or on another remote system) ; to upload and download text, possibly onto the device’s local filesystem; and to view the contents of the mentioned download. The app’s general navigation and behavior will be controlled by metadata, which will be provided by our API server. (Hopefully, we’ll also rarely need to rebuild or to resubmit/redeploy our apps ever again!) For this example, we will choose to implement our API server using the .NET platform. In particular, we will create a Web service using the standard template provided by the ASP.NET MVC 4 framework in Visual Studio. However, before we start coding our Web service, we need to employ MDD and determine the metadata that is needed by our application. Based on our simple requirements, we would need something like the following:

Loading this metadata into a cache that is used by our Web service, we can now employ its use when we receive requests from our iOS app. In accordance with the HTTP specification4, our ApiController methods should react appropriately to specific types of HTTP requests; namely, we will implement the handlers for GET and POST methods to create a RESTful API. Much like Mr. Perera’s example, we want to create generic function signatures for these two handlers that are extensible, in that they can accommodate the alternate payloads of different actions sent to them. Now that we have taken all of these necessary factors into account, we are able to write the code that can handle and respond to these requests:

By simplifying the interface to just two methods inside one controller, we have streamlined our flow of communication when it comes to all interactions between the client and the server. Also, we can now modify certain properties of the metadata in order to dynamically update the service’s available functionality. In the case of a new POST action, we can provide an additional method by adding another row to the metadata and by placing a new DLL in the specified directory. More importantly, this simple application eliminates our need to ever rebuild and redeploy the Web service! By not requiring a redeployment of the Web service (and essentially molding it into a part of our general network infrastructure), we have reaped a few consequential benefits. One, we can be assured that the communication mechanism between our client and server has an incredibly low risk of developer error; by removing that variable, our future debugging efforts will be much simpler. Consequently, our unit tests can be focused on the new/updated functionality in the new DLLs, since we will have less effort needed for regression testing of the Web methods (like detecting whether any Web method signatures have been broken). Two, we will save a significant amount of time and resources that would have been required by the efforts of a Web deployment to production. Three, by funneling our online traffic into only a few Web methods, the maintenance and scrutiny of our methods’ security paradigm demands less oversight. Finally, by subscribing and committing to such a RESTful formula, we have guaranteed the scalability of our Web service. At this point, we may have only implemented the server portion of this architecture, but we have also now finished the first step towards creating a dynamic client as well.

How exactly have we done that? As stated earlier, our simple iOS app will allow the user to navigate through a few screens. As with the development of any mobile app, it’s usually desirable for every screen to be customized in some way. (On Android, customization exists on another level, where the permutations increase with every screen on every different device.) This customization requires a certain level of meticulousness on the developer’s part. Let’s take the simple example of an ordinary UITableView screen that simply displays a scrollable, interactive list of rows; these rows usually display text and/or images. Each row (i.e., cell) in this list can be customized in terms of its layout, font, dimensions, etc. Usually, it’s favorable to standardize the rows so that they have an identical structure, in order for the list to appear more uniform (i.e., more organized). However, we might want to change some aspect about them in the future. Better yet, we might want to change the contents of this displayed list! As with many existing iOS projects, we can simply make these alterations for the rows’ appearance in the actual code...but that would be troublesome due to the need for redeployment or resubmission. Instead, let’s aim to make our mobile apps into clandestine agents that are readily available at our disposal.

Our simple iOS app will have an UITableViewController as its startup screen, and we will create a class called “MainView” that will inherit from UITableViewController. Normally, during the initialization section of the app, we could find the specific rows for our view declared explicitly within the “viewDidLoad()” method. In our case, though, we will do something different: we will use a NSURLConnectionDelegate in order to make a GET call to our server, providing the parameter “name” with the value “main”. The resulting payload will be a JSON message that provides us with descriptive information about the content and appearance that should be displayed by our main screen (i.e., startup screen):

Observe that we have not included all of the possible options from our metadata; for example, we have omitted the metadata option to start a batch job.  In the future, though, we could always add that option back into the roster. Of course, you should always bundle your app with a default JSON configuration, in the event that network diagnostics indicate a general lack of connectivity. Normally, though, this returned payload will deliver an up-to-date configuration for our main screen. In the case that you need to support other types of mobile apps (like Android), it might behoove you to create different payloads that address the specific display needs of that platform. If you observe our sample payload, you can find the rows defined for an instance of UITableViewController, especially the important values needed for display and interaction. The one property “enabled” is for our own usage within the logic of the mobile app’s code, allowing us to present or remove certain options to the user with a simple edit to our payload’s file. (In the case above, the “Advanced Settings” row will not be a presented option for the app’s user when “viewDidLoad()” is called by our main screen.) By having metadata drive both our server and our client, we have created multiple layers of dynamic behavior within our paradigm.

In some cases, though, it may not be desirable to rely upon a filesystem, either for the payloads of our screen configurations or for the DLLs that contain our supplemental functionality. For example, due to potential security issues, some cloud vendors discourage the deployment of web services to their PaaS if those services attempt to access a local filesystem. In that case, we would want that same functionality without file usage, and we could achieve it through only using table columns in our database. Obviously, we could easily save the contents of our screen configurations to a CLOB column in our database table. The dynamic functionality, on the other hand, presents us with a bit more of an obstacle. The safest route would be to simply provide a forwarding URL; this redirection would point to a new Web service somewhere on the network behind the firewall. However, there are other options. We can also use a CLOB column in this situation to accomplish our goal. Depending on the JIT capability of your chosen platform (and how much you enjoy courting a certain amount of danger), you might be able to compile and then utilize code that’s simply resting inside a row of our metadata table. In our example above, we have appointed the .NET environment as our chosen platform, and in this case, it is completely possible to utilize JIT compilation by making use of the System.CodeDom.Compiler namespace (and, in particular, the CSharpCodeProvider class). As long as our dynamic functionality doesn’t require a project’s set of classes, we can easily embed a C# class (or equivalent block of code) within a CLOB column and then execute it at runtime, all without the need of a filesystem.

However, this one example cannot showcase the entire range of potential value that MDD can enable us to attain and capitalize upon. At this point, you might even be thinking that the previous scenario is basically the exhibition of a multidimensional configuration file...and if we stopped our design here, that would be true. However, we have not truly taken advantage of MDD yet. For one, this Web service does not need to exclusively handle just iOS devices; further metadata could allow other platforms (Android, Xamarin, etc.) to be supported by the very same flow of code. Following the advice of Mr. Perera, we can also populate our metadata with method signatures and data layouts, and by depending all builds upon it, we can ensure that the code of the server and the client are synchronized through the same source. (Obviously, this commonality for builds would have the side benefit of enforcing more transfers of knowledge between the server team and the client team.) A further enhancement would be to implement additional functionality within the client, so that it could request up-to-date options listed within the server’s cache. Such an additional layer would be the next step to creating the final iteration of the solution: truly dynamic interaction between the two sides. As shown in one of my previous articles 5, it is possible for us to reiterate through this design and implementation repeatedly, in order to truly unite the client and the server. As long as the number of exchanged properties remains relatively small, we can invest a small effort into our client’s intelligence, and in return, we can enable the client to assemble and utilize the Web service’s methods and data structures, all at runtime.

About the Author

Aaron Kendall is a software engineer in New York City, with nearly 20 years of experience in the design and implementation of enterprise data systems. After beginning as a developer of device drivers and professional software, he became passionate about software design and architecture. He has created innovative business solutions using a variety of platforms and languages, as well as numerous freelance software projects that range from open source packages to game design and mobile apps. If you would like to read more about his work, you are encouraged to visit LinkedIn and his blog.

References


1Perera, Kevin S. (January 2004) Metadata-Driven Application Design and Development MSDN Developer Network

2 Kendall, Aaron (13 April 2015) Metadata-Driven Design: Designing a Flexible Engine for API Data Retrieval InfoQ

3 Fielding, Roy T. (20 October 2008) REST APIs must be hypertext-driven

4 World Wide Web Consortium (9 July 2003) URIs, Addressability, and the use of HTTP GET and POST RFC 2616

5 Kendall, Aaron (19 February 2015) Metadata Driven Design - An Agile Bridge Between Design and Development InfoQ

Rate this Article

Adoption Stage
Style

Hello stranger!

You need to Register an InfoQ account or or login to post comments. But there's so much more behind being registered.

Get the most out of the InfoQ experience.

Tell us what you think

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread
Community comments

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Allowed html: a,b,br,blockquote,i,li,pre,u,ul,p

Email me replies to any of my messages in this thread

Discuss

Login to InfoQ to interact with what matters most to you.


Recover your password...

Follow

Follow your favorite topics and editors

Quick overview of most important highlights in the industry and on the site.

Like

More signal, less noise

Build your own feed by choosing topics you want to read about and editors you want to hear from.

Notifications

Stay up-to-date

Set up your notifications and don't miss out on content that matters to you

BT