Restify and Mobilize Your Data
Enterprises are striving to make their corporate data available on mobile devices via RESTful interfaces, but building mobile apps and REST APIs takes serious time.
In this piece, we’ll briefly review existing key technologies, to build on their advances and to benefit from lessons learned in the marketplace. We’ll then provide specifics for Executable Schema and Declarative Behavior.
The promise of accelerating delivery time is enticing and has spawned some great work over the past decade. First, let’s briefly examine the current approaches to draw some lessons learned.
Model Driven Development
Approaches such as Model Driven Architecture1 and Model Driven Engineering2 have met with some success in constructing partial elements of systems. And while promising, the real opportunity lies in:
- Building executable systems directly, rather than elements (e.g, Beans, Transfer Objects) that require understanding and integration
- Leveraging Cloud/Mobile technology
- Eliminating model/schema redundancy and synchronization complexity
Convention over Configuration: Ruby, Grails…
The notion of generating executable systems has been taken up by popular products such as Ruby, Grails, ROO, etc, that provide screen and code generation from models using Convention Over Configuration3. Here is a simple Grails example:
And while it’s great to build Web Apps quickly, significantly greater value would be achieved if:
- The apps were substantially more complete – for example with Master/Details (show list of Lineitems for Order) updatable grids (update the Lineitems in place) and automatic joins to show product names instead of numbers
- Support for multi-table business logic were provided (update and security business logic is tedious and verbose)
- Code generation issues were avoided – for example regeneration and loss of customizations when the originating input (the data model) changes
A new category of Mobile Backend as a Service (MBaaS) is focused on cloud technology, such as REST / JSON. Products like DreamFactory and SlashDB create a complete RESTful API for your SQL data, with convenient table browser User Interfaces.
While it’s certainly a start, imagine the value if you could:
- Extend the table browsers to full, multi-table applications with master/details, navigations, etc.
- Provide for business logic, to ensure updates trigger the the proper computations and validations
- Provisions for Custom Multi-Table Resources, with alias/projection
These approaches could be significantly enhanced by introducing notions like
- Executable Schemas, where starting with the schema as the model, you could create a REST/Cloud service that pre-supplies the common behavioral patterns for Multi-Table UI and API - an instantly executable system.
- Declarative, business-oriented behavior could provide declarative support for API extensions (such as multi-table resources), logic and security behavior. The real value is delivering these behaviors at a very high level of abstraction, more like business-oriented requirements than code.
So, the development team can operate and deliver at a near-business level of abstraction, with meaningful increases in speed. They can translate BDD stories into running software to confirm understanding and uncover the next set of behaviors, resulting in dramatically reduced miscommunication since stakeholders are seeing running software and can read the “code.”
If we are to meet the business demands for instant results, we need to be running right out of the box. That is, just by connecting our system to a schema we should have a completely executable API and Application as described below.
It is quite possible to build a richly functional mobile-ready back-office app. The presumption here is that while “front office” apps require a handcrafted UI, there is a large class of data maintenance apps that follow a set of well-known patterns, and that these can be created from the schema.
The sample screens below illustrates these common patterns for a fully functional data maintenance application:
- Search / Filter, with paginated List / Form view, sortable column headers
- Foreign Keys drive Multi-Table application behavior:
- Master/Detail (Orders of Customer)
- Drill-down Navigations (Screen 2 – Order with Items, via the magnifying glass or the zoom button at the lower right)
- Automatic Joins: show Product Name, not Number
- Update support, including updatable grids, lookups (find part), and error handling
The join functionality is interesting, because it is quite common. In the example below, the Product foreign key is a Product Number, not Name - a very common pattern. In the ideal case, the system would recognize this case (numeric foreign keys, e.g., Product Number) and automatically join in a more suitable column (the Name). We would have not only an executable schema, but an instantly useful one as well.
(Click on the image to enlarge it)
(Click on the image to enlarge it)
Default REST API
The architecture of choice for cloud / mobile apps is a RESTful API. Such an API not only provides mobile access, but also provides network capable web services that can be accessed by other systems.
And the mapping from database to REST is straightforward… just by connecting:
- Create a REST Endpoint for each base/view table, and Stored Procedure
- Provide Get/Put/Post/Delete support (familiar CRUD operations), with support for
- Filtering and sorting
- Well-known patterns such as pagination, discovery for structure and related data, and optimistic locking
The result is a “flat” relational API. A good start, but we’ll need to extend it as described below.
While we are proposing that the model not replicate the schema, there are clearly requirements to extend it for the key behaviors below.
Rich REST API: Multi-Table Resources, Alias, Projection
Our “flat” API is a good start, but a richer “document oriented” API is required. This requirement is both as a convenient API model for mobile developers and to minimize latency due to multiple queries.
Such objects are somewhat like relational views, with support for aliasing and projection. However unlike views, they are not flat; they support nested “Sub Resource” objects, such as an Order object with LineItems and Shipment information.
Such “multi-table views” might be defined via point and click or syntactically such as:
Create Resource OrdersResource
from Orders (id as orderNumber, customer.name as customerName),
SubResource Items as LineItems via <fkName>
(name as product.name, qty as qtyOrdered...)
SubResource Shipment as Shipents via <FkName>
Logic: Spreadsheet-Like Reactive
Business Logic is a significant element of most systems. It is tedious to code, consisting of significant amounts of change detection and propagation logic, SQL (with caching for performance), etc.
The prevailing assumption is that such logic is domain specific (true) and that it must therefore be coded in an imperative, procedural language (false). A declarative approach for such logic would be a significant advance in accelerating software delivery and maintenance. And there is such a solution.
Reactive is currently getting attention in handling UI events, like sync model to view. But the most common use is the spreadsheet; cell formulas react to changes in referenced values by recomputing themselves, which of course can chain. While simple enough for non-programmers, spreadsheets provide sufficient power to address non-trivial complexity.
Reactive fits very well for database transaction logic:
- Assign expressions to database columns, or “virtual columns” (these become part of the model for transaction processing, but are not physically stored)
- RESTful Put/Post/Delete operations are watched for changes in referenced values, which trigger adjustments to the referencing value
- Validations are addresse`d as well (balance < creditLimit).
Most interesting transactions are multi-table, so it follows that our expressions be able to reference related data (customer.balance := sum(orders.amount_total where paid = false). This enables the system to take responsibility for not only the change detection, but also persistence (reading / writing related data, caching, optimizations, etc.).
Note that encapsulating logic to columns results in inherent re-use. In the example above our declarative system watches and responds to changes in the orders’ amount_total or paid flag.
Declarative logic is by definition un-ordered. So the system must deduce the execution order, for example via a dependency graph. This has substantial value in maintenance. Automatic ordering means you add / change logic without worrying about the details of execution order.
By eliminating the dependency management, propagation and SQL handling, meaningful advances in conciseness and readability are realized. Meaningful here means in excess of an order of magnitude, where the following logic - virtually a requirements spec - being directly executable:
lineitem.amount = row.product_price * row.qty_ordered
lineitem.product_price = copy(row.product.price)
orders.amount_total = sum(lineitem.amount)
customer.balance = sum(orders.amount_total where paid = false)
validate customer as row.balance <= row.credit_limit
Of course there is no silver bullet. Not all logic can be declarative, so critical provisions are required for proper integration of procedural logic. These are described below.
Another key requirement of most systems is security. Beyond familiar endpoint access, the system must enforce security at the row and column instance level. For example, you might want folks to see Orders only in their own region.
Many systems provide such functionality in views, but that leads to a proliferation of views to define and maintain. A better approach is to encapsulate the security into the table as a role-based permission and ensure it is reused across Multi-Table Resources.
Role SalesRep authorized for Orders
Rows (region = $user.region)
The notation $user is meant to suggest mechanisms to associate named values (eg., region) with users that can be as parameters in filter expressions.
Executable Schemas and declarative behavior can provide value, but only if they are integrated with familiar procedural extensions so there are no boundaries.
Extensibility must also be addressed on the client side. Default UIs are powerful and useful, but most appropriate for back office applications. For front-office applications there must be provisions to develop any User Interface.
There are two reasonable approaches. First, the Default UI can be forward engineered (code generated) into your favorite platform / IDE, so you can use it as a starting point.
Alternatively, it is reasonable to turn the model inside out and craft the default UI objects as components that can be snapped into a larger app as parameterized iFrames. This retains the model linkages, avoiding the “you own it” implications of code generation.
Finally, declarative API creation in no way obviates the capacity and need to build custom APIs.
We have outlined here a realistic approach for:
- Instant creation of a mobile app and a default RESTful API from an existing schema
- Extending the API to define multi-table hierarchical resources
- Adding behavior using declarative “reactive” expressions to govern update logic and row access, with the simplicity and power of a spreadsheet
Author’s Note: Espresso Logic has been executing on this vision and now provides many of the elements described above. If you’d like to check it out, we offer a free eval with zero install that you can try on your own database. And we’re always eager to exchange views about how we can improve the technology!
1 The OMG effort
About the Author
Val Huber has over 20 years of experience building technology to accelerate the development and maintenance of business applications. Prior to Espresso, Val was the CTO of Versata; a company that was one of the top 5 IPOs of 2000 and was initially funded by visionaries such as Paul Allen and Hasso Plattner of SAP. At Versata, he led the development of a J2EE development environment for large-scale transactional and workflow applications. Versata supported tens of thousands of concurrent users and was deployed by many Fortune 500 firms including ADP, Equifax, Fidelity, IBM and Sears. Val worked as an Architect at Sybase (now part of SAP) before Versata. He was also the driving force behind Wang’s highly regarded PACE (Professional Application Development Environment). PACE provided a relational DBMS, with a fully automated tool suite for reports and interactive applications. Val holds various patents and has enjoyed considerable commercial success of the systems developed by his teams.
Marlin De Paul
Also, I'm intrigued by the "default app", but I don't see how you can possibly address every possible use case.
Re: Interesting but...
The trouble with metaphors is that they convey a lot, not always what you intended. The spreadsheet metaphor was intended only at the conceptual level. The actual technology does use spreadsheets at all. Spreadsheets basically don't address foreign keys (the basis of most aggregations), and the considerations of performance for caching and pruning are significant.
And you are quite right about the "default app" - it does not address every Use Case. My experience is that a system has several (expensive) custom-built interfaces. These are not addressed by the default app, though the API, Logic and Security are a help. Systems also have many "back office" data maintenance apps, where the default app might save quite a lot of time.
Again, thanks for the interest, and love to talk - you can guess my email ;)
How Can We Use Our Creative Power and Technological Opportunity to Address the Challenges of the 21st Century?
Gyorgyi Galik Feb 26, 2015