This document explains how the Espresso Logic executes your business logic in response to REST post
requests. This is not an internals document, rather, it is intended to help you understand how the server operates so you can debug
Before reading this, you should be familiar with the Overview
and the Architecture
. In particular, you will need to understand the Object Model
to be a handy reference.
Row Events are passed row, oldRow and logicContext.
What makes Espresso unique is Reactive Programming: declarative, spreadsheet-like rules. These enable you to deliver update business logic 10X faster.
While rules hold enormous promise, they operate differently than conventional programming. Shown conceptually in this video, this page shows how to use them, how they work, and how they integrate with events.
You declare rules, which the system then uses to process RESTful updates as outlined below.
Connect and Declare Rules
The declarative rules are expressions for deriving / validating data. We'll look into more detail later, but here are two rules. The first Validation rule ensures that the balance does not exceed the credit limit (else the transaction is rolled back), and the second rule computes the balance by Summing the related Orders.
Validation: return row.CreditLimit >= row.Balance
Derive Balance as Sum(Orders.AmountTotal where ShippedDate === null)
These rules are fully executable. Once entered, you don't need to worry with the details of calling them, ordering them, reading/writing data - that is all addressed with system RESTful Update Processing.
RESTful Update Processing
The Sequence Diagram summarizes how your rules are automatically applied to incoming RESTful update requests (Put, Post, Delete):
- A database transaction is started. Each Request is a Transaction.
- For each JSON Object (recall you can update multiple objects in a single update request), the system
- Creates a Declarative Persistence Object
row. This combines the JSON data with current disk data (JSON rows might be a subset of columns), and performs Optimistic Locking checks based on a Hash. The
oldRow is also made available to your logic.
- Initiates Row Logic:
- Early Row Event are called, supplied with
logicContext. Your event can inspect / alter the row before rules fire.
- Executes (only) rules whose dependent row data has changed (based on comparing
oldRow). Rule execution order is computed based on rule dependencies, discovered by automatic rule parsing. The rule updates the
row (state), so it's visible to ensuing rules.
- Row Events are executed, enabling you to do whatever is not expressed in declarative logic (send email, post data to other systems, credit card checks, etc etc). You are passed
logicContext. The effects of rule changes are visible. You can alter the
row, but must resave it.
- Row Chaining. If the logic has altered data referenced by rules in related objects, the system instantiates rows for them, and invokes their logic. For example, altering an OrderTotal would therefore update the related Customer's Balance, whose rules would verify the CreditLimit. Note this is an efficient 1-row update - not an aggregate query.
- Rows are saved to disk from Transaction Cache. Updates are buffered into a Transaction Cache so that only a single update is required for each row.
- Commit Row Events and Validations. You can handle events that run after all rows are processed, so you can see the results of rule execution on all rows (e.g., the OrderDetails of an Order are reflected in the Orders AmountTotal). Your commit validations are invoked after logic processing is complete for all rows. This is useful when your validation needs to see the results of aggregations, such as the count of children as explained in the link.
- The Transaction is committed.
- Request Post Events are raised. You can alter the response message, or perform other functions such as logging.
RESTful Update Processing Notes
In addition to the overview above, please take note of the special topics below.
Recall that you can define multiple custom Resources on the same underlying Base Table, and that these can represent projects and aliases of table columns. Integrity demands that the Base Table logic is enforced regardless of the Resource used.
Resource/Object Mapping is therefore required to map the Resource objects onto their respective Row Objects, including column name de-aliasing and materialization of columns not projected (sometimes called "hydrated"). This means that the declarative and event logic is always dealing with a full row, and that logic is shared over all Custom Resources. This of course requires a database read, with concurrency control supplied by the DBMS.
So that you can define logic based on data changes, the system builds 2 row objects:
row: reflects client changes from JSON, and all logic processing
oldRow: reflects the row before the transaction started (this is from the database read noted above)
Optimistic locking checks are performed to ensure that updated rows will not overlay the updates of other users. This check is on a time-stamp field if provided, otherwise it is the hash of all attributes in the Resource row.
Generated Primary Key Handling
Databases support system-generated primary keys. There are special requirements when processing JSON POSTs that include child rows for such tables, such as the items for an order - the system needs to "stamp" the generated order# into each item.
The write-cache is flushed to the database at the end of the transaction, after all rows have been processed. In addition to the flush, there is a very important logic provision.
Your logic specifications for commit and action rules can stipulate that they run during normal per-row execution, or be deferred until commit. If you elect that option, such logic is executed just prior to transaction flush.
To see what this is important, consider the example where we wish to ensure Purchase Orders have at least 1 Line Item. A good approach would be to define a Purchaseorder.item_count, along with a Purchaseorder validation that item_count > 0.
While a good approach, this would fail. Why? The system will process the Purchaseorder insert first, before Line Items. At this point, the count will be 0, so the validation will fail.
We therefore provide the commit option for validations
that need to operate on the end-result of logic processing for all the rows in the entire transaction.
Forward Chaining is an often-used term for Dependency Management. It simply means that when referenced values are changed, all the derived referencing attributes are recomputed. The term chaining correctly infers that a derived attribute (e.g.,
Purchaseorder.amount_total ) is itself referenced in another derivation (
Customer.balance). It is of course the systems' responsibility to track these references and perform the forward chaining, automatically.
For formulas (e.g, price * quantity), this simply entails evaluating the expression (though see ordering, below). It is much more complicated for dependencies and multi-table derivations, as discussed in the sub-sections below.
Columns dependent on changed columns may themselves have interdependencies. For example:
It is clear that
a depends on
b, so if
x is altered, we must recompute
b before we recompute
a. Again, this is the systems' responsibility - you are not required to state these rules in any particular order. This means you can change the rules during maintenance, without concern for ordering.
Continuing our Customer.balance example, imagine a simple update where a Purchaseorder is marked paid. We need to recompute the new balance.
A dreadful approach would be to issue a SQL Sum query to add all the existing orders. In general, there could be thousands! And worse, this could be chained, where the summed attributes depend on further summed attributes. That is, in fact, just the case here: the
Purchaseorder.amount_total is itself a sum of the
Lineitem.amount. This is a very significant performance factor; ORM (Object Relational Mapping) products are often blamed for poor performance due to excessive use of chained aggregate queries.
So, the system adjusts the parent sum by the change in the child. The result is a one-row update (unless it was pruned per the discussion above).
There are analogous considerations where the client alters a parent attribute referred to in child logic (e.g., a Formula). When this occurs, the system visits each related child to run the dependent logic. This may update the child, and trigger further adjustment / cascade processing to other related data.
Adjustment and Cascade processing both make updates to related data. So, you will observe that the system will often issue SQL updates for data beyond that originally sent from the client. This is a natural consequence of your logic, and exactly what business logic is supposed to do.
Such triggered updates are subjected to the full logic analysis / chaining process, so will often result in still other updates. For example: consider a simple update to a Lineitem.quantity:
- Lineitem.amount is derived as price*quantity, so is recomputed
- Purchaseorder.amount_total is derived as Sum(Lineitem.amount), so it is recomputed (adjusted)
- Customer.balance is derived as Sum(Purchaseorder.amount_total where Paid = false), so is is adjusted
- The customer logic re-evaluates the credit limit check - if not passed, the entire transaction is rolled back, and an exception is returned to the client.
Observe that chaining means your logic may be executed more than once on the same row multiple times within a transaction. Consider a transaction comprised of a Purchase Order with multiple Line Items. So the Purchase Order logic is clearly executed on insertion. Now consider that each Line Item would adjust the Purchase Order's amount_total. This re-executes the Purchase Order logic, now as an update. Your logic can determine
initialVerb via the LogicContext.
We can now make some key observation about some fundamental characteristics that distinguish Reactive Programming from conventional Procedural (Imperative) programming:
- No Control flow: you do not invoke your rules - they are invoked by the system, and only in reaction to actual changes. You do not order their execution.
- Elimination of Boiler Plate code: in a conventional approach, the bulk of your code is Change Detection, Change Propagation, and Persistence handling (SQL commands). Reactive Programming automates all of this, so the logic shown above is fully executable.
Simple Example: Check Credit
So, let's continue our example, to devise a solution of Check Credit. Building on the two rules above, we have:
return row.CreditLimit >= row.Balance
Derive Customer.Balance as
(Orders.AmountTotal where ShippedDate === null)
Derive Orders.AmountTotal as Sum(OrderDetails.Amount)
Derive OrderDetails.Price as copy(Product.Price)
And that represents the complete, executable solution; note:
- Ordering: The rules above can be entered in any order, since they are automatically ordered per dependency management, above
- Re-use: The rules are applied to all incoming transactions, automatically invoking the (relevant) logic above
- Automatic Persistence: The system provides all the SQL to process incoming transactions. So, adjusting a Quantity automatically reads / adjusts the Orders and Customer rows, and it does so efficiently (a 1 row update, not an expensive select sum query)
These are common Logic Patterns
This simple "Balance < CreditLimit" example illustrates one of the most common Logic Patterns:
Constrain derived Result
Other examples of the pattern:
- Rollup employee salaries to department, constrain to budget
- Rollup departments, constrain to budget
- Rollup Student Course Credit, constrain to max for student, max for course
A similar pattern is:
Existence Checks: validations on [qualified] counts
- Order must have items
- Department must have employees
Scaling to Complexity
- Copy - invoke copy, including deep copy, for patterns like auditing, or cloning. It even automates a Bill of Materials explosion.
- Allocation - allocate an amount to a set of recipient - a bonus to department employees, a payment to outstanding orders, etc.
You can explore these examples, which range for simple to quite complex - all handled with a few rules.
Business Perspective: agility, transparency, quality
Declarative logic is remarkably more expressive than imperative code. The 5 lines of logic above equates to over 200 lines
of triggers, or 500 lines of Java. It's also far more readable - in fact, understandable to Business Users.
In an industry where we walk over hot coals for a 30% gain, this is a remarkable 40X improvement in expression factor. That's what delivers the 10X reduction in delivery.
So what, exactly, drives this compression factor? It stems from 2 key factors: removing boilerplate code, and automatic re-use.
We noted above:
- ORM creation - considerable code is saved in the automatic creation of the Object Model
- change detection - most of the alternative code noted above is detecting changes to determine when to propagate updates. This is eliminated in the Declarative Reactive approach
- sql (caching) - we're all painfully aware that sql handling is tedious; rules automate the sql, including the critical underlying services for caching.
Rules are bound to the data, not a specific Use Case, so they apply to all in-coming transactions. In other words, the logic above automatically processes all of these transactions:
- Order inserted - balance increased
- Order deleted - balance decreased (if not paid)
- Order unshipped - balance decreased
- Order shipped - balance decreased
- Order amountTotal changed - balance adjusted
- Order reassigned to different customer - balance increased for new customer, decreased for old
- OrderDetail inserted - obtain price, adjust Order and Customer (and check credit)
- OrderDetail Deleted - reduce Order and Customer totals
- OrderDetail Quantity increased - adjust Order and Customer (and check credit)
- OrderDetail Product Changed - obtain price, adjust Order and Customer (and check credit)
- OrderDetail Quantity and Product Changed - obtain price, adjust Order and Customer (and check credit)
- Customer CreditLimit changed - check credit
This results in a meaningful improvement in quality. Reactive Programming eliminates of an entire class of programming error (e.g., balance checked when adding an order, but overlooked on unshipping an order).