Methodology

Repeatable success can be achieved by employing a methodology that applies encapsulation principles to data integrations. For example, the data application to be architected, developed, deployed, and managed may be modeled as a single object.

An object is a black box that contains defined data inputs, defined data outputs, and a defined specification that precisely describes how data moves through the object. The implementation of the specification can be any technology like IBM Datastage, Informatica, Oracle PL/SQL, or even C code.

The methodology becomes infinitely more helpful when the overall application object is decomposed into ever smaller process objects, each having its own defined data interfaces and processing specifications, and these objects have a defined relationship to one another (execution order). Through the decomposition process, bits of the overall application requirements can be distributed across multiple developers, and each object may developed and tested independently in any timezone. No needs to be aware of the contents of any other developer's object.

After all of the decomposed objects are independently developed and tested, we have an application that is ready to run having the following characteristics:

An application that can migrate from development, to test, and to production environments without programmatic changes and user errors.
Automatic point-of-failure recovery without user intervention.
Automatic rollback to any prior point of execution without user intervention.
Control of the processing resources used by the application for each execution.
Automatic dynamic distribution of encapsulated objects across dynamically available processing resources.

Summary

The steps for designing, developing, testing, and deploying a successful data application every time are:

Create a top level object with completely defined data input and output interfaces. This means that at the output side we know the tables/files, columns, and types for each value delivered. On the input side we have an idea of the source data required to deliver the output. These source requirements will evolve as the design phase progresses.
Drill into the top level object decomposing it into more smaller objects, each having their own data interfaces. An example of this could be a top level ETL object that is decomposed into three objects: an Extraction object, a Transformation object, and a Load object. Each of these objects may be decomposed further. For example the Extraction object may be decomposed into 3 objects: an East Region object, a West Region object, and a South Region object. The goal is to reach a level of decomposition where the resulting objects contain simple ETL specifications, and these simple specifications can be developed into simple maintainable ETL processes/programs.
Objects may be developed and tested independently by multiple developers living in multiple localities.
The application is complete when all of the individual objects are developed and tested. This feature of the methodology eliminates scope-creep because problem objects can be identified throughout the development and testing process, not at the end.
At this point the application can be deployed in any environment and executed. The Mozart engine automatically takes care of parameterization, point-of-failure recovery of a failed job, resource utilization, and load distribution of individual application objects across available process hardware and resources.

Mozart 4 Visio

Data Management from First Principles

Methodology

Summary

Mozart 4 Visio

Data Management from First Principles

Methodology

Summary

This website uses cookies.