Batch Framework‏

Java Batch Job Framework - http://jbjf.sourceforge.net/documentation.html#

TERASOLUNA Batch解惑 http://www.offshore-jp.com/html/49/t-549.html

Enterprise Batch Server  http://batchserver.sourceforge.net/

CA 7® Workload Automation 参考

Batch Application Frameworks

 

How do I do bulk processing? For example, how do I get my ETL done to load the warehouse?

 

Here are some batch application frameworks, in no particular order.

Vestigo JavaBatch - a zOS (IBM Mainframe) framework for running batch applications written in Java.
CleverSoft Sodiax includes some batch control capability.
Xebia J2EE Batch Framework - depends on WebLogic and Flux for job scheduling.
BMC Control-M - a sophisticated batch control product.
Microsoft Job Scheduling SMF - A windows framework for running batch applications.
Taricon Xi-Batch - job scheduling and control.
CA Unicenter CA-7 - very, very complete scheduling solution.
OpenPBS.ORG Portable Batch System - Looks reasonably complete, and well-established.
Digipede http://www.digipede.net/
http://www.digipede.net/" target="NewWindow">Digipede Network - which raises the question, do grid controllers count?

What do these frameworks require? Almost nothing... and that's a problem.

What Makes Batch Processing Complex?

In the web framework world, the directory structure for web applications is tightly constrained by the web server and the security considerations around what files can be served. In the desktop GUI world -- Windows mostly -- your application's location is defined by convention, and your data's location is often left to the user to make wise choices.

In the batch control applications listed above, however, directory structures are left unspecified. In my experience, this leaves people foundering, unaware of precisely what they must specify at a broad, architectural level. These products leave too many degrees of freedom.

This foundering can be compounded by enterprise's shift to midrange processing. Some places are growing out of low-end Windows boxes. Other places are not expanding their zOS mainframes and are adding midrange servers instead. Whether growing or down-sizing, they find themselves lost in the UNIX directory tree, unable to determine where files should be located.

The batch-design foundering is further compounded by the implementation of a data warehouse, which is typified by long-running, complex batch processing. In many cases, programmers have more experience with interactive applications and small, simple batch interfaces. This kind of long-running, complex batch job is new to some programmers.

Filling in the Solution.

To prevent foundering, and get the project afloat, we must manage the complexity of batch processing. Data warehouse loads, for example, are already complex; too much freedom in the batch environment adds complexity without creating real value.

This is a four-step process:

1. Document the audit and control requirements. These include non-functional features of batch processing, and functional requirements outside the business purpose of the application software. These are the worst kind of requirements to capture because there's so little user involvement -- it's all IT navel-gazing.

2. Develop a Framework. The important thing is to pare down the degrees of freedom to arrive at the simplest possible batch processing environment that fills audit and control objectives. This is actually hard to do because IT folks tend to over-write their internal requirements.

3. Catalog your basic Design Patterns. Batch jobs are all very similar, and a few key superclasses will assure that every job fits the framework and supports the essential design patterns.

4. Supplement with Programming in the Large patterns. Avoid the shell; it's trouble waiting to happen. These additional design patterns add considerable flexibility and avoid the performance and maintenance costs of doing too much shell programming.

The issue is that the framework is elusive. Many things can work, but what's the simplest approach that assures compliance with the control objectives?