Architecture - Overview

The nevisDataPorter can be run as a standalone Java application which can be controlled using an op-interface. In addition, it can be run as a server enabling time or event controlled provisioning job execution.

It is also possible to integrate nevisDataPorter into your own application, regardless whether your application is a Web-, Java EE- or standalone Java application.

Plug-Ins

The core is built around a simple plug-in architecture. As mentioned earlier, there are different types of plug-ins: DataSources, DataFilters, DataSinks, DataObjects and DataFunctions. The nevisDataPorter itself already comes with a set of plug-in implementations. Custom implementations can be registered programmatically or configurationally to extend the nevisDataPorter without the need of a new release.

Configuration

Due to the plug-in architecture, the XML Schema describing the configuration is very generic and is thus weakly typed. The XML Schema validation is limited to syntax verification. To be able to do strong typed configuration validation each plug-in has to provide a description of its type, its mandatory and optional configuration parameters.

Modules

The engine of nevisDataPorter loads a list of modules from the configuration file. Each module contains a set of entities. Modules group entities and represent a unit of work that can be executed.

Entities

Each entity describes one object on the target system. E.g., if you have to export users from your IDM database to an LDAP directory, then your target system is the LDAP directory. A user's data on the IDM database may be split over multiple tables, whereas in the LDAP directory a user is represented by a single directory object. Since entities represent objects/records on the target system you'll have one entity that represents the user in the LDAP directory in the nevisDataPorter's configuration file.

Each entity in a module consists of a set of optional DataSources and a set of filter chains. A filter chain is an optional list of DataFilters ending by a DataSink. Data from multiple DataSources is merged and is treated as if it would come from a single DataSource, i.e. all filter chains are executed for every piece of data obtained from the sources. A DataSink always marks the end of a filter chain. A filter chain can contain no (e.g. for deletion), one or many DataFilters.

State Engine

The nevisDataPorter clearly distinguishes between the initialization and the execution phase. In the initialization phase the configuration is verified. The corresponding init sections will be executed where DataObjects like connection pools may be initialized. In the execution phase nevisDataPorter processes the list of entities contained in the requested module. For each entity it iterates over the source records using the DataSource, pipes each source record through the DataFilter chain and finally uses the DataSink to store the resulting record. The following pseudocode illustrates the execution phase for a given module:

foreach entity do {
    foreach dataSource do {
        while(dataSource.hasMoreObjects) {
            src = dataSource.getNextObject();
            foreach dataSink, dataFilter do {
                dst = dataFilter.filter( src ); // process entire chain
                dataSink.export( dst );
                dataSink.flush();
            }
        }
        foreach dataSink do {
            dataSink.flush();
        }
    }
}

The flushing behavior can be controlled using the chunkSize attribute on the entity. It defaults to the value of 1, which means the data sinks are flushed after every record. If it is set to a positive value, the data sinks are flushed after every specified record. If it is set to a negative value, the data sinks are only flushed at the end of an export.