Reducing Audit levels of Oracle SOA Suite for Production Mode

Reducing Audit levels in Production Mode

In this post will talk about: How to reduce the audit level for composites and thus the data written to the SOA schema.

All production environments should have the Audit level set to Production. This should be standard setting for all SOA domains in production. Please note that each component can be set independently of the composite level however Production should be the minimal of settings. Various performance tests have shown that changing from the audit level from Development to Production results in an average of 34 % - 46 % performance improvement (without tuning the individual component audit settings).

If an organization can turn off auditing (provided the business allows for it) there are substantial performance improvements. Internal performance results have shown the changing from Production to Off can result in an average improvement of over 50%-60% (without tuning the individual component audit settings). However since most organizations cannot turn off auditing we recommend the following steps to be considered for reducing the audit levels.

For synchronous BPEL components if the audit trail is not required then use the following settings:

•         inmemoryOptimization = true

•         completionPersistPolicy = faulted

With these settings the synchronous process only shows in the audit trail if the instance faults. Please note these settings should only be used for synchronous BPEL processes and not for any type of asynchronous BPEL processes.

Another way to reduce the audit level is to turn instance state tracking. On the FMW console Uncheck the "Capture Composite Instance State" this has two benefits

1.        A performance improvement since the state is not captured in the DB

2.        Less data in the DB for purge purposes.

The implication is although you may still see the composite instances in the FMW console, the "state" of composite instances will be shown as a question mark (?), so you cannot decipher from the console whether the composite has completed or not. Please note that the state column will not reflect the states of the individual components (BPM, BPEL, UserTask, Mediator, etc) within the composite. The state of the individual components can be tracked by going to the respective component pages for e.g. to soa-infra à Service Engine à BPEL in FMW console.

Each component also has its own audit level which can be set since the initial setting is Inherit, which will inherit the global audit setting. These settings can be set via FMW console soa-infra àSOA Administration à BPMN (BPEL, Mediator) Properties. Auditing can be further tuned by setting the level of the composite audit level. It is also set to Inherit by default and it inherits the respective component auditing level. So the following settings are being used at several customers:

•         Set global audit level = Production

•         Capture Composite Instance State = Off

•         BPEL Engine Audit Level = Minimal

•         Composite Audit Level = Inherit

•         InMemoryOptimization/completionPersistPolicy = true/faulted (for synchronous processes only!)

The above settings can reduce the composite data traffic by 50% and drastically improved purging times.

References: http://www.oracle.com/technetwork/database/features/availability/soa11gstrategy-1508335.pdf

Dehydration Store Database Growth Management Challenges and Purging Strategies

Dehydration Store Database Growth Management Challenges 

SOA tables may grow excessively due to an ineffective management strategy.  When an ineffective purging strategy has been implemented the SOA suite tables may grow very large leading to an urgent need to reclaim space. The larger the tables the harder it will be to delete rows and reclaim space.

This situation highlights the need to constantly monitor the effectiveness of the growth management strategy and to take corrective actions as soon as possible before the tables become very large.

 Recommended Strategies

The growth management strategies recommended is based on the SOA database profile.

The strategies require testing in order to determine their effectiveness. The testing needs to ensure that the purge scripts have headroom grow; that partitioned tables have sufficient disk space; that growth is monitored and that maintenance is performed with minimal impact to OLTP performance.

Recommendations for Large Database profiles

Installation with large database profile must seriously consider table partitioning as part of their strategy. Table range partitioning is a proven method for managing large tables:

1.       Execute the parallel purge excluding the tables that will be partitioned

2.       Partition the bottleneck tables to the purge script

3.       Reclaim space as part of maintenance cycle

4.       Drop eligible partitions as part of the maintenance cycle

5.       Monitor the inflow and outflow

6.       Monitor database and hardware resources

Recommendations for Medium Database profiles

The installations with a medium database profiles have the potential to become large thus the recommendations are the same as that for large profiles. It is important that medium installation perform rigorous QA testing and monitoring, as table partitioning post go-live might require downtime.

The only additional recommendation is that the CTAS (Create Table as Select) scripts when available may be an alternative to purging. This depends on the size of the database being truly medium and the availability of downtime during the maintenance windows.

Recommendations for Small Database profiles

Small installation with limited CPU, memory and disk space should start with the single threaded purge and then move to the parallel purge:

1.       Execute the single threaded purge

2.       Execute the parallel threaded purge. If the single threaded purge is not performing and then the parallel purge should be tested.

3.       Monitor the inflow and outflow

4.       Monitor database and hardware resources

For more information please refer below whitepaper published by Oracle.

References: http://www.oracle.com/technetwork/database/features/availability/soa11gstrategy-1508335.pdf

Oracle SOA Suite Components 

Service Oriented architecture interview questions

What is the basic life cycle of a SOA composite application?

Use Oracle JDeveloper to design a SOA composite application with various SOA components.

Package the composite application for deployment.

Deploy the SOA composite application to the SOA Infrastructure. The SOA Infrastructure is a Java EE-compliant application running in Oracle WebLogic Server. The application manages composites and their life cycle, service engines, and binding components.

Use Oracle Enterprise Manager Fusion Middleware Control to monitor and manage the composite application for a farm's SOA infrastructure.

What is the Service Component Architecture (SCA) used for?

Service Component Architecture (SCA) assembly model abstracts the implementation and allows assembly of components, with little implementation details. SCA enables you to represent business logic as reusable service components that can be easily integrated into any SCA-compliant application. The resulting application is known as a SOA composite application. The specification for the SCA standard is maintained by the Organization for the Advancement of Structured Information Standards (OASIS).

What is the purpose of Service Data Objects (SDO)?

Service Data Objects (SDO) provides a data programming architecture. It provides a standardized view on data, and provides efficient transportation, as well as change capture, in form of a change summary. More specifically, it collects a data graph of related business objects, called DataObjects. This graph tracks the schema that describes the DataObjects. Knowledge is not required about how to access a particular back-end data source to use SDO in a SOA composite application. Consequently, you can use static or dynamic programming styles and obtain connected and disconnected access.

What is the purpose of Business Process Execution Language (BPEL)?

BPEL provides enterprises with an industry standard for business process orchestration and execution. Using BPEL, you design a business process that integrates a series of discrete services into an end-to-end process flow. This integration reduces process cost and complexity.

What is the purpose of XSL Transformations?

XSLT processes XML documents and transforms document data from one XML schema to another.

What is the purpose of Java Connector Architecture?

Java Connector Architecture JCA provides a Java technology solution to the problem of connectivity between the many application servers in Enterprise Information Systems (EIS).

What is the purpose of Java Messaging Service?

(JMS) provides a messaging standard that allows application components based on the Java Enterprise Edition (JEE) to access business logic distributed among heterogeneous systems.

What is the purpose of Web Service Description Language?

Web Service Description Language (WSDL) file provides a standardized view on the capabilities of a service. Bindings provide the entry points into the composite at runtime.

What is the purpose of SOAP over HTTP?

SOAP provides the default network protocol for message delivery.

What are the components comprise an Oracle SOA Suite installation?

The following components comprise an Oracle SOA Suite installation:

Service Infrastructure

Oracle Mediator

Oracle Adapters

Business Events and Events Delivery Network

Oracle Metadata Repository

Oracle Business Rules

Oracle WSM Policy Manager

Oracle BPEL Process Manager (Business Process Execution Language)

Human Workflow

Oracle Business Activity Monitoring

Oracle User Messaging Service

Oracle B2B

Oracle JDeveloper

Oracle Enterprise Manager

The following components are included with Oracle SOA Suite, but available as a separate download:

Oracle Service Bus (provides service virtualization and protocol transformations for oracle SOA appl)

Oracle Complex Event Processing

What is the purpose of Oracle Mediator?

Oracle mediator is used for route, validate, filter and transform data from service providers to external partners.

Route: Determines the service component (BPEL process, business rule, human task, and mediator) to which to send the messages.

Validate: Provides support for validating the incoming message payload by using a schematron or an XSD file.

Filter: If specified in the rules, applies a filter expression that specifies the contents (payload) of a message be analyzed before any service is invoked.

Transformation: If specified in the rules, transforms document data from one XML schema to another, thus enabling data interchange among applications using different schemas.

What is the purpose of Oracle Service Bus?

Oracle Service Bus provides standalone service bus capabilities, enabling separation between application developers and target systems or services. Oracle Service Bus receives messages through a transport protocol such as HTTP(S), JMS, File, and

FTP, and sends messages through the same or a different transport protocol. Service response messages follow the inverse path. Oracle Service Bus handles the deployment, management, mediation, messaging, security and governance of implementing SOA to enterprise applications.

What is the purpose of Oracle Adapters?

Oracle Adapters use JCA (Java Connector Architecture) technology to connect external systems to the Oracle SOA Suite. Oracle SOA Suite provides the following technology adapters to integrate with transport protocols, data stores, and messaging middleware:

BAM

FTP

Java Messaging Service (JMS)

Advanced Queuing (AQ)

Files

Message Queuing (MQ) Series

Legacy Adapters

Application Adapters

What is the purpose of Business events?

Business events are messages sent as the result of an occurrence or situation, such as a new order or completion of an order. In Oracle SOA Suite, the mediator service component subscribes or publishes events. When an event is published, other applications can subscribe to it.

What is Oracle Metadata Repository ?

Oracle Metadata Repository MDS stores business events, rulesets for use by Oracle Business Rules, XSLT files for Oracle Service Bus and Oracle Mediator, XSD XML schema files for Oracle BPEL Process Manager, WSDL files, and metadata files for Complex Event Processing.

What is Oracle Business rules used for?

Oracle Business Rules, initiated by a BPEL process service component, enable dynamic decisions at runtime. In addition, the human task and mediator service components can make use of rules for dynamic routing.

What is the purpose of Oracle WSM Policy Manager?

Oracle WSM Policy Manager provides the infrastructure for enforcing global security and auditing policies in the Service Infrastructure.

What is the purpose of Oracle BPEL Process Manager?

Oracle BPEL Process Manager enables you to orchestrate synchronous and asynchronous services into end-to-end BPEL process flows. You also integrate technology adapters and services, such as human tasks, transformations, notifications, and business rules within the process.

Oracle Business Activity Monitoring (Oracle BAM) is a complete solution for building real-time operational dashboards and monitoring and alerting applications over the Web. Oracle SOA Suite makes it easy to expose SOA events, such as BPEL processes, to the BAM engine.

Oracle Complex Event Processing (Oracle CEP), a data management infrastructure that supports the notion of streams of structured data records together with stored relations. Oracle CEP is included with Oracle SOA Suite, but available as a separate download. It is optimized to handle very large volumes of events, such as those found in bank transactions, that cannot be managed by Oracle BAM. In addition, Oracle Complex Event Processing can perform complex correlations and pattern matching.

Oracle B2B is an eCommerce gateway that enables the secure and reliable exchange of messages between an enterprise and its trading partners.

COM application initiates the purchase order.

A mediator service component receives the purchase order. It validates, performs code conversion, transforms the purchase order to a canonical, and routes the document. (Canonical refers to a canonical data model that is used to transition between different document standards.)

A BPEL process service component receives the purchase order, orchestrates any required business process, and can invoke a human task, business rule, and error handling as required.

A mediator service component receives the purchase order, validates, performs code conversion, transforms the canonical to the target purchase order, and routes the document.

B2B receives the purchase order, identifies the partner, identifies the agreement, validates the purchase order, translates the purchase order to EDI, generates the EDI envelope, generates acknowledgments, and manages the secure exchange of the purchase order with the external trading partner.

Oracle JDeveloper is the development component of Oracle SOA Suite. It forms a comprehensive Integrated Service Environment (ISE) for creating and deploying composite applications and managing the composite. Oracle JDeveloper enables developers to model, create, discover, assemble, orchestrate, test, deploy, and maintain composite applications based on services.

The SOA Composite Editor enables you to create, edit, and deploy services, and also to assemble them in a composite application, all from a single location. These components are integrated together into one application and communicate with the outside world through binding components such as Web services and JCA adapters.

The SOA Composite Editor enables you to use either of two approaches for designing SOA composite applications:

The top-down approach of building a composite application puts interfaces first and implementation next. For example, you first add BPEL processes, human tasks, business rules, and Oracle Mediator routing services components to an application, and later define the specific content of these service components.

The bottom-up approach takes existing implementations of service components and wraps them with Web service interfaces for assembly into a composite application. For example, you first create and define the specific content of BPEL processes, human tasks, business rules, and Oracle Mediator routing services components, and later create a SOA composite application to which you add these service components.

Additional editors provided by J editor

Oracle BPEL Designer: You can create a BPEL process service component in the SOA composite application of Oracle JDeveloper and then design it by using the BPEL Designer, which is displayed, when you double-click a BPEL process in the SOA Composite Editor.

Oracle Mediator Editor: You can create a mediator service component in the SOA composite application of Oracle JDeveloper, and then design it by using the Mediator Editor, which is displayed when you double-click a Mediator in SOA Composite Editor.

Human Task Editor: You can create a human task service component in the SOA composite application of Oracle JDeveloper and then design it by using the Human Task Editor, which is displayed when you double-click a human task in the SOA Composite Editor.

Business Rules Designer: You can create a business rules service component in the SOA composite application of Oracle JDeveloper and then design it by using the Business Rules Designer, which is displayed when you double-click a business rule in the SOA Composite Editor.

Oracle Enterprise Manager

You can configure, monitor, and manage your SOA composite application during run time from Oracle Enterprise Manager Fusion Middleware Control Console. Fusion Middleware Control is a Web browser-based, graphical user interface that you can use to monitor and administer a farm.

A farm is a collection of components managed by Fusion Middleware Control. It contains Oracle WebLogic Server domains, one Administration Server, one or more Managed Servers, clusters, and the Oracle Fusion Middleware components that are installed, configured, and running in the domain.

You deploy SOA composite applications designed in Oracle JDeveloper to the SOA Infrastructure. The SOA Infrastructure is a Java EE-compliant application running in Oracle WebLogic Server. The application manages composites and their life cycle, service engines, and binding components.

The SOA Infrastructure provides you with access to all deployed SOA composite applications, service engines, service components, business events, and other elements. Fusion Middleware Control organizes a wide variety of performance data and administrative functions into distinct, Web-based home pages for the farm, domain, servers, components, and applications.

Fusion Middleware Control provides a wide variety of administrative and performance data for the SOA components, composite applications, and composite instances within the SOA infrastructure, enabling you to administer and pinpoint issues.

The basic life cycle of a SOA composite application is as follows:

Use Oracle JDeveloper to design a SOA composite application with various SOA components.

Package the composite application for deployment.

Deploy the SOA composite application to the SOA Infrastructure. The SOA Infrastructure is a Java EE-compliant application running in Oracle WebLogic Server. The application manages composites and their life cycle, service engines, and binding components.

Use Oracle Enterprise Manager Fusion Middleware Control to monitor and manage the composite application for a farm's SOA infrastructure.

Services provide the outside world with an entry point to the SOA composite application. The WSDL file of the service advertises its capabilities to external applications. These capabilities are used for contacting the SOA composite application components. The binding connectivity of the service describes the protocols that can communicate with the service, for example, SOAP/HTTP or a JCA adapter.

Performance drop when inserting large Payload using DBAdapter

In BPEL and OSB when  inserting a lot of data (big payload messages) into the database by using jca db adapter.
We noticed performance would drop hugely when the db adapter needed to insert large payloads (CLOBS) and when the stream of messages on these services would continue even more performance would dramastically go down on the system.

After disabling “wrap data types” on the datasource in Weblogic we had a pretty good performance win on processing big clob messages into the database.

To disable wrapping of data type objects do the next :

·       Login the Weblogic Console

·       Click yourdomain > Services > Data Sources > your_datasource > Connection Pool > Advanced > disable Wrap Data Types

Difference between Oracle SOA Suite 10g and 11g

 

Main difference between Oracle SOA Suite 10g to 11g

SCA architecture was followed in 11g and not in 10g

In 11g one can put all our project SOA components in composite.xml file and deploy to single server, where in 10g we have to deploy each component to the respective server(i.e ESB to ESB server, BPEL to BPEL Server)          

1. Basically all the SOA components like BPEL, ESB (Called Mediator in 11g), & OWSM are brought into one place in 11g using SCA composite concept.
2. The major difference between 10g & 11g would be the app server container. 10g by default runs on OC4J while 11g runs on Weblogic Server.
3. In 10g every BPEL is a separate project, but in 11g several components can make 1 project as SCA.
4.  In 10g consoles are separate for BPEL and ESB, but in 11g Enterprise Manager contains all.
5.  In 10g we have to deploy each project separately, but in 11g we can deploy SCA which contains all.
6. In 10g BAM and business rules are outside SOA Suite, but in 11g they are in SOA Suite

ESB is now Mediator

• Oracle Enterprise Service Bus (OESB) was the Service Bus for SOA 10g
• In SOA 11g ESB is known as “Mediator” and acts as a component in a SCA assembly
• The role of ESB in SOA 11g now is to provide The role of ESB in SOA 11g now is to provide mediation services between SOA Suite components

Transaction handling within OSB

Lots of people get confused about the transaction handling capabilities in  OSB. This is because transaction enlisting and demarcations are implicit in OSB. This means OSB developer have no explicit actions to start/commit/rollback transactions. However it is possible to influence transaction via code. Lets explain this with JMS transport as example as it will be used as the de-facto transactional transport in asynchronous interfaces.

Enabling Transactions
1. Make sure the source JMS transport URI is using an XA enabled Connection Factory.

2. In Proxy Service Configuration –> JMS Transport –> Advanced Settings –> Check Is XA Required

3. In Proxy Service Configuration –>Message Handling Configuration –>CheckTransaction Required.

This will instruct the underlying OSB JMS Transport to start a transaction before executing the proxy service code. The transaction will be committed/rolled back based on how the proxy service code executes.

Committing transaction on non-recoverable exception
1. Transaction will commit implicitly if proxy service code executes successfully.
2. In case of error and if proxy flow enters an error handler , the transaction can still be committed by using a reply action ( “Reply With Success” or “Reply with Failure”)

It can look contradicting that a reply with failure action can in fact commit a transaction, but that’s how it works!
Request Transaction Manager to rollback the transaction
A transaction can be marked to be rolled back by the transaction manager by allowing an error to propagate all the way up to the system error handler.

OSB’s error pipeline can consist of either of the following sequence :

§  Route Error Handler –> Service Error Handler –> System Error Handler (Comes into picture when an error occurs in a route node)

§  Stage Error Handler –> Pipeline (request or response) Error Handler –> Service Error Handler –> System Error Handler (Comes into picture when an error occurs in a stage).

If a particular error handler is not present the error is propagated to the next available parent in the chain until it reaches system error handler. This propagation can be stopped by using a ‘reply’ action in any of the error handlers. The developer can mark a transaction to rollback by allowing an error to propagate all the way to the system error handler byNOT configuring a reply action in any of the intermediate error handlers.

Enlist the target business service within the same transaction
1. The target business service to which the message is routed (using Route Node) or published (using Publish Action) can be enlisted within the same transaction. For this the business service endpoint should be transactional i.e. if JMS should use a XA enabled connection factory.
2. Within the route node or publish action request action, set Quality of Service to “Exactly-Once” using a routing options action.

If the outbound transport is HTTP, there will be a design decision you will need to make on what type of errors to be marked for rollback. OSB’s HTTP transport commits all transaction if it gets a HTTP response code back from the server. So even if it gets a 404 Page Not Found or 500 Internal Server Error , HTTP transport doesn’t mark the transaction for rollback. This will result in message loss if the target service is frontended by webservers. You will get a http error back from the webserver even if the backend app servers hosting the service is down. The transaction then gets committed as HTTP transport didn’t mark it for rollback.

On the other end if you are sending a faulty message and service keeps on returning a soap:Fault which will be returned with a HTTP response code of 500, this will be non retriable fault and you would want to commit the transaction after publishing to an error destination. As noted before this is a designer decision to be taken depending upon your error handling requirements.

To override the default behaviour and make the HTTP transport commit the transaction only on receiving a success response from server ( HTTP response code 200 or 202) , make sure you check the ‘Same Transaction for Response’ checkbox.