thread management?
Is thread management going to be discussed in the next release of the spec? It seems dangerous to me that a component can create threads which can interfere with the container.
I embedded Jetty into a binding component within OpenESB. Because I did not properly shut Jetty down, OpenESB could not uninstall the component because the servlet engine thread was still running. And restarting OpenOSB didn't help because upon restart the component is started again. Deleting the component from the OpenESB repository isn't easy - you can't just delete the deployed directories/files. If that's all you do, the component won't be loaded, OpenESB still thinks it is. You also have to delete the component references in registry.xml and ComponentData.xml. I realize this last part is an OpenESB issue, but the spec should address threading in some way - either manage it or disallow it.
-tim
[909 byte] By [
tdrurya] at [2007-10-2 21:11:02]
Tim,
At the "What's next for JBI?" birds-of-a-feather session held at JavaOne less that two weeks ago, thread management was indeed on the list of things to consider for JBI dot next.
The chief danger of letting components manage their own threads is the possibility that they will not shut them down properly. If such threads are not daemon threads, the JVM cannot shut them down, making orderly shutdown impossible. Aside from this, allowing components to manage their own threads isn't "dangerous."
Open ESB has a couple of weaknesses in its internal state management that removing uncooperative components quite difficult. This is being addressed.
What form do you think thread management ought to take in JBI? A work item pool, like Java connectors, or simply a centrally managed thread pool? Or perhaps you have other ideas?
-Ron
Consider a manufacturing execution system (MES) which runs a factory floor. It interfaces to hundreds, if not thousands, of machines. It interfaces to ERP, inventory, etc. It must run 24x7, be fault tolerant, and clustered. How can a single binding component handle messages from a thousand machines with a period of seconds to minutes and message sizes from tens of bytes (high freq) to tens of megabytes (lower freq)? They cannot be synchronized through a single threaded component without destroying the
transaction rate. How can a single transform service engine convert those messages from their native format to the MES format?Again, In a J2EE container, the container creates new threads for the session beans that are needed under load. It can also collect and deactivate those beans when their loads are low. Clustering is built in.
I know we can all write thread management into our components, but why should we have to?
I'd like to see the message listener interface changed from a Runnable interface to something more formalized and managed such as, um, MessageHandler. This class is responsible for handling messages that are serviced by this component. This class should _not_ be explicitly threaded, but instead managed by a thread pool within the JBI container. If the message bandwidth increases, the container can draw from the thread pool to handle them. As the rate decreases, the container can
recover the threads and place them back into the pool.
There may not be much that can be done with binding components since there are a myriad ways in which interfaces connected to the JBI container. Most physical interfaces must be synchronized/serialized, but not all. For example, email and file polling interfaces can be handled the same way addressed above. In addition, it would be nice to have some clear hand off
where a message is guaranteed to be handled without loss. How about an InterfaceHandler which is given access to a database-backed JMS queue? The handler can be thread-managed by the container and its sole responsibility to read from an interface and place the raw data into the queue. Once queued, JBI guarantees the message will be delivered to its destination (if one is defined).
Comments?
Ron, addressing your comment that managing your own threads isn't dangerous, I have to disagree. While it sure would be nice to work with a group of extremely skilled and experienced developers, that just doesn't happen. It's also possible that third-parties and/or customers will write components that run on your bus that you "own". If one of these components is poorly written, it can bring the bus down. While it's possible to do this in a J2EE container also, it's a bit harder. Some containers had removed the Thread and File I/O classes from their managed classloaders although I don't think that happens much any more. In any event, it was pounded into J2EE developers heads to avoid Threads and File I/O but I don't see the same warnings in JBI. Are they to be avoided or are they allowed? If allowed what happens when the JBI container is clustered? When happens when a thread cannot be collected by the container? I've already seen what happens in OpenESB - you have to shutdown the container and manually edit XML files and delete other files. We can't have that in an MES system.
I think the JBI spec really does fill a gap in J2EE, but it is, understandably, immature. I think I'll stick around and help it grow up! I'm tired of EAI systems - vender lock in, proprietary APIs and methodologies, outrageous license fees, etc. I want JBI or something like it to succeed.
-tim
Tim,
Thanks for the thoughtful comments. I used to work in industrial automation, so I am very familiar with MES scenario.
Please note that while the DeliveryChannel must support multi-threaded use, it does not have to be implemented as (to delegate to) a single synchronized object. You are correct that this is a difficult area to get right, in order to support proper scalability, along with other performance and reliability characteristics. JBI 2.0 will take a closer look at clustering and thread management with these needs in mind. As an aside, I must comment that in industrial automation systems where determinism is key to achieving key performance goals, the main stategy is to provision for the worst-case scenario: the resources needed for peak load should be preallocated, or available is such quantities that peak load will not exhaust them. The best you can do with sharing resources is try to ensure graceful degradation of performance under load.
Your suggestions on how to structure a JBI component to avoid building your own thread management are good ones. We have heard a similar story from many users of JBI technology who wish to write their own components: components ought to be easier to write. The JBI 1.0 specification assumes that component authors are, for the most part, container authors: they are, as you say, extremely skilled and experienced developers. This is not a good fit in many development organisations. JBI components are often containers themselves. I wouldn't advise the implementors of an EJB container, for example, to avoid using threads or file I/O! Yet the EJB container itself presents a very different programming model to the ultimate user.
This picture of the JBI component developer has proven to be a poor fit in some organisations, partly due to skill sets, and partly due to simple misunderstanding of the technology itself. It is really in infrastructure standard, to be built on. It is designed to enable service-oriented integration, but does not provide the particular technologies needed in every integration (or other service provision) situation. For example, JBI doesn't define a Modbus/TCP binding component, yet this is necessary if you have some Modicon gear in your plant. As JBI is "built out", you'll be able to purchase a binding component for such protocols, but in these early days you may well have to "roll your own."
Obviously a framework can be built atop the JBI 1.0 APIs and SPIs to make development tasks simpler. I've suggested to the Project Open ESB folks that they take this on. In addition, the JBI 2.0 expert group will look at this issue. If you feel strongly about this issue, you should consider joining the expert group (the JSR should be filed with the JCP soon).
-Ron
Tim,
One other thought on the subject of scaling. JBI 1.0 is really built with the enterprise service bus (ESB) in mind. ServiceMix and Project Open ESB are two examples of how JBI can be used as a foundation of a distributed SOA.
By distributing the system, and employing intelligent message routing, one can avoid running all message traffic through a single binding component, or having all business logic supplied by a single instance of a particular service engine. The ESB approach provides scalability, flexibility, and fault tolerance.
Instead of a single JBI instance running in an application server (perhaps clustered), one has several JBI instances, tied together through a service bus such that they make services available to each other in a fashion that looks like a larger, single JBI instance.
I won't get into a full description of ESBs here. Suffice to say that it is an alternative approach to designing your system to create a scalable, fault tolerant, flexible SOA.
> [...] In addition, it would be nice to have some
> clear hand off
> where a message is guaranteed to be handled without
> loss. How about an InterfaceHandler which is given
> access to a database-backed JMS queue? The handler
> can be thread-managed by the container and its sole
> responsibility to read from an interface and place
> the raw data into the queue. Once queued, JBI
> guarantees the message will be delivered to its
> destination (if one is defined).
Reliable message exchange can be accomplished within the JBI 1.0 environment by using its support for transactions. This includes transactions with the JMS component's interactions with the backing store for queue. The JBI 1.0 specification has an appendix (starting at page 207) that discusses how transactions can be so used. For example, the JMS binding component will, when reading a message from the queue and normalizing it, include the transaction context for reliably reading the message and taking it off the queue. This context "flows through" to the service provider, which can use it to co-ordinate its own state changes with the queue's, giving us reliable message exchange from JMS queue to the SE data store.
This is just an example. The point is that JBI enables such behaviour on the part of components; it is up to the component (and the component writer) to take advantage of these features as appropriate. It isn't auto-magic reliable message exchange, but if the component is a container, such behaviour could be provided to the container user. For example, if the SE is an EJB container, the transaction context "flowing through" from the service consumer could transparently be used to wrap database updates in the transaction. The EJB writer would not have to do anything to enlist in the transaction; the container would do that automatically.
-Ron
ServiceMix provides some features that help writing JBI components:
* a framework which take care of thread management, transactions, deployment, etc...
* a push model for components, in which case all this stuff is handled by the JBI Container
* a lightweight deployment model for wiring simple components
> ServiceMix provides some features that help writing
> JBI components:
> * a framework which take care of thread management,
> transactions, deployment, etc...
> * a push model for components, in which case all
> this stuff is handled by the JBI Container
> * a lightweight deployment model for wiring simple
> components
A quick question about the ServiceMix component framework: is it portable to other JBI implementations (ones supporting transactions, at least)?
Thanks!
-ron
The appendix in the spec about transactions is really light and does not cover all the possible transaction use cases.
The main problem is that the transaction can only be conveyed by the JBI exchange if the exchange is send synchronously. And such behavior can not work with transactional transports such as JMS: if a component send an InOut MEP to a JMS binding component, this can not be done inside the same transaction (because the jms message can not be delivered until the transaction is commited). You may also want to send multiple messages inside the same transaction, but send them asynchronously.
Transactions inside JBI is a tough part and I think it deserves a real work. Hopefully this will be addressed in JBI 2.0.
ServiceMix framework should be easily ported to other JBI containers as it is not strongly tied to ServiceMix container, but uses a few classes that could be extracted. But it leverages some features that are not specified by the spec (such as transaction management, clustering, etc).
Ron, sorry this is such a late response but I was away from my computer for a couple weeks (*gasp*).
Actually, determinism isn't an issue. Any real-time requirements would be handled at the PLC, not by the MES. However, there are some performance requirements that should be met and which can only be met by processing these messages in parallel. It is typical to deploy an MES on a server with many processors and/or a cluster. We can't have all the manufacturing-floor machinery talking to a single, synchronized port on the integration server, but this appears to be the case with JBI 1.0. If all the machines talk to, for example, an HTTP binding component and even if that binding component spawns many threads to pull in the data from many sockets, those threads will still have to compete for the single synchronized DeliveryChannel assigned to that binding component. All the messages from the shop floor end up being synchronized.
What really raises my eyebrows is your statement that JBI 1.0 assumes that component authors are also the container authors. Surely this cannot be? Can you elaborate on this? If true, then JBI is nothing more than EAI rehashed and we are, once again, at the mercy of the JBI/EAI vendors. I thought JBI was to provide a framework and infrastructure similar to EJB where all the common, horizontal services would be handled by the container and customers could buy/write components which plugin into the container to handle their vertical needs. Perhaps I misunderstood...
-tim
I suppose this request was unrealistic at this time. I was hoping the spec would dictate some sort of handshake so that an external system can know with 100% certainty that its data was successfully passed to the binding component (and, by virtue of JBI transactions, to its destination component). I can see how you are limited to whatever functionality a binding component author deems appropriate.
-tim
We've been prototyping with ServiceMix, Petals, and OpenESB. While I'm sure some people find helper classes useful, I try to avoid them. This stems from my original assumption (at the top of this thread) that I can author my own JBI components and move them from container to container in a similar fashion to EJBs. I realize this doesn't always work with EJBs and we typically have to have deployment descriptors for each container we deploy on, but the basic premise remains. And I cannot move JBI components from ServiceMix to anything else if I build them via the ServiceMix helper classes.
Petals actually has some very clean, implementation-agnostic, example components which almost all deploy on all the JBI containers. The only exception is their swing-based client test component.
Tim,
Welcome back. I hope that two weeks away from your computer was spend enjoyably.
But getting back to the DeliveryChannel topic. I think you are misinterpreting what I said earlier in this thread. The DeliveryChannel must allow multi-threaded access by the Component. That is not the same as saying it is a single synchronized object. While marking each method of the DeliveryChannel implementation as synchronized would be a fairly easy way to fulfill this requirement, it is not the only way. For example, I have seen a DeliveryChannel implementation that farms out send() calls to a pool of procecessing objects. The only time such an implementation will block the component's thread will be when the pool of processing objects is empty. It would be rather useless to serialize all concurrent DC requests, although naive implementations are certainly free to do so. If you look at the Open ESB implementation of DeliveryChannel (runtime/src/com/sun/jbi/messaging/DeliveryChannelImpl.java) you'll see a rather elegant implementation that is thread-safe and performant.
(Sun has spent a lot of time over the last few years making chip-level multi-threading (CMT) a reality. Soon will will have SPARC processors than execute 64 separate threads concurrently. We obviously want to take advantage of cool iron like that!)
> What really raises my eyebrows is your statement that JBI 1.0 assumes that component authors are also the container authors.
That not quite what I meant. I meant that similar skill sets are often needed, especially if the components are sophisticated. JBI components often are containers themselves. The GlassFish Java EE JBI service engine, for instance, is a container for servlets and EJBs. The Open ESB WS-BPEL engine is a container for BPEL processes. In this sense the component author is also a container author. Just as you wouldn't roll your own EJB container, you won't want to create your own BPEL engine, SOAP/HTTP binding, etc. You should be able to purchase the technologies you need, often in the form of service engines that act as containers.
JBI components *should* be portable. A major aim of the specification was to break the old EAI tradition of vendor lock-in by standardizing service-oriented integration. We cannot prevent component authors from deliberately tying their components to a particular JBI implementation, but that is something of a risky approach for a component vendor. Portability still costs, but between the use of Java and the use of JBI, portability should be, in most cases, easy to achieve.
Ron, thanks for the response. That makes much more sense. Most JBI containers will probably implement a threaded DeliveryChannel, but the buyer should research it first.
With regard to the JBI container/component authoring issue, what you are saying makes sense (again). I misinterpreted your meaning. I think binding components should remain simple and within the realm of most competent developers. This is important since these are the components that are most likely to require customization - for example, to talk to your custom apps or non-JBI supported apps.
I agree that a fully-loaded service engine which can be managed dynamically can be a daunting task.
I think, perhaps, one source of confusion is the role of JBI within an enterprise architecture. I find myself wanting to place business logic within a service engine as if the JBI container is some substitute for a J2EE container. I suppose you _could_ do this if your business logic is a set of BPEL processes run by the BPEL engine within JBI. Are there any papers out there that discuss the role of JBI within an enterprise architecture? We're using to replace our EAI and to marry our J2EE applications to external systems. Is this its primary role? Does JBI have other roles?
Somewhere I had read that JBI would be a "container of containers" and I envisioned a J2EE container would be just another service engine. But then reality set in and, at this time, I cannot see the fairly fragile implementations of JBI wrapping an extremely robust J2EE container like Weblogic. I'm constantly having to stop and restart OpenESB/Sun AS9 which would retart the J2EE container also. We almost _never_ restart Weblogic in production. Do you agree with the "container of containers" description or maybe there's an alternate interpretation?
Ron - I really appreciate the time you've spent contributing to this forum.
-tim
answering my own question, I found this paper on SOA, J2EE, and JBI from Sun: http://java.sun.com/developer/technicalArticles/WebServices/soa3/
