Your MOM sucks and doesn't belong here
Anyone doing any development for a financial institute will no doubt have come across any number of projects that have huge budgets and lofty goals, all of which involve incredible amounts of wild hand waving and some mumbo jumbo about moving from point to point connections to some kind of magical fairy land run on top of a centralised message bus that makes everyone giggle with glee.
Now that said software has been procured (and appropriate droves of ‘experts’ hired to install said software), the next step is to fumble about wildly trying to find a use for the damn thing. Thus, the ‘integration project’ (complete with suitably impressive codenames) starts its long tortuous life.
The next step is in identifying all the legacy systems that will now be integrated. Invariably, a bunch of these have all been working very well for over a decade, and involve the mystical and mysterious art of ftp and end of day batched processes.
Here of course is where such projects will come to a grinding halt. The reality is that all these systems could not possibly be any less interested in a realtime approach. They equally have next to no interest in moving away from ftp. It works, it’s worked for years, everyone is happy with it, except for those clever folk who just spent a few million dollars/euros on MOM infrastructure.
The next step is the Big Compromise step. Here the new cheerful kids realise that the old grumpy ftp folk are not jumping onto their bandwagon. So in order to make everyone happy, the bandwagon gets some modifications, like square wheels, pegs that nail it to the ground, and suchlike. The perfect solution manifests itself….let’s just send the files across our message bus! That way you get all the benefits of point to point (one EOD file per system), and the MOM obsessed get to justify their existence since said file is sent across the message bus.
Having done all that, one cannot help but wonder….why bother? Exactly what has the million dollar piece of software you’ve stuck in the middle of it all done for you?
November 24th, 2003 at 2:19 pm
1) MOM’s are much better for high granularity workflow engines.
2) pseudo-realtime systems allow you to process requests throughout the day rather than hoping all your requests can be processed between the hours of 3am-6am. And large end-of-day batch jobs may force you into buying a really big chunk of iron that runs full-tilt for three hours and then does little but generate BTU’s and burns holes in the ozone layer during the rest of the day.
3) ftp isn’t known for its transactional integrity, but it’s true that it’s possible to get it right if you make a determined effort.
November 24th, 2003 at 2:36 pm
Often MOMs are foisted on people for no good purpose, and that could well be the case here (just like some people may use Oracle when a flat file would be overkill for their problems). But for some industries, what worked in the past won’t necessarily work in the future. For example, in the financial industry for years all biz transactions were T+3, meaning that you had 3 days to process the sucker. We’re now down to T+1 (you’ve got a day), and moving towards T+0 (you gotta do it now!). What this means is that the biz is now forcing IT systems to move from batch operation to real time, asynchronous operation. To summarize, FTP just isn’t cutting it anymore.
November 24th, 2003 at 2:40 pm
It has created work for an army of high priced contractors that this large financial institution inevitably has on the payroll.
The project was pitched by the ‘architects’ came up with this MOM infrastructure because they were bored silly maintaining legacy applications that do the *real* work in said company (but written the language of the year winner from 1997). Aside from boredom, this new project gives them opportunity to learn new stuff and add a few more buzzwords to the C.V.
Oh did I mention the wining and dining treatment that the I.T. Director got from the MOM vendor’s salesforce?
November 24th, 2003 at 2:45 pm
The pb of T+0 is not link to the FTP technology. It is more a pb of how you build your software.
November 24th, 2003 at 3:05 pm
The pb of T+0 is not linked (sorry for the mistake) to the FTP technology. It is more a pb of how you build your software. Integrate non transactional applications (legacy) and want transactional behaviour is a prowess !
—
to fate: if you don’t get it, you can try to apply for another job ! I did it…
November 24th, 2003 at 3:06 pm
This is one of your best posts ever. I read most of it and thought “I wonder if he’ll bring up the fact that most MOM systems end up carrying files across the bus, in imitation of FTP” and there it was! It’s like magic.
November 24th, 2003 at 4:04 pm
The funniest thing is when the manager of one of these projects has the briliant idea that your application should be able to connect to different MQ servers for sending and recieving messages. These would be different servers in the same company. So you just spent millions on software that can route from any app (via it’s local MQ server) to any other (that only talks to it’s local server) and rewrote that functionality yourself. It’s like cutting out your local SMTP server and Outlook connecting directly to the recipients MTA. Brilliant.
November 24th, 2003 at 4:05 pm
With every MOM there is the concept of magic buses with glittering onramps and offramps. It really just makes me want to punch the clown at every exit.
Applications should really be in the business of providing appropriate interfaces (i.e. SOAP or XML-RPC) to expose the means for any client to access the necessary business function. If other “middleware” type components are needed to broker transactions or consume services and provide aggregate services than so be it. They would act as any other application would providing their own “standard” interface for others to consume.
The message here is not too create some bloated clown punching proprietary MOM system to introduce yet another tangled mess but to subscribe to a standards-based approach where systems provide the interfaces. To that end, I suggest punching the clown before pursuing a million dollar MOM boondoggle.
November 24th, 2003 at 4:20 pm
With every MOM there is the concept of magic buses with glittering onramps and offramps. It really just makes me want to punch the clown at every exit.
Applications should really be in the business of providing appropriate interfaces (i.e. SOAP or XML-RPC) to expose the means for any client to access the necessary business function. If other “middleware” type components are needed to broker transactions or consume services and provide aggregate services than so be it. They would act as any other application would providing their own “standard” interface for others to consume.
The message here is not too create some bloated clown punching proprietary MOM system to introduce yet another tangled mess but to subscribe to a standards-based approach where systems provide the interfaces. To that end, I suggest punching the clown before pursuing a million dollar MOM boondoggle.
November 25th, 2003 at 1:46 am
i just stumbled across this blog somehow and i have to say its the BEST JAVA BLOG i’ve ever read. fate, you are a damn genius. i have forwarded the url on to many of my dev friends.
November 25th, 2003 at 5:36 am
Amen, brother!
My company is stuck in EAI/MOM hell at the moment. Having to sit quietly while managers and their pet consultants yak endlessly about Message Brokers/ Enterprise Busses/ Content-Free Gartner Buzzword of the Week without flipping out and ripping out their larynxes is proving extremely trying.
November 25th, 2003 at 5:37 am
You really don’t get it do you Hani ?
MOM’s are but one part of that game they play in the financial world …. “wank word bingo”. ALl the senior staff have their bingo cards and try to get these words in during a meeting, ticking them off when they do. All building themselves up into a frenzy.
They’re not making choices on technological grounds …. really, they’re not!!
November 25th, 2003 at 6:26 am
Ah Hani,
But when you see the terrifying mess of scripts, little purpose built “applications”, typically written by support people with no development experience, and the sheer beurocracy that goes into keeping this system of moving files around by ftp limping along…. you will grovel on your knees and beg for forgiveness and kiss the CD on which the MOM is shipped.
FTP may have worked for years… for kiddies downloading porn, but there is an entire cottage industry around making ftp into something like a JMS Queue. All of it as complex as a swiss watch.
And for the poor unfortunate downstream systems, rather than a simple onMessage() to implement, they have to scan a file system for new files. However, its not as simple as that, because the new files might be currently written to (thats why they are new). So you have to have a signal file that is transfered once the main file transfer is finished. So you have to look for all new files – but not ones that are newer than the signal file….
Then, if the waters werent muddy enough already, you have to think about what happens when a failure occurs. “Oh shit, we already have a file of that name – what are we going to do now?”. “Oops the host went down. Did the file transfer complete? What do I do now?” “My database has just gone down – do I leave this file here, or move it to an error directory?”
Bottom line: FTP is a fucking big mess and a sad substitute for a message queue.
Dont get me wrong, I’ll be the first volunteer to lop off the heads owning the mouths that spout shite about a “message bus”. But, ftp is NOT the way to move information around…
-P
November 25th, 2003 at 6:55 am
“messagebus” ? Have you worked at UBS ?
November 25th, 2003 at 7:09 am
The problem with MOM is that some senior staff want it everywhere for inter-application integration. There are place where a message queue can be used and there are place where other technologies are better, MOM are not everywhere !
November 25th, 2003 at 8:31 am
messagebus? don’t you mean bangbus?
November 25th, 2003 at 8:39 am
all your moms are belong to us
November 25th, 2003 at 10:12 am
I find it humorous when Tibco tries to sell itself as an EAI tool. It was developed for real-time market updates. This means that messaging is RELIABLE, not TRANSACTIONAL. EAI in the business world most often needs transactional, once and only once delivery. Tibco was never designed for this. Trying to make Tibco transactional is like trying to make your favorite porn 3d by crossing your eyes.
Tibco consultants are the biggest group of no-talent ass clowns I have ever met.
November 25th, 2003 at 10:40 am
Of course, if when I talked about MOM above, I DIDNT mean Tibco Rendezvous. RV is completely unsuited to this kind of task.
Hani, out of curiosity, are you also using Tibco Business(doesnt)Works? I know your pain…
Steer well clear of it, is my advice.
-Nick
November 25th, 2003 at 5:56 pm
>>Maybe I’m young and naive
It’s a 10-4 little hani.
November 25th, 2003 at 7:58 pm
I used to work at a UK company that had bought £3 million worth of Tibco…and they couldn’t find a use for it. The architects went on an on about how all the new products they bought ‘had’ to integrate with the bus…but it was amazing, when you pinned them down in a corner they didn’t know why!!!
November 26th, 2003 at 3:29 am
The only problem with buses is that they never come along when you want them, and when they do 2 come at the same time.
November 26th, 2003 at 5:37 am
having worked extensively in financial companies there are plenty of good applications for message-ware, the problem is ‘one size fits all’ the particular size (tool) is not important it’s the rest of that statement.
message-ware is tried and proven in financial apps, what do you think FIX is (albeit over a synchronous connection)? of course it’s messaging protocol was designed by fat bankers on acid on holiday in barbados, but once you get your head around it’s quirks (and your connection partner’s non standard use of it) it’s possible to do at least place trades with the damn thing.
what’s really demented is the current project i work on where i watch the poor EAI team tear their hair out trying to implement business logic in webmethods! hah haa ha! the poor sods.
November 26th, 2003 at 1:16 pm
Why spend millions on TIBCO, when spread is free?
http://www.spread.org. Spread additionally provides safe messaging guarantees, which are required to implement virtually synchronous applications across a network.
November 28th, 2003 at 12:57 pm
yo fate, I just wanna get me some of those financial bitches holmes.
December 2nd, 2003 at 9:47 am
Message Queueing is better than ftp, but more expensive by a factor of infinity.
December 2nd, 2003 at 3:16 pm
|Message Queueing is better than ftp, but more
|expensive by a factor of infinity.
|
Depends how you measure it.
Keeping the ftp mess running isnt free.
Its common that some kind of MOM (usually MQSeries) is already laying about the place…
December 4th, 2003 at 4:20 am
Remember Information?