Is my approach at building middleware using a queue a good approach?

Question

I want to build a middleware system that my company can use and expand on to handle the integration points between our various bespoke and 3rd party systems as well as our hosted websites.

I am having some trouble with the design/architecture and would like some advice (please let me know if there is a better site to ask these questions).

The middleware will basically be the broker between our ERP and our other systems such as our website/3rd party retailers etc. It will be mainly dealing with API calls and XML file exports.

I was thinking of:

• Using RabbitMQ as a queuing mechanism.
• Using Quartz.NET as a scheduler contained within a Windows Service. It would be very lightweight and would simply load up a list of jobs from a database, dropping these into the queue when their schedule is reached.
• Using a windows service (I'd call this the agent) which would be a consumer of the RabbitMQ queue. It would monitor the queue and process the jobs as they would come in. Ideally, I would want to be able to run separate jobs at the same time so that one job doesn't necessarily tie up another job from running. We can always scale this portion out if required.

Does anyone see any issue with this approach? I can't find a lot of example implementations on the internet about using RabbitMQ as a worker service in .NET (using multiple threads) and am not sure if using a scheduler service to simply drop items on the queue is the right approach.

Answer 1

To elaborate a little bit more on our comment exchanges.

Advantages of your queue based approach

Your queue based approach has definitively some advantages:

• Producers ("your db feeding jobs") and consumers (your agents) are decoupled: They don't have to know each others internals. They only have to know your queue and the data structure therein. THis will make your whole architecture more maintainable.
• You could scale the producer side and enrich the data sources by connecting other applications to the queue.
• You could cope with production peaks, when producer proce fasters than consumers can consume, by spreading the processing over time thantks to the asynchronous behavior allowed by your queue.
• You could scale the consumer side by multiplying the processing nodes and distribute them across servers, if there'd be any lack of capacity.
• depending on the queuing system, you could even scale the queue through distribution (With RabbitMQ you can set up a distributed installation, same for other systems like for example Kafka which aims specifically at big data).

Here a picture to highlight all this: :

Disadvantage of your queue

The queue adds a level of complexity to your architecture and has some impacts:

• The queuing system is an additional system to be installed, maintained and operated.
• Everything is dependent of the queue. New releases could require to update all the producers and consumers. And you might find yourself dependent of your queuing technology vendor.
• In terms of processing power, the queuing architecture adds some overhead, adding more processes and requiring processing power for the intermediary.

Conclusion

In my opinion, the inconvenience are marginal compared to the benefits. The flexibility and scalability enabled by the queuing architecture makes it a future proof option in many scenarios.

However you don't seem to really take advantage of the queueing in your system. If your architecture is designed to stay like that, one could ask why you do batch processing by batch-loading your queue and not directly process the records from the database without additional overhead. If everything is driven by your db, your db will anyway be the bottleneck.