Communication between two apps

Question

I am thinking of creating two applications, one of which (App 1) will be in Django (DRF) and other (App 2)might be Django but might be another more lightweight framework (maybe Flask or plain Django or maybe different language altoghether depending of how efficient will be the script in Python). Anyway the idea is that the App 2 needs to be standalone.

I am thinking of creating a connection between them that would be working in both directions.

App 1 (Main Django app) would send request to the App 2 to start calculation (which would take quite long time). Response for this request would be some information that App 2 received request and now is doing calculation in background, so that App 1 would not be waiting for this request to finish. Then when App 2 finishes it would then send request to App 1 to inform App 1 that it(App 2) had finished calculating and updated database (which is shared). After that App 1 sends information to the FrontEnd that everything is ready (updated database, App 2 finished calculations).

What would be best way to enable this back and forth communications?

I read about celery-redis, but should I implent this in App 1 or in App 2 in this case? As far as I understand I could create task queue in App 1 and when App 1 sends first request to App 2 (like calling external api), the request is moved to the task queue and the response will be later delivered when App 2 finishes calculation, is that the "go to" approach?

I also started reading into websockets (django channels) and Im wondering if they would be better fit for this ase.

Answer 1

The idea that App 2 makes a request back to App 1 is a common architecture to get web services to work together. Usually, this would work like this:

• App 1 makes a request to App 2
  • the request contains an URL on App 1 to be called upon completion
• App 2 responds to the requests, and begins background work
• eventually, App 2 requests the URL provided by App 1's initial request
  • this request can either contain necessary status data, or App 2 might have an API from which App 1 can retrieve necessary data

Such a design is e.g. common on many authorization or payment flows on the web, where multiple parties must work together. However, there's a lot of back-and-forth here, which might be inefficient.

A completely different approach uses message queues.

• App 1 publishes a message on a topic
  • this happens asynchronously, so there is no initial response by App 2
  • each message has an ID
• App 2 is listening on that topic and will eventually process the message
• App 2 publishes a new message with results
  • the results message will have a correlation-ID that matches the ID of the request
• App 1 is listening for results and will eventually receive a results message with the desired correlation ID

Such message queue based architectures are especially interesting in high-throughput scenarios, or in an enterprise context where you want to be able to easily add more listeners for events, instead of HTTP-style 1:1 communication.

Celery is a Python tool that uses such message queues, but simplifies them. With Celery, Celery is the App 2 – you must start a celery worker server. This lets App 1 easily perform some function call as a background task. This is similar to using async functions or to starting a thread, but the Celery worker doesn't have to run on the same server.

So if you just want asynchronous background tasks, Celery is a good fit. If you already have an App 2, Celery won't help. Instead, you should implement one of the HTTP-request or message queue based approaches.

Answer 2

I would do it with a queue in both directions. App1 -> queue -> App2 -> queue -> App1. I would use RabbitMQ and have multiple instances of App2 if the load increased.

The problem you could have is with notifying front end, but I guess you could have some sort of polling or push notifications when the operation finished.