2

I have a service A that is essentially doing this:

  1. Fetches batch of events from external service B.
  2. Based on these events, transforms data from the database and persists it back to the database.
  3. Sends a confirmation that batch was successfully processed to service B.

I'm trying to implement this in a "clean architecture" manner.

Right now, I'm having trouble figuring out correct abstraction for the second step. Should this be a use case? How to call the transformation process (is it a Gateway/Adapter/Mapper)? Does clean architecture even apply to an infrastructure service?

1
  • To me, this sounds like more of an ETL pipeline use-case than a traditional API service? Depending on where your data lies, cloud providers often provide solutions for this. Eg., if you hold data on AWS, then data pipeline, can fetch, transform and load the data back to your DB of choice. aws.amazon.com/datapipeline
    – Sam
    Commented Mar 21, 2022 at 9:19

2 Answers 2

1

It can be considered as an adapter :

Interfaces / Adapters

  • Retrieve and store data from and to a number of sources (database, network devices, file system, 3rd parties, and so on.)
  • Define interfaces for the data that they need in order to apply some logic. One or more data providers will implement the interface, but the use case doesn’t know where the data is coming from
  • Implement the interfaces defined by the use case
  • There are ways to interact with the application, and typically involve a delivery mechanism (for example, REST APIs, scheduled jobs, GUI, other systems)
  • Trigger a use case and convert the result to the appropriate format for the delivery mechanism ( if your case this matches)
  • the controller for a MVC
1

Should this be a use case?

What is the Use Case as per Clean Architecture?

A use case is a description of the way an automated system is used. It specifies the input to be provided by the user, the output to be returned to the user, and the processing steps involved in producing that output. Those elements will be classes or functions or modules that have prominent positions within the architecture, and they will have names that clearly describe their function. From the use case, it is impossible to tell whether the application is delivered on the web, or on a thick client, a console, or a pure service.

Think about your step number two. Does it fit this description? From your description, it looks like your second step matches with this part

...processing steps involved in producing that output.

where it played as one step of the Use Case. So Use Case in your case would be all three steps that you mentioned. But you want to know how to deal only with the second step right? Let's think about what are these steps according to Clean Architecture. How does Use Case work if it does not have (direct) access to the database?

Let's take a look at this definition of the "interface adapters" layer.

The software in the interface adapters layer is a set of adapters that convert data from the format most convenient for the use cases and entities to the format most convenient for some external agencies such as the database or the web. No code inward of this circle should know anything at all about the database. Between the use case interactors and the database are the database gateways. These gateways are polymorphic interfaces that contain methods for every create, read, update, or delete operation that can be performed by the application on the database. Those gateways are implemented by classes in the database layer.

This sounds more closely to what you want to achieve. So what conclusion we can make from that? The implementation of the Use Cases uses gateway interfaces to interact with the database in the outer layer.

Here is one possible way of how pseudo code for your particular use case could look like:

@override
  Confirmation processAndConfirmBatchOfEventsUseCase([
    ProcessBatchOfEventsParams input = const ProcessBatchOfEventsParams(),
  ]) {
    // Step 1: load the batch of events from service B
    BatchOfEvents batchOfEvents = _eventsGateway.loadBatch();
    // Step 2.1: transform the data from the database based on the batch of events
    TransformedData transformedData = _dataGateway.transformData(batchOfEvents);
    // Step 2.2: persist the transformed data back to the database
    Confirmation confirmation = _dataGateway.persistData(transformedData);
    // Step 3: send the confirmation to service B
    return confirmation;
  }

Regarding your next question:

How to call the transformation process.

It depends on the type of "transformation" you perform. If it is a simple operation it will be implemented inside a use case, if it is something related to the database then it will be another method taken from one of the available gateways. Do you think it could be a "mapper"? Take a look at the following chapter from the book (it is very short) and think if that aligns with what you are going to implement.

DATA MAPPERS

Going back to the topic of databases, in which layer do you think ORMs like Hibernate belong? First, let’s get something straight: There is no such thing as an object-relational mapper (ORM). The reason is simple: Objects are not data structures. At least, they are not data structures from their users’ point of view. The users of an object cannot see the data, since it is all private. Those users see only the public methods of that object. So, from the user’s point of view, an object is simply a set of operations. A data structure, in contrast, is a set of public data variables that have no implied behaviour. ORMs would be better named “data mappers,” because they load data into data structures from relational database tables.

Where should such ORM systems reside? In the database layer of course.

Indeed, ORMs form another kind of Humble Object boundary between the gateway interfaces and the database.

Not the answer you're looking for? Browse other questions tagged or ask your own question.