Success Metrics of system that is time-based

Question

We have a internal system (lets call it system B) that responds "No" as a response to a request from external-facing system (lets call system A) based on elapse of 48 hours from earlier request if a token was not attached with the earlier request.
. More Context: We have an external system which receives requests from users. A user account can contain multiple profiles. Profiles are attached to requests being sent to the external system to be executed. If the user requires actions to be executed in prioritized manner, they need to 'register' their profile after which they get token. The external system communicates with internal system to decide if request should be executed in prioritized manner. If this token is provided as part of first request, the internal system keeps track of it and respond "YES" to external system, thereby allowing the external system to execute subsequent steps immediately. If the token was not provided, the internal system responds "NO" to external system till 48 hour window has elapsed.

The response of "YES" or "NO" is primarily dependent on 48 hour time period. It does not check for any process completion behind the scene ( eventually consistent ) or number of requests in a certain period (throttling) or such any additional criteria.

The internal system was not built as throttling mechanism but with goal to prevent subsequent steps to be executed by external system for 48 hours if prior steps were executed with request not including specific parameter. The user may also choose not send the request to execute the subsequent steps after 48 hours. In such case, our system would have "prevented" the execution.

What can be considered a good metric to measure the effectiveness of this system since it does not actually prevent but rather delay the execution of subsequent steps (unless the user does not make request after 48 hours)?

Also, is there a name such systems are called where only "time" passed is the only factor that determines the response provided?

Answer 1

So if I got this right, the 48 hours delay shall prevent unregistered users from making too many requests in a short time, and the goal is to encourage them to register their profile (which may have a price tag?). So this is something like a sales system, or a try-and-buy system.

I doubt there is special technical name for such systems, where only passed time is involved. Still you can lookup certain keywords which describe certain aspects of it, like

• temporal
• artificial delay
• throttling

I think the effectiveness of this approach can only be assessed by comparing it to modes with a different delay, or no delay at all. You can measure the ratio

#requests of registered users / #total requests.

over a certain period and compare the value for different modes.

This will require some time, of course. With a 48 hours delay, the period should be at least a few weeks, maybe a few months. Then change the delay, run the system for the same time again and see what happens. When you observe a change in the request ratio, you will have to make sure it wasn't caused by different external factors, which makes a useful assessment quite hard.

Hence, I think your should consider whether it would not be more effective to simply ask users how they feel about the delay (for example, in some forum).