Create Asynchronous Processing Flows

The FintechOS Platform provides an asynchronous processing engine that allows you to address scenarios that require time-consuming logic or a high volume of requests through asynchronous processing. Some use cases include:

• Large scale data processing
• Time consuming operations
• Rate limiting and throttling
• Resilient processing with redelivery and Dead Letter Queues (DLQ)
• Scalable workload distribution
• Service decoupling

The Async Engine uses message queues that collect incoming inputs for each step (on-demand automation script) in a processing flow. Each step can operate independently as it consumes inputs from its queue, allowing for parallel processing and efficient resource utilization. Additionally, the message queues provide a buffer that can handle bursts of incoming requests, ensuring smooth processing and improved resilience in case of spikes in load.

1 Create the Asynchronous Steps

1. Create On-demand Server Automation Scripts for each step in your asynchronous flow.
2. Customize the Input Parameters of each script so that they match the following structure:
   Copy

   {
       "correlationId": "a66e0eef-fa3d-421c-83f6-e7e7c8caf4a5",
       "payload": {
           "your-data": "here"
       }
       "additional-properties: optional"
       "...."    
   }

    
   When an asynchronous flow is called, the Async Engine receives the payload, generates a correlation-id for tracing, and passes them to the first step of the flow.
3. Customize the Output Structure for each step in the flow except the last one to also match the above input parameters (as their outputs become inputs for the next step in the flow).
   IMPORTANT!  
   Make sure to propagate the correlation-id in the output, otherwise the Async Engine will lose track of the requests.

2 Configure the Message Queues and Flow in the Configuration Manager

See the Administration Guide for instructions on how to configure message queues for each step and how to set up the flow. Follow the instructions to configure queues' sizes and polling, flow name, sequence of steps, processing threads, etc.

Call an Asynchronous Flow via API

The endpoint for calling an asynchronous flow is available at the following address:

<platform_address>/async/api/start?flowName=<flow_name>

The payload (which will be propagated by the Async Engine to the first step of the flow) must be provided in JSON format:

{
   "your_data": "here"
}

A Swagger UI interface with all the asynchronous flow endpoints is also available at:

<platform_address>/async/specifications/swagger-ui/index.html

Call an Asynchronous Flow via Service Pipes

To set an asynchronous flow as the destination endpoint of a Service Pipes route, add the following properties to the route:

.setProperty("flowName", constant("your-flow-name"))
.to("direct:callAsyncEngine");

For more information about working with Service Pipes, see Call External Services via Service Pipes and the Service Pipes tutorial.