External Request Processing

To improve the platform's overall performance through parallel processing, Distribution Management sends the requests generated for processing outside of the platform, instead of being processed within the platform, through JobServer. Thus, Distribution Management processes the requests in a parallel asynchronous way, through Async Engine.

Configuring the Async Engine

The configuration for Distribution Management-Async Engine is as follows:

The GenerateInvoiceDetailsFlowName, GenerateInvoicesFlowName and PayInvoicesFlowName Distribution Management system parameters must be used when a request for the specific flow is triggered.
The Async Engine web API client library has to be loaded on the platform using the Async Engine swagger file.
The Vault settings for the Studio, Portal, and JobServer components must contain a key named AsyncEngineURL_DistributionManagement with the value of the Async Engine's URL.

Read the following section for more detailed information:

Calling the External Request Processor

The asynchronous processing configuration is kept in the Vault.

IMPORTANT!

Configure the external request processor app service to use the async-engine folder in Vault.

During implementation, create a web API client library to enable Distribution Management to call these configurations, making sure the library uses the latest swagger file found at <server_address>/services/specifications/swagger-ui.

IMPORTANT!

Set the Is Used for service pipes attribute to True.

Send the AsyncEngineURL parameter, stored in Vault, with the URL of the external request processor when importing the web api client library in the Distribution Management context.

When the external request processor calls the Portal, the portal user client must be part of the Role defined in rbac.config vault parameter.

IMPORTANT!

Make sure you create a user for running the OpenAPI pipelines. The user must be part of the Registered users group.

While calling the external request processor using the startProcess operation, send the following parameters:

flowName - mandatory;
correlationId - optional, automatically generated if missing;
payload - mandatory.

Configuring the Vault Parameters

You must configure a series of parameters so that your external request processor can receive and forward requests.

Add the AsyncEngineURL_DistributionManagement Vault parameter to Distribution Management Portal/ Studio/ JobServer. This is used when importing the web API client library.

Copy

AsyncEngineURL_DistributionManagement

"AsyncEngineURL_DistributionManagement": "https://app-<async-engine-instance_Distribution>-<project_name>-<env>.azurewebsites.net"

The following parameters are used in the external request processor's Vault parameters:

Copy

threadpool.config

{
                "poolSize": 10,
                "maxPoolSize": "1000",
                "maxQueueSize": "100"
            }

Copy

azure.storage.config

{
                "accountName": "<storage account name>",
                "accountKey": "<storage acount key>",
                "suffix": "core.windows.net"
            }

Copy

openid.config

{
                "use-resource-role-mappings":  "false",
                "autodetect-bearer-only":  "true",
                "ssl-required":  "external",
                "auth-server-url":  "<auth server url>",
                "principal-attribute":  "preferred_username",
                "resource":  "<client used for authentication>",
                "credentials":  {
                "secret":  "<client secret>"
                },
                "realm":  "<realm>"
            }

Copy

openapi.url

https://<open api url>.azurewebsites.net/ftosapi/automation-processors/actions/

Copy

openapi.username

<username>

Copy

openapi.password

<password>

Copy

flow.config

[
    {
        "flowName": "Flow_TPM_GenerateInvoiceDetails_AsyncEngine",
        "queueMappings": [
            {
                "sourceQueueName": "ftos-tpm-find-agrements",
                "endpointName": "FTOS_TPM_FindMatchingAgreements_AsyncEngine",
                "targetQueueName": "ftos-tpm-build-invoice-details",
                "initial": true
            },
            {
                "sourceQueueName": "ftos-tpm-build-invoice-details",
                "endpointName": "FTOS_TPM_BuildInvoiceDetails_AsyncEngine",
                "targetQueueName": "ftos-tpm-calculate-pricings"
            },
            {
                "sourceQueueName": "ftos-tpm-calculate-pricings",
                "endpointName": "FTOS_TPM_CalculatePricings_AsyncEngine",
                "targetQueueName": "ftos-tpm-generate-invoice-details"
            },
            {
                "sourceQueueName": "ftos-tpm-generate-invoice-details",
                "endpointName": "FTOS_TPM_GenerateInvoiceDetails_AsyncEngine"
            }
        ]
    },
    {
        "flowName": "Flow_TPM_GenerateInvoices_AsyncEngine",
        "queueMappings": [
            {
                "sourceQueueName": "ftos-tpm-generate-invoice",
                "endpointName": "FTOS_TPM_GenerateInvoice_AsyncEngine",
                "initial": true
            }
        ]
    },
    {
        "flowName": "Flow_TPM_PayInvoices_AsyncEngine",
        "queueMappings": [
            {
                "sourceQueueName": "ftos-tpm-pay-invoice",
                "endpointName": "FTOS_TPM_PayInvoice_AsyncEngine",
                "initial": true,
                "maxThreads": 10
            }
        ]
    }
]

Copy

queue.config

[
                {
                "name": "ftos-tpm-find-agrements",
                "maxMessages": 10,
                "timeToLive": 1
                },
                {
                "name": "ftos-tpm-build-invoice-details",
                "maxMessages": 10,
                "timeToLive": 1
                },
                {
                "name": "ftos-tpm-calculate-pricings",
                "maxMessages": 10,
                "timeToLive": 1
                },
                {
                "name": "ftos-tpm-generate-invoice-details",
                "maxMessages": 10,
                "timeToLive": 1
                },
                {
                "name": "ftos-tpm-generate-invoice",
                "maxMessages": 10,
                "timeToLive": 1
                },
                {
                "name": "ftos-tpm-pay-invoice",
                "maxMessages": 10,
                "timeToLive": 1
                }
            ]

Copy

rbac.config

{
                "apiMappings": [
                {
                "url": "/api",
                "roles": [
                "<user profile created and used for service account roles> "
                ]
                }
                ],
                "applicationMappings": [
                {
                "url": "/actuator",
                "roles": [
                "<user profile created and used for service account roles>"
                ]
                }
                ],
                "swaggerMappings": [
                {
                "url": "/specifications",
                "roles": [
                "<user profile created and used for service account roles>"
                ]
                }
                ]
            }

Copy

redelivery.config

{
    "default": {
        "count": 2,
        "delay": 2
    },
    "additional": [
        {
            "httpStatus": 502,
            "count": 2,
            "delay": 1
        }
    ]
}

Copy

table.config

{
                "statusTable": "statusTableAsyncEngine",
                "dlqTable": "dlqTableAsyncEngine"
            }

Copy

http.connection.config

{
    "requestTimeout": "180000",
    "connectionTimeout": "120000",
    "socketTimeout": "230000",
    "totalConnections": "1000000",
    "routeConnections": "500000",
    "connectionTimeToLive": "1000"
}

Managing Unprocessed Requests

If a request is not processed by the Async Engine regardless of the reason, you can view a message in the dedicated DLQ dashboard in your FintechOS Portal. For accessing the dashboard, perform the needed configurations during the implementation. Send the request for reprocessing by the Async Engine from the same dashboard, simply clicking a button next to that message.

Configuring the Messages and Access Rights for the DLQ Dashboard

The DLQ Messages (dead letter queue) dashboard stores messages about the requests that the Async Engine could not process for any reason. To view the messages and to send the requests back to processing, make sure you perform the following configurations:

In FintechOS Studio, select the Admin > System Parameters menu.
On the newly displayed System Parameters list page, find and double-click the DLQInterfaceQueuesToBeShown system parameter to open it for editing. This parameter represents the list of queues from which the messages are displayed in the DLQ Messages dashboard in FintechOS Portal.
On the Edit System Parameter page, edit the parameter value to display the messages from the desired flows. Enter the names of the flows one by one, separated by commas and with no spaces between them. For example:

ftos-cb-loan-approval,ftos-cb-repayment-notification,ftos-cb-bankaccount-transaction,ftos-tpm-find-agrements,ftos-tpm-build-invoice-details,ftos-tpm-calculate-pricings,ftos-tpm-generate-invoice-details,ftos-tpm-generate-invoice,ftos-tpm-pay-invoice,ftos-cb-customer,ftos-cb-customerlimit-create,ftos-cb-customerlimit-approve,ftos-cb-settlementaccount-create,ftos-cb-settlementaccount-approve,ftos-cb-topup-create,ftos-cb-topup-approve,ftos-cb-loancontract-create,ftos-cb-loancontract-approve

The flows must be specified in Vault.

NOTE
The order of messages displayed in the dashboard depends on the order of the flows you mention here, so make sure you enter the most desired flow name first and then add them according to your priorities.
Click the Save and Close button.
Select the Security > System Users menu.
On the newly displayed System Users list page, find and double-click each user who should have access to the DLQ Messages dashboard.
Edit the system users, assigning them the Async Engine Admin security role.
Click the Save and Close button.

Now you are ready to view the DLQ Messages dashboard and send any requests for reprocessing.