External Request Processing
To improve the platform's overall performance through parallel processing, Distribution Management sends the requests generated for processing outside of the platform, instead of being processed within the platform, through JobServer. Thus, Distribution Management processes the requests in a parallel asynchronous way, through Async Engine.
Configuring the Async Engine
The configuration for Distribution Management-Async Engine is as follows:
-
The GenerateInvoiceDetailsFlowName, GenerateInvoicesFlowName and PayInvoicesFlowName Distribution Management system parameters must be used when a request for the specific flow is triggered.
-
The Async Engine web API client library has to be loaded on the platform using the Async Engine swagger file.
-
The Vault settings for the Studio, Portal, and JobServer components must contain a key named AsyncEngineURL_DistributionManagement with the value of the Async Engine's URL.
Read the following section for more detailed information:
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
The asynchronous processing configuration is kept in the Vault.
Configure the external request processor app service to use the
async-engine
folder in Vault.During implementation, create a web API client library to enable Distribution Management to call these configurations, making sure the library uses the latest swagger file found at <server_address>/services/specifications/swagger-ui
.
Set the
Is Used for service pipes
attribute to True
.Send the AsyncEngineURL
parameter, stored in Vault, with the URL of the external request processor when importing the web api client library in the Distribution Management context.
When the external request processor calls the Portal, the portal user client must be part of the Role defined in rbac.config
vault parameter.
Make sure you create a user for running the OpenAPI pipelines. The user must be part of the
Registered users
group.While calling the external request processor using the startProcess
operation, send the following parameters:
-
flowName
- mandatory; -
correlationId
- optional, automatically generated if missing; -
payload
- mandatory.
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
You must configure a series of parameters so that your external request processor can receive and forward requests.
Add the AsyncEngineURL_DistributionManagement
Vault parameter to Distribution Management Portal/ Studio/ JobServer. This is used when importing the web API client library.
"AsyncEngineURL_DistributionManagement": "https://app-<async-engine-instance_Distribution>-<project_name>-<env>.azurewebsites.net"
The following parameters are used in the external request processor's Vault parameters:
azure.storage.config
{
"accountName": "<storage account name>",
"accountKey": "<storage acount key>",
"suffix": "core.windows.net"
}
{
"use-resource-role-mappings": "false",
"autodetect-bearer-only": "true",
"ssl-required": "external",
"auth-server-url": "<auth server url>",
"principal-attribute": "preferred_username",
"resource": "<client used for authentication>",
"credentials": {
"secret": "<client secret>"
},
"realm": "<realm>"
}
[
{
"flowName": "Flow_TPM_GenerateInvoiceDetails_AsyncEngine",
"queueMappings": [
{
"sourceQueueName": "ftos-tpm-find-agrements",
"endpointName": "FTOS_TPM_FindMatchingAgreements_AsyncEngine",
"targetQueueName": "ftos-tpm-build-invoice-details",
"initial": true
},
{
"sourceQueueName": "ftos-tpm-build-invoice-details",
"endpointName": "FTOS_TPM_BuildInvoiceDetails_AsyncEngine",
"targetQueueName": "ftos-tpm-calculate-pricings"
},
{
"sourceQueueName": "ftos-tpm-calculate-pricings",
"endpointName": "FTOS_TPM_CalculatePricings_AsyncEngine",
"targetQueueName": "ftos-tpm-generate-invoice-details"
},
{
"sourceQueueName": "ftos-tpm-generate-invoice-details",
"endpointName": "FTOS_TPM_GenerateInvoiceDetails_AsyncEngine"
}
]
},
{
"flowName": "Flow_TPM_GenerateInvoices_AsyncEngine",
"queueMappings": [
{
"sourceQueueName": "ftos-tpm-generate-invoice",
"endpointName": "FTOS_TPM_GenerateInvoice_AsyncEngine",
"initial": true
}
]
},
{
"flowName": "Flow_TPM_PayInvoices_AsyncEngine",
"queueMappings": [
{
"sourceQueueName": "ftos-tpm-pay-invoice",
"endpointName": "FTOS_TPM_PayInvoice_AsyncEngine",
"initial": true,
"maxThreads": 10
}
]
}
]
[
{
"name": "ftos-tpm-find-agrements",
"maxMessages": 10,
"timeToLive": 1
},
{
"name": "ftos-tpm-build-invoice-details",
"maxMessages": 10,
"timeToLive": 1
},
{
"name": "ftos-tpm-calculate-pricings",
"maxMessages": 10,
"timeToLive": 1
},
{
"name": "ftos-tpm-generate-invoice-details",
"maxMessages": 10,
"timeToLive": 1
},
{
"name": "ftos-tpm-generate-invoice",
"maxMessages": 10,
"timeToLive": 1
},
{
"name": "ftos-tpm-pay-invoice",
"maxMessages": 10,
"timeToLive": 1
}
]
{
"apiMappings": [
{
"url": "/api",
"roles": [
"<user profile created and used for service account roles> "
]
}
],
"applicationMappings": [
{
"url": "/actuator",
"roles": [
"<user profile created and used for service account roles>"
]
}
],
"swaggerMappings": [
{
"url": "/specifications",
"roles": [
"<user profile created and used for service account roles>"
]
}
]
}
{
"default": {
"count": 2,
"delay": 2
},
"additional": [
{
"httpStatus": 502,
"count": 2,
"delay": 1
}
]
}
{
"requestTimeout": "180000",
"connectionTimeout": "120000",
"socketTimeout": "230000",
"totalConnections": "1000000",
"routeConnections": "500000",
"connectionTimeToLive": "1000"
}
Managing Unprocessed Requests
If a request is not processed by the Async Engine regardless of the reason, you can view a message in the dedicated DLQ dashboard in your FintechOS Portal. For accessing the dashboard, perform the needed configurations during the implementation. Send the request for reprocessing by the Async Engine from the same dashboard, simply clicking a button next to that message.
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
The DLQ Messages (dead letter queue) dashboard stores messages about the requests that the Async Engine could not process for any reason. To view the messages and to send the requests back to processing, make sure you perform the following configurations:
-
In FintechOS Studio, select the Admin > System Parameters menu.
-
On the newly displayed System Parameters list page, find and double-click the DLQInterfaceQueuesToBeShown system parameter to open it for editing. This parameter represents the list of queues from which the messages are displayed in the DLQ Messages dashboard in FintechOS Portal.
-
On the Edit System Parameter page, edit the parameter value to display the messages from the desired flows. Enter the names of the flows one by one, separated by commas and with no spaces between them. For example:
ftos-cb-loan-approval,ftos-cb-repayment-notification,ftos-cb-bankaccount-transaction,ftos-tpm-find-agrements,ftos-tpm-build-invoice-details,ftos-tpm-calculate-pricings,ftos-tpm-generate-invoice-details,ftos-tpm-generate-invoice,ftos-tpm-pay-invoice,ftos-cb-customer,ftos-cb-customerlimit-create,ftos-cb-customerlimit-approve,ftos-cb-settlementaccount-create,ftos-cb-settlementaccount-approve,ftos-cb-topup-create,ftos-cb-topup-approve,ftos-cb-loancontract-create,ftos-cb-loancontract-approve
The flows must be specified in Vault.
NOTE
The order of messages displayed in the dashboard depends on the order of the flows you mention here, so make sure you enter the most desired flow name first and then add them according to your priorities. -
Click the Save and Close button.
-
Select the Security > System Users menu.
-
On the newly displayed System Users list page, find and double-click each user who should have access to the DLQ Messages dashboard.
-
Edit the system users, assigning them the Async Engine Admin security role.
-
Click the Save and Close button.
Now you are ready to view the DLQ Messages dashboard and send any requests for reprocessing.
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
Follow these steps to view the requests that the Async Engine could not process:
-
In FintechOS Portal's Home page, select the DLQ Messages dashboard.
NOTE
Your system user must have the Async Engine Admin security role to access the DLQ Messages dashboard.
![Closed](../../Skins/Default/Stylesheets/Images/transparent.gif)
You can send for processing each of the failed requests from the DLQ Messages dashboard one by one, as follows:
-
Click the Reprocess button in line with the desired request.
-
The request is sent back to the Async Engine for processing and an informative message is displayed on your screen.