Log Management

A log is a record of the events occurring within an organization’s systems and networks. Logs are composed of log entries. Each entry contains information related to a specific event that has occurred within a system or network.

FintechOS logs contain records related to computer security (which are generated by sources such as antivirus software, firewalls, and intrusion detection and prevention systems), as well as logs generated by operating systems (on servers, workstations, and networking equipment) and applications.

Security Logs

Irrespective of the device or application, it is imperative that log data has accurate time stamps.

In FintechOS, logs are generated at the following levels:

  1. Applications

    Applications log their activity in correlation with the business processes they support, particularly those operations that modify permissions or access rights.

    These logs generally include:

    • The business operation that was requested.
    • Whether the request was accepted or denied.
    • The time and date the operation was performed (Start and end times may be appropriate for long operations).
    • Who initiated the operation.
    • System and network resources used.
    • Any information needed for business process controls.
    • Client hardware and software characteristics.

  2. Systems

    Many components of the IT infrastructure generate logs. Examples of these components include:

    • Operating Systems.
    • Web servers.
    • Database servers.
    • Print servers.
    • File servers.
    • Authentication servers.
    • DHCP servers.
    • DNS servers.
    • E-mail server logs.

  3. Network devices

    Many components of the network infrastructure generate logs. Examples of these components include:

    • Routers.
    • Switches.
    • Wireless access points.
    • Network-based firewalls, intrusion detection and prevention systems, next-generation firewalls.

    These logs typically describe flows of information through the network, but not the individual packets contained in that flow. Information logged for a flow should include:

    • Network (IP) addresses or telephone numbers of the end points.
    • Service identifiers (port numbers) for each of the end points.
    • Whether the flow was accepted or denied.
    • Date, time, and duration of the flow.
    • Number of packets and bytes used by the flow.

  4. Microsoft Azure

    Three types of platform logs are generated which record the following types of actions:

    • Azure Active Directory Reports – detailed changes made in Azure AD and login activity.
    • Activity logs – record operations performed on an Azure resource – such as creating a VM.
    • Resource logs – capture operations performed within an Azure resource, such as querying a database or writing to a storage bucket.

    Both Activity Logs and Resource Logs use the Log Analytics workspace to store and access these logs. Also, Log Analytics data is used/analyzed by SIEM to generate alerts/incidents based on defined analytics rules.

Operating system logs and application logs

FintechOS operating system logs and application logs typically hold a variety of information, including computer security-related data.

  1. Operating System logs

    Operating systems (OS) for servers, workstations, and networking devices (e.g., routers, switches) usually log a variety of information related to security. FintechOS is logging the most common types of security-related OS data, as follows:

    • System Events - System events are operational actions performed by OS components, such as shutting down the system or starting a service. Each event is timestamped and other supporting information may include event, status, and error codes; service name; and user or system account associated with an event.
    • Audit Records - Audit records hold security event information such as successful and failed authentication attempts, file accesses, security policy changes, account changes (e.g. account creation and deletion, account privilege assignment), and use of privileges.

  2. Applications/platform logs

    Logging level can take one of the following values: Verbose, Debug, Information, Warning, Error, Fatal. A minimum level of logging for each logging sink can be set.

    FintechOS is actively using two logging sinks which can be enabled together or separately: file logging and Application Insights logging. We also have SEQ logging already added to the platform and can add more Serilog sinks in future FintechOS releases (based on request).

    Type of logs that we expect to meet:

    • Platform

      • Exceptions in the execution of the FTOS platform logic.
      • Warning/Info messages added by the platform developers to have a better view on what happened in a certain workflow.
      • Errors where the exceptions are handled, and information is provided about the cause on why a specific logic failed.
      • Digital developers can decide to add their own logging with the desired log level in the automation scripts written for client specific implementations.
      • Items related to observability on the application performance/statistics are not handled at the platform level and should be extracted using, for example, AppInsights out-of-the-box capabilities.
      • Trace.log files configured by default with periodical switching to new files of logs. Trace log files are grabbed by external sources. (SIEM, log server).

    • Database

      There are four types of logging events which can be stored in the DB:

      • FintechOS Platform API Logging

        • Optional setting – based on customer requirements

        • Can be filtered by source type (i.e. OpenApi, Data Service) and method name

        • Logs are stored in EbsLogs.ApiLog

        • Attention needs to be paid to filtering and cleaning the logging table, as the high number of requests can increase the DB size considerably

        • Starting with 22.1.0 this can also be sent directly to the same logging sink as platform logging (no filtering on security, it is all or nothing)

      • CRUD Operations Logging

        • Optional setting – based on customer requirements

        • It can only be turned on and off without any other granularity settings – logging the CRUD events on all entities

        • Logs stored in EbsLogs.UniversalLog

        • Starting with 22.1.0 this can also be sent directly to the same logging sink as platform logging (no filtering on security, it is all or nothing)

      • Authentication Events logging

        • Logs stored in EbsMetadata.SystemAudit

        • Optional setting starting with 22.1.0, always enabled on older versions

        • Starting with 22.1.0 this can also be sent directly to the same logging sink as platform logging (no filtering on security, it is all or nothing)

      • Data Audit

        Optional setting which by default is turned off

      NOTE  
      FintechOS is logging all CRUD and API operations executed in the platform, by default, in a separate database schema named EbsLogs. Database administrators can restrict read access for this schema and grant insert rights only for the SQL login used by the FintechOS platform. Granular rights can be set up.

  3. DCS/DCI (Digital Cloud Services and Digital Cloud Infrastructure)

    Logs generated by integration with FTOS technology partners hold data needed for solution debugging, sometimes files sent to the partners and, in some cases, initial requests and responses. Logs are stored on virtual machines where solutions reside. Restricted access to these virtual machines is enforced.

Anonymize Logs

The log entries that contain sensitive information, such as usernames, passwords, credit card numbers, personal identification numbers, can be anonymized in the platform, thus assuring compliance with GDPR standards. You can anonymize the properties when a JSON object or array is logged using the log SDK method. The sensitive information is replaced with N/A in anonymized logs.

Enable or Disable Log Anonymization

Log anonymization is enabled by default on environments, and you can check in the Configuration Manager.

  1. Go to App Features in Configuration Manager on your environment.

  2. Add the value 1 to the feature-log-anonymization-enabled to enable anonymization. Add 0 to disable it.

Define Logs to be Anonymized

Once the log anonymization feature was enabled in Configuration Manager, you must now define which logs to be anonymized.

  1. In FintechOS Studio, go to System Parameters.

  2. Double click on sys-log-anonymization-settings to open this system parameter.

  3. In the Paramenter Value field, add the logs to be anonymized, eg.: username, password, credit card number. Make sure to separate parameters with a comma.

  4. Click Save and close when done.

In this example, we are anonymizing multiple customer emails and first names using a script that contains an array with 2 objects. In this scenario, Log anonymization is already enabled.

  1. Edit the sys-log-anonymization-settings and add email, firstName in the Parameter Value field.

  2. In Studio, create a new server script with an array with 2 objects:

    Copy 
    example1 = new Object();
    example.email = "example1@mail.com";
    example.firstName = "Name";

    example2 = new Object();
    example.email = "example2@mail.com";
    example.firstName = "First";

    log([example1, example2]);

  3. Create an endpoint for the script.

  4. Open the console from the browser and execute the script: ebs.callActionByName("example_script").

  5. Check the logs in App Insights and notice that the elements are anonymized.

You can anonymize multiple customer emails and first names using a script that contains 2 elements in Json:

  1. Edit the sys-log-anonymization-settings and add email, firstName in the Parameter Value field.

  2. In Studio, create a new server script that has 2 elements in Json:

    Copy 
    log([{"email":"example1@mail.com", "firstName" = "Name"}, {"email":"example2@mail.com", "firstName" = "First"}]))

  3. Create an endpoint for the script.

  4. Open the console from the browser and execute the script: ebs.callActionByName("example_script",(res)=>console.log(res)).

  5. Check the logs in App Insights and notice that the elements are anonymized.

Filter SDK logs

You can set a different level for SDK logs so that you can view certain logs on development environments. In this way, you can differentiate between logs at platform level and logs from server automation scripts. You can filter logs by automation script, digital asset, machine name, user ID, correlation ID.

Enable Filters for Logs

Follow the steps below to enable or disable the filters for logs:

  1. In Configuration Manager, go to Studio > App Settings.

  2. Add sdkLogLevel: Info; to the azure-appinsights-logging entry. You can change the value from Info to any other log level that you need, such as Debug or Error.

Define Filters

You need to add some settings for the logs. Follow the steps below to define which property you want to filter out:

  1. In Configuration Manager, go to Studio > App Settings.

  2. Edit sdk-log-filters to add {"automationscript":["automation_script_name"], "digital-asset":["digital_asset_name"]}.

  3. After the changes are done, wait for this change to take effect.

NOTE  
Automation scripts part of digital assets are displayed if the digital asset is specified in sdk-log-filters.

In the following example we're filtering by two automation scripts, test_script_1 and test_script_2, and one digital asset, named test_da.

  1. In Configuration Manager, go to Studio > App Settings.

  2. Edit sdk-log-filters to add sdk-log-filters: {"automationscript":["test_script_1", "test_script_2"], "digital-asset":["test_da"]}.

  3. Open Studio and run the following commands from the Console, one-by-one:

    Copy 
    ebs.callActionByName('test_script_1')
    ebs.callActionByName('test_script_2')
    ebs.callActionByName('test_da')

  4. Check the logs from App Insights. The scripts and the digital asset are displayed in the logs.