BT

Facilitating the Spread of Knowledge and Innovation in Professional Software Development

Write for InfoQ

Topics

Choose your language

InfoQ Homepage Articles Monitoring Critical User Journeys in Azure

Monitoring Critical User Journeys in Azure

Key Takeaways

  • ​​A critical user journey (CUJ) is an approach that maps out the key interactions between users and a product. CUJs are a great way to understand the effectiveness of application flows and identify bottlenecks.
  • Tools like Prometheus and Grafana provide a standardized way to collect and visualize these metrics and monitor CUJs.
  • In the Flowe technology stack, fulfilling a CUJ often involves a user's request being handled by a mix of PaaS and Serverless technology.
  • In the current economic climate, pricing is a critical factor for any monitoring solution. The decision of build vs buy must be analyzed closely, even when running initial experiments.
  • Infrastructure as Code (IaC) frameworks like Azure Bicep help provision resources and organize CUJ metric collection as resources are deployed.

 

The Need for Application Monitoring

I work as an SRE  for Flowe , an Italian challenger digital bank where the software, rather than physical bank branches, is the main product. Everything is in the mobile app, and professionally speaking, ensuring continued operation of this service is my main concern. Unlike a traditional bank, customers rely on this mobile app as a key point of interaction with our services.

Flowe established a monitoring team to ensure proper integration between the bank platform and its numerous third-party services (core banking, card issuer, etc.). This team is available 24/7, and when something goes wrong on the third-party systems (i.e., callbacks are not working), they open a ticket on third-party systems.

Although they do an excellent job, the monitoring team doesn’t have deep knowledge about the system architecture, business logic, or even all the components of the bank platform. Their scope is limited to third parties.

This means that if a third party is not responding, they are quick to open tickets and monitor them until the incident is closed. However, the monitoring team lacks the development skills and knowledge to catch bugs, improve availability/deploy systems, measure performances, monitor dead letter queues (DLQs), etc. For this reason, at least initially, when Flowe launched, senior developers were in charge of these tasks.

However, after the first year of life, we realized developers were too busy building new features, etc., and they didn’t have time for day-to-day platform observation. So we ended up creating our SRE team with the primary goal of making sure the banking platform ran smoothly.

SRE Team Duties

What Flowe needed from an SRE team changed over time. As explained in the previous paragraph, the first necessity was covering what developers and the monitoring team couldn’t do: monitor exceptions and API response status code, find bugs, watch the Azure Service Bus  DLQs, measure performances, adopt infrastructure as code (IaC), improve deployment systems and ensure high availability.

The transition of responsibilities toward the SRE team had been slow but efficient, and over time the SRE team has grown, expanding the skill set and covering more and more aspects of Flowe’s daily routine. We started to assist the "caring team" (customer service) and put what is called Critical User Journeys (CUJ) in place.

CUJs are a great way to understand the effectiveness of application flows and identify bottlenecks. One example of a CUJ in our case is the "card top up process", an asynchronous flow that involves different actors and services owned by many organizations. This CUJ gives us the context of the transaction and enables us to understand where users encounter issues and what the bottlenecks are. Solving issues rapidly is extremely important. Most users that get stuck in some process don’t chat with the caring team but simply leave a low app rating.

SRE Team Tools

Azure Application Insights  is an amazing APM tool, and we use it intensively for diagnostic purposes within our native iOS/Android SDK. However, Although we had decided to use Application Insights, the integration into the core Azure Monitor suite lacked some features that were critical to our usage.

For example, you can only send alerts via emails and SMS, and there is no native integration with other services such as PagerDuty, Slack, or anything else. Moreover, creating a custom dashboard using an Azure Workbook is only flexible and scalable in some environments because of their limited flexibility.

For all of the mentioned reasons, we, as the SRE team, decided to put in place two well-known open-source products to help us with monitoring and alerting tasks: Prometheus  and Grafana.

Prometheus is an open-source project hosted by the Cloud Native Computing Foundation (CNCF). Prometheus uses the pull model to collect metrics on configured targets and save data in its time series database. Prometheus has its data model - a plain text format - and as long as you can convert your data to this format, you can use Prometheus to collect and store data.

Grafana is another CNCF project that allows you to create dashboards to visualize data collected from hundreds of different places (data sources). Usually, Grafana is used with Prometheus since it can understand and display its data format. Still, many other data sources, such as Azure Monitor, Azure DevOps, DataDog, and GitHub, can be used. Grafana handles alerts, integrating with many services such as Teams or Slack.

Monitoring Solution

As we adopted Prometheus and Grafana, we needed to ensure that we did this in a cost-effective manner - the two key metrics were the size of our team and the amount of data processed/stored. So we did some proof of concept using an Azure Virtual Machine (VM)  with a Docker Compose file to start both Prometheus and Grafana containers and practice with them. The solution was cheap, but managing an entire VM to run two containers wastes time.

For this reason, we looked at the managed version offered by Azure:

  • Grafana Managed Instance costs 6€/month/active user - a bit more expensive than the 30€ for the monthly VM as the SRE team consists of six people.
  • Prometheus Managed pricing model  is not straightforward, especially if you are starting from scratch and don’t have any available metrics to rely on. Moreover, you have to pay for notifications (push, emails, webhooks, etc.).

After some experiments and research on all possible solutions, we saw that a combination of SaaS and serverless Azure solutions seemed the best option for our use case. Let’s see which ones, how, and why they’ve been integrated with each other.

Azure Managed Grafana

Azure Managed Grafana is a fully managed service for monitoring solutions. So, actually, we found what we were looking for: automatic software upgrades, SLA guarantees, availability zones, Single Sign-On with AAD  and integration with Azure Monitor (via Managed Identity ) ready out of the box.

Provisioning Grafana using Bicep

Bicep provides a simple way of provisioning our cloud resources through Infrastructure as Code principles. This allows repeatable deployments as well as a way to record the resource’s place along the CUJ. The Bicep definition of a Grafana Managed instance is simple.

resource grafana 'Microsoft.Dashboard/grafana@2022-08-01' = {
  name: name
  location: resourceGroup().location
  sku: {
    name: ‘Standard’
  }
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    apiKey: 'Disabled'
    autoGeneratedDomainNameLabelScope: 'TenantReuse'
    deterministicOutboundIP: 'Enabled'
    publicNetworkAccess: 'Enabled'
    zoneRedundancy: 'Enabled'
  }
}

In this configuration, it is worth highlighting in detail: the `deterministicOutbountIP` is set to `Enabled`. This allows us to have two static outbound IPs that we will use later to isolate the Prometheus instance from Grafana.

Finally, we needed to grant the Grafana Admin Role to our AAD group and the Monitoring Reader Role to the Grafana Managed Identity to get access to Azure Monitor logs.

@description('This is the built-in Grafana Admin role. See https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles')
resource grafanaAdminRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: '22926164-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}

@description('This is the built-in Monitoring Reader role. See https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#monitoring-reader')
resource monitoringReaderRole 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  scope: subscription()
  name: '43d0d8ad-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}

@description('This is the SRE AD group.')
resource sreGroup 'Microsoft.Authorization/roleDefinitions@2022-04-01' existing = {
  name: 'aed71f3f-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
}

resource adminRoleSreGroupAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(subscription().id, sreGroup.id, grafanaAdminRole.id)
  properties: {
    roleDefinitionId: grafanaAdminRole.id
    principalId: sreGroup.name
    principalType: 'Group'
  }

  dependsOn: [
    grafana
  ]
}

resource monitoringRoleSubscription 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(subscription().id, grafana.name, monitoringReaderRole.id)
  properties: {
    roleDefinitionId: monitoringReaderRole.id
    principalId: grafana.outputs.principalId
    principalType: 'ServicePrincipal'
  }

  dependsOn: [
    grafana
  ]
}

Azure Container Apps (ACA) for Prometheus

Azure Container Apps is a serverless solution to run containers on Azure. Under the hood, it runs containers on top of Kubernetes; it is completely managed by Azure and the Kubernetes API (and any associated complexity) is never exposed to the users. Plus, it can scale from 0 to 30 replicas (0 is for free!), and you can attach volumes via a Files Share mounted on a Storage Account (a necessary option to run Prometheus as it is a time series database). We chose this service for its simplicity and flexibility, within the cost of around 20€/month when turned on 24/7.

Provisioning Prometheus on ACA using Bicep

We start to define a Storage Account that will be used to mount a volume on the container, initially allowing connections for outside.

resource storageAccount 'Microsoft.Storage/storageAccounts@2022-09-01' = {
  name: storageAccountName
  location: resourceGroup().location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_ZRS'
  }
  properties: {
    supportsHttpsTrafficOnly: true
    minimumTlsVersion: 'TLS1_2'
    networkAcls: {
      defaultAction: 'Allow'
    }
    largeFileSharesState: 'Enabled'
  }
}

And then the related File Share using the SMB protocol.

resource fileServices 'Microsoft.Storage/storageAccounts/fileServices@2022-09-01' = {
  name: 'default'
  parent: storageAccount
}

resource fileShare 'Microsoft.Storage/storageAccounts/fileServices/shares@2022-09-01' = {
  name: name
  parent: fileServices
  properties: {
    accessTier: 'TransactionOptimized'
    shareQuota: 2
    enabledProtocols: 'SMB'
  }
}

Selecting the right `accessTier` here is important: we chose the `Hot` option, but it was an expensive choice with no performance gain. `TransactionOptimized` is much cheaper and more suited to Prometheus’s work. 

This File Share resource will be mounted on the container, so it shall arrange the local environment for Prometheus by provisioning two folders: `data` and `config`. In my case, the latter must contain the Prometheus configuration file named `prometheus.yml`. 

The former is used to store the time series database. In our Bicep file, we launch a shell script through a Bicep Deployment Script  to ensure these prerequisites exist at each pipeline run. And finally, the container app with the accessory resources - environment and log analytics workspace.

resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2022-10-01' = {
  name: containerAppLogAnalyticsName
  location: resourceGroup().location
  properties: {
    sku: {
      name: 'PerGB2018'
    }
  }
}

var vnetConfiguration = {
  internal: false
  infrastructureSubnetId: subnetId
}

resource containerAppEnv 'Microsoft.App/managedEnvironments@2022-10-01' = {
  name: containerAppEnvName
  location: resourceGroup().location
  sku: {
    name: 'Consumption'
  }
  properties: {
    appLogsConfiguration: {
      destination: 'log-analytics'
      logAnalyticsConfiguration: {
        customerId: logAnalytics.properties.customerId
        sharedKey: logAnalytics.listKeys().primarySharedKey
      }
    }
    vnetConfiguration: vnetConfiguration
  }
}

resource permanentStorageMount 'Microsoft.App/managedEnvironments/storages@2022-10-01' = {
  name: storageAccountName
  parent: containerAppEnv
  properties: {
    azureFile: {
      accountName: storageAccountName
      accountKey: storageAccountKey
      shareName: fileShareName
      accessMode: 'ReadWrite'
    }
  }
}

resource containerApp 'Microsoft.App/containerApps@2022-10-01' = {
  name: containerAppName
  location: resourceGroup().location
  properties: {
    managedEnvironmentId: containerAppEnv.id
    configuration: {
      ingress: {
        external: true
        targetPort: 9090
        allowInsecure: false
        ipSecurityRestrictions: [for (ip, index) in ipAllowRules: {
          action: 'Allow'
          description: 'Allow access'
          name: 'GrantRule${index}'
          ipAddressRange: '${ip}/32'
        }]
        traffic: [
          {
            latestRevision: true
            weight: 100
          }
        ]
      }
    }
    template: {
      revisionSuffix: toLower(utcNow())
      containers: [
      {
        name: 'prometheus'
        probes: [
          {
            type: 'liveness'
            httpGet: {
              path: '/-/healthy'
              port: 9090
              scheme: 'HTTP'
            }
            periodSeconds: 120
            timeoutSeconds: 5
            initialDelaySeconds: 10
            successThreshold: 1
            failureThreshold: 3
          }
          {
            type: 'readiness'
            httpGet: {
              path: '/-/ready'
              port: 9090
              scheme: 'HTTP'
            }
            periodSeconds: 120
            timeoutSeconds: 5
            initialDelaySeconds: 10
            successThreshold: 1
            failureThreshold: 3
          }
        ]
        image: 'prom/prometheus:latest'
        resources: {
          cpu: json('0.75')
          memory: '1.5Gi'
        }
        command: [
          '/bin/prometheus'
          '--config.file=config/prometheus.yml'
          '--storage.tsdb.path=data'
          '--storage.tsdb.retention.time=60d'
          '--storage.tsdb.no-lockfile'
        ]
        volumeMounts: [
          {
            mountPath: '/prometheus'
            volumeName: 'azurefilemount'
          }
        ]
      }
    ]

      volumes: [
      {
        name: 'azurefilemount'
        storageType: 'AzureFile'
        storageName: storageAccountName
      }
    ]
      scale: {
        minReplicas: 1
        maxReplicas: 1
      }
    }
  }
  dependsOn: [
    permanentStorageMount
  ]
}

The above script is a bit long but hopefully still easy to understand. (If not, check out the documentation.) However, some details are worth highlighting and explaining.

Since we don’t need to browse the Prometheus dashboard, the ACA firewall should be enabled to block traffic from anything except Grafana, which is configured to use two outbound static IPs (passed as parameter `ipAllowRules`). 

To achieve this result, ingress must be enabled (`ingress`:`external` equals `true`). The same must be done for the underlying Storage Account. However, at the time of writing, isolation between ACA and Storage Accounts is not supported yet. This option is available just for a few services, such as VNets. 

For this reason, we had to isolate ACA in a VNet (unfortunately /23 size is a requirement ). Due to a Bicep bug, this option will not work if defined at the first Storage Account definition already written above. Still, it must be repeated at the end of the Bicep script.

resource storageAccount 'Microsoft.Storage/storageAccounts@2022-09-01' = {
  name: storageAccountName
  location: resourceGroup().location
  kind: 'StorageV2'
  sku: {
    name: storageAccountSku
  }
  properties: {
    supportsHttpsTrafficOnly: true
    minimumTlsVersion: 'TLS1_2'
    networkAcls: {
      defaultAction: 'Deny'
      bypass: 'AzureServices'
      virtualNetworkRules: [for i in range(0, length(storageAccountAllowedSubnets)): {
        id: virtualNetworkSubnets[i].id
      }]
    }
    largeFileSharesState: 'Enabled'
  }
}

Wrapping out the monitoring solution

What is described above so far could be represented diagrammatically in this way:

[Click on the image to view full-size]

  • The SRE AD group and related Managed Identity used for automation can access Grafana through AAD
  • Grafana can access Azure Monitor using Managed Identity: metrics, logs, and resource graphs can be queried using Kusto
  • Grafana IPs are allowed to connect with Prometheus ACA, hosted in a custom VNet
  • File Share mounted on a Storage Account is used as volume to run Prometheus container
  • Potentially, Prometheus can be scaled up and down - until now, we haven’t needed to do this

At this point a question arises: how can Prometheus access the Bank Platform - hosted in a closed VNet - to collect aggregate data?

This can be addressed with a different serverless solution: Azure Functions.

Monitoring Function App

An Azure Function app hosted in the Bank Platform VNet would be able to collect data from all the productive components around the platform: it has access to Azure SQL Database, Cosmos DB, Azure Batch, Service Bus, and even the Kubernetes cluster. 

It would be possible to query different data using a combination of tools such as Azure SDKs, and so why did we choose to use a Function app? Because it can expose REST APIs to the scheduled Prometheus jobs, and when these jobs are paused or stopped, Function apps are for free, being a serverless solution. Moreover, we could configure the Functions app to accept connections from a specific VNet only, the Prometheus VNet in this case.

Then, the complete diagram appears this way:

[Click on the image to view full-size]

In our case, the Monitoring Function app runs on .NET 7  using the new isolated worker process.

In `Program.cs`, create and run the host.

var host = Host.CreateDefaultBuilder()
    .ConfigureAppConfiguration((ctx, builder) =>
    {
        if (ctx.HostingEnvironment.IsDevelopment())
        {
            builder.AddUserSecrets(Assembly.GetExecutingAssembly(), optional: false);
            return;
        }

        // On Net7, it is fast enough to be used
        var configuration = builder.Build()!;
        // This logger is useful when diagnostic startup issues on Azure Portal
        var logger = LoggerFactory.Create(config =>
        {
            config.AddConsole();
            config.AddConfiguration(configuration.GetSection("Logging"));
        })
        .CreateLogger("Program");

        logger.LogInformation("Environment: {env}", ctx.HostingEnvironment.EnvironmentName);

        builder.AddAzureAppConfiguration(options =>
        {
            
            options.ConfigureRefresh(opt =>
            {
                // Auto app settings refresh
            });

            options.ConfigureKeyVault(opt =>
            {
                // KeyVault integration
            });
        }, false);
    })
    .ConfigureServices((ctx, services) =>
    {
        // Register services in IoC container
    })
    .ConfigureFunctionsWorkerDefaults((ctx, builder) =>
    {
        builder.UseDefaultWorkerMiddleware();

        if (ctx.HostingEnvironment.IsDevelopment())
            return;

        string? connectionString = ctx.Configuration.GetConnectionString("ApplicationInsights:SRE");
        if (string.IsNullOrWhiteSpace(connectionString))
            return;

        builder
            .AddApplicationInsights(opt =>
            {
                opt.ConnectionString = connectionString;
            })
            .AddApplicationInsightsLogger();

        builder.UseAzureAppConfiguration();
    })
    .Build();

await host.RunAsync();

Each function namespace represents a different monitoring context, so we have, for example, a namespace dedicated to the Azure Service Bus and another for Azure Batch, and so on. All namespaces provide an extension method to register into `IServiceCollection` all the requirements it needs. These extension methods are called from the `ConfigureServices`.

Monitoring Examples

Before concluding this, I want to provide some real usage examples.

Application Insights Availability Integration

Ping availability tests provided by Application Insights (AI) cost 0.0006€ per test. However, you can ping your services with a custom code and send the result to Application Insights using the official SDK for free.

Here is the code of the Availability section of the Monitoring Function app.

private async Task PingRegionAsync(
        string url,
        string testName)
    {
        const string LOCATION = "westeurope";

        string operationId = Guid.NewGuid().ToString("N");

        var availabilityTelemetry = new AvailabilityTelemetry
        {
            Id = operationId,
            Name = testName,
            RunLocation = LOCATION,
            Success = false,
            Timestamp = DateTime.UtcNow,
        };

        // not ideal, but we just need an estimation
        var stopwatch = Stopwatch.StartNew();

        try
        {
            await ExecuteTestAsync(url);

            stopwatch.Stop();

            availabilityTelemetry.Success = true;
        }
        catch (Exception ex)
        {
            stopwatch.Stop();

            if (ex is HttpRequestException reqEx && reqEx.StatusCode == System.Net.HttpStatusCode.NotFound)
            {
                _logger.LogError(reqEx, "Probably a route is missing");
            }

            HandleError(availabilityTelemetry, ex);
        }
        finally
        {
            availabilityTelemetry.Duration = stopwatch.Elapsed;

            _telemetryClient.TrackAvailability(availabilityTelemetry);
            _telemetryClient.Flush();
        }
    }

    private async Task ExecuteTestAsync(string url)
    {
        using var cancelAfterDelay = new CancellationTokenSource(TimeSpan.FromSeconds(20));

        string response;

        try
        {
            response = await _httpClient.GetStringAsync(url, cancelAfterDelay.Token);
        }
        catch (OperationCanceledException)
        {
            throw new TimeoutException();
        }

        switch (response.ToLowerInvariant())
        {
            case "healthy":
                break;
            default:
                _logger.LogCritical("Something is wrong");
                throw new Exception("Unknown error");
        }
    }

    private void HandleError(AvailabilityTelemetry availabilityTelemetry, Exception ex)
    {
        availabilityTelemetry.Message = ex.Message;

        var exceptionTelemetry = new ExceptionTelemetry(ex);
        exceptionTelemetry.Context.Operation.Id = availabilityTelemetry.Id;
        exceptionTelemetry.Properties.Add("TestName", availabilityTelemetry.Name);
        exceptionTelemetry.Properties.Add("TestLocation", availabilityTelemetry.RunLocation);
        _telemetryClient.TrackException(exceptionTelemetry);
    }

Initially, an `AvailabilityTelemetry` object is created and set up. Then, the ping operation is performed; depending on the result, different information is stored in AI using the SDK’s objects.

Note that the `stopwatch` object is not accurate, but it is enough for our use case.

Card Top up Critical User Journey

This is an example of a Critical User Journey (CUJ) where a user wants to top up their bank account through an external service. Behind the scenes, a third-party service notifies Flowe about the top up via REST APIs. Having access to Azure Monitor and from Grafana, it’s simply displaying the count of callbacks received through a Kusto query against the Application Insights resource.

requests
| where (url == "<callback-url>")
| where timestamp >= $__timeFrom and timestamp < $__timeTo
| summarize Total = count()

[Click on the image to view full-size]

It is also possible to display the same data as a time series chart but group callbacks by their status code.

requests
| where (url == "<callback-url>")
| where timestamp >= $__timeFrom and timestamp < $__timeTo
| summarize response = dcount(id) by resultCode, bin(timestamp, 1m)
| order by timestamp asc

[Click on the image to view full-size]

After the callback is received, a Flowe internal asynchronous flow is triggered to let microservices communicate with each other through integration events .

To complete this CUJ, the same Grafana dashboard shows the number of items that ended up in DLQ due to failures. Azure Monitor does not expose this kind of data directly, so custom code had to be written. The Monitor Functions app exposes an endpoint to return aggregate data of the items stuck in the Azure Service Bus DLQs.

[Function(nameof(DLQFunction))]
    public async Task<HttpResponseData> RunAsync(
        [HttpTrigger(
            AuthorizationLevel.Function,
            methods: "GET",
            Route = "exporter"
        )] HttpRequestData req)
    {
        return await ProcessSubscriptionsAsync(req);
    }

    private async Task<HttpResponseData> ProcessSubscriptionsAsync(HttpRequestData req)
    {
        var registry = Metrics.NewCustomRegistry();
        _gauge = PrometheusFactory.ProduceGauge(
            registry,
            PrometheusExporterConstants.SERVICEBUS_GAUGE_NAME,
            "Number of DLQs grouped by subscription and subject",
            labelNames: new[]
                {
                    PrometheusExporterConstants.SERVICEBUS_TYPE_LABEL,
                    PrometheusExporterConstants.SERVICEBUS_TOPIC_LABEL,
                    PrometheusExporterConstants.SERVICEBUS_SUBSCRIPTION_LABEL,
                    PrometheusExporterConstants.SERVICEBUS_SUBJECT_LABEL,
                });

        foreach (Topic topic in _sbOptions.serviceBus.Topics!)
        {
            foreach (var subscription in topic.Subscriptions!)
            {
                await ProcessSubscriptionDlqs(_sbOptions.serviceBus.Name!, topic.Name!, subscription, _gauge);
            }
        }

        return await GenerateResponseAsync(req, registry);
    }

    private async Task ProcessSubscriptionDlqs(string serviceBus, string topic, string subscription, Gauge gauge)
    {
        var stats = await _serviceBusService.GetDeadLetterMessagesRecordsAsync(serviceBus, topic, subscription);

        var groupedStats = stats
            .GroupBy(x => x.Subject, (key, group) => new { Subject = key, Count = group.Count() })
            .ToList();

        foreach (var stat in groupedStats)
        {
            gauge!
                .WithLabels("dlqs", topic, subscription, stat.Subject)
                .Set(stat.Count);
        }
    }

    private static async Task<HttpResponseData> GenerateResponseAsync(HttpRequestData req, CollectorRegistry registry)
    {
        var result = await PrometheusHelper.GenerateResponseFromRegistryAsync(registry);

        var response = req.CreateResponse(HttpStatusCode.OK);
        response.Headers.Add("Content-Type", $"{MediaTypeNames.Text.Plain}; charset=utf-8");
        response.WriteString(result);

        return response;
    }

In this case, the Function App doesn’t query the Azure Service Bus instance directly; it is instead wrapped by another custom service through the `_serviceBusService`.

P.S. We are working to publish this service on GitHub!

Once data is returned to Prometheus, Grafana can show them using PromQL.

service_bus{subject="CriticalEventName"}

[Click on the image to view full-size]

OnBoarding Critical User Journey

The "OnBoarding" CUJ starts when customers open the app for the first time and finishes when the customer successfully opens a new bank account. It is a complicated journey because the user’s digital identity must be confirmed, and personal data is processed by Know Your Customer (KYC) and anti-money laundering services. To complete the process, a lot of third parties are involved.

Here, I want to share a piece of this CUJ dashboard where a sequence of APIs is monitored, and a funnel is built on top of them, among other data.

[Click on the image to view full-size]

 

The queries used to build this dashboard are similar to those described above.

Conclusions

"Critical User Journeys are an effective way to decide what metrics monitor and guide their collection. Bringing these metrics together in tools like Prometheus and Grafana simplifies the way in which SREs and Architects can exchange responsibilities to oversee operations. Custom code may be needed to collect certain metrics about different CUJs, but all teams benefit from the resulting simplicity in monitoring the overall workflow.

Usually, Prometheus and Grafana are used to monitor Kubernetes and application metrics such as API average response time and CPU under use.

Instead, this post shows how to calculate and display CUJs over aggregated data, necessarily collected using custom code.

Architectural choices point to a cost-effective solution in Azure, but keep in mind the deployment simplicity and requirements that an organization may have (such as SSO and security concerns). Plus, the need for maintenance is almost eliminated."

Note: All the numbers and names of the screenshots shown in this article were taken from test environments (and sometimes mixed as well). There is no way to get real data and real flows from the charts above.

About the Author

Rate this Article

Adoption
Style

BT