System memory health check for ASP.NET Core

I found temporary cross-platform solution for .NET Core to read system memory metrics until framework level libraries appear. This blog post shows how to build ASP.NET Core health check for system memory metrics.

Getting started

We are using MemoryMetricsClient class defined in blog post referred above. The class with MemoryMetrics result is given here again.

public class MemoryMetrics
{
    public double Total;
    public double Used;
    public double Free;
}
 
public class MemoryMetricsClient
{
    public MemoryMetrics GetMetrics()
    {
        MemoryMetrics metrics;
 
        if (IsUnix())
        {
            metrics = GetUnixMetrics();
        }
        else
        {
            metrics = GetWindowsMetrics();
        }
  
        return metrics;
    }
 
    private bool IsUnix()
    {
        var isUnix = RuntimeInformation.IsOSPlatform(OSPlatform.OSX) ||
                     RuntimeInformation.IsOSPlatform(OSPlatform.Linux);
 
        return isUnix;
    }
 
    private MemoryMetrics GetWindowsMetrics()
    {
        var output = "";
 
        var info = new ProcessStartInfo();
        info.FileName = "wmic";
        info.Arguments = "OS get FreePhysicalMemory,TotalVisibleMemorySize /Value";
        info.RedirectStandardOutput = true;
 
        using (var process = Process.Start(info))
        {
            output = process.StandardOutput.ReadToEnd();
        }
 
        var lines = output.Trim().Split("\n");
        var freeMemoryParts = lines[0].Split("=", StringSplitOptions.RemoveEmptyEntries);
        var totalMemoryParts = lines[1].Split("=", StringSplitOptions.RemoveEmptyEntries);
 
        var metrics = new MemoryMetrics();
        metrics.Total = Math.Round(double.Parse(totalMemoryParts[1]) / 1024, 0);
        metrics.Free = Math.Round(double.Parse(freeMemoryParts[1]) / 1024, 0);
        metrics.Used = metrics.Total - metrics.Free;
 
        return metrics;
    }
 
    private MemoryMetrics GetUnixMetrics()
    {
        var output = "";
 
        var info = new ProcessStartInfo("free -m");
        info.FileName = "/bin/bash";
        info.Arguments = "-c \"free -m\"";
        info.RedirectStandardOutput = true;
 
        using (var process = Process.Start(info))
        {
            output = process.StandardOutput.ReadToEnd();
            Console.WriteLine(output);
        }
 
        var lines = output.Split("\n");
        var memory = lines[1].Split(" ", StringSplitOptions.RemoveEmptyEntries);
 
        var metrics = new MemoryMetrics();
        metrics.Total = double.Parse(memory[1]);
        metrics.Used = double.Parse(memory[2]);
        metrics.Free = double.Parse(memory[3]);
 
        return metrics;
    }
}

For Windows we are using wimc to get memory metrics. On Linux we are using command line utility called “free”.

NB! This memory metrics client works also on Windows Subsystem for Linux (WSL). Solutions using Process class and may return incorrect results under WSL.

Health status

As we are writing health check it’s good idea to stop for a moment and think how to translate memory metrics to health status. We can return memory metrics with healthy status no matter what numbers say but it doesn’t seem very intelligent to me.

I took rough numbers I have seen in online conversations by admin guys and defined health status based on used memory like shown here:

• Healthy – up to 80%
• Degraded – 80% – 90%
• Unhealthy – over 90%

System memory health check

Here is the health check. Notice how I use data dictionary for memory metrics.

public class SystemMemoryHealthcheck : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        var client = new MemoryMetricsClient();
        var metrics = client.GetMetrics();
        var percentUsed = 100 * metrics.Used / metrics.Total;
 
        var status = HealthStatus.Healthy;
        if(percentUsed > 80)
        {
            status = HealthStatus.Degraded;
        }
        if(percentUsed > 90)
        {
            status = HealthStatus.Unhealthy;
        }
 
        var data = new Dictionary<string, object>();
        data.Add("Total", metrics.Total);
        data.Add("Used", metrics.Used);
        data.Add("Free", metrics.Free);
 
        var result = new HealthCheckResult(status, null, null, data);
 
        return await Task.FromResult(result);
    }
}

NB! If you see Append() extension method on data dictionary when adding metrics then don’t use it. It doesn’t generate error but it doesn’t also work.

Enabling ASP.NET Core health checks

To make health checks work we have to add them to ASP.NET Core request pipeline. In ConfigureServices() method of Startup class we add services to request pipeline and configure them.

public void ConfigureServices(IServiceCollection services)
{
    services.AddHealthChecks()
            .AddCheck("ERP", new PingHealthCheck("www.google.com", 100))
            .AddCheck("Accounting", new PingHealthCheck("www.bing.com", 10))
            .AddCheck("Database", new PingHealthCheck("www.__Dbing1.com", 100))
            .AddCheck<SystemMemoryHealthcheck>("Memory");
 
    services.AddControllersWithViews();
    services.AddRazorPages();
}

You can comment out PingHealthCheck rows. If you need ping health checks too then take a look at these blog posts:

• ASP.NET Core health checks quick start
• Avoiding ping flood in ASP.NET Core health checks

In Configure() method of Startup class needs also few changes. Take a careful look at options.ResponseWriter line. This is where formatting of health check result happens.

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }
    else
    {
        app.UseExceptionHandler("/Home/Error");
    }
 
    var options = new HealthCheckOptions();
    options.ResponseWriter = async (c, r) => {
 
        c.Response.ContentType = "application/json";
        var result = new List<ServiceStatus>();
        result.Add(new ServiceStatus { Service = "OverAll", Status = (int)r.Status });
        result.AddRange(
            r.Entries.Select(
                e => new ServiceStatus
                {
                    Service = e.Key,
                    Status = (int)e.Value.Status,
                    Data = e.Value.Data.Select(k => k).ToList()
                }
            )
        );
        
        var json = JsonConvert.SerializeObject(result);
 
        await c.Response.WriteAsync(json);
    };
 
    app.UseHealthChecks("/hc", options);
    app.UseStaticFiles();
    app.UseRouting();
    app.UseAuthorization();
 
    app.UseEndpoints(endpoints =>
    {
        endpoints.MapControllerRoute(
            name: "default",
            pattern: "{controller=Home}/{action=Index}/{id?}");
        endpoints.MapRazorPages();
    });
}

Health checks are available as /hc end-point in our application. Let’s run application and navigate to health checks page.

[
  {
    "Service": "OverAll",
    "Status": 0,
    "Data": null
  },
  {
    "Service": "ERP",
    "Status": 2,
    "Data": []
  },
  {
    "Service": "Accounting",
    "Status": 1,
    "Data": []
  },
  {
    "Service": "Database",
    "Status": 0,
    "Data": []
  },
  {
    "Service": "Memory",
    "Status": 0,
    "Data": [
      {
        "Key": "Total",
        "Value": 8117.0
      },
      {
        "Key": "Used",
        "Value": 7462.0
      },
      {
        "Key": "Free",
        "Value": 655.0
      }
    ]
  }
]

We can visualize health checks like shown in my blog post Displaying ASP.NET Core health checks with Grafana and InfluxDB. It’s not much effort to add support for memory metrics to data collector.

How long it took to get memory metrics?

As our current solution makes use of external shell applications I would like to see how long it took to get system memory metrics. To get this data to metrics I add Duration property to Memory metrics class.

public class MemoryMetrics
{
    public double Total;
    public double Used;
    public double Free;
    public long Duration;
}

To get time it took to get memory metrics we modify GetMetrics() method of MemoryMetricsClient class().

public MemoryMetrics GetMetrics()
{
    MemoryMetrics metrics;
    var watch = new Stopwatch();
 
    watch.Start();
    if (IsUnix())
    {
        metrics = GetUnixMetrics();
    }
    else
    {
        metrics = GetWindowsMetrics();
    }
    watch.Stop();
 
    metrics.Duration = watch.ElapsedMilliseconds;
 
    return metrics;
}

To make duration appear in memory health check data we have to add it to data collection in healt check class.

var data = new Dictionary<string, object>();
data.Add("Total", metrics.Total);
data.Add("Used", metrics.Used);
data.Add("Free", metrics.Free);
data.Add("Duration", metrics.Duration);

It’s time to check out health check results again and see what it the approximate duration of system memory health check.

[
  {
    "Service": "OverAll",
    "Status": 0,
    "Data": null
  },
  {
    "Service": "ERP",
    "Status": 2,
    "Data": []
  },
  {
    "Service": "Accounting",
    "Status": 1,
    "Data": []
  },
  {
    "Service": "Database",
    "Status": 0,
    "Data": []
  },
  {
    "Service": "Memory",
    "Status": 0,
    "Data": [
      {
        "Key": "Total",
        "Value": 8117.0
      },
      {
        "Key": "Used",
        "Value": 7462.0
      },
      {
        "Key": "Free",
        "Value": 655.0
      },
      {
        "Key": "Duration",
        "Value": 186
      }
    ]
  }
]

Getting of system memory metrics is the matter of few hundred milliseconds on my machine and it clearly tells us there is minimal overhead getting memory metrics.

Wrapping up

Like often happens in this blog then one thing lead to another. This time I wrote first cross-platform class to get system memory metrics with .NET Core and then I took the client class to ASP.NET Core web application and used it to write memory metrics health check. I use this health check in some Azure VM-s that run Linux where guest OS agent is unstable on reporting memory metrics. With this health check I get always valid data about VM-s memory.