Detecting faces on photos using Microsoft Cognitive Services

I started playing with Microsoft Cognitive Services and discovered that it makes many otherwise complex tasks very easy. This blog post shows how to build web application that detects faces on uploaded photo.

Cognitive services account. You need Microsoft Cognitive Services account before you can use services. Only thing you need is Microsoft Account or Office 365 account. After authentication you need to acquire service key and wait some moments while it is generated and ready to use. Most of services can be used for free in small scale.

Building cognitive services client

To use cognitive services API-s we need two settings we will keep in web.config file:

• CognitiveServicesKey – this is the key we get when joining with cognitive services
• CognitiveServicesApiUrl – API end-point, by example: https://westus.api.cognitive.microsoft.com

Sample appication is available in my Github repository CognitiveServicesDemo. Be warned that code of this application is very raw and experimental right now but I will improve it over time.

Client class with call to face detection service is here. It reads settings from web application config files, builds URL and uploads image to analyze to cognitive services API.

public static class CognitiveServicesClient
{
    private static readonly string ApiKey = ConfigurationManager.AppSettings["CognitiveServicesKey"];
    private static readonly string ApiUrl = ConfigurationManager.AppSettings["CognitiveServicesApiUrl"];
 
    public static async Task<List<DetectedFace>> DetectFaces(Stream image)
    {
        using (var client = new HttpClient())
        {
            var content = new StreamContent(image);
            var url = ApiUrl + "/face/v1.0/detect?returnFaceId=true&returnFaceLandmarks=true";
 
            client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", ApiKey);
            content.Headers.ContentType = new MediaTypeHeaderValue("application/octet-stream");
            var httpResponse = await client.PostAsync(url, content);
            
            if (httpResponse.StatusCode == HttpStatusCode.OK)
            {
                var responseBody = await httpResponse.Content.ReadAsStringAsync();
                return JsonConvert.DeserializeObject<List<DetectedFace>>(responseBody);
            }
        }
 
        return null;
    }
}

The code doesn’t build until we add DTO classes. First, we need some class to represent points in double or float format.

public class FacePoint
{
    public double X;
    public double Y;
}

From API we get back also a special object that describes different points in face found by API. Cognitive services use term face landmarksfor these points.

public class FaceLandmarks
{
    public FacePoint PupilLeft;
    public FacePoint PupilRight;
    public FacePoint NoseTip;
    public FacePoint MouthLeft;
    public FacePoint MouthRight;
    public FacePoint EyebrowLeftOuter;
    public FacePoint EyebrowLeftInner;
    public FacePoint EyeLeftOuter;
    public FacePoint EyeLeftTop;
    public FacePoint EyeLeftBottom;
    public FacePoint EyeLeftInner;
    public FacePoint EyebrowRightInner;
    public FacePoint EyebrowRightOuter;
    public FacePoint EyeRightInner;
    public FacePoint EyeRightTop;
    public FacePoint EyeRightBottom;
    public FacePoint EyeRightOuter;
    public FacePoint NoseRootLeft;
    public FacePoint NoseRootRight;
    public FacePoint NoseLeftAlarTop;
    public FacePoint NoseRightAlarTop;
    public FacePoint NoseLeftAlarOutTip;
    public FacePoint NoseRightAlarOutTip;
    public FacePoint UpperLipTop;
    public FacePoint UpperLipBottom;
    public FacePoint UnderLipTop;
    public FacePoint UnderLipBottom;
}

We get also back rectangle around the face. This is called face rectangle in cognitive services.

public class FaceRectangle
{
    public int Left; 
    public int Top; 
    public int Width;

    public int Height;
 
    public Rectangle ToRectangle()
    {
        return new Rectangle(Left, Top, Width, Height);
    }
}

I added here ToRectangle() method that converts face rectangle to rectangle used by System.Drawing classes. We need it later. As API returns the list of faces detected from photo we need one general DTO to keep everything about face together.

public class DetectedFace
{
    public string FaceId;
    public FaceRectangle FaceRectangle;
    public FaceLandmarks FaceLandmarks;
}

No we have all classes needed to communicate with face detection service and it’s time to build UI for service client.

Building face detection web page

I created simple ASP.NET MVC 5 application as there we have stable support for processing images. Our face detection page works as follows:

• show photo upload form and submit button
• if photo was uploaded then analyze it
• if there were faces on photo then draw rectangles around faces
• return photo as image string
• show photo on page

Here is the DetectFaces method of HomeController.

public async Task<ActionResult> DetectFaces()
{
    if(Request.HttpMethod == "GET")
    {
        return View();
    }
 
    var imageResult = "";
    var file = Request.Files[0];
    IList<DetectedFace> faces = null;
 
    // will be actually disposed by stream content
    using (var analyzeCopyBuffer = new MemoryStream()) 
    {
        file.InputStream.CopyTo(analyzeCopyBuffer);
        file.InputStream.Seek(0, SeekOrigin.Begin);
        analyzeCopyBuffer.Seek(0, SeekOrigin.Begin);
        faces = await CognitiveServicesClient.DetectFaces(analyzeCopyBuffer);
    }
 
 
    using (var img = new Bitmap(file.InputStream))
    // make copy, drawing on indexed pixel format image is not supported
    using (var nonIndexedImg = new Bitmap(img.Width, img.Height))
    using (var g = Graphics.FromImage(nonIndexedImg))
    using (var mem = new MemoryStream())
    {
        g.DrawImage(img, 0, 0, img.Width, img.Height);
 
        var pen = new Pen(Color.Red, 5);
 
        foreach (var face in faces)
        {
            var rectangle = face.FaceRectangle.ToRectangle();
 
            g.DrawRectangle(pen, rectangle);
        }
 
        nonIndexedImg.Save(mem, ImageFormat.Png);
 
        var base64 = Convert.ToBase64String(mem.ToArray());
        imageResult = String.Format("data:image/png;base64,{0}", base64);
    }
 
    return View((object)imageResult);
}

I had to add some nasty hacks here as for some reason StreamContent class used in cognitive services client automatically disposes input stream. Before we process the image we will make copy of it. This is because drawing on photos that have indexed picel format is not supported by System.Drawing. In the end we convert the image to inline string we can use with img tag.

We also need a view with upload form and image.

@model string
@{
    ViewBag.Title = "DetectFace";
}
 
<h2>DetectFace</h2>
 
<form method="post" enctype="multipart/form-data">
    Upload photo to analyze: <br />
    <input type="file" name="ImageToAnalyze" /><button type="submit">Send</button>
</form>
 
@if(Model != null)
{
    <img src="@Model" />
}

Now we are done with web application and it’s time to try it out.

Detecting faces

To try web application out I took The Beatles photo from Google search and the source of photo is The Beatles page at Pitchfork.

Let’s run web application and move to DetectFaces page. Then let’s upload The Beatles photo, click send and see what happens.

My result is here:

If keys are working, API end-point URL is correct and we have not exceeded out daily limit we get back photo where there are rectangles around all detected faces.

Wrapping up

Face detection and other image analyzing solutions sound like something big, resource intensive and complex at first place. Using Microsoft Cognitive Services that run on Azure cloud wipes all these complexities away from us. We have to write some client code and visualize the results. That’s it. The big work is done in Azure servers, we just consume these services over simple web API. Our web application that seems sort of intelligent is actually a simple web based UI that works as mediator between user and cognitive services.

References

• Microsoft Cognitive Services (Official homepage)
• Face API (Official documentation)
• Cognitive services demo application (Gunnar Peipman)