Detecting faces on photos using Microsoft Cognitive Services

I started playing with Microsoft Cognitive Services and discovered that it makes many otherwise complex tasks very easy. This blog post shows how to build web application that detects faces on uploaded photo.

Cognitive services account. You need Microsoft Cognitive Services account before you can use services. Only thing you need is Microsoft Account or Office 365 account. After authentication you need to acquire service key and wait some moments while it is generated and ready to use. Most of services can be used for free in small scale.

Building cognitive services client

To use cognitive services API-s we need two settings we will keep in web.config file:

Sample appication is available in my Github repository CognitiveServicesDemo. Be warned that code of this application is very raw and experimental right now but I will improve it over time.

Client class with call to face detection service is here. It reads settings from web application config files, builds URL and uploads image to analyze to cognitive services API.


public static class CognitiveServicesClient
{
   
private static readonly string ApiKey = ConfigurationManager.AppSettings["CognitiveServicesKey"
];
   
private static readonly string ApiUrl = ConfigurationManager.AppSettings["CognitiveServicesApiUrl"
];

   
public static async Task<List<DetectedFace>> DetectFaces(Stream
image)
    {
       
using (var client = new HttpClient
())
        {
           
var content = new StreamContent
(image);
           
var url = ApiUrl + "/face/v1.0/detect?returnFaceId=true&returnFaceLandmarks=true"
;

            client.DefaultRequestHeaders.Add(
"Ocp-Apim-Subscription-Key", ApiKey
);
            content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream"
);
           
var httpResponse = await
client.PostAsync(url, content);
           
           
if (httpResponse.StatusCode == HttpStatusCode
.OK)
            {
               
var responseBody = await
httpResponse.Content.ReadAsStringAsync();
               
return JsonConvert.DeserializeObject<List<DetectedFace
>>(responseBody);
            }
        }

       
return null;
    }
}

The code doesn’t build until we add DTO classes. First, we need some class to represent points in double or float format.


public class FacePoint
{
   
public double
X;
   
public double Y;
}

From API we get back also a special object that describes different points in face found by API. Cognitive services use term face landmarksfor these points.


public class FaceLandmarks
{
   
public FacePoint
PupilLeft;
   
public FacePoint
PupilRight;
   
public FacePoint
NoseTip;
   
public FacePoint
MouthLeft;
   
public FacePoint
MouthRight;
   
public FacePoint
EyebrowLeftOuter;
   
public FacePoint
EyebrowLeftInner;
   
public FacePoint
EyeLeftOuter;
   
public FacePoint
EyeLeftTop;
   
public FacePoint
EyeLeftBottom;
   
public FacePoint
EyeLeftInner;
   
public FacePoint
EyebrowRightInner;
   
public FacePoint
EyebrowRightOuter;
   
public FacePoint
EyeRightInner;
   
public FacePoint
EyeRightTop;
   
public FacePoint
EyeRightBottom;
   
public FacePoint
EyeRightOuter;
   
public FacePoint
NoseRootLeft;
   
public FacePoint
NoseRootRight;
   
public FacePoint
NoseLeftAlarTop;
   
public FacePoint
NoseRightAlarTop;
   
public FacePoint
NoseLeftAlarOutTip;
   
public FacePoint
NoseRightAlarOutTip;
   
public FacePoint
UpperLipTop;
   
public FacePoint
UpperLipBottom;
   
public FacePoint
UnderLipTop;
   
public FacePoint UnderLipBottom;
}

We get also back rectangle around the face. This is called face rectangle in cognitive services.


public class FaceRectangle
{
   
public int Left;

   
public int Top;

   
public int Width;


   
public int Height;


   
public Rectangle
ToRectangle()
    {
       
return new Rectangle(Left, Top, Width, Height);
    }
}

I added here ToRectangle() method that converts face rectangle to rectangle used by System.Drawing classes. We need it later. As API returns the list of faces detected from photo we need one general DTO to keep everything about face together.


public class DetectedFace
{
   
public string
FaceId;
   
public FaceRectangle
FaceRectangle;
   
public FaceLandmarks FaceLandmarks;
}

No we have all classes needed to communicate with face detection service and it’s time to build UI for service client.

Building face detection web page

I created simple ASP.NET MVC 5 application as there we have stable support for processing images. Our face detection page works as follows:

  • show photo upload form and submit button
  • if photo was uploaded then analyze it
  • if there were faces on photo then draw rectangles around faces
  • return photo as image string
  • show photo on page

Here is the DetectFaces method of HomeController.


public async Task<ActionResult> DetectFaces()
{
   
if(Request.HttpMethod == "GET"
)
    {
       
return
View();
    }

   
var imageResult = ""
;
   
var
file = Request.Files[0];
   
IList<DetectedFace> faces = null
;

   
// will be actually disposed by stream content
    using (var analyzeCopyBuffer = new MemoryStream
())
    {
        file.InputStream.CopyTo(analyzeCopyBuffer);
        file.InputStream.Seek(0,
SeekOrigin
.Begin);
analyzeCopyBuffer.Seek(0, SeekOrigin.Begin);
        faces =
await CognitiveServicesClient
.DetectFaces(analyzeCopyBuffer);
    }


   
using (var img = new Bitmap
(file.InputStream))
   
// make copy, drawing on indexed pixel format image is not supported
    using (var nonIndexedImg = new Bitmap
(img.Width, img.Height))
   
using (var g = Graphics
.FromImage(nonIndexedImg))
   
using (var mem = new MemoryStream
())
    {
        g.DrawImage(img, 0, 0, img.Width, img.Height);

       
var pen = new Pen(Color
.Red, 5);

       
foreach (var face in
faces)
        {
           
var
rectangle = face.FaceRectangle.ToRectangle();

            g.DrawRectangle(pen, rectangle);
        }

        nonIndexedImg.Save(mem,
ImageFormat
.Png);

       
var base64 = Convert
.ToBase64String(mem.ToArray());
        imageResult =
String.Format("data:image/png;base64,{0}"
, base64);
    }

   
return View((object)imageResult);
}

I had to add some nasty hacks here as for some reason StreamContent class used in cognitive services client automatically disposes input stream. Before we process the image we will make copy of it. This is because drawing on photos that have indexed picel format is not supported by System.Drawing. In the end we convert the image to inline string we can use with img tag.

We also need a view with upload form and image.


@model string
@{
    ViewBag.Title =
"DetectFace"
;
}

<h2>DetectFace</h2>

<form method="post" enctype="multipart/form-data">
    Upload photo to analyze: <br />
    <input type="file" name="ImageToAnalyze" /><button type="submit">Send</button
>
</
form>

@
if(Model != null
)
{
   
<img src="@Model" />
}

Now we are done with web application and it’s time to try it out.

Detecting faces

To try web application out I took The Beatles photo from Google search and the source of photo is The Beatles page at Pitchfork.

The Beatles photo from Pitchfork

Let’s run web application and move to DetectFaces page. Then let’s upload The Beatles photo, click send and see what happens.

My result is here:

Faces of The Beatles are detected by face detection API

If keys are working, API end-point URL is correct and we have not exceeded out daily limit we get back photo where there are rectangles around all detected faces.

Wrapping up

Face detection and other image analyzing solutions sound like something big, resource intensive and complex at first place. Using Microsoft Cognitive Services that run on Azure cloud wipes all these complexities away from us. We have to write some client code and visualize the results. That’s it. The big work is done in Azure servers, we just consume these services over simple web API. Our web application that seems sort of intelligent is actually a simple web based UI that works as mediator between user and cognitive services.

References



See also

One thought on “Detecting faces on photos using Microsoft Cognitive Services

Leave a Reply

Your email address will not be published. Required fields are marked *