Detecting faces on photos using Microsoft Cognitive Services

I started playing with Microsoft Cognitive Services and discovered that it makes many otherwise complex tasks very easy. This blog post shows how to build web application that detects faces on uploaded photo.

Cognitive services account. You need Microsoft Cognitive Services account before you can use services. Only thing you need is Microsoft Account or Office 365 account. After authentication you need to acquire service key and wait some moments while it is generated and ready to use. Most of services can be used for free in small scale.

Building cognitive services client

To use cognitive services API-s we need two settings we will keep in web.config file:

Sample appication is available in my Github repository CognitiveServicesDemo. Be warned that code of this application is very raw and experimental right now but I will improve it over time.

Client class with call to face detection service is here. It reads settings from web application config files, builds URL and uploads image to analyze to cognitive services API.

public static class CognitiveServicesClient
private static readonly string ApiKey = ConfigurationManager.AppSettings["CognitiveServicesKey"
private static readonly string ApiUrl = ConfigurationManager.AppSettings["CognitiveServicesApiUrl"

public static async Task<List<DetectedFace>> DetectFaces(Stream
using (var client = new HttpClient
var content = new StreamContent
var url = ApiUrl + "/face/v1.0/detect?returnFaceId=true&returnFaceLandmarks=true"

"Ocp-Apim-Subscription-Key", ApiKey
            content.Headers.ContentType =
new MediaTypeHeaderValue("application/octet-stream"
var httpResponse = await
client.PostAsync(url, content);
if (httpResponse.StatusCode == HttpStatusCode
var responseBody = await
return JsonConvert.DeserializeObject<List<DetectedFace

return null;

The code doesn’t build until we add DTO classes. First, we need some class to represent points in double or float format.

public class FacePoint
public double
public double Y;

From API we get back also a special object that describes different points in face found by API. Cognitive services use term face landmarksfor these points.

public class FaceLandmarks
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint
public FacePoint UnderLipBottom;

We get also back rectangle around the face. This is called face rectangle in cognitive services.

public class FaceRectangle
public int Left;

public int Top;

public int Width;

public int Height;

public Rectangle
return new Rectangle(Left, Top, Width, Height);

I added here ToRectangle() method that converts face rectangle to rectangle used by System.Drawing classes. We need it later. As API returns the list of faces detected from photo we need one general DTO to keep everything about face together.

public class DetectedFace
public string
public FaceRectangle
public FaceLandmarks FaceLandmarks;

No we have all classes needed to communicate with face detection service and it’s time to build UI for service client.

Building face detection web page

I created simple ASP.NET MVC 5 application as there we have stable support for processing images. Our face detection page works as follows:

  • show photo upload form and submit button
  • if photo was uploaded then analyze it
  • if there were faces on photo then draw rectangles around faces
  • return photo as image string
  • show photo on page

Here is the DetectFaces method of HomeController.

public async Task<ActionResult> DetectFaces()
if(Request.HttpMethod == "GET"

var imageResult = ""
file = Request.Files[0];
IList<DetectedFace> faces = null

// will be actually disposed by stream content
    using (var analyzeCopyBuffer = new MemoryStream
analyzeCopyBuffer.Seek(0, SeekOrigin.Begin);
        faces =
await CognitiveServicesClient

using (var img = new Bitmap
// make copy, drawing on indexed pixel format image is not supported
    using (var nonIndexedImg = new Bitmap
(img.Width, img.Height))
using (var g = Graphics
using (var mem = new MemoryStream
        g.DrawImage(img, 0, 0, img.Width, img.Height);

var pen = new Pen(Color
.Red, 5);

foreach (var face in
rectangle = face.FaceRectangle.ToRectangle();

            g.DrawRectangle(pen, rectangle);


var base64 = Convert
        imageResult =
, base64);

return View((object)imageResult);

I had to add some nasty hacks here as for some reason StreamContent class used in cognitive services client automatically disposes input stream. Before we process the image we will make copy of it. This is because drawing on photos that have indexed picel format is not supported by System.Drawing. In the end we convert the image to inline string we can use with img tag.

We also need a view with upload form and image.

@model string
    ViewBag.Title =


<form method="post" enctype="multipart/form-data">
    Upload photo to analyze: <br />
    <input type="file" name="ImageToAnalyze" /><button type="submit">Send</button

if(Model != null
<img src="@Model" />

Now we are done with web application and it’s time to try it out.

Detecting faces

To try web application out I took The Beatles photo from Google search and the source of photo is The Beatles page at Pitchfork.

The Beatles photo from Pitchfork

Let’s run web application and move to DetectFaces page. Then let’s upload The Beatles photo, click send and see what happens.

My result is here:

Faces of The Beatles are detected by face detection API

If keys are working, API end-point URL is correct and we have not exceeded out daily limit we get back photo where there are rectangles around all detected faces.

Wrapping up

Face detection and other image analyzing solutions sound like something big, resource intensive and complex at first place. Using Microsoft Cognitive Services that run on Azure cloud wipes all these complexities away from us. We have to write some client code and visualize the results. That’s it. The big work is done in Azure servers, we just consume these services over simple web API. Our web application that seems sort of intelligent is actually a simple web based UI that works as mediator between user and cognitive services.


See also

5 thoughts on “Detecting faces on photos using Microsoft Cognitive Services

  • […] Detecting faces on photos using Microsoft Cognitive Services – Gunnar Peipman […]

  • Sachin says:

    Hi Gunnar Peipman
    Hope you are doing good.
    I’d like to know about FaceApi image storage on the cloud.My concern is how did I identify the uploaded images in Azure, Does it located on a specific path somewhere.
    Though in AWS we can see uploaded images easly.
    Kindly share your valuable thoughts

  • Gunnar says:

    Hi Sachin,
    Cognitive services actually don’t seem to hold copies of images. AFAIK they just detect faces with all important points and save this meta data. Keeping up the link between image file and data in cognitive services is up to developers.

  • H.Chloe says:


    I downloaded your codes from Github! it works wonder. But when I try to copy your code and do from scratch I am having errors like reference/assembly runtime. which I failed to solve it.

    Do you have a project on how to create a person group and person for face API in MVC web application?

  • Gunnar says:

    Sample application supports managing person groups and people too. Just select Faces from top menu and Person groups from left menu.

Leave a Reply

Your email address will not be published. Required fields are marked *