Create thumbnails using Azure Cognitive Services

Azure Cognitive Services is set of powerful and intelligent cloud services to analyze photos and images. This blog post shows how to create smart thumbnails using Azure cognitive services and ASP.NET Core.

Why thumbnail service?

Back in time I had to argue with one customer about how thumbnail should be done. There was popular misconception that GetThumbnailImage() method in .NET framework is the way to go… until those using it ran in to image quality issues. Why? The method worked well until there’s no thumbnail image added to photo by camera. If thumbnail image was there in JPG file then GetThumbnailImage() operated on it instead. If thumbail in photo was 100×100 and GetThumbnailImage() was used to get 250×250 thumbnail then the end result was awful low quality mess.

Find out more about photo thumbnail topic of ASP.NET Forms era and ASP.NET MVC early days from my blog post Resizing images without loss of quality.

If image is not square and we need square thumbnail then there’s one question – which part from photo to take? We can show image to user and ask to move cropping box to position that user wants to thumbnail but it’s annoying for users. We can go with intelligent algorithms but it’s complex task and it takes time.

Thumbnail service in Azure Computer Vision services family solves both problems for us. It generates thumbnails with good quality and it’s also bit intelligent so we can let is choose which part of photo is the best for thumbnail.

Setting up cognitive services account

To get started you need Azure subscription and Cognitive Services or Computer Vision account. Cognitive Services account is overall account giving an access also to other services. Depending on account there are different pricing tiers available. Take free one (F0) if available as it is enough for experimenting.

When account is ready then open it in portal and move to Keys page like show on the following screenshot.

One of these keys (you choose) with service endpoint is used by API to communicate with Azure thumbnail service.

Configuring web application

First thing to do is to register IComputerVisionClient to ASP.NET Core dependency injection. We must do it in ConfigureServices() method of Startup class. I’m using factory method to create new instance of ComputerVisionClient.

public void ConfigureServices(IServiceCollection services)
{
    services.AddControllersWithViews();
 
    services.AddScoped<IComputerVisionClient>(factory => {
        var key = Configuration["ComputerVisionKey"];
        var host = Configuration["ComputerVisionEndpoint"];
 
        var credentials = new ApiKeyServiceClientCredentials(key);                
        var client = new ComputerVisionClient(credentials);
        client.Endpoint = host;
 
        return client;
    });
}

We have to modify appSettings.json file and add computer vision service parameters there.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft": "Warning",
      "Microsoft.Hosting.Lifetime": "Information"
    }
  },
  "AllowedHosts": "*",
  "ComputerVisionKey": "0c010fa5424f561de1e5e42d34111a",
  "ComputerVisionEndpoint": "https://westeurope.api.cognitive.microsoft.com/"
}

Now we are ready to start using computer vision service.

Creating simple UI for thumbnail service

Before going to code let’s build a view to try out how thumbnail service works. This is how my Index view of HomeController looks like.

<style>
    .thumbnail {
        margin:20px;
        border:1px solid darkgrey
    }
</style>
 
<div class="text-center">
    <h1 class="display-4">Get thumbnail</h1>
 
    @if(ViewBag.Error != null)
    {
        <div class="alert alert-danger">
            @ViewBag.Error
        </div>
    }
 
    <form method="post" enctype="multipart/form-data">
        <input type="file" name="file" /><input type="submit" value="Send" />
    </form>
    
    @if(ViewBag.Image != null)
    {
        <div style="text-align:center">
            <p><img src="@ViewBag.Image" class="thumbnail" /></p>
        </div>
    }
</div>

Important things to notice in view:

• if something went wrong with thumbnail service then we will show error message to user,
• we have file upload form (don’t forget enctype attribute),
• if we got thumbnail then we will show it as an inline image.

We need two Index actions in HomeController. First one just returns view when we move to page. Second one takes picture, processes it and gives results to same view.

public class HomeController : Controller
{
    private readonly IComputerVisionClient _visionClient;
 
    public HomeController(IComputerVisionClient visionClient)
    {
        _visionClient = visionClient;
    }
 
    public IActionResult Index()
    {
        return View();
    }
 
    [HttpPost]
    public async Task<IActionResult> Index(IFormFile file)
    {
        try
        {
            using var stream = file.OpenReadStream();
            using var result = await _visionClient.GenerateThumbnailInStreamAsync(200, 200, stream);
            using var mem = new MemoryStream();
 
            await result.CopyToAsync(mem);
 
            var base64 = Convert.ToBase64String(mem.ToArray());
            ViewBag.Image = String.Format("data:image/png;base64,{0}", base64);
        }
        catch(HttpOperationException opex)
        {
            ViewBag.Error = opex.Response.Content;
        }
 
        return View();
    }
 
    [ResponseCache(Duration = 0, Location = ResponseCacheLocation.None, NoStore = true)]
    public IActionResult Error()
    {
        return View(new ErrorViewModel { RequestId = Activity.Current?.Id ?? HttpContext.TraceIdentifier });
    }
}

One may ask why I’m copying image bytes from result stream to MemoryStream? What we get back is ContentLengthReadStream and it is internal class for ASP.NET Core. It doesn’t have Position property implemented. If we want to get array then we have to write methods that read bytes from result array and copy these to some other array.

MemoryStream is the shortest way to byte array. Anyway if you use this code in real project you have requirement to save thumbnails somewhere. No matter if it’s local file system or cloud storage you need stream where contents of result stream are copied.

Creating thumbnails

There’s small old trains museum in Estonia, near summer capitol Pärnu. Nothing big or glorious but still cool for those interested in history or trains. For test drive I take on photo made there.

This photo is good candidate because for square thumbnail I want right part of photo where my dear daughter is sitting with her toy bunny that I used on the way to museum to scare away flying bugs. Anyway, sorry for a little bit morbid picture :)

With default settings we get center part of image as thumbnail.

I don’t have any complaints on thumbnail quality but I want better positioning.

Using smart cropping

Azure thumbnail service has a feature called smart cropping. Smart cropping is about finding the best part of image for thumbail. Think about it as an automatic moving of cropping rectangle.

All we need to do is to enable smart cropping in our code.

[HttpPost]
public async Task<IActionResult> Index(IFormFile file)
{
    try
    {
        using var stream = file.OpenReadStream();
        // use smart cropping
        using var result = await _visionClient.GenerateThumbnailInStreamAsync(200, 200, stream, true);
        using var mem = new MemoryStream();
 
        await result.CopyToAsync(mem);
 
        var base64 = Convert.ToBase64String(mem.ToArray());
        ViewBag.Image = String.Format("data:image/png;base64,{0}", base64);
    }
    catch(HttpOperationException opex)
    {
        ViewBag.Error = opex.Response.Content;
    }
 
    return View();
}

Now we get better and more intelligent result.

It’s exactly the thumbnail I want for photo above.

Handling thumbnail service errors

Errors from thumbnail service are given us as HttpOperationException. There’s Response.Content property where details about error are stored. It’s just a string and in case of computer vision service the error is given as a JSON string.

Let’s download too big image to see some real error in service.

As we see then we can show message attribute (“Input image is too large”) to end users and write message with all other details to system log.

For this specific error there’s also hack to use to make image smaller so thumbnail service accepts it. We can check if image width is greater than let’s say 1200 and if it is then we can just make it smaller so it gets also smaller in size.

If you want to apply this hack then you can start from my blog post Recognizing printed text on images using Azure Computer Vision API. It may also help to see ImageUsingBaseController class I wrote for demo application (classic ASP.NET MVC).

Wrapping up

With Azure Cognitive Services account and few lines of code we created powerful solution to generate thumbnails for images and photos. And even better – most of users doesn’t need to open view with original image and cropping box as thumbnail service makes its best bet on which part of photo should get to thumbnail. For those who just want to play with Azure Computer Vision services there are also free accounts available in Azure Portal.