ASP.NET Search Sitemaps – something for SEO

ASP.NET Futures introduced support for search engine sitemaps. This is content detection protocol introduced originally by Google and later accepted also by Microsoft and Yahoo! ASP.NET makes sitemaps generating easy for us and as a bonus their model is perfect for more complex web applications like e-commerce and e-services sites. Let’s see now what we can do.

Couple of word about search sitemaps

If you have small site with low number of links then managing sitemaps manually is not a problem. But if your site has many links and content that changes often you may want to automate the task of generating sitemap.

Sitemap is important thing for each site because it tells to search engine spiders where to find information from your site. If you give this information there is something you will get back – Google gives you very rich set of information about your site through their Webmasters service.

ASP.NET 2.0 introduced us the new concept – sitemaps. Besides new site navigation definition format we got also providers that were used by navigation controls. ASP.NET search sitemaps technology extends this model and makes our site navigation understandable for search engine spiders.

ASP.NET Search Sitemaps

Let’s take a closer look at search sitemaps now. Suppose we have a web application that uses search sitemaps. As a first thing we will look at sitemap file that defines static navigation of our application.

<?xml version="1.0" encoding="utf-8" ?>
<
sitemap xmlns="http://schemas.microsoft.com/AspNet/SiteMap-File-1.0"
>
  <
sitemapnode url="/" title="Home"  description="Home page"
>
    <
sitemapnode url="about-us.aspx" title="About us" description=""
>
      <
sitemapnode url="corporate.aspx" title="Corporate information"  description=""
/>
      <
sitemapnode url="values.aspx" title="Corporate values" description=""
/>
    </
sitemapnode
>
  </
sitemapnode
>
</
sitemap
>

Now let’s see section from application’s web.config where dynamic part of site’s navigation is defined.

<searchSiteMap enabled="true">
  <
providers
>
    <
add name="Navigation"type="Microsoft.Web.Preview.Search.AspNetSiteMapSearchSiteMapProvider, Microsoft.Web.Preview"
/>
    <
add name="Product"
           type="ASPNETFuturesEnabledWebApplication1.MySitemapProvider, ASPNETFuturesEnabledWebApplication1"
            targetUrl="Products.aspx"
            targetUrlseparator="?"
            pathInfoFormat="false"
            queryStringDataFields="ProductId"
            queryStringDataFormatString="ProductId={0}"
         />
  </
providers
>
</
searchSiteMap
>

As we can see there are two providers given that provide the sitemap system with dynamic data about navigation. First provider is the default one that generates search sitemap based on site’s static navigation.

The second provider was added by me. This provider returns information about my products section navigation. When asked, it will return all links that products section contains. The code is as follows.

public class ProductEntry
{
   
public string
ProductId;
   
public string
ProductName;
   
public String
SiteMapLastModified;
   
public String
SiteMapChangeFrequency;
   
public String
SiteMapPriority;
}

public class ProductSitemapProvider : DynamicDataSearchSiteMapProvider
{
   
public override IEnumerable
DataQuery()
    {
       
List<ProductEntry> products = new List<ProductEntry
>();
       
ProductEntry
entry;

       
foreach (Product product in DAL
.GetProducts())
        {
            entry =
new ProductEntry
();
            entry.ProductId = product.Id.ToString();
            entry.ProductName = product.Name;
            entry.SiteMapLastModified = product.Modified.ToString(
"yyyy-MM-ddThh:mm:ss.fffZ"
);
            products.Add(entry);
        }
       
return products;
    }
}


The class called ProductSitemapEntry defines search sitemap entry that is used with products. Second class, ProductSitemapProvider, is the one used by search sitemaps engine to get information about products. You can handle ProductSitemapEntry as one of the DTO classes for Products.

Search sitemaps structure

Search sitemaps protocol allows also sitemaps hierarchy to be defined. It is specially useful for large sites because we can say on higher level sitemaps which subsitemaps are changed meanwhile. This way we can lower our spidering traffic also.

Search sitemaps HttpHandler that we can find from web.config generates hierarchical search sitemaps. Let’s see what happens when we make request to this handler.

<?xml version="1.0" encoding="utf-8"?>
<
sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
>
  <
sitemap
>
    <
loc>http://localhost:49781/SearchSiteMaps.axd?sitemap=Navigation</loc
>
    <
lastmod>2008-04-20T01:58:41.694Z</lastmod
>
  </
sitemap
>
  <
sitemap
>
    <
loc>http://localhost:49781/SearchSiteMaps.axd?sitemap=Product</loc
>
    <
lastmod>2008-04-20T01:58:41.694Z</lastmod
>
  </
sitemap
>
</
sitemapindex
>

This is pure hierarchical search sitemap that refers to two subsitemaps. These subsitemaps are asked through same handler and name of search sitemap is given by query parameter. When you look at search sitemaps configuration section above you may notice that these query parameters have same values as search sitemaps providers names.

Let’s see now what happens when we ask first sitemap. Remember that the first one corresponds to site’s static navigation shown before. Just copy and paste URL from above XML to your browser.

<?xml version="1.0" encoding="utf-8"?>
<
urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
>
  <
url
>
    <
loc>http://localhost:49781/</loc
>
  </
url
>
  <
url
>
    <
loc>http://localhost:49781/about-us.aspx</loc
>
  </
url
>
  <
url
>
    <
loc>http://localhost:49781/corporate.aspx</loc
>
  </
url
>
  <
url
>
    <
loc>http://localhost:49781/values.aspx</loc
>
  </
url
>
</
urlset
>

Now let’s see what is returned by products search sitemap provider.

<?xml version="1.0" encoding="utf-8"?>
<
urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
>
  <
url
>
    <
loc>http://localhost:49781/Products.aspx?ProductId=100</loc
>
    <
lastmod>2008-04-20T02:02:02.544Z</lastmod
>
    <
priority>0.4</priority
>
  </
url
>
  <
url
>
    <
loc>http://localhost:49781/Products.aspx?ProductId=102</loc
>
    <
lastmod>2008-04-20T02:02:02.544Z</lastmod
>
    <
priority>0.4</priority
>
  </
url
>
</
urlset
>

As we can see there are two products (oh, lazy me!) introduced to search engines spiders.

Conclusion

As we can see the search sitemaps technology is very powerful feature that makes our sites SEO much simpler than it was before. We can add as many search sitemaps providers to our application’s configuration and let them generate search sitemaps. Besides static navigation we may have many different sections in our site. By example, we may have blog, product vatalog, support forum, FAQ pages and so on.

ASP.NET search sitemaps will provide us with common and flexible ways to make our applications support search sitemaps protocol.

Gunnar Peipman

Gunnar Peipman is ASP.NET, Azure and SharePoint fan, Estonian Microsoft user group leader, blogger, conference speaker, teacher, and tech maniac. Since 2008 he is Microsoft MVP specialized on ASP.NET.

    8 thoughts on “ASP.NET Search Sitemaps – something for SEO

    • February 5, 2008 at 11:18 pm
      Permalink

      Well… you have to write your own search sitemap provider that is able to use SqlSitemapMapProvider. All sitemap providers are using same interface and as long as you use this interface you can create search sitemaps based on all providers you have.

      When you are creating search sitemap you have to step through all nodes in given sitemap and add links you got to search sitemap.

    • May 13, 2008 at 5:17 am
      Permalink

      You can also add a line to your robots.txt file to let the search engines know where to find your sitemap. Live Search now has webmaster tools which is where I found the robots.txt trick.

    • May 13, 2008 at 8:52 am
      Permalink

      Yes, robots.txt should say to spiders where sitemap file locates. There are no restrictions set on sitemap file name.

    • May 14, 2008 at 11:38 am
      Permalink

      interesting post.

      I built a site wide solution that you can plug into any existing asp.net site.

      See link under my name.

    • May 14, 2008 at 12:04 pm
      Permalink

      For larger sites you need to use some automatics that generates all the links available. The pages that are called “landing pages” are the only ones that need very careful handling and strong propagation in web. The other pages usually are not considered so important and to get best out of them you should use automatics.

      The interesting thing comes from code design side if we want to integrate search sitemaps support to our applications. It is possible to create common interface for search sitemap entries that all publicly exposed objects implement. In this case we should be able to automate also sitemaps manual management through system’s user interface.

      I mean we may have a custom control that let’s us handle the sitemaps specific information on insert and edit forms.

    • July 23, 2008 at 12:35 pm
      Permalink

      Does anyone know when the Search SiteMaps feature is going to make it into production code rather than being part of the mysterious “Futures” release? The last bit of information I found on futures was July 2007 – since then nothing it seems.

      Thanks in advance.

    • July 23, 2008 at 5:26 pm
      Permalink

      I don’t know exactly but let’s hope that Sitemaps will be available with .Net 3.5 SP1

    • November 3, 2008 at 10:49 pm
      Permalink

      Hi,

      Great article! I know that the AspNetSiteMapSearchSiteMapProvider is used when you have a static “web.sitemap” sitting in your application root, but what if I wanted to use the DynamicDataSearchSiteMapProvider instead for navigation URLs? The reason is becuse I am using Jeff Prosise’s SqlSitemapMapProvider, and have no idea where to go from here! Could you shed a little light on this please.

      Thanx!

    Leave a Reply

    Your email address will not be published. Required fields are marked *