Last Updated: 15 August 2023

Asp.Net Core 6.0 Sitemap.xml & Robots.txt Generator

Background

This package provides automatic generation of sitemap.xml and robots.txt for ASP.NET Core 6 applications. Both of these files are considered essential for good Search Engine Optimization (SEO) and are looked upon favourably by search engines such as Google and Bing. The latest version of the package targets .Net 6.

Save the time and hassle of manually maintaining your sitemap files, by using the ASP.NET routing tables to generate a standards-compliant sitemap automatically. If you have routes you do not want to appear in your sitemap, simply decorate the Controller class or Action method with the exclusion attribute [SitemapExclude].

Extensibility

Asp.Net Core 6.0 Sitemap and Robots Generator is extensible. Out of the box, you will get URLs that are automatically discovered from your routing table. You can also create your own URL providers either to augment these generated URLs, or to replace them completely. For example, you can manually add URLs to your sitemap, including Video Sitemaps.

By default, requesting robots.txt will return you a basic file pointing to your generated site map. You can easily modify this behaviour by implementing your own robots provider classes.

Http or Https

By default, your sitemap URLs will use the same scheme (i.e. http or https) as the incoming request. In some circumstances, such as running behind nginx or a reverse proxy, the request may appear to your app as if it is http, but the end user sees https. In which case, you can use the 'force https' option to always use secure URLs in your sitemap.

Getting Started

Sitemap is implemented as ASP.NET middleware, and works by looking at all controller routes for valid pages that should be included in the xml output. By default this will only include GET verbs. If no explicit attribute is defined on your controller method, it will be assumed to be a GET.

Any requests to /sitemap.xml or /robots.txt will be handled by the Sitemap middleware. The robots file will contain a link directing robots (e.g. Googlebot) to your sitemap.

Once you have installed the Nuget package, add this line to Startup.cs to add Sitemap to your application:

public void ConfigureServices(IServiceCollection services)
{
    // ... other code omitted ...

    services.AddSitemap();

    // ... other code omitted ...
}

This will register the Sitemap middleware, along with any custom URL providers and config.

Now add this line to Startup.cs to enable Sitemap:

public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{
    // ... other code omitted ...

    app.UseSitemap();

    // ... other code omitted ...
}

Excluding Controller From Sitemap

To exclude a controller and all actions within it, such as an error or login controller, decorate the controller class with the [SitemapExclude] attribute:

// All routes in this controller will be ignored
[SitemapExclude]
public class BlahController : Controller
{
    [Route("some-route")]
    public IActionResult Something()
    {
        return View();
    }
}

Excluding Specific Actions/Routes From Sitemap

To exclude only certain actions, routes, or pages from your sitemap, decorate the action method with the [SitemapExclude] attribute:

public class BlahController : Controller
{
    [SitemapExclude]
    [Route("some-route")]
    public IActionResult Ignored()
    {
        return View();
    }

    [Route("some-other-route")]
    public IActionResult NotIgnored()
    {  
        return View();
    }
}

Excluding Razor Page From Sitemap

To exclude a Razor page from appearing in the sitemap, you can also add the [SitemapExclude] attribute there too:

[SitemapExclude]
public class ErrorModel : PageModel
{
    // Other code...
}

Customizing The Config

Default config is created with sensible defaults. If you wish to provide your own config, e.g. if you want to force https on all sitemap URLs, then create a class that implements ISitemapOptions. You do not need to register it with your DI container, this is done automatically for you.

public class MySitemapOptions : SitemapOptions
{
    public bool ForceHttps => true;
}

Create Custom Url Providers

By default, a Url Provider called RouteBasedSitemapProvider is registered, which uses the routing table to automatically figure out your site's URLs.

You can implement your own providers by creating classes that implement ISitemapUrlProvider or ISitemapVideoUrlProvider. Urls returned by ISitemapUrlProvider will be rendered in the conventional way. Urls returned by ISitemapVideoUrlProvider have more detail around video content and will be rendered as a Video Sitemap Element.

ISitemapUrlProvider

public class DemoSitemapUrlProvider : ISitemapUrlProvider
{
    public IReadOnlyCollection<ISitemapElement> GetElements(string baseUrl)
    {
        return new List<SitemapElement> {new SitemapElement($"{baseUrl}/demo/some-url")};
    }
}

ISitemapVideoUrlProvider

Note that ThumbnailUrl, Title, Description and VideoUrl are required and cannot be empty.

public class DemoSitemapVideoUrlProvider : ISitemapVideoUrlProvider
{
    public IReadOnlyCollection<ISitemapVideoElement> GetElements(string baseUrl)
    {
        return new List<ISitemapVideoElement>
        {
            new SitemapVideoElement($"{baseUrl}/video-element-demo")
            {
                // Required
                ThumbnailUrl = $"{baseUrl}/media/thumbnail.jpg",
                Title = "This is a demo video",
                Description = "A demonstration of how to add video elements to your sitemap",
                VideoUrl = $"{baseUrl}/media/video.mp4",
                // Optional
                DurationSeconds = 180,
                FamilyFriendly = true,
                Live = true,
                PublicationDate = DateTime.UtcNow,
                Rating = 4.9M,
                RequiresSubscription = true,
                ViewCount = 133041,
                Price = 12.99M,
                PriceCurrency = "EUR"
            }
        };
    }
}

By implementing your own providers, you can get URLs for your sitemap from any data source, for example CRM system, JSON file, XML file, SQL database, or even manually hardcode them.

Create Custom Robots.txt Providers

By default, a Robots Provider called DefaultRobotsProvider is registered, which returns the following basic file, pointing to your generated site map:

User-agent: *
Allow: /
Sitemap: <YOUR SITEMAP URL> 

You can implement your own providers by creating classes that implement IRobotsProvider. You can remove the default provider from your DI container if you wish. Providers will be asked for their text entry to the returned robots.txt file in the order they are registered.