TL/DR?

If you don't have much time you can skip to the conclusion

Watch the video

Look through the slides

You can download the slides

App Service

Each environment was hosted in their own Azure App service plan to keep them isolated from each other. We had three app service plans in total, one for Umbraco 8 and two for Umbraco 9 (Windows/Linux).

Due to slight differences in Linux and Windows App Service plans near the cheaper tiers, we only used B1, S1, P1V2 and P2V2. You could get cheaper hosting in Windows with D1:Shared but Linux doesn’t have an equivalent tier.

Database

Each site has the same database in SQL Azure running at S2 Tier as recommended by the Umbraco documentation.

Deployment

For Umbraco 9 we had different build/deployment pipelines for Linux and Windows the main difference being the host operating system for the pipeline.

Umbraco Content

Each Umbraco site was running Paul Seal’s ‘Clean Starter Kit’ and each site had 100 content nodes.

K6 We used k6.io to run our stress tests, K6 allows you to write your tests using JavaScript, the same script was used for each environment. Here is the JavaScript we used: import { sleep, group } from "k6"; import http from "k6/http";

export const options = {
ext: {
    loadimpact: {
    distribution: {
        "amazon:gb:london": { loadZone: "amazon:gb:london", percent: 100 },
    },
    },
},
scenarios: {
    login: {
    executor: 'ramping-vus',
    startVUs: 0,
    stages: [
        { duration: '5m', target: 100 },
        { duration: '10m', target: 200 },
        { duration: '15m', target: 500 },
        { duration: '15m', target: 1000 },
        { duration: '15m', target: 0 }
    ],
    gracefulRampDown: '0s',
    },
}
};

export default function main() {
let response;

var appServiceName = "name-of-web-app";

group(
    "main pages - https://" + appServiceName + ".azurewebsites.net/",
    function () {
    var pages = ["/", "/about/", "/blog/", "/contact/", "/search/", "/blog/popular-blogs/", "/blog/conferences/", "/blog/meetups/", "/blog/community/"]
    var pageCount = pages.length;

    for (var i = 0; i < pageCount; i++) {
        response = http.get(
        "https://" + appServiceName + ".azurewebsites.net" + pages[i],
        {
            headers: {
            "upgrade-insecure-requests": "1",
            "sec-ch-ua":
                '"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"',
            "sec-ch-ua-mobile": "?0",
            "sec-ch-ua-platform": '"Windows"',
            },
        }
        );
        sleep(1);
    }
    }
);

group(
    "test pages - https://" + appServiceName + ".azurewebsites.net/",
    function () {
    for (var i = 1; i <= 200; i++) {
        response = http.get(
        "https://" + appServiceName + ".azurewebsites.net/about/" + i + "/",
        {
            headers: {
            "upgrade-insecure-requests": "1",
            "sec-ch-ua":
                '"Google Chrome";v="95", "Chromium";v="95", ";Not A Brand";v="99"',
            "sec-ch-ua-mobile": "?0",
            "sec-ch-ua-platform": '"Windows"',
            },
        }
        );
        sleep(1);
    }
    }
);

// Automatically added sleep
sleep(1);
}

The test ran the following stages one after the other:

• Ramp up to 100 users in 5 minutes

• Ramp up to 200 users in 10 minutes

• Ramp up to 500 users in 15 minutes

• Ramp up to 1000 user in 15 minutes

• The last 15 minutes we gracefully ramp down to 0 users.

Each user was making 200 requests to different pages of Umbraco content.

Conclusion

• Overall Umbraco v9 is between 1.5x and 2x more performant than Umbraco v8.
• If you upgrade your v8 site to v9 you can halve your hosting costs.
• On the B1 tier, Linux was 4x cheaper than Windows but it had very similar results.
• On the S1 tier, both Windows and Linux were similar on performance and price.
• On P1V2, Windows was almost double the cost of Linux but only 10 - 20% more performant. 
• On P2V2 both Windows and Linux performed almost the same but Windows was nearly double the cost of Linux.
• During setup, we noticed Linux has a longer cold boot time so that is worth bearing in mind.