SDP — optimising hosting infrastructure clusters for improved performance

Providing Victorians with better website performance and stability

Case study summary: Salsa and amazee.io worked with SDP to optimise the SDP architecture by separating production and non-production (or development) workloads into separate clusters. This ultimately delivers better performance on production websites coupled with significant cost savings because non-production clusters can be scaled down or idled more easily.

On this page:

At a glance

2022
2 to 4 months
Completed
Drupal, Single Digital Presence, LAGOON
State government
Build & migration, Hosting & maintenance, Support & optimisation, Technical advisory
GovTech, Whole of government, Open source, Web development, Headless CMS, Open platform, Content management systems
Multidisciplinary teams, Agile delivery, Tools & systems, Security, Open standards & common platforms, Open source, Measure performance

Overview

SDP’s challenge

Victoria’s Single Digital PresenceExternal Link was migrating from OpenShift AWS to Microsoft Azure AKSExternal Link . However, with production and development websites on one cluster, production sites’ performance could sometimes be affected by high activity on development websites. In addition, the platform’s ability to scale down (and save money) wasn’t being maximised.

More about the challenge

SDP’s transformation

Salsa and amazee.ioExternal Link worked with SDPExternal Link to create two separate clusters — production and non-production. Now all SDP production sites are on a separate cluster to development sites. We also upgraded the database to Azure’s Flexible ServerExternal Link , a more resilient database service.

More about the transformation

The outcomes

  • Improved reliability and performance of live/production websites

  • Cost savings through the ability to scale down, using less nodes overall and less powerful nodes for non-production sites

More about the outcomes

Full case study

Below is more information on the challenge, transformation and outcomes.

SDP’s challenge — competing workloads

Victoria’s Single Digital PresenceExternal Link was migrating from OpenShift AWS to Microsoft Azure AKSExternal Link . While that project was driven by OpenShift 3 coming to end-of-life (and moving to OpenShift 4 would have been quite a big task), it also presented an opportunity to improve the SDP cluster architecture.

The SDP platformExternal Link was built with a scalable architecture. This means when demand is low, some servers could be scaled down or idled, which ultimately reduces costs. However, with both production and development sites running on one cluster the ability to significantly scale down was limited. In addition, if development sites were very active, sometimes the performance on production sites could be impacted. In computer science we call this the ‘noisy neighbour’ effect. If neighbours are taking up a lot of bandwidth (creating noise), this impacts you (your site).

There was an opportunity to improve the architecture by splitting the production and non-production sites into separate clusters.

We drafted a project plan for SDP’s review. The proposal also coincided with some recent changes to Azure, and the project plan included using Azure Flexible ServerExternal Link and Terraform support for Flexible Server.

SDP’s transformation — separate clusters

After our proposal was approved, Salsa worked with amazee.ioExternal Link (our platform partners) and key SDP staff to complete the cluster optimisation and to ensure sites were cleaned-up or migrated with little or no impact on end-users.

The work was broken into 5 main stages:

  1. Planning and costings — to fully scope and cost the project
  2. Migration preparation
  3. Cluster provisioning
  4. Migration
  5. Post-migration tasks

We migrated fixed non-production environments (e.g. develop and master) as there are other parts of the platform that rely on being able to route to the project (CDN) and it was simpler if we handled the whole process for those environments. PR environments were not migrated and were left to developers to recreate as required.

The new architecture can be seen in the diagram below.

The migration effort was quite involved from a project management perspective, with content freezes required while sites were being migrated from the old cluster to one of the new clusters. It was also quite involved technically, with many discrete steps required. To help reduce risk and streamline the migration process, we created migration scripts to automate as much of the process as possible. This reduces the time taken for migration and increases the reliability of the migration process.

During the cluster architecture we also upgraded the database to Azure’s Flexible ServerExternal Link , a more resilient database service.

The outcomes — a better platform

  • More predictability and consistency in the workloads in the production cluster so we can tune the resources better (one benefit of that is improved performance)
  • Improved performance of live government websites (due to the workload changes in the production cluster)
  • Cost savings via a dedicated development cluster that can be scaled-down or idled when not in use

About SDP

Victoria’s Single Digital PresenceExternal Link is the whole-of-government digital platform run by Victoria's Department of Premier and CabinetExternal Link (DPC). DPC is responsible for several elements of Victoria’s digital engagement, including SDP, vic.gov.auExternal Link , data.vic.gov.auExternal Link and engage.vic.gov.auExternal Link .