Policy consortium — Rules as Code policy modelling and anomaly detection

The project’s challenge

Anna and Pia wanted to bring together a group of academics, government representatives and industry experts to explore the intended and unintended impact of policy for adaptive policy management. The project was split into two streams:

1. Research (academia led by Anna)
2. Prototypes/demonstrators (government and industry led by Pia)

Pia needed industry vendors to explore different use cases and create prototypes.

The transformation

Salsa Digital worked on an anomaly detection use case. We leveraged our existing Rules as Code experience and a digital policy twin of the student visa (500) that we’d created in collaboration with the Department of Home Affairs as part of the GovCMS Rules as Code sandpit program. Salsa then built a policy monitoring feature on top of the native out-of-the-box OpenFisca simulations. Our demonstrator detects if student visa submissions fall above or below the expected thresholds. If they do, an email alert is sent to highlight the unintended consequences of policy changes.

The outcomes

• A demonstrator/prototype that supports the consortium’s theme for adaptive policy management
• A policy monitoring prototype that flags any unintended consequences of policy
• An invaluable tool for data-driven decision making
• Email alerts so relevant stakeholders know when anomalies occur
• A proof of concept that can be expanded across many other use cases

Diagram showing the prototype’s process flow: digital policy twin, then policy simulations using synthetic data (in a real-world scenario you’d use real data), then if the results are outside expected thresholds an email alert is sent to the appropriate policy team.

“When I was the Chief Data Officer at Home Affairs, we wanted to innovate with adaptive policy management and improve our capability for modelling policy options and monitoring policy impact. Our data scientists and services team worked well with Salsa Digital on this prototyping project, bringing together different skillsets and expertise to create a meaningful example of what is possible when you combine rules as code, data science and adaptive policy methods. I hope it inspires more public servants and departments to innovate and evolve their approach to proactively optimising policy settings for the best possible public good and policy outcomes. There is a growing appetite around the world for improving policy efficacy and investing for impact.”

Pia Andrews
Rules as Code advocate

Detailed case study

Below is detailed information on the challenge, transformation and final outcomes/benefits.

The challenge — showing the art of the possible

The project leaders wanted to bring together a group of academics, government representatives and industry experts to explore the intended and unintended impact of policy for adaptive policy management.

The project was split into two streams:

1. Research (academia led by Anna)
2. Prototypes/demonstrators (government and industry led by Pia)

Pia needed industry vendors to explore different use cases and create prototypes.

The project’s transformation — a prototype of policy monitoring

Salsa Digital worked on an anomaly detection use case. We started the project thinking about the flow from policy, to modelling, to monitoring, to anomaly detection and alerts.

Next, we needed to find a good use case. This proved a little challenging, but eventually we decided to approach the Department of Home Affairs to see if we could use its digital policy twin of the student visa (500). This digital policy twin was created by Salsa and the Department of Home Affairs as part of the GovCMS Rules as Code sandpit program.

Once we had permission, the next step was to set up the OpenFisca simulations (modelling) on top of the policy twin. We modelled/simulated expected changes in eligibility based on:

• Change in English language standards
• Change in financial capacity
• Change in health requirements

For example, perhaps increasing the English language requirements is expected to lead to a 10% decrease in student visa eligibility (from the policy modelling) but then through policy monitoring you discover it’s led to 22% decrease.

For the prototype we generated synthetic data, but in a real-world scenario you’d use historic data. The process followed three steps:

1. Run simulations
2. Aggregate requirements
3. Set expected (baseline) values and acceptable threshold percentages

The next step was to build the policy monitoring tool. The policy monitoring workflow is:

• Get baseline values and thresholds
• Collect submission data (or generate synthetic data)
• Run policy simulations (OpenFisca)
• Detect anomalies (custom Python)
• Generate alerts (if anomalies detected)
• Generate reports

As part of the setup, you can choose when you run the monitoring process, e.g. monthly, daily, etc. The monitoring compares the expected results (from policy modelling/simulations) to the actual results.

You can also calculate deviation severity. For example, in the prototype we nominated +/- 5% as the threshold with the following severity alerts:

• Critical: if the deviation percent is greater than or equal to 3 times the threshold
• High: if the deviation percent is greater than or equal to 2 times the threshold
• Medium: if the deviation percent is greater than or equal to 1.5 times the threshold

Email alerts indicate the exact % deviation and the status.

For example, perhaps you expected the student visa eligibility to decrease by 10% with higher English language requirements, but it actually decreased by 22%. This deviation would be picked up by our policy monitoring.

The outcomes — highlighting unintended consequences of policy

• A demonstrator/prototype that supports the consortium’s theme of intended and unintended impact of policy for adaptive policy management
• A policy monitoring prototype that clearly shows unintended consequences of policy
• Email alerts so relevant stakeholders know when anomalies are detected
• An invaluable tool for data-driven decision making
• The ability to set the policy monitoring timeframe (e.g. check submissions against expected outcomes daily, weekly, monthly, etc.)
• The ability to tailor thresholds and email alert severity levels
• A proof of concept with the potential to be expanded across many other use cases