The project
Salsa updated Queensland’s data portal ( ) for the DHPW. The new CKAN portal is now more secure, more stable, and easier to maintain — while also providing a more interactive experience to drive open data use. View case study
The giveback pledge
Salsa’s giveback pledge is that we will contribute 20% of open source work back to the client. This can be used by the client towards the project or for any future work. For Queensland’s data portal, the 20% ‘credit’ was used for additional hours towards modifications DHPW requested at the user testing stage, and to generally provide a more comprehensive solution than the initial scope.
The giveback contributions
The giveback contributions for Data Qld fall into three main categories:
- New CKAN extensions
- Feature improvements to existing extensions
- Snippets and tid-bits
New CKAN extensions
Salsa created four new extensions:
- ODI Certificate extension — Salsa developed a CKAN extension that queries the ODI certificate and displays relevant information, if available.
- Google Analytics and APIs — This extension tracks API requests via events in Google Analytics.
- New default order — When a new resource is added it goes to the top of the list.
- CKAN package group permission — Allows any organisational admin or editor to assign datasets to groups without needing to belong to each group.
Feature improvements to existing extensions
Most of our giveback activities were feature improvements to existing extensions, with 10 contributions in this area. A lot of this work focused on the comments feature, including a profanity filter, the ability to mute/unmute comments, and email notifications.
Snippets and tid-bits
These are useful snippets of code that don't always warrant being a separate CKAN extension. We made two code contributions that fall into this category.
Full list of CKAN contributions
1. New CKAN extensions
In cases where a new CKAN feature, or enhancement provides functionality that was non-site specific and could be isolated, we have created a new CKAN extension to contribute back to the CKAN community.
DQL-154 : ODI Certificate Extension
The Open Data Institute (ODI) provides a free online tool for users to verify dataset quality. Datasets submitted to the ODI certificate tool are given one of four certificates (badges) depending on the level of assessment information provided.
Salsa developed a CKAN extension that queries the ODI certificate and displays relevant certificate information on the dataset detail page in CKAN if available:
This extension has been submitted to the CKAN extension registry via:
DQL-164 : GA event tracking of API endpoints with additional data for category, label & value
Our client’s project uses Google Tag Manager (GTM) to track frontend page views and activities. Unfortunately this does not allow tracking of API requests as those occur in the backend.
The standard ckanext-googleanalytics extension ( ) contains functionality to track API requests. The relevant code for this functionality was extracted and re-purposed into a new CKAN extension that has been contributed back to the CKAN community via this repository:
This extension tracks API requests via events in Google Analytics.
DQL-166 : When a new resource is added it goes to the top of the list
By default, when a new resource is added to a dataset in CKAN it appears last in the list of resources associated with that dataset. From a user perspective this means for datasets with many resources, the user has to scroll to the bottom of the resource list to see the latest resources.
Our client expressed a preference to have any newly added resources appear at the top of the list automatically, without having to manually reorder a dataset's resources.
The functionality to achieve this was documented in a Github Gist here:
The code from this Gist was also separated out into a new CKAN extension that can be utilised by other CKAN developers with the same requirement here:
DQL-165 : CKAN Package Group Permission - Allow any org admin or editor to assign datasets to groups without needing to belong to each group
Datasets in CKAN can be logically grouped into categories (groups) by associating the dataset with one or more groups. Out of the box, a user must be assigned a role within a group in order to be able to assign a dataset to that group.
Our client required the ability for any admin or editor users of an organisation to be able to assign their datasets to any group with CKAN, without the user having to be specifically assigned to all groups.
This feature has also been requested by users in the CKAN community, for example in this Stack Overflow question:
Salsa developed a new CKAN extension to solve this requirement:
The new extension was submitted to the CKAN extension registry via:
2. Feature improvements to existing extensions
This category forms the bulk of our giveback activities. Based on functional requirements for our client's project, many 3rd party extensions were utilised.
In some cases an existing extension provided 90% of the functionality required, with additional functionality added to meet the requirements.
Where possible we have genericized functional enhancements in order to submit them back to the original extension via pull requests.
ckanext-ytp-comments
The most commonly used approach for adding commenting functionality to CKAN is to use the CKAN Disqus extension. Disqus is a 3rd party service that provides a platform for adding commenting functionality. Due to the need for our client to retain ownership of comment data, Disqus was not feasible for their project.
An existing CKAN comment extension was identified (ckanext-ytp-comments), with an understanding that it was no longer actively maintained and functionally incomplete.
Salsa implemented numerous functional enhancements and additions to meet our client's requirements, implementing a basic suite of features common to commercial comments extensions.
Where possible these have been contributed back to the original extension via a series of pull requests.
DQL-157 : Frontend validation for comment submission forms
One of the functionally incomplete aspects of YTP Comments was that there was no front-end validation to prevent submission of empty comments.
Salsa addressed this and submitted a pull request via:
This pull request implements basic front-end validation was added to prevent submission of empty comments.
Our pull request was accepted and merged into the master branch of the upstream extension repository on 05/02/2020.
Approach
Comment submissions are validated on the front-end via jQuery to prevent empty, or all whitespace entries from being submitted.
Error message HTML element added to both submit and edit comment entry forms to display error relating to empty submission,e.g.
DQL-156 : Comments - Restricting thread levels
One of the functionally incomplete aspects of YTP Comments was that there was no limit to the number of threads/sub-comments a comment could contain.
Salsa addressed this and submitted a pull request via:
This pull request implements a configuration setting and supporting code to limit the depth of a comment thread based on the configuration setting.
Approach
- Add a configuration option to the CKAN .ini file (ckan.comments.maximum_thread_depth) to restrict the number of threads comments can extend to.
- Control the thread depth limit on the front-end based on the limit set in the CKAN .ini configuration file, e.g. ckan.comments.maximum_thread_depth = 3, e.g.
DQL-158 : Profanity filtering on submitted comments
One of the functionally incomplete aspects of YTP Comments was that there was no prevention of submitting or editing of comments containing profanity.
Salsa addressed this and submitted a pull request via:
This pull request implements blocking of submission comments or edits that contain profanity and displays an error message to the user if their submission is blocked.
Approach
The Python profanityfilter library is used to validate comment submissions and edits against a list of bad words.
If a user submits a comment, or edits a comment to contain profanity in the subject or comment text, it is blocked and an error message appears, e.g.
DQL-159 : Comment submission email notifications
This project required that dataset owners be emailed when new comments were added to their datasets.
Salsa addressed this functional requirement and submitted a pull request via:
This pull request adds functionality to email dataset owners when a new comment has been added to their datasets.
Approach
Two new configuration settings were added to the CKAN .ini file:
- to enable email notifications: ckan.comments.email_notifications_enabled (Defaults to False)
- to set who should receive the email notification: ckan.comments.email_notification_to (can be set to author, or maintainer. Defaults to author)
A notification email is sent to the author or maintainer of the dataset based on the CKAN .ini file setting ckan.comments.email_notification_to, to inform them that a new comment has been added to their dataset.
Email notification subject and body templates were added to the extension in the /templates/emails/ directory.
The CKAN job is used to send emails to avoid latency between comment submission and front-end display of updated comments.
DQL-155 : Extend Comments extension to support comments for Data Requests
This project required that the same commenting functionality used for datasets would also be used for commenting on data requests.
Salsa addressed this functional requirement and submitted a pull request via:
This pull request adds functionality to support comments on data requests using the YTP comments extension..
Approach
The concept of `content_type` was introduced to the extension to support content types such as data requests, in addition to existing dataset commenting.
Helper functions were created to address the differences between routes and permissions of datasets and data requests.
Template overrides were added for the data requests comment and show templates.
CKAN routes were added to handle generic display, submission, editing and deleting of comments, with `dataset` being the default content type.
DQL-160 : Mute and follow comments
This feature allows users to follow or mute comments at the content item level or for a specific comment thread on the content item.
When a user follows comments on a content item or content item thread they will receive email notifications when new comments or replies are posted.
Comment notifications (via email) are managed by opt-in, i.e. without opting in to receive comment notifications at the content item or thread level, only authors or organisation admins will receive email notifications.
Salsa developed this functionality and contributed back via a pull request to the original YTP extension:
Approach
- Added a new setting in CKAN `.ini` file `ckan.comments.follow_mute_enabled` to enable follow/mute UI (default: False).
- A new database model (`comment_notification_recipient`) was added to the extension. A paster command is also added to initialise the database table in the CKAN postgres database.
- Comment follow and mute routes were added to the extension, along with a controller class to handle requests via those routes.
- A helper function file was created to isolate required functionality and enable reuse of those functions.
- The comment list template was adjusted to add follow/mute related markup via template snippets, along with the necessary CSS, Javascript code and email notification templates.
- When a user adds a new comment, or replies to an existing comment, they will automatically follow that thread.
ckanext-datarequests
The CKAN Data Requests extension allows users to ask for datasets that are not already published in a CKAN instance.
This extension satisfied 95% of our client’s requirements out of the box.
Additional functionality and enhancements were developed by Salsa to fulfill remaining requirements and contributed back to the CKAN community.
DQL-161 : Data Requests - Configuration to make description mandatory
One of the functionally incomplete aspects of Data Requests was that there was no option to set the description field as mandatory.
Salsa addressed this functional requirement and submitted a pull request via:
Purpose
This pull request adds functionality to set description field as mandatory.
Approach
Configurable 'description_required' option in CKAN config file (default is False)
If 'description_required' is set to True, when a user is creating or updating a data request, the description field is mandatory.
An error message is displayed on front end for invalid entries.
DQL-162 : Ability to Re-open Data Request
One of the functionally incomplete aspects of Data Requests was that there was no option to reopen data requests once they were closed.
Salsa addressed this functional requirement and submitted a pull request via:
Purpose
This pull request adds functionality for data request creator's to reopen their closed data requests.
Approach
Configurable 'allow_datarequest_reopening' option in CKAN config file (default is False)
If allow_datarequest_reopening is set to True, the user is viewing a closed data request, a 'Reopen' button is displayed beside the 'Manage' button.
Clicking the "Reopen' button will open the data request and send a notification email to admin users of that organisation.
3. Snippets and tid-bits
These are useful snippets of code that don't always warrant being a separate CKAN extension.
These have been provided as Github Gists and/or answers to Stack Overflow questions.
DQL-153 : Hide schema metadata if schema not attached to a resource
ckanext-validation is an incredibly useful, and flexible extension that provides data validation functionality when adding resources to a dataset.
By default, when no validation schema is added to a resource, the validation related fields are still displayed with the resource on the CKAN front-end.
This simple Github Gist details how to conditionally hide those validation fields if they have not been utilised:
DQL-163 : Google Analytics - Identify what dataset/resources belong to which organisations
For organisations that utilise GTM tracking on their CKAN website it can be difficult to associate a dataset or resource with its respective organisation in GA.
By adding a simple meta tag to the HTML output on the dataset and resource detail pages, a GTM administrator can make this connection in their tracking statistics.
Our method for doing this has been provided as a simple Github Gist here:
