Mueez Khan
Building a CKAN Release Timeline
Setting up an interactive timeline site through Python scripting &
TimelineJS.
Published: 2023-07-20
Last updated: 2024-01-05
work
python
data-engineering
scripting
A version of this blog post has also been republished on datHere's blog.
Let's walk through the process of setting up an interactive timeline (linked to at the bottom of this blog post) below) based on the CKAN data management system's version release history.
Background
While exploring new tools for developing a presentation of my work during June as a Data Engineering Intern at datHere, I found TimelineJS while searching on GitHub for showcasing my work in a timeline format. I customized the timeline styles locally and integrated the timeline as the final slide in my presentation I built with sli.dev (you may view the archived slides here). During the presentation, our team at datHere decided to make a timeline using TimelineJS for the open-source data management system CKAN as part of the Pathways to Enable Open-Source Ecosystems (POSE) project.Setting Up the Timeline Template
TimelineJS has step-by-step instructions on how to make a timeline website using a spreadsheet from Google Sheets as your data source, so I simply followed the steps.the steps: Your timeline website also updates after making changes to the spreadsheet, and by following the steps you'll get a link to your hosted timeline and embed code for inserting the timeline in blog posts, websites, and more.Considering Our Options
There are two main ways to import data into a timeline: Google Sheets or JSON. Considering Google Sheets to be more user-friendly and collaborative than working on a JSON file locally, we decided on the former as our data source. But two questions remained:- From where should we source the CKAN release history?
- How do we convert the changelog data to a usable format for our spreadsheet?
Sourcing the Release Data
An initial suggestion from the team was to gather the commit history from the repository and port them into the timeline. As of the time of writing this article, there are 24,906 commits in the master branch of the ckan/ckan GitHub Repository. This would result in a timeline that could be hard to follow (commit messages aren't always the most descriptive) were we to continue with this idea without further improvements (e.g. using a large-language model to generate release summaries based on the commit data). Conveniently, there is aCHANGELOG.rst
file in the GitHub repository for CKAN consolidating all release versions including each of their dates and changes made in a more readable format. The file is written in a format similar to Markdown, so I decided to build a Python script.
Converting the Changelog Data to CSV format
The following is the final iteration of a Python script to convert theCHANGELOG.rst
data into CSV format which we could import into our spreadsheet.
The first iteration of the script did not convert the data into HTML format but instead provided the raw changelog output. We learned that TimelineJS allows for rendering HTML in certain columns including a row's
Text
field, which includes the description for a release of CKAN. Therefore, I used the markdown
package to convert each line into HTML format since the input was akin to Markdown styling.
Caveats
There are some caveats with this method of conversion (i.e., improvements may be made to the script and overall process).- Most of the timeline had formatting but there are places with custom formatting and other syntax (perhaps related to
.rst
format or for rendering the data on a changelog page on CKAN's docs site) that were not covered in the conversion properly. - Since we're converting the document to HTML line by line, the output can result in redundancy (like using a separate
<ul>
for each<li>
), though this is fine since we can reuse the script by writing the release description in Markdown if revising it. - Versions 0.1 and 0.2 did not have a
day
value so they were removed from CHANGELOG.rst and manually inputted into the timeline (though this can be fixed by editing the script). - The timeline is not in sync with the
CHANGELOG.rst
file, so any modifications made would need to be updated in the Google Sheet including new version releases. An automation system is definitely possible though. - Not all releases have been manually verified, and there may be formatting issues due to .rst and .md having similarities and differences in syntax.