Building a CKAN Release Timeline
Setting up an interactive timeline site through Python scripting & TimelineJS.
Last updated: 2024-01-05
A version of this blog post has also been republished on datHere's blog.
Let's walk through the process of setting up an interactive timeline (linked to at the bottom of this blog post) below) based on the CKAN data management system's version release history.
BackgroundWhile exploring new tools for developing a presentation of my work during June as a Data Engineering Intern at datHere, I found TimelineJS while searching on GitHub for showcasing my work in a timeline format. I customized the timeline styles locally and integrated the timeline as the final slide in my presentation I built with sli.dev (you may view the archived slides here). During the presentation, our team at datHere decided to make a timeline using TimelineJS for the open-source data management system CKAN as part of the Pathways to Enable Open-Source Ecosystems (POSE) project.
Setting Up the Timeline TemplateTimelineJS has step-by-step instructions on how to make a timeline website using a spreadsheet from Google Sheets as your data source, so I simply followed the steps.the steps: Your timeline website also updates after making changes to the spreadsheet, and by following the steps you'll get a link to your hosted timeline and embed code for inserting the timeline in blog posts, websites, and more.
Considering Our OptionsThere are two main ways to import data into a timeline: Google Sheets or JSON. Considering Google Sheets to be more user-friendly and collaborative than working on a JSON file locally, we decided on the former as our data source. But two questions remained:
- From where should we source the CKAN release history?
- How do we convert the changelog data to a usable format for our spreadsheet?
Sourcing the Release DataAn initial suggestion from the team was to gather the commit history from the repository and port them into the timeline. As of the time of writing this article, there are 24,906 commits in the master branch of the ckan/ckan GitHub Repository. This would result in a timeline that could be hard to follow (commit messages aren't always the most descriptive) were we to continue with this idea without further improvements (e.g. using a large-language model to generate release summaries based on the commit data). Conveniently, there is a
CHANGELOG.rst file in the GitHub repository for CKAN consolidating all release versions including each of their dates and changes made in a more readable format. The file is written in a format similar to Markdown, so I decided to build a Python script.
Converting the Changelog Data to CSV formatThe following is the final iteration of a Python script to convert the
CHANGELOG.rst data into CSV format which we could import into our spreadsheet.
The first iteration of the script did not convert the data into HTML format but instead provided the raw changelog output. We learned that TimelineJS allows for rendering HTML in certain columns including a row's
Text field, which includes the description for a release of CKAN. Therefore, I used the
markdown package to convert each line into HTML format since the input was akin to Markdown styling.
CaveatsThere are some caveats with this method of conversion (i.e., improvements may be made to the script and overall process).
- Most of the timeline had formatting but there are places with custom formatting and other syntax (perhaps related to
.rstformat or for rendering the data on a changelog page on CKAN's docs site) that were not covered in the conversion properly.
- Since we're converting the document to HTML line by line, the output can result in redundancy (like using a separate
<li>), though this is fine since we can reuse the script by writing the release description in Markdown if revising it.
- Versions 0.1 and 0.2 did not have a
dayvalue so they were removed from CHANGELOG.rst and manually inputted into the timeline (though this can be fixed by editing the script).
- The timeline is not in sync with the
CHANGELOG.rstfile, so any modifications made would need to be updated in the Google Sheet including new version releases. An automation system is definitely possible though.
- Not all releases have been manually verified, and there may be formatting issues due to .rst and .md having similarities and differences in syntax.