The evolution of the SciPy developer CLI
Published May 03, 2022
🤔 What is a command-line interface (CLI)?
Imagine a situation, where there is a massive system with various tools and functionalities, and every functionality requires a special command or an input from the user. A CLI is designed to tackle such situations. Like a catalog or menu, it lists all the options available, thus helping the user to navigate a complex system.
Now that we understand what a
CLI is, how about we dive into the world of
🤓 SciPy CLI journey
An open-source project undergoes multiple iterations, and each iteration adds layers of tools and functionalities to it. Modules like build, test, benchmarking, release notes, etc are the building blocks of an open source project. With time, the contribution guides start to span over multiple pages, and the over all effort to maintain the project grows exponentially.
The two homegrown CLI for SciPy are
runtests.py(for distutils-based builds) and
dev.py(for Meson-based builds) developed using python tooling
Both of these tools have long chains of conditional statements, which are notorious for reducing code readability drastically. It is harder to find the right code block in the chain to modify, hence code maintainability becomes a challenge. The documentation runs into an infinite loop of updates, requiring additional efforts from maintainers. Another issue that remains is the lack of task grouping. A group of tasks linking to a single objective helps reduce confusion, super helpful for new contributors (like me :D).
Thus the idea of a developer command-line interface (CLI) was born, easing the development experience with an intuitive and informative CLI. In addition, we also removed dependency on legacy tooling like
paver, which added great value to the overall experience. More details could be found under issue-#15489.
✍️ Planning and objective
Like any development activity, the plan was to experiment with available tools. doit and Typer were the first ones we picked. The two components of our interests were a task runner, and a command-line interface tool. doit satisfied the requirements of a task runner along with added functionality, like maintaining a task dependency graph as a DAG. While
Typer is quick to get started with building CLI applications.
As a starting point, I began experimenting with existing
dev.py options, wrapped around individual
Typer tasks. Both the doit, Proof of concept (POC) and Typer POC were developed by wrapping a few selected options available under
As I progressed with the development of POCs using both tools, I experienced certain shortcomings. A better way to integrate an external library for exposing a CLI was the missing piece. Eduardo Naufel Schettino the author of doit developed an architecture combining the core elements of
doit along with
click. Henceforth we continued with a
doit-click approach, the journey is captured in detail under issue-#133.
💁🏽♀️ More about the architecture and core components
Combining these tools wasn’t a straightforward journey; after multiple iterations, we were able to achieve a stable state. doit underwent updates to incorporate added functionalities, helping the pieces come together. Below are the core components for the
doit-click based task definition along with an illustration (code snippet 01 and 02).
- ✍️ Click based approach to input arguments/options/parameters
- ☑️ Base class to define doit task and/or click command
- 🏃 Method to execute a task
- 🌟 Additional utilities like
task dependencyand metadata definition using class attribute
- The code snippet below initiates a class based
- A command may make use of a
Click.Groupcontext defining a
- The command options are also defined as class attributes
To incorporate the look and feel, I designed a layer on top of the existing CLI architecture with the help of rich-click. It offers a variety of style options, markdown settings and flexibility to group tasks and options. Together it adds that perfect richness to the CLI command pallet. Below is a simple example which demonstrates the grouping of options and tasks along with style settings.
🎥 The developer CLI in action
Current list of implemented tasks
Below are the lists of tasks currently implemented as part of the developer CLI. This list is dynamic and subject to change in the coming months.
Build & testing tasks
- build (build & install package on path)
- test (Run tests along with options to run tests for a given submodule)
Static checker tasks
- pep8 ( Perform pep8 check with flake8)
- mypy ( Run mypy on the codebase)
- shell (Start Unix shell with PYTHONPATH set)
- python (Start a Python shell with PYTHONPATH set)
- ipython (Start IPython shell with PYTHONPATH set )
- doc (Build documentation)
- refguide-check (Run refguide check)
- notes (Release notes and log generation)
- authors (Task to generate list the authors who contributed within a given revision interval)
- bench & bench compare
👏🏽 Great Collaboration and teamwork
From an idea to developing a successful POC, the journey was a great learning opportunity for me. Planning, coordination, teamwork and clear communication played a very important role. Huge thanks to Ralf Gommers and Eduardo Naufel Schettino for the amazing collaboration and support.
As a newcomer to the SciPy codebase, it was a steep learning curve. I asked a ton of questions, at times drifted to the ocean searching for answers. Learning something completely new can be overwhelming, different emotions brush past. Thanks to Ralf and Pamphile Roy for addressing my questions. The learnings and achievements will stay with me for a long-long time.
😇 The next steps
The experimental CLI is available under
scipy/do.py for the wider community to test and provide us with valuable feedback.
Handy commands to quickly try out the CLI
- Enabling the GUI:
- Listing all the available arguments/options for a task:
python do.py <task_name> --help
- 📜 Self-documentation and hierarchical help option
- 🧭 Easy to navigate and intuitive interface
- ⏩ Clear and concise examples to get started quickly
- ⏱ Reduction in the time spent navigating documentation
With a great start comes possibilities. In the coming weeks, the CLI will become mature and stable. After we receive wider usage and acceptance from the community, support for
runtests.py will be paused and
do.py will be renamed to
dev.py. The user documentation for the CLI components and usage will be made available for clear and concise understanding.
To foster reusability, Eduardo has developed a package named
pydevtool. The reusable elements will be incorporated into the SciPy developer CLI code. We will also be adding support for
act, which will enable users to run GitHub CI jobs locally.
🙂 Parting thoughts
Many thanks to the wonderful community for all the support and guidance. We are excited to collaborate with projects looking forward to adapting a similar developer command-line interface.