Motivation

I have thinking about how to package and distribute my own libraries. Working in a large institution, leveraging the huge infrastructure is easy, but we also lose the skills and interests to figure out how these are done.

As an experienced but not maybe not serious programmer, i.e. I use Python more on the application level, like algorithms, data science. I have little to say about software engineering. This motivates me to learn more.

The major content of this post comes from https://py-pkgs.org/. This is a GREAT resource. I will just add some of my own experience and notes here to accompany your study. And this is by no means to be comprehensive.

Virtual Environment

Please create a virtual environment for your project. I always use the base environment, but as time goes by, it is a mess and can be corrupted. A virtual environment creates an isolated field for code testings, and dependencies can be clearly tracked, which mimics that new users environment. Before install your own package, please create a virtual environment, here it is named as pycounts.

conda create --name pycounts python=3.9 -y
conda activate pycounts

To deactivate the virtual environment, just type conda deactivate pycounts. You should see the prompt changes back to (base) in the terminal.

Suppose you have the following project structure, (Note that the root name is also pycounts, and our source code is in the src/pycounts folder.)

pycounts
├── .readthedocs.yml
├── CHANGELOG.md
├── CONDUCT.md
├── CONTRIBUTING.md
├── docs
│   └── ...
├── LICENSE
├── pyproject.toml
├── README.md
├── src
│   └── pycounts
│       ├── __init__.py
│       └── pycounts.py
└── tests
    └── ...

After this, at the root of the project, you can install the package using,

poetry install

Now, a Python interpreter could import this package. One good thing is that, the package is installed in the “editable” mode, which means that, if you change the source code, you do not need to reinstall the package, the change will be reflected immediately. This is extremely useful for development.

This install adds the path to the project to the sys.path variable, so that changes to the source code will be reflected immediately.

Add Dependencies

To add dependencies, you can use the following command, this will do two things,

1. add the dependency to the pyproject.toml file
2. install the specified dependency into the virtual environment

poetry add matplotlib

Tests and Coverage

To test the code, we need two more packages, pytest and pytest-cov. Note that, we only need these packages for development, so we add them as dev dependencies.

poetry add --dev pytest
poetry add --dev pytest-cov

By default, the test files should be in the tests folder. The test files should be named as test_*.py.

pytest tests/ --cov=pycounts

Package Documentation

To add the usage examples, you could add a jupyter notebook to the docs folder.

poetry add --dev jupyter
jupyter notebook

To build the documentation using sphinx, you need to add the following extensions.

poetry add --dev myst-nb sphinx-autoapi sphinx-rtd-theme

And then run the following command in the docs folder. Here, make command works because we have a make.bat ready in the docs folder. This is automatically generated by cookiecutter when we create the project.

cd docs
make html
cd ..

Build the Package

To build the package, you use the following command,

poetry build

This will create a dist folder under the root of the project. Two distribution types are created.

1. A wheel file, which is a built distribution, which could be installed using pip install pycounts-0.1.0-py3-none-any.whl
2. A sdists file, which is a source distribution, which could be installed by first unzipping the file, and then install. Like the following,

tar xzf pycounts-0.1.0.tar.gz
pip install pycounts-0.1.0/

Appendix

In my local desktop, since a conda virtual environment including python is created, then poetry will not install the package to a new environment. But in the Github Codespace, it will create its own virtual environment, and install the package there. So, you need to activate it first,

poetry shell

You will see a prefix <package_name> in the terminal.

If you do not want poetry to create the environment to isolate the python installation for you, you can use the following command to config it before use.

poetry config virtualenvs.create false

If a virtual environment is already there, to make this effective, please first delete and reinstall.

poetry env list  # shows the name of the current environment
poetry env remove <current environment>
poetry install  # will create a new environment using your updated configuration

References:

1. https://stackoverflow.com/questions/59882884/vscode-doesnt-show-poetry-virtualenvs-in-select-interpreter-option