Is there a way to use Git and Jupyter notebooks together while committing py files instead of ipynb seamlessly?

Question

I like my github repositories to include py files instead of ipynb files, but I much prefer working in Jupyter when doing projects.

I'd ideally like to work on my project by:

• Performing a git pull from the repository I'm working on, which contains a .py file that I've converted into a .py file from my Jupyter notebook
• Update the Jupyter notebook with changes before a commit
• Download the notebook as a py file and then make the commit, with the notebook file ignored by git

Is there a way to do this 'updating of the py file' without manually downloading my notebook in to a py file, taking the py file I downloaded in my Downloads folder, dragging it in my work directory and manually replacing the extant py file before finally making a commit? It just seems very unclean.

Perhaps I should have my ipynb in the repository as well, so that I can make sure that when I pull the notebook is up to date and is the exact same as the py file? But I would want it hidden from the public.

I just want to be able to make things as seamless as working with just py files, while doing actual work in Jupyter and then for the public having the final code be py files, but I want to do it relatively fluidly without a lot of manual dragging and dropping of files and replacing old ones. Kind of ruins the point of using git as I'm no longer using version control "elegantly".

Answer 1

Yes. jupytext can keep .ipynb and .py files in sync for you via so-called "paired notebooks", as long as you edit those in one of supported clients, such as Jupyter Notebook or JupyterLab.

Alternatively, you could configure pre-commit git hook to run jupytext conversion from .ipynb to .py just before every commit, which is documented here. In short, you would need to install pre-commit - e.g. with:

pip install pre-commit

and then create/modify .pre-commit-config.yaml with:

repos:
-   repo: https://github.com/mwouts/jupytext
    rev: v1.11.2  # (replace with the latest stable version)
    hooks:
    - id: jupytext
      args: [--from, ipynb, --to, "py"]

It would be also possible to configure a pre-commit hook using the built-in jupyter conversion capabilities if you do no want to depend on jupytext, but this will provide fewer configuration options

Answer 2

I do store Jupyter notebooks in Git. I clear all outputs, save notebook, then commit. Don't use pre-commit hook as the process is simple to follow manually. We do review notebooks changes in pull requests when needed - easy to do if outputs are cleaned.

Check out this video for demo on how I put notebooks under Git.