Notebook support

• Composable Logs supports tasks defined in Jupyter Python notebooks, but can also run pure Python tasks (to verify)
• All tasks in the demo pipeline are notebooks.
• In git, notebooks source files (with code) are assumed to be in the Jupytext format.

• Jupytext notebook source files:

  • can be version controlled and code reviewed as usual Python text files.
  • support interactive work/editing with Jupyter and VS Code.
  • do not contain any evaluated outputs. Thus any images and data tables (that could contain sensitive information) are not commited to git.
  • can be Black formatted and type checked with Mypy.

• Notebook tasks are executed using the Papermill library and can be parameterized.

• After a notebook task has run, the evaluated notebook (that include all output cells) is emitted to the OpenTelemetry log. From the UI, evaluated notebooks can be inspected, or downloaded in ipynb or html format.

Info

As motivation for supporting notebooks, there are already Python libraries like Evidently and Pandas profiling that generate interactive dashboards/UIs inside Jupyter. Thus, deploying a public pipeline using Composable Logs can be used to deploy a first version of a public dashboard.

Further: JupyterLite make it possible to run Jupyter server in the browser; see Possible Futures for Jupyter, PyData 7/2022 (YouTube, 20 mins). JupyterLite does not yet seem to be supported by existing Jupyter dashboard libraries, but there are initial work in this direction, see eg link. This could make it possible to deploy interactive dashboards using Composable Logs.