add file_to_run=file.ipynb capability to the notebook#1675

Labels

Milestone

Assignee

8 participants

over at ipython/nbconvert#13, @Krastanov requested the ability to re-run all cells in a notebook before doing a conversion. @Krastanov said:

Is there a way to evaluate all cells of a notebook before we use nbconvert?

Why:
I am writing a script that will convert our example notebooks to some static form to be part of the documentation. But many of > the examples contain plots, which take too much space to be kept in git so we are keeping only the unevaluated notebooks in the tree. However we want to generate documentation containing these images, so we must generate them somehow?

I suggested this can legitimately live in the ipython notebook entry point (hence this enhancement request), but can also see the advantages of having that functionality in the (forthcoming) ipython nbconvert

This would be the equivalent of clicking on "Run All" under the "Cell" menu of a notebook and saving it, without needing a web browser.

The question is what are the desired semantics: do we want to run the cells and update the nb contents in-place? That's currently more or less impossible, since the code responsible for much of that is actually in the browser. Now, being able to re-execute the code in a notebook by itself, that's a different story. But that would be only to re-generate figures on-disk, for example...

Given how much of the document logic is in JavaScript, I think this is a no-go for the foreseeable future, and I'm not sure it's worth keeping it open since it's likely unrealistic to change even in the long run. @ellisonbg, @minrk, thoughts on this? I'm trying to be fairly aggressive in issue triage so we keep open things we can realistically work on, and I'm not quite sure this falls in that category...

I think writing a Python script that runs code and records output is actually very straightforward. Remember that the output format in the notebook is basically a transcript of the contents of pyout/display_pub messages. The part of the js that records outputs and submits execution is actually extremely basic, and largely duplicated by code in the KernelManager object.

True, it may not be so bad after all... We'd want to preserve all text cells in-place, so it has to be written with a bit of care.

I think it would actually be harder to *not* preserve text cells and all othet metadata if you follow the pattern I mentioned. Run cells in order, editing in-place, rather than generating a new notebook struct.

This would be absolutely great for the sympy documentation (the "examples" part that is being ported to notebooks at the moment).

Regrettably, I do not know the internal structure of Ipython very well. If you do not plan to implement this anytime soon (ie in the next few weeks), could you point me to the submodules in question (I really have no idea of your submodule structure, so an example preamble with the needed imports for the @minrk pseudo-code will be greatly appreciated. I should be able to understand the rest by tracing the docstrings)?

I find the scripts mentioned in the last comment very useful to automatically run and test my IPython notebooks.
Would it be possible to include them in the nbconvert repo to make them easier accessible?
They don't do notebook conversion, but if I understand correctly the nbconvert repo is actually the breeding ground for any notebook related tools.

Make them executable

Maybe combine them in one command line tool with options whether to test the output or exceptions?

The gists have been updated, but actually need PR #3222 to be merged before they work in master. Not much has changed, just some reorganization and PR #3011, which split the KernelManager into two objects - the KernelManager, which creates / starts / restarts Kernels, and the KernelClient, which communicates with a Kernel via channels.

If anyone is still interested in this functionality, I wrote an external program to run an IPython notebook as a script and optionally export the HTML through nbconvert: https://github.com/paulgb/runipy