Tag: orgmode

Over the years, I’ve built up quite a collection of notes as Org mode text files. So far, it has proven to be the most expressive and the most robust note-taking modality out of a long list of candidates that I’ve tried.

In other words, consulting one’s org mode notes database from a mobile device is painful. This should not be the case; notes should be always and instantly available, even on mobile.

In this blog post, I show you how to import your complete org mode notes database into Apple Notes, including typesetting and attachments, using the org mode publishing functionality.

To be clear: Org mode on the desktop remains my primary note-taking system. The goal of importing all of my notes into Apple Notes is only to have my personal knowledge base accessible from my mobile devices.

End result

After configuring and running org-publish, and then importing the whole directory of exported HTML files and attachments into Apple Notes on macOS, your notes will look like one of the two examples below: First macOS, then iOS on the phone.

Note the Emacs-supplied syntax highlighting, and the inline image.

If you import these to your icloud account (the default) the notes will be available on all of your iOS mobile devices.

These imported notes are fully indexed, and hence searchable from all of your devices.

An example note in Apple Notes on macOS.The same note as above, but now in Apple Notes on iOS.

org-publish configuration

In order to make this happen, we make use of the org-publish functionality. We also configure one or two Apple Notes-specific changes to improve rendering.

Add the configuration below to your init.el.

There are two publish targets: One for the org files (called pkb4000 below), and one for all of the attachments (called pkb4000-static below).

As an aside, pkb4000 is short for Personal Knowledge Base 4000. Ichose the name as a joke, as this synced directory of org mode filesfelt like just the Nth in a long series of knowledge baseiterations. Little did I know how well this one would stick.

Remember to change both the :base-directory properties to the top-level directory of your notes database. :publishing-directory is anywhere convenient where you would like to store the published HTML files and attachments.

This process can be easily repeated when you want to refresh the database on your Apple Notes, but unfortunately Notes will create a new Imported Notes N folder.

Alternatively, you can re-publish, and then import just changed HTML files one by one, after which you’ll have to move them back into the correct Apple Notes folder.

Conclusion

This solves the problem of being able to search rapidly and consult your whole org mode notes database using your iOS mobile device.

However, it does not yet solve the problem of importing Apple Notes you create on the mobile device back into Orgmode. This is something one could consider trying to solve using AppleScript.

Whatever the case may be, this is still a nice improvement over my previous workflow!

Bonus round: Convert orgmode buffer to Apple Note using AppleScript

Before I tried org-publish, I worked on some emacs-lisp and AppleScript to convert the current org mode buffer to HTML, and then to inject that into Apple Notes using Apple Script.

I am posting this here in case it might be useful to someone. However it is NOT required for the org-publish workflow described above.

This assumes that you have an orgmode folder.

Although far more humble than org-publish, this code will only create a new note if it does not exist yet. If the note already exists, it will simply update its contents to the current org file.

;; https://www.emacswiki.org/emacs/string-utils.el(defunstring-utils-escape-double-quotes(str-val)"Return STR-VAL with every double-quote escaped with backslash."(save-match-data(replace-regexp-in-string "\"""\\\\\"" str-val)))(defunstring-utils-escape-backslash(str-val)"Return STR-VAL with every backslash escaped with an additional backslash."(save-match-data(replace-regexp-in-string "\\\\""\\\\\\\\" str-val)))(setq as-tmpl "set TITLE to \"%s\"set NBODY to \"%s\"tell application \"Notes\" tell folder \"orgmode\" if not (note named TITLE exists) then make new note with properties {name:TITLE} end if set body of note TITLE to NBODY end tellend tell")(defunoan-export()(interactive)(let((title (file-name-base (buffer-file-name))))(with-current-buffer(org-export-to-buffer 'html "*orgmode-to-apple-notes*")(let((body (string-utils-escape-double-quotes
(string-utils-escape-backslash (buffer-string)))));; install title + body into template above and send to notes(do-applescript (format as-tmpl title body));; get rid of temp orgmode-to-apple-notes buffer(kill-buffer)))))

In this post, I’ll show you how you can use Emacs and orgmode to query live data from any RESTful webservice, and then use that data in orgmode tables, a really great way to get live table-based calculation and display functionality in your rich orgmode-based documentation.

As an example, we will query live ticker data from the Kraken cryptocurrency exchange, and then use the current trading values of two different cryptocurrencies to calculate a fictitious investor’s position.

There’s a short YouTube video accompanying this post that demonstrates how the whole business works. Read on for more details!

I can specify any number of pairs, but for the sake of exposition we’re going to work with Bitcoin in Euro, and Ethereum in Euro.

Query the ticker webservice using emacs-lisp

Next we’ll write some emacs-lisp code to embed in our orgmode file. In this case, this blog post is actually an orgmode file which I shall later export to wordpress.

The code makes use of TKF’s great request.el package, installable from MELPA. I started with one of the examples on the request.el github, and then used the shiny new let-alist macro to extract the first element of the c key (the last traded price) of the result - PAIR hierarchy.

The eth-eur and btc-eur pairs are stored in the cpb-kraken-etheur and cpb-kraken-xbteur variables respectively.

Whenever I press C-c C-c with my cursor anywhere over the code, it will retrieve the current values into emacs-lisp variables, and the recalculate the table shown below.

Use the ticker data in a spreadsheet-like table

Next, we construct the orgmode table. If you have never done this with Emacs orgmode, you should try it. The UX for quickly making and maintaining text-mode tables is breathtaking. The table as you see it below is the HTML representation (generated by Emacs and orgmode) of the plaintext table as it lives in this org file:

coin

units

curr unit price

curr val

frac

eth

15.231

197.000000

3000.507

0.36

btc

2.335113

2252.100000

5258.9080

0.64

2017-06-03T14:10:14

8259.415

Everything in column 3 and to the right, and in the last row, is calculated based on the ticker values we have pulled in using the emacs-lisp code above. The table will update whenever I press C-c C-c on the embedded code block, as it ends with (org-table-iterate-buffer-tables), meaning to recalculate all tables in this file, until convergence.

The orgmode formula editor (shortcut C-c '), enables you to edit all cell formulas with live highlighting of the references. The editor interface looks like this:

As you can see, I pull in the three variables retrieved and calculated by the emacs-lisp code using snippets of lisp. The other formulas are the standard spreadsheet fair with an orgmode flavour. @2$3 for example refers to the third column in the second row.

I can make any number of tables in this same file, depending on values from either the lisp code, or other tables. As per usual, I can export the file to PDF, HTML, ODF or even to a wordpress site, as I’m doing right now.

Screenshot of orgmode source, PDF preview on the right, interactive citation selection in the minibuffer. Click for full resolution.

I have (co-)written a few LaTeX documents in my time.

However, as I have been writing my life and lab notes and many of my technical blog posts in Emacs orgmode for the past few years, I wanted to see how one would go about using BiBTeX references in orgmode files (using John Kitchin’sorg-ref package) such that they would render correctly in orgmode’s export LaTeX and PDF.

My use case is of course slightly different than the norm: I maintain my master bibliography using the wonderful Zotero software, and I prefer exporting document-specific BiBTeX files from that main database.

This post shows you how to do it.

Pre-requisites

Make sure that latexmk is installed on your system. This is one of the easiest ways to make sure that pdflatex and bibtex are executed in the correct sequence, and a sufficient number of times.

With M-x package-install RET org-ref install the org-ref package. This will enable interactive selection and insertion of bibtex references from your local bib file.

You should also have the ox-latex Emacs package installed. This is what gives orgmode its LaTeX output powers.

Emacs setup

In your init.el somewhere, use the following code to ensure that the orgmode LaTeX exporter invokes latexmk in the correct way:

A minimal but complete orgmode example

The usual org-ref workflow is that you use it to manage a single main BiBTeX file, which is configured in your init.el. As mentioned above, I prefer using small document-specific BiBTeX files.

This is what the first three lines sets up using an Emacs file variable. Note that the local bib file is specified without quotes of any kind. Also, because there’s no path specified, it lives together with the orgmode document itself, which I prefer.

After this, I have two LaTeX-specific lines ensuring that the output PDF looks prettier than default. You can thank me later.

The third LATEX_HEADER line imports the natbib package, which means I have extra cite commands to differentiate between for example textual citing (the first example) and parenthetical citing (the second example).

#+LATEX_CLASS_OPTIONS: [a4paper, 11pt, colorlinks=true, citecolor=., linkcolor=black, urlcolor=black]#+LATEX_HEADER: \usepackage{fourier}#+LATEX_HEADER: \usepackage{natbib}#+TITLE:Example orgmode to LaTeX* Introduction
This is that really difficult intro spection, as was also
demonstrated by cite:huang_identification_2016.
* Conclusions
This is the even more difficult final section. We will now use a
parenthetical citation citep:meyer_four-level_2012.

Finally, very importantly, I have the orgmode orgmode markup to select a suitable bibliography style, and again to specify the bib file. This will get automatically translated to the suitable LaTeX or even HTML versions.

Putting it all together

With something like the example above in your orgmode file, and a suitable bib-file, you can do C-c C-e l o to export to LaTeX, execute the latexmk command we specified above, and view the PDF.

In the screenshot at the start of this post, you can see what this looks like on my setup. I am using PDFTools for viewing the PDF inside of Emacs (this gets updated automatically whenever I rebuild the PDF).

You can also see the ivy-based completion as I’m in the process of interactively inserting a new citation into the document.

The default helm-based completion is more attractive, but it takes over the whole display, which is why I made use of ivy for instruction’s sake.

Org mode is great for authoring rich documents with syntax highlighted source code, LaTeX math and images. It even supports evaluating live snippets of code embedded in the text. It does all of this whilst remaining a plain text format.

Imagine how useful it would be to author programming-related or technical emails using this functionality?

Imagine no more! org-mime, part of the org mode contrib, does this for a number of emacs-based mail clients. However, our preference is for mu4e, which is not part of that list.

Fortunately, it seems mu4e has orgmode support integrated. To get this working, configure the following in your ~/.emacs.d/init.el:

When composing a new email, switch on the special mu4e / orgmode mode with M-x org~mu4e-mime-switch-headers-or-body (it will automatically switch between compose mode and orgmode depending on whether your cursor is on the headers or in the body).

You should add the following options to your org mode body. The first configures LaTeX (math) to be converted using imagemagick (instead of MathJax or dvipng), and the second removes the table of contents.

#+OPTIONS: tex:imagemagick#+OPTIONS: toc:0

When you want to send the email, move to the headers, and then do C-c C-c. Because org-mu4e-convert-to-html is set to true, the orgmode body will automatically be converted to HTML before being sent.

Until my pull request is merged in, you will have to make the following change to function org~mu4e-mime-convert-to-html() in org-mu4e.el:

With Emacs, org mode and org-babel, it’s possible to evaluate source code samples embedded in your org files and have the output of said evaluation appear inline. This makes for a beautiful literate programming environment. It also enables one to include graphs in one’s documents (org mode, PDF, HTML presentations or blog posts) by using for example GraphViz.

Now press C-c C-c to evaluate this code. Emacs will generate the configured output file dot_success.png and then link to it in an automatically created #+RESULTS section right below it.

You can press C-c C-x C-v to toggle display of inline images to see it directly in Emacs. Alternatively, use M-x org-display-inline-images to switch this on. Whenever you change the DOT source code (press C-c ‘ to edit the dot source code in a separate buffer), just press C-c C-c to re-execute the updated source.

In any exported documents (for examlpe this blog post), only the output graph itself will appear, like this: