… I came to realize that many Emacs users seem to spend a great deal of time learning about Emacs, tweaking it, and writing new extensions, rather than getting non-Emacs-related work done. Sometimes it feels as though heavy Emacs users actually get less done overall, if you consider only non-Emacs-related tasks. My question is, is it possible to get work done in Emacs, without most of that work being Emacs-related?

It got me thinking about skills or tools that can be used to improve themselves, and the balance between using and improving tools.

Not all skills or tools can be used to improve themselves. I’m learning how to sew, but that doesn’t lead to making my sewing machine better (aside from fiddling with the dials).

Here are some skills that can be used reflexively:

Philosophy asks questions about good questions to ask

Learning about learning helps you learn more effectively

Woodworkers and machinists have a tradition of making their own tools

3D printers can print parts for their own models

You can program tools to help you program better: testing, version control, project management, etc.

Although making your own tools takes time, here are some advantages of doing so instead of buying them off the shelf:

You understand the internals better, and you can appreciate the subtleties

You can customize it to fit the way you work

You can create different variants for greater flexibility. Mass customization can’t anticipate or cost-effectively provide all the different types of things people may want.

As your skills and needs increase, you can create better and better tools for yourself.

Many programmers spend time deliberately improving their toolkits; if they don’t, they stagnate. At the basic level, people try programs or frameworks that other people have created. The next level might be scripting things to work together. A third level might be writing customizations or extensions, and a fourth level might be creating entirely new tools or frameworks. Beginner programmers might start at the first level of reusing other people’s code, but wizardly performance often involves a mix of the other levels.

So the question is: How can we balance doing things and improving things?

No one can answer this for you.

Me, I tend to avoid hard deadlines and I do things faster than people expect them to be done, so I have plenty of leeway to improve my tools – which helps me be even more effective, so it’s a virtuous cycle.

You’ll need to find your own balance. You might get urgent stuff out of the way first, and then figure out how to balance smaller requests with investing in capabilities.

Here’s something I put together to help you figure out where you might be in terms of balance. Alternatively, if you’re thinking about whether to pick up a skill or tool that can be used to improve itself, you can use this to evaluate what you read from people sharing their experiences with the tool. Can they find a good balance for themselves, or are they frustrated by the challenges of getting something to work?

“I have what I need in order to work.” This is the basic scenario. People focus on doing things instead of improving things.

“I can keep pushing, but performance is dropping, so I should invest time in maintenance.” It’s like the way a knife or a saw dulls over time. When you notice diminishing returns, it might be good to invest some time in maintenance. It’s not an urgent need, but it can pay off.

“I’d better take care of this now before it becomes a problem.” This is like maintaining a car or taking care of your health. A little time now can avoid big problems later.

“Grr, it’s broken. I have to fix it before I can work.” If you let things go for too long, or if you’re working with something finicky, you’ll be forced into maintenance mode. For example, some 3D printers require a lot of fiddling. Watch out for this scenario.

“It’s fine the way it is, but I know I can make it better.” The way you’re currently doing things is okay, but you know (from your experience or from what you’ve read of other people) that you can invest a little time to work more effectively. You might even know the return on investment. It’s easy to decide whether you should just go ahead with the status quo or invest the time in improving.

“It’s fine the way it is, but I think I can make it better.” The way you’re currently doing things is okay, but you have some ideas that might make it even better. If you think those ideas might be worth it, it might be good to give yourself a time limit for exploring those ideas so that you don’t get distracted. Alternatively, you can save it for a slower time.

“I’m waiting or stuck, so I might as well work on tools.” Maybe you’re waiting for feedback from someone else. Maybe you’re waiting for programs to compile or tests to pass. Why not spend a little time exploring how to make your tools a little better?

“I’m doing this for fun/learning.” Tool improvement can become more enjoyable than some of the other ways you used to like spending time. For example, you might find yourself wanting to watch a screencast or try out a tweak instead of watching TV or browsing random sites on the Internet. You don’t have to completely replace other activities, you just have to shift a little time from things that have less value to you.

“I can’t write about my actual work, but I can write about this.” If you’re wondering about yak-shaving propensity based on the blog posts you’re reading, consider: do people write about their improvements instead of the work that they’re doing because their work is confidential or hard to explain? Maybe they think blog posts about improvements are more interesting. Maybe they’re writing about improvements in the process of figuring things out (which in an excellent process, by the way). All these things can skew your perception of how much time people spend doing things versus improving things, and how much they accomplish within that time.

In terms of Emacs, these things mostly apply to me:

“I’m doing this for fun/learning” – Emacs tickles my brain, and the community is wonderful.

“I can’t write about my actual work, but I can write about this” – I suppose I could write more about the other stuff I’m interested in (sewing? cooking?), so there’s that. However, the consulting stuff is covered by agreements, and that’s a small fraction of my life anyway.

I assume other geeks are rational, especially if they have a lot of experience with it and other tools. Therefore, if people spend time tweaking (while avoiding the consequences of low performance), I assume it’s because they see the value of doing so (whether the pay-off is certain or not). On the surface, an effective person’s behaviour might resemble an ineffective person’s behaviour – six hours sharpening the saw for two hours of work, or six hours procrastinating and two hours of cramming? But if you look at:

if they get stuff done

whether other people are happy with their performance, or if they generally appear successful in their endeavours

how happy they are about the process

then you can get a better idea of whether it’s working for them.

As you think about your own balance or read other people’s blogs, can you identify what scenarios you and other people might resonate with? Am I missing any that I should add to the list? Please comment below!

1:31:39 watching over people’s shoulders when it comes to Emacs; Hangouts, virtual conference? Moderators are helpful. Handing a virtual microphone out – moderator names someone who can then check in, ask the question, etc. monitoring comments. (Could use help reaching out to speakers, organizing schedule, etc.)

Sacha (or anyone): this is a much earlier comment you made, but I’d be interested in hearing about your experience using the “standard” Emacs keybindings with Dvorak some time. Maybe during the next Emacs Meetup!

Jonathan Hill

9:54 PM

/me E I gather that’s likely not what you actually said. I seem to have gotten stuck on it, though

Will Monroe

9:56 PM

Thanks for the advice, everyone. This was the first time I’ve been able to interact with so many Emacs users. Hope to return next month and perhaps again at the possible conference. All the best!

When I build a tool for other people to use and I want to store data, I usually have to think in terms of relational databases: tables, fields, and queries. There are other kinds of databases out there, like ones with flexible documents or ones that are optimized for graphs, but I haven’t gotten the hang of them yet.

When I build a tool for myself and I want to store data, I usually use plain text. (Or maybe a spreadsheet, but now that I’m getting the hang of Org Mode tables, I’m leaning more and more towards text.)

I like the flexibility of plain text. Sometimes I want to organize my thoughts in an outline or an index. Sometimes I want to make a graph, like the way I wanted to visualize how my goals are related to each other. Sometimes I change my mind about what I want. (All the time, actually. =) ) Plain text lets me add structure the way I want to. It’s all in my text editor, so I can move things around or reorganize things using the tools in Emacs.

Sure, sometimes I mess up because of formatting mistakes or the lack of validation. For example, typos in my personal ledger show up when the numbers don’t match my bank balances or there’s a new category with a misspelled name. But these are easy enough to catch and fix, and I can’t completely guard against them with a database anyway. And it’s nice to know that version control can let me visually step through the changes or recover from mistakes.

What about speed? Databases can be much faster than plain text for large quantities of data, for sure. I tend to work with pretty small quantities of data. For example, my blog index has 3257 lines, and the file that I’m drafting this in is under a megabyte. Even with whatever Emacs Lisp I’ve written to extract or cross-reference data, I’m still mostly bottlenecked by my brain instead of my computer. Sure, it took me a little longer to figure out how to do table calculations using Org Mode, but now that I have some notes on that, I should be able to come up with future calculations more easily. Besides, if I need to analyze things quickly, I can export and then crunch the numbers using a different tool.

Speaking of tools, staying with lightly-structured plain text lets me build a toolkit of text manipulation techniques. When I’m editing things in Emacs, scripting with Emacs Lisp, searching with grep, or writing Javascript/Ruby/Perl code to work with text, I’m developing skills that I can use in a wide range of situations.

If you’re interested in keeping your data in plain text with Org Mode, here are some tips that can help you learn how to work with your information.

Start with tables

Learn how to use keyboard shortcuts to create, move, or delete rows and columns.

Learn how to sort tables.

Learn how to use the column with specifier (ex: <10>) to limit the displayed size of your column while still being able to add more information.

Use Org Mode’s support for calculations to do math or perform other operations on your table.

Consider using properties

Org tables don’t do well with paragraphs or more complex information, so you might want to use Org subtrees with properties.

You can use Org columns to display property values, or use Org dynamic blocks to put a summary of the values into your Org Mode file. See org-collector.el for a propview report.

If you want more control, you can work with the information using Emacs Lisp. You can use org-entry-get, org-entry-get-multivalued-property, or org-entry-get-with-inheritance to get the value of the property. If you want to go through all the subtrees (or a subset of them), use org-map-entries to call your own function at each of the matching headlines in the scope. org-heading-components will give you the information from the current heading, and you can use org-end-of-subtree to give you the boundary of the subtree if you want to process it further.

You can parse Org Mode lists with org-list-struct. I haven’t dug into this deeply yet, but it looks interesting.

Parse free-form text

In addition to working with tables and properties, you can write functions that use regular expressions or other techniques to extract data from text. re-builder can be useful for visual feedback while you’re figuring out the right regular expression to build. Remember, you’re in Emacs, so you don’t have to come up with the perfect regular expression that extracts all the data in one go. You can search for a regular expression, use a command like forward-line, save something to a variable, and so forth. Try thinking about how you would do something by hand, and then using repeat-complex-command to see what functions Emacs called when you did that.

save-restriction, narrow-to-region, and save-excursion are very useful when it comes to limiting the scope of your processing or saving your position, so check them out in the Emacs Lisp manual.

I find plain text to be really useful when I’m figuring things out (so, all the time), since I don’t have to build a complex interface for working with it. As I learn more about Org Mode’s features, I find myself using it for more and more of my data. Org’s slogan is “Organize your life in plain text!” – and I think it just might be serious about that!

I’ve been thinking about how to improve the way that I navigate to, clock in, and create tasks in Org Mode. If the task is one of the ones I’ve planned for today, I use my Org agenda. If I know that the task exists, I use C-u C-c C-w (org-refile) to jump to it, and then ! (one of my org-speed-commands-user options) to clock in and track it on Quantified Awesome. If I want to resume an interrupted task, I use C-u C-c j (my shortcut for org-clock-goto). For new tasks, I go to the appropriate project entry and create it, although I really should be using org-capture instead.

I thought about how I can reduce some of these distinctions. For example, what if it didn’t matter whether or not a task already exists? I can modify the org-refile interface to make it easier for me to create tasks if my description doesn’t match anything. To make things simpler, I’ll just reuse one of my org-capture-templates, and I’ll pre-fill it with the candidate from Helm.

Next, I want to add this to the way that Helm prompts me to refile. That means that my creation task should return something ready for org-refile. Actually, maybe I don’t have to do that if I know I’m always going to call it when I want to jump to something. I might as well add that bit of code that sets up clocking in, too.

To make it easier to draw using Autodesk Sketchbook Pro on my laptop (a Lenovo X220 tablet PC), I’ve created several templates with consistent dot grids and sizes. Since I want to minimize typing when I’m drawing, I wrote a couple of functions to make it easier to copy these templates and set up appropriately-named files. That way, I can save them without the grid layer, flip between files using Sketchbook Pro’s next/previous file commands, and then process them all when I’m ready.

Index cards

I’ve been experimenting with a habit of drawing at least five index cards every day. Here’s a function that creates five index cards (or a specified number of them) and then opens the last one for me to edit.

I might tweak the files a little more after I rename them, so I don’t automatically upload them. When I’m happy with the files, I use a Node script to upload the files to Flickr, move them to my To blog directory, and copy Org-formatted text that I can paste into my learning outline.

Automatically resize images

The image+ package is handy for displaying the images so that they’re scaled to the window size.

By using Emacs Lisp functions to set up files that I’m going to use in an external application, I minimize fussing about with the keyboard while still being able to take advantage of structured information.

Do you work with external applications? Where does it make sense to use Emacs Lisp to make setup or processing easier?

How do you take notes in Org? Are you buried in a heap of uncategorized notes? Do you manually open the right file and navigate to the right heading? Are you mystified by org-capture and org-refile? Here’s a path that can help you learn how to more efficiently take (and organize!) notes in Org Mode.

Set up a keyboard shortcut to go to your main Org file

Use org-refile to file or jump to headings

Use org-capture to write notes quickly

Define your own org-capture templates for greater convenience

Pull in additional information

Step 1. Set up a keyboard shortcut to go to your main Org file

Instead of using C-x C-f (find-file) all the time, set up shortcuts to jump to the Org files you use the most. This way, you can easily type that keyboard shortcut, go to the end of the file, and add your note. Here’s some sample code that sets the C-c o shortcut to open organizer.org in your home directory. You can add it to your ~/.emacs.d/init.el and then call M-x eval-buffer to load the changes.

Alternatively, you can use registers, which are Emacs data structures that can hold text, file references, and more. The following code sets the o register to organizer.org in your home directory:

(set-register ?o (cons 'file "~/organizer.org"))

You can then jump to it with C-x r j (jump-to-register), specifying o at the prompt.

Once you’re in your Org file, you can use M-> (end-of-buffer) to go to the end of the file, or you can use C-s (isearch-forward) to search for some text.

You’ll still need to switch back to your original buffer or window configuration when you’re done, but that’s something you can fix when you learn how to use org-capture.

Step 2. Use org-refile to file or jump to headings

The next improvement is to use org-refile to move the current subtree to a specified heading, or jump to one without moving any text. This will let you quickly go to a project or task from anywhere in Org Mode.

By default, org-refile will show you only the top-level headings of the current file. Let’s configure it to show you headings from all your agenda files. You can use M-x customize-variable to change org-refile-targets. Click on the INS button, then click on Value menu next to Identify target headline by. Change this to Max level number. In the Integer field, fill in a suitably high number, like 6. This is the maximum depth of headings that will be shown.

If you prefer to set your variables using Emacs Lisp, here’s the code that you can add it to your ~/.emacs.d/init.el. Call M-x eval-buffer to load the changes.

(setq org-refile-targets '((org-agenda-files . (:maxlevel . 6))))

Be sure to add your main Org Mode file to your agenda list. You can do so by going to the file and typing C-c [ (org-agenda-file-to-front), or by setting the org-agenda-files variable.

The standard Emacs completion interface isn’t as friendly as it could be. I use the Helm package to make it easier to select and complete input. Since Helm can be a little complex, you may want to start with ido-mode instead. Here’s how you can set Ido up to use with Org Mode:

(ido-mode)
(setq org-completion-use-ido t)

Once you’ve set up your org-refile-targets, your agenda files, and either Helm or Ido, you can get the hang of using org-refile. The standard keyboard shortcut for org-refile is C-c C-w when you’re in an Org Mode buffer. org-refile can do different things depending on how you call it:

By default, it moves the current subtree to the specified location.

If you call it with the prefix argument C-u (like so: C-u C-c C-w), it jumps to the specified location instead of moving the current subtree.

If you call it as C-u C-u C-c C-w, it jumps to the previous refiling location.

First, practise using it with the prefix argument (C-u C-c C-w) to jump to a location. Once you’ve gotten the hang of that, go to some of your uncategorized entries and use org-refile without the prefix argument (just C-c C-w) to move the entries to the right place.

org-refile gives you a quick way to jump to a heading, but you still have to find your way back to whatever you were working on before you wanted to take a note. After you’re comfortable with refiling notes to the right place, move on to learning how to use org-capture to quickly take notes from anywhere.

Step 3. Use org-capture to write notes quickly

org-capture can help you take notes quickly by popping up a window or leading you through prompts. When you’re done taking the note, it will return you to whatever you were looking at before you started. In order to take advantage of this, though, you’ll need to customize org-capture.

You can use M-x customize-variable to set org-default-notes-file to the filename you would like notes to be saved to, or set it in Emacs Lisp code like this:

(setq org-default-notes-file "~/organizer.org")

Make sure that the file exists and is automatically opened in Org Mode.

If you type C-c c, org-capture will display a prompt. t is a simple task template, and C will show you the customization interface for org-capture-templates.

Let’s start with t. It will show you a buffer with a simple TODO entry. You can fill in the rest of the details, use C-c C-s (org-schedule) to schedule it for a particular day, set the deadline with C-c C-d (org-deadline), etc. You can change the TODO keyword or delete it.

When you’re done, type C-c C-c to automatically save it to your default notes file (as specified by org-default-notes-file). Changed your mind? Cancel with C-c C-k. After either C-c C-c or C-c C-k, you should be back to whatever it was that you were working on.

Practise using C-c c (org-capture) to quickly jot down several tasks or notes. Then go to your notes file and use C-c C-w (org-refile) to move the notes to the right place.

You can also refile the notes right from the capture buffer. Instead of typing C-c C-c to finish your note, use C-c C-w to refile it.

Get the hang of using org-capture to take notes, organizing them every so often (maybe at the end of your day, or once a week?) or refiling them as you go.

Step 4. Define your own org-capture-templates for greater convenience

If you find yourself capturing different kinds of notes often or you want to capture in another format (table entry? list item?), invest the time in customizing org-capture-templates. In the beginning, you might find the Customize interface you get from M-x customize-variableorg-capture-templates to be easier to work with than setting the values in plain Emacs Lisp, since the Customize interface lists the options. Read the documentation and look at examples of how other people have configured their org-capture-templates for more ideas. I have quite a few templates defined in my config, and http://doc.norang.ca/org-mode.html has a number of templates too.

Step 5. Pull in additional information

org-capture and org-refile are great when you’re at your computer, but what if you’re away? Quite a few people use MobileOrg to take quick notes on the go. I haven’t gotten around to setting that up for my workflow properly; instead, I use Evernote to jot quick notes on my phone. As part of my weekly review process, I look at the notes in my Evernote inbox and copy them into Emacs as needed.

You can manually copy information from your preferred non-Emacs note-taking tools, or you can figure out an automatic way of doing so. For example, I have some code to copy Evernote notes titled “Journal” into an Org Mode file structured by year-month-day.

Tweak your workflow!

Here’s a quick sketch showing some of your workflow options when it comes to capturing and organizing information with Org Mode. Which combination do you prefer, and how could you make it even better?