++ In Pursuit of an Efficient Org-Agenda +
+ +Table of Contents
+ ++After beginning my Emacs and Org-Mode adventures I quickly realized the insufficiency of the standard agenda. +It is wonderful at producing a list, but I am quickly approaching over 1000 notes. +The agenda tooling is incapable of searching this many files efficiently, and it took over thirty seconds to generate an agenda. +
+ ++This was intolerable. I had two options: +
+-
+
- Place all my TODO items into a single file. +
- Narrow the number of files the agenda mechanism needs to search +
+I find the first option undesireable for reasons I mentioned in my post about zettelkasten tools. +I like to have my todo items mixed with the context where they were born. +A student of the Getting Things Done methodology might ask, "Doesn't this violate the central todo-list principle?" +Yes, it does. +Org-agenda allows me to have my cake and eat it too. +I can create a centralized todo-list out of all my todo items, and then immediately jump into the context of my next task. +
+ ++It took several iterations of configuration to reach a seamless workflow. +If you'd like to jump straight to the final setup, here's the link. +
+ +1. First Iteration
+
+The first piece of tooling I used came from this post and this Gist.
+Essentially it creates a function vulpea-project-files that can easily query for a tag, and then sets the org-agenda files to all the files containing this tag.
+
+Coupled with a helper-function that adds the tag hastodos to any file containing a TODO entry, it functioned well.
+
(defun vulpea-project-files () + "Return a list of note files containing 'hastodos' tag." ; + (seq-uniq + (seq-map + #'car + (org-roam-db-query + [:select [nodes:file] + :from tags + :left-join nodes + :on (= tags:node-id nodes:id) + :where (like tag (quote "%\"hastodos\"%"))])))) +(setq org-agenda-files (vulpea-project-files)) ++
+This solution is great because it is pure elisp.
+Anywhere you're running Emacs it should function just fine.
+It also works well with one of my needs: transparent file encryption.
+Emacs has extensions for both gpg and age encryption that allow files to be transparently encrypted and decrypted.
+This solution can make use of that where a standard grep could not.
+
2. Second Iteration
++Eventually, I grew tired of seeing the entire solution in my config file. +All told, it is about 170 lines. +This is a lot for a small utility! +
+ +
+I decided to try using ripgrep to fix this:
+
(defun set-org-agenda-files-ripgrep () + (setq org-agenda-files (split-string (shell-command-to-string "rg -torg -l TODO /home/user/org")))) ++
+This worked nicely. +It takes just a few lines, runs just as fast–probably faster–than pure elisp, and is much easier to read and understand. +
+ +
+The only difficulty is encryption.
+ripgrep does not operate on open emacs buffers, instead it operates on what is saved to disk.
+Naturally, this means that it only sees the encrypted files.
+
3. Final
++The solution to file encryption is ripgrep preprocessing. +I owe this solution to this reddit post. +
+ +
+Ripgrep can run a command (or shell script!) to files before processing.
+While the original poster was using gpg to encrypt files, just a few modifications allowed me to use age.
+
#!/usr/bin/env zsh +case "$1" in +*.age) + # The -s flag ensures that the file is non-empty. + if [ -s "$1" ]; then + exec /usr/bin/age --decrypt -i ~/.age/personal $1 + else + exec cat + fi + ;; +*) + ;; +esac ++
+This script operates on all .age files, decrypting them to stdout.
+When using a preprocessor, ripgrep will simply search the output of the command.
+
+I often have a mix of files in my notes directory, so I use filetypes to restrict ripgrep.
+This means I have to add a new filetype to allow .age files through the filter.
+
rg --type-add 'aorg:*.org.age' \ + -torg -taorg \ + --pre ~/age-preprocessor.zsh --pre-glob '*.age' -l TODO /home/user/org ++
+Finally, we have our completed function: +
+(defun set-org-agenda-files-ripgrep () + (setq org-agenda-files (split-string (shell-command-to-string "rg --type-add \'aorg:*.org.age\' -torg -taorg --pre ~/age-preprocessor.zsh --pre-glob \'*.age\' -l TODO /home/user/org ")))) ++
+Hope this helps! +
+