Exporting Wikis with Orgmode

One of the major boons I've found of using org-roam since my rework of my note taking environment is that I've been finding more ways to get use out of backlinks. I figured this would actually be a pretty useless feature, and I never spent so much time with it previously, but now that I'm more actively taking notes (and especially noting down random thoughts) I've found it very helpful. Being able to explore backwards through index cards has really been great for ideation, and particularly for worldbuilding.

Unfortunately, while it's great for notes, I still prefer to have a proper project structure when I'm building up the world. Generally, I've done this via Kanka because it's free, but as I've started a new worldbuilding project recently I figured I should give it a shot in Org directly. I already had some use with org-novelist (which I have stopped using), and while there isn't a direct equivalent for worldbuilding, I figured that if I structured my articles as if I was using Zettelkasten, then it may work out.

The main reason that I was using Kanka previously is because the end result is easily shareable. I won't go into the project structure of my worldbuilding setup in depth, but I want to show off the "sharing" part - publishing it as a wiki with ox-publish.

As for why I chose to use basic ox-publish over weblorg, which I use for this blog: I want to do some hacky stuff to get some wiki functionality working here, and I want a more free-form directory structure.

One other thing: I'm not using org-roam here, because I don't want to mess with having multiple DBs and I don't want to accidentally "cross-contaminate" from my personal notes to my public facing projects. That's a bad idea security-wise.

Anyhow, System Crafters has a good page on setting up a basic web project, and that's the basis of my build script. The only addition is this list to the org-publish-project-list:

(list "org-static"
      :base-directory "."
      :base-extension "css\\|js\\|json\\|png\\|jpg\\|gif\\|pdf\\|mp3\\|ogg\\|swf"
      :publishing-directory "./export"
      :recursive t
      :publishing-function 'org-publish-attachment
      )

This capture's attachments, and crucially it also captures CSS files, JavaScript files, and JSON files. The CSS is less important here, it's just for styling which can be set like this in the HTML head (base-path here is the base URL of the website):

;; Customize the HTML output
(setq org-html-head (concat "<link rel=\"stylesheet\" href=\"" base-path "/styles.css\" type=\"text/css\">"))

Just this on it's own already gives pretty exports (well, with a good stylesheet anyway). But there are two things which any good wiki needs: searching, and backlinks.

I implemented searching with Fuse: it's one JavaScript file that sits at the root of the project next to the build script. And now we get to the most disgusting Elisp code you'll ever see. I welcome any attempts to clean this up, I know it is terrible. Fuse needs a JSON file containing all the contents it should search, and I'm going to construct one by recursively going down through each org file in the project, fetching the title, filetags, and contents, then writing that to a JSON file.

(with-temp-file
  "search.json"
(insert "searchData = ")
(json-insert (vconcat (seq-filter
                       (lambda (x) x)
                       (mapcar
                        (lambda (f)
                          (when (org-publish-find-property f :title nil)
                            (list
                             :path (concat base-path (string-trim-left f "."))
                             :title (car (org-publish-find-property f :title nil))
                             :tags (vconcat (org-publish-find-property f :filetags nil))
                             :contents (with-temp-buffer
                                         (insert-file-contents f)
                                         (goto-char (point-min))
                                         (flush-lines "^#+")
                                         (org-replace-all-links-by-description)
                                         (goto-char (point-min))
                                         (flush-lines "^$")
                                         (flush-lines "^- $")
                                         (buffer-string))
                             )))
                        (directory-files-recursively "." "\\.org$"))))))

I'm prepending searchData = so that the file can be included in a script tag in the postamble, we'll see this in a bit. The :contents property is a bit confusing, so step by step:

In a temporary buffer, insert the current org file, then go to the start of it.
Get rid of all property lines.
Replace all links with their descriptions, so getting rid of the plaintext syntax.
Go back to the start of the buffer, and get rid of all empty lines.
Get rid of all lines that are just a single dash - this happens in index cards where I have incomplete bullet lists.
Return the whole buffer as a string.

org-replace-all-links-by-description looks like this:

(defun org-replace-all-links-by-description (&optional start end)
  "Find all org links and replace by their descriptions."
  (goto-char (point-min))
  (while (re-search-forward org-link-bracket-re nil t)
    (replace-match (match-string-no-properties 
                    (if (match-end 2) 2 1)))))

And I got that from here. Also, in case it seems a bit weird to be using vconcat to turn the list into an array, json-insert doesn't work on lists. And the seq-filter - it's for getting rid of any random null values.

Ok, all told that gives me a nice JSON file containing all the metadata I want to search. Now to search it.

In the org-html-preamble (I'll come back to this in a second), I have a search box and an empty unordered list.

<input id="search_input" type="text" placeholder="Search for...">
<ul id="search_results">

A simple JavaScript script can read when the search input is submitted, and then searches with Fuse. It will push results to the search results list:

document.getElementById("search_input").addEventListener("keypress", search);

const fuse = new Fuse(searchData, {includeScore: true, ignoreDiacritics: true, includeMatches: true, ignoreLocation: true, useExtendedSearch: true, threshold: 0.3, keys: [{name: 'title', weight: 2},{name: 'tags', weight: 3},{name: 'contents', weight: 1}]});

function search(event) {
    if (event.key == "Enter") {
      var x = document.getElementById("search_input");
      var r = document.getElementById("search_results");
      r.innerHTML = "";
      var sr = fuse.search(x.value);
      console.log(fuse.search(x.value));
      sr.forEach((result) => {
          var c = document.createElement('li');
          var a = document.createElement('a');
          var b = document.createElement('p');
          var h = document.createTextNode(result.item.title);

          a.appendChild(h);
          a.title = result.item.title;
          a.href = result.item.path.replace(/\.[^/.]+$/, ".html");

          const truncate = (input) => input.length > 125 ? `${input.substring(0, 125)}...` : input;

          b.appendChild(document.createTextNode(truncate(result.item.contents)));

          c.appendChild(a);
          c.appendChild(b);
          r.appendChild(c);
      });
    }
}

Again I am aware that this is terrible. But, when I add this, Fuse, and the search data JSON to the post-amble like so:

(setq org-html-postamble (concat
                        (concat "<script type=\"text/javascript\" src=\"" base-path "/search.json\"></script>")
                        (concat "<script src=\"" base-path "/fuse.js\"></script>")
                        (concat "<script src=\"" base-path "/search.js\"></script>")))

It all works! Search results get dredged up from the depths and shown underneath the search input bar.

Now backlinks.

This is actually conceptually simple: iterate through each org file, get a list of all the links inside it, then transpose it so that ((source-1 (target-1 target-2))) becomes ((target-1 (source-1)) (target-2 (source-1))), and write that to the page somewhere.

;; Construct backlink tracker
(defvar backlink-hashmap (make-hash-table :test #'equal))

(dolist (i (mapcar
          (lambda (f)
            (list
             (list :file (string-trim-left f "\\."))
             (list :links (mapcar
                           (lambda (l)
                             (string-trim-left (concat "/" (file-relative-name (expand-file-name l (file-name-parent-directory f)) ".")) "\\."))
                           (with-temp-buffer
                             (insert-file-contents f)
                             (org-element-map (org-element-parse-buffer) 'link
                               (lambda (link)
                                 (when (string= (org-element-property :type link) "file")
                                   (org-element-property :path link)))))))))
          (directory-files-recursively "." "\\.org$")))
  (mapcar
   (lambda (b)
     (let ((new-lst (gethash b backlink-hashmap '())))
       (remhash b backlink-hashmap)
       (puthash b
              (append new-lst (list (cadr (assoc :file i))))
              backlink-hashmap)))
   (cadr (assoc :links i))))

;; Remove duplicates
(maphash
 (lambda (x y)
   (delq nil (delete-dups y)))
 backlink-hashmap)

For the nth time, this is terrible code. You'll particularly cringe at the file name transformation crap. Regardless, it does work, and backlink-hashmap gets populated in the way we want. One thing of note here: I'm using the relative file name as the key for the hash map, which means that setting the test to equal is necessary for queries to work.

Now we can get the backlinks for anyfile with gethash and the name of our file, so we can fetch all of this data in a function for org-html-preamble:

(defun build-navigation (info)
  "Constructs navigation HTML. INFO contain all export options."
  (concat "<nav>
  <ul>
  <li>Project Name</li>
  <li class=\"float-right sticky\"><input id=\"search_input\" type=\"text\" placeholder=\"Search for...\"></li>
  <li><a href=\""
        base-path
        "/home.html\">Home </a></li>
<li><a href=\"#\">Backlinks ▾</a>
      <ul>"
        (mapconcat
         (lambda (x)
           (concat "<li><a href=\"" (concat base-path (string-trim-right x "\\.org") ".html") "\">" (car (org-publish-find-property (expand-file-name (concat "." x) (file-name-directory load-file-name)) :title nil)) "</a></li>"))
         (gethash
          (concat "/" (file-relative-name (plist-get info :input-file) (file-name-directory load-file-name)))
          backlink-hashmap))
        "</ul>
    </li>
  </ul>
  </nav>
<div>
<ul id=\"search_results\">
</ul>
</div>"))

(setq org-html-preamble 'build-navigation)

And that's it. Each page now has a drop down in the navigation bar showing all backlinks. Hopefully you learned something in this process, even if that's that I'm terrible at writing Elisp.