Site Generation - Matt Whipple

This site is generated with pandoc(1). Each html file can be produced by the corresponding source markdown(2). The source files can be collected using a wildcard/glob which can then be used to derive the output files. The pattern itself can be used to call pandoc with the desired invocation.

To support execution in CI or on any system without a haskell environment, this will also support execution through a container.

gen_site

Here a script will be defined which drives building the site from the relevant source files.

Set Some Flags

To enforce stricter bash behavior we’ll abort on unhandled failures and require variables to be defined.

set -euo pipefail

Transform a File

The site itself will be generated by producing individual files with pandoc. If this function is called with a source and an output file it will invoke pandoc with some standard options.

Here the call to pandoc is abstracted so that the presence of pandoc can be validated or alternative execution methods can be provided.

gen_site::xform_file() {
  local -r source=${1:?'Source file must be specified!'}
  local -r output=${2:?'Output file must be specified!'}
  gen_site::call_pandoc --defaults pandoc.defaults --standalone --output ${output} ${source}
}

Call Pandoc

The pandoc indirection introduced above can now be defined. The initial implementation to match the development system will call Pandoc through Docker and it will relay any provided parameters.

TODO Detect options or die trying

gen_site::call_pandoc() {
  docker run --rm --volume $(pwd):/host -w /host pandoc/core:2.14.0.1 ${@}
}

Main

The main build process will consist locating all of the source files and transofming each. This will expect a directory where *.md source files exist and a directory to which output will be written.

gen_site::main() {
  local -r src_dir=${1:?'Source directory must be specified!'}
  local -r out_dir=${2:?'Output directory must be specified!'}
  mkdir -p "${out_dir}"
  for src in "${src_dir}"/*md; do
    gen_site::xform_file "${src}" "${out_dir}/${src%.md}.html"
  done
}

gen_site::main ${@}

Required Commands

This section will define some of the invoked commands using the required command Make recipe. For the time being this will just be copied around.

missing-command = $(error $(or ${${1}_INSTALL}, '${1}; is missing; please install ${1}))
required-command = $(or ${_${1}_which},          \
    $(eval _${1}_which=$(shell which ${1})), \
    ${_${1}_which},                          \
    $(call missing-command,${1}))

DOCKER = $(call required-command,docker)

Basic Project Structure

Next to define some basic project structure with inputs and outputs. A site PHONY target will be defined to enable building of all of the outputs.

OUT_DIR  := public/

SOURCES   = $(wildcard *.md)
OUTPUTS   = $(addprefix ${OUT_DIR},${SOURCES:%.md=%.html})

site: ${OUTPUTS}
.PHONY: site

Define Docker Image

Running within a container requires selection of a Docker image. I’ll make use of one of the standard pandoc images (3).

PANDOC_IMAGE := pandoc/core:2.14.0.1

Run Within Container if Specified

I typically use one of two patterns to run things in containers, either a prefixed target pattern (i.e. docker-<target>) or accepting a flag which swaps out how things are done. Here I’ll use the latter where the passing of a defined IN_DOCKER flag (often provided as an enviornment variable) modifies targets to run within defined containers rather than directly on the host.

If that IN_DOCKER flag is defined then pandoc should be run in a container which works within a bind mount of the local directory.

TODO: Extract and use current user

ifdef IN_DOCKER
  PANDOC = ${DOCKER} run --rm --volume $(abspath .):/host -w /host ${PANDOC_IMAGE}
else
  PANDOC = $(call required-command,pandoc)
endif

Store Commands in File

While it seems somewhat heavy, this project will also support a third means of execution. The pandoc image used above does not include make and therefore cannot readily use the invocations derived by the rules within this file. Since these steps can be done independently I’d rather go that route than introduce a customized image when wanting to use a pipeline of containers (for CI). It seems likely I’ll find a one-stop image at some point which will render this unnecessary, but for now this file will also support generation of a shell script containing what it would otherwise do.

Generation of such a file is fairly straightforward as it can amount to overriding the already defined PANDOC variable to echo to the specified file rather than call the produced command. The final functionality is very simple but is built on top of a fairly high level of comfort with the underlying tools.

GEN_SCRIPT:= gen_site

clean-script: ; rm -f ${GEN_SCRIPT}
.PHONY: clean_script

${GEN_SCRIPT}: PANDOC = >> ${GEN_SCRIPT} echo pandoc
${GEN_SCRIPT}: clean-script site
    chmod +x ${@}

Define Conversion Rule

With all of the building blocks defined, the rule to pass the sourcefiles through pandoc to produce the HTML output can be defined.

${OUT_DIR}%.html: %.md
    ${PANDOC} --defaults pandoc.defaults -s -o ${@} ${<}

Pandoc defaults file

Pandoc will be configured through the use of the defaults file (1 sec. #default-files). supports which is pointed to above.

Define input and output formats

I’ll be using Pandoc enhanced Markdown as input and generating modern HTML. At some point in the future I may want to swap the output over to something more component/React based but that’s not particularly likely to actually happen and if so not for a bit yet.

from: markdown
to: html5

Configure Citations

I want to use citations with a global bibliography. The link-citations metadata field (1 sec. #other-relevant-metadata-fields) is helpful to provide appropriate hyperlinking. With the linking I also prefer more of a footnote citation style so this uses the numeric ISO 690 CSL style retrieved from the Zotero Style Repository(4). Without an appropriate background, sticking to an ISO standard seemed a safe choice.

citeproc: true
bibliography:
- sources.bib
metadata:
  citation-style: iso690-numeric-en.csl
  link-citations: true

Configure Math Rendering

At some point I’ll likely be incorporating some interesting path but in the short term I’ve already run up against the limits of the raw rendering so I’ll start with what seems like the default better option of mathjax.

html-math-method:
  method: mathjax
1.
Pandoc - pandoc user’s guide [online]. June 2021. Available from: https://pandoc.org/MANUAL.html
2.
Daring fireball: markdown [online]. 17 April 2021. Available from: https://daringfireball.net/projects/markdown
3.
Pandoc/dockerfiles: Dockerfiles for various pandoc images [online]. May 2021. Available from: https://github.com/pandoc/dockerfiles
4.
Zotero style repository [online]. Available from: https://zotero.org/styles