Literate programming in Go | InfoWorld

During a modern firm hackathon, my team’s constitution was to strengthen the documentation for the Steampipe plugin SDK. Like other elements of the Steampipe procedure, the plugin SDK is prepared in Go and printed to pkg.go.dev. The edition that existed when we begun is in this article. As is regular, the documentation was an autogenerated catalog of features and varieties. To make clear how to use individuals features and types, we delivered assistance on the steampipe.io web-site.

Our hackathon problem was to weave this kind of direction into the produced documentation. If you might be creating a Steampipe plugin, your listing of tasks appears like this:

1. Define the plugin
2. Generate the plugin entry level
3. Determine your initial desk
4. …

These jobs involve that you use functions and varieties, but whilst comments attached to all those features and forms can enhance the generated documentation, they are as well granular for the higher-level exposition we aimed for. Searching for inspiration, Steampipe guide developer Kai Daguerre recognized that our top-level web site lacked the overview section he saw in the documentation for pgx, a Go driver for Postgres.

That overview comes from https://github.com/jackc/pgx/blob/grasp/doc.go, which is one very long remark (that makes use of Go comment syntax) adopted by a offer declaration.

So we included a doc.go at the prime stage of our repo to develop this overview.

IDG

Producing a wiki in Go responses

We made use of the identify doc.go because that seems to be common, but it could have been termed foo.go. What is actually salient is that it truly is a legitimate Go file that belongs to a deal. The bundle contains no code, only documentation. We wrote headers to produce the sections of the overview, and in each individual portion we described a plugin writer’s activity making use of narrative, inline code examples, inner one-way links to capabilities and varieties, and external back links to examples elsewhere.

The skill to backlink within just the code’s namespace was a revelation. To describe the process called Increase hydrate features, for case in point, we wrote this:

# Add hydrate features

A column may well be populated by a Checklist or Get get in touch with. If a column requires knowledge not offer by Record 
or Get, it may determine a [plugin.HydrateFunc] that helps make an additional API phone for each individual row.
Insert a hydrate purpose for a column by environment [plugin.Column.Hydrate].

The area defined by the Incorporate hydrate features header is a hyperlink focus on: Include hydrate features. And the bracketed things render as backlinks to a kind, plugin.HydrateFunc, and to a assets, plugin.Column.Hydrate. This was starting off to feel like wiki creating!

What we even now lacked, though, was the capability to build new wiki internet pages wherever we could describe larger-level principles. The overview was a person position to do that, but we desired to reserve that for narration of the plugin writer’s journey. In which could we discuss a principle like dynamic tables, an innovative attribute that permits plugins like CSV which have no mounted schema and have to determine columns on the fly?

Kai realized that not only could we produce new documentation-only packages for these types of topics, we could also import them so that their names had been obtainable for the identical form of shorthand linking we could do with, e.g., [plugin.HydrateFunc]. In the prime-stage doc.go he did this:

offer steampipe_plugin_sdk

import (
"github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables"
)

var forceImportDynamicPlugin dynamic_tables.ForceImport

And in /docs/dynamic_tables/doc.go he did this:

bundle dynamic_tables

kind ForceImport string

Now we could produce a sentence in a comment like this:

/*
Use [dynamic_tables] when you cannot know a table's schema in progress.
*/
package deal steampipe-plugin-sdk

And the bracketed time period autolinks just like any other title in the code’s namespace.

If that would seem like far more hassle than it can be worthy of, you can skip the import gymnastics and just use an external backlink:

/*
Use [dynamic_tables] when you simply cannot know a table's schema in advance.


[dynamic_tables]: https://pkg.go.dev/github.com/turbot/steampipe-plugin-sdk/v5/docs/dynamic_tables
*/
deal steampipe-plugin-sdk

Either way, the authoring working experience now feels far more absolutely wiki-like. You can develop new internet pages as required, and weave them into the hypertextual documentation that lives inside of the code foundation.

It was, admittedly, a bit of a struggle to use the Go program in the means explained in this article. The comment syntax is Markdown-like but frustratingly not Markdown it is dependent on lots of implicit formatting conventions. If you go this route you can expect to will need a nearby previewing resource, and it is not tremendous-evident that the a person you want is not godoc but relatively pkgsite. Had we uncovered the command go set up golang.org/x/pkgsite/cmd/pkgsite@most up-to-date we would have saved ourselves a large amount of grief.

How is this literate programming?

It isn’t really! In his eponymous paper Knuth wrote:

The practitioner of literate programming can be regarded as an essayist, whose primary worry is with exposition and excellence of type. This kind of an writer, with thesaurus in hand, chooses the names of variables cautiously and points out what every single variable implies. He or she strives for a method that is comprehensible because its concepts have been launched in an purchase that is ideal for human knowing, utilizing a mixture of official and casual strategies that strengthen just about every other.

To guidance this exercise he invented a procedure termed website whose components, tangle and weave, enabled the creator of a method to convey to its story in a language that combined code (initially, Pascal) and documentation (initially, TeX) in a narrative-initially way. Our modern strategies of mixing code and documentation, and creating docs from embedded responses, only superficially resemble Knuth’s exercise, as Mark Jason Dominus pointed out in his essay POD is not Literate Programming (2000).

And nevertheless, now that our hackathon work out has provided me a flavor of what code-as-wiki can be, it does feel like a phase toward the storytelling tactic that Knuth advocates. Remember that he named his procedure world wide web before there was the web we now inhabit, by no means mind wikis! I are unable to assert that we’re now literate programmers in the Knuthian sense, and I have often puzzled if only a Knuthian head could employ that initial vision. But this way of improving upon the narrative top quality of autogenerated docs does really feel like a phase in the right way.