Outside of kernel symbols Mathematica provides access to lots of usage info, e.g. DownValues, Options, Messages, etc.
Most languages, if given this sort of info, would be able to provide an automatic documentation generation scheme, e.g. Python from doc strings.
Is there a good way to turn all this info into Mathematica-style docs?
Note: the main function is no longer GenerateRefPages but rather GenerateSymbolPages
The paclet can be gotten by running:
PacletInstall["DocGen",
"Site" ->
"http://www.wolframcloud.com/objects/b3m2a1.paclets/PacletServer"
]
Or by going here and just downloading it.
For those interested there's a GitHub repo for it here but I am not actively going to develop it.
It provides the interface described below plus functions for actually building pages. The answer below just provides the core information and functions for auto-generation.
The online documentation pages for it built like this are here
Mathematica docs are split into a usage section, a details section, and an examples section.
We'll assume (and yes, this is a rather large assumption) that we have a function that will take these three things for us and convert it into Mathematica docs, if we can provide the appropriate stuff.
The first thing we need is our usage patterns. Generally only DownValues usages are provided, but for an automatic documentation generator it's best to take all of the callable *Values (i.e. not OwnValues, FormatValues or NValues)
We don't want all of the particulars of our definitions, though, so we'll define a pair of functions to extract just those parts of the pattern that make sense as definitions:
usagePatternReplace[
vals_,
reps_: {}
] :=
With[{
names = AssociationMap[Null &, {}],
conts = Alternatives @@ {"System`", "FrontEnd`", "PacletManager`",
"Internal`"}
},
ReplaceRepeated[
FixedPoint[
Replace[#,
{
Verbatim[Pattern][_, e_] :>
e,
Verbatim[HoldPattern][Verbatim[Pattern][_, e_]] :>
HoldPattern[e],
Verbatim[HoldPattern][Verbatim[HoldPattern][e_]] :>
HoldPattern[e]
},
1
] &,
vals
],
Flatten@{
reps,
Verbatim[Optional][name_, _] :>
name,
Verbatim[Pattern][name_, _] :>
name,
Verbatim[PatternTest][p_, _] :>
p,
Verbatim[Alternatives][a_, ___] :>
a,
Verbatim[Blank][] :>
expr,
Verbatim[Blank][
t : Integer | Real | String | List | Association] :>
RuleCondition[
Replace[t, {
Integer :> int,
Real :> float,
String :> str,
List :> list,
Association :> assoc
}],
True
],
Verbatim[Blank][t_] :>
t[],
Verbatim[BlankSequence][] :>
Sequence @@ ToExpression[{"expr1", "expr2"}],
Verbatim[BlankNullSequence][] :>
Sequence[],
symbolUsageReplacementPattern[names, conts]
}
]
]
symbolUsageReplacementPattern[names_, conts_] :=
s_Symbol?(
GeneralUtilities`HoldFunction[
! MatchQ[Context[#], conts] &&
!
MemberQ[$ContextPath, Context[#]] &&
!
KeyMemberQ[names, SymbolName@Unevaluated[#]]
]
) :>
RuleCondition[
ToExpression@
Evaluate[$Context <>
With[{name = SymbolName@Unevaluated[s]},
If[StringLength@StringTrim[name, "$"] > 0,
StringTrim[name, "$"],
name
]
]
],
True]
With these we can then make a function that will align these cleaned up usages with a usage string, coming from sym::usage:
AutoGenerateUsage[sym_Symbol] :=
With[{
vals =
Replace[
usagePatternReplace[
Map[First,
Join @@
Map[#[sym] &,
{UpValues, DownValues, SubValues}
]
]
], {
l : {__} :>
Map[
Cell[
Replace[#, {
None :>
BoxData@ToString@Unevaluated@sym,
Verbatim[HoldPattern][Verbatim[HoldPattern][e_]] :>
Replace[
FixedPoint[
Replace[
Verbatim[HoldPattern][Verbatim[HoldPattern][p_]] :>
HoldPattern[p]
],
HoldPattern[e]
],
Verbatim[HoldPattern][e2_] :>
BoxData@toSafeBoxes[Unevaluated[e2], StandardForm]
],
Verbatim[HoldPattern][e_] :>
BoxData@toSafeBoxes[Unevaluated[e], StandardForm]
}],
"UsageInput"
] &,
l
],
{} -> {
Cell[BoxData@toSafeBoxes[Unevaluated[sym], StandardForm],
"UsageInput"]
}
}]
},
With[{s =
StringSplit[
StringTrim@
Replace[sym::usages,
_MessageName :>
Replace[sym::usage,
_MessageName :> "No usage message..."]
],
"\n"
]
},
If[Length@s >= Length@vals,
With[{u = Flatten@ConstantArray[vals, Length@s]~Take~Length@s},
Riffle[u, Cell[#, "UsageText"] & /@ s]
],
Append[
Riffle[vals,
Cell[StringTrim@StringJoin@Riffle[s, "\n"], "UsageText"]
],
Cell[StringTrim@StringJoin@Riffle[s, "\n"], "UsageText"]
]
]
]
];
AutoGenerateUsage[s : Except[_Symbol]?(MatchQ[#, _Symbol] &)] :=
AutoGenerateUsage@System`Evaluate@s;
AutoGenerateUsage~SetAttributes~HoldFirst;
This function can be mostly ignored. It converts usages and strings into cells that are put in a template notebook, which is what is actually parsed into the doc notebook.
The crucial bits of info in the details are really, how many call patterns are there, are there any FormatValues, and what are the Options, Messages, and Attributes
One wrinkle arises, here, though in that Options are often inherited from other functions and so there can be a huge number of useless options and generally we just want the useful ones.
So we need a function that will find only the new, useful options (and a corresponding one to find which functions the function at hand inherits from):
$filterOutOps =
{
Graphics, Graphics3D,
Framed, Style, Cell,
CloudDeploy, Button,
Pane, Panel, ActionMenu,
PopupMenu, Column,
Row, Grid, CreateDocument,
Notebook, Plot, Plot3D,
DialogInput
};
novelFunctionOptions[sym_, others_: $filterOutOps] :=
With[{
o = Options@Unevaluated[sym],
tests =
Replace[First /@ Options@#,
s_Symbol :>
SymbolName@s,
1] & /@ others},
With[{selected =
Flatten@
Select[tests,
With[{k =
Replace[Keys@o,
s_Symbol :>
SymbolName@s,
1]},
Complement[#, k] == {} &
]
]
},
DeleteCases[o,
_[_?(MemberQ[selected,
Replace[#, _Symbol :> SymbolName[#]]] &), _]
]
]
];
novelFunctionOptions[s : Except[_Symbol]?(MatchQ[#, _Symbol] &)] :=
novelFunctionOptions@Evaluate@s;
novelFunctionOptions~SetAttributes~HoldFirst
containedFunctionOptions[sym_, others_: $filterOutOps] :=
With[{o = Options@Unevaluated[sym]},
Select[others,
With[{k =
Replace[Keys@o,
s_Symbol :>
SymbolName@s,
1]},
With[{ops =
Replace[First /@ Options@#,
s_Symbol :>
SymbolName@s,
1]
},
Complement[ops, k] == {}
] &
]
]
];
containedFunctionOptions[s : Except[_Symbol]?(MatchQ[#, _Symbol] &)] :=
containedFunctionOptions@Evaluate@s;
containedFunctionOptions~SetAttributes~HoldFirst
With that we can build our details section (once again, the function is provided, but there's nothing interesting about it):
AutoGenerateDetails[sym_Symbol] :=
With[{
ovs = OwnValues[sym],
dvs = DownValues[sym],
uvs = UpValues[sym],
svs = SubValues[sym],
fvs = FormatValues[sym],
hovs = System`Private`HasOwnCodeQ[sym],
hdvs = System`Private`HasDownCodeQ[sym],
huvs = System`Private`HasUpCodeQ[sym],
hsvs = System`Private`HasSubCodeQ[sym],
hfvs = System`Private`HasSubCodeQ[sym],
ops =
If[Length@OwnValues[sym] == 0, novelFunctionOptions[sym], {}],
conts =
If[Length@OwnValues[sym] == 0, containedFunctionOptions[sym], {}],
attrs = Flatten@{Attributes[sym]},
msgs =
DeleteCases[Messages[sym],
Verbatim[HoldPattern][
HoldPattern@MessageName[_, "usage" | "usages"]
] :> _
]
},
Flatten@{
Replace[Length@ovs, {
0 -> Nothing,
1 ->
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has an immediate value"
}, "DetailsItem"],
n_ :>
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has ",
ToString@n,
" immediate values"
}, "DetailsItem"]
}],
If[hovs,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has an internal immediate value"
},
"DetailsItem"
],
Nothing
],
If[Length@dvs > 0,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has ",
ToString@Length@dvs,
" call",
If[Length@dvs > 1, " patterns", " pattern"]
}, "DetailsItem"],
Nothing
],
If[hdvs,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has an internal call pattern"
},
"DetailsItem"
],
Nothing
],
If[Length@uvs > 0,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has ",
ToString@Length@uvs,
" ",
inlineRefBox@"UpValues",
If[Length@uvs > 1, " patterns", " pattern"]
}, "DetailsItem"],
Nothing
],
If[hdvs,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has an internal ",
inlineRefBox@"UpValues",
" pattern"
},
"DetailsItem"
],
Nothing
],
If[Length@svs > 0,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has ",
ToString@Length@svs,
" ",
inlineRefBox@"SubValues",
If[Length@svs > 1, " patterns", " pattern"]
}, "DetailsItem"],
Nothing
],
If[hsvs,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has an internal ",
inlineRefBox@"SubValues",
" pattern"
},
"DetailsItem"
],
Nothing
],
If[Length@ops > 0,
{
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has the following ",
inlineRefBox@"Options"
}, "DetailsItem"],
Map[
{
Cell[TextData@inlineRefBox@
toSafeBoxes[Evaluate@First@#, StandardForm],
"DetailsRow"],
Cell[TextData@inlineRefBox@
Extract[#, 2, toSafeBoxes],
"DetailsColumn"]
} &,
ops
]
},
Nothing
],
If[Length@conts > 0,
{
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" can take any ", inlineRefBox@"Options",
" from the following"
}, "DetailsItem"],
Map[
{
Cell[TextData@inlineRefBox@ToBoxes[#, StandardForm],
"DetailsRow"]
} &,
conts
]
},
Nothing
],
If[Length@msgs > 0,
{
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has the following ",
inlineRefBox@"Messages"
}, "DetailsItem"],
Map[
{
Cell[TextData@inlineRefBox@
Replace[First@#,
Verbatim[HoldPattern][m_MessageName] :>
ToBoxes[Unevaluated[m]]
],
"DetailsRow"],
Cell[TextData@inlineRefBox@ToBoxes[Last@#, StandardForm],
"DetailsColumn"]
} &,
msgs
]
},
Nothing
],
If[Length@attrs > 0,
{
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" has the following ",
inlineRefBox@"Attributes"
}, "DetailsItem"],
Map[
Cell[TextData@inlineRefBox@ToBoxes[#, StandardForm],
"DetailsRow"] &,
attrs
]
},
Nothing
],
If[Length@fvs > 0 || hfvs,
Cell[TextData@{
inlineRefBox@toSafeBoxes[Unevaluated@sym, StandardForm],
" is a formatted symbol"
}],
Nothing
]
}
];
AutoGenerateDetails[s : Except[_Symbol]?(MatchQ[#, _Symbol] &)] :=
AutoGenerateDetails@System`Evaluate@s;
AutoGenerateDetails~SetAttributes~HoldFirst;
Finally we have our examples section. We'll generate a basic example for each usage we defined previously and a simple usage of each novel option in the first usage that contains an OptionsPattern[]. We'll need a new replacement mechanism, though, as the usage replacements aren't necessarily what we want. Instead we'll provide default values according to any type information provided in the patterns:
callPatternReplace[vals_,
reps_: {}
] :=
DeleteCases[
ReplaceRepeated[
FixedPoint[
Replace[
#, {
Verbatim[Pattern][_, e_] :>
e,
Verbatim[HoldPattern][Verbatim[Pattern][_, e_]] :>
HoldPattern[e],
Verbatim[HoldPattern][Verbatim[HoldPattern][e_]] :>
HoldPattern[e]
},
1
] &,
vals
],
Flatten@
{
reps,
Verbatim[Pattern][p_Symbol, Verbatim[Blank][]] :>
RuleCondition[
ToExpression[
SymbolName@Unevaluated[p]
],
True
],
Verbatim[Pattern][p_, Verbatim[Blank][]] :>
RuleCondition[
ToString@Unevaluated[p],
True
],
Verbatim[Pattern][p_,
Verbatim[Blank][
t : Integer | String | List | Association]] :>
RuleCondition[
With[{pstring =
Replace[Unevaluated[p], {
_Symbol :>
SymbolName[p],
_ :>
ToString[Unevaluated[p]]
}]
},
Replace[Unevaluated@t, {
Integer -> 1,
String -> pstring,
List -> {pstring},
Association :>
<|
pstring -> pstring|>
}]
],
True],
Verbatim[Pattern][p_, Verbatim[Blank][t_]] :>
t[],
Verbatim[Pattern][p_,
Verbatim[BlankSequence][
t : Integer | String | List | Association]] :>
RuleCondition[
With[{pstring =
Replace[Unevaluated[p], {
_Symbol :>
SymbolName[p],
_ :>
ToString[Unevaluated[p]]
}]
},
Replace[t, {
Integer -> Sequence[0, 1],
String :>
Sequence[
pstring <> "1",
pstring <> "2"
],
List :>
Sequence[{pstring}, {pstring}],
Association -> Sequence[<||>, <||>]
}]
],
True],
Verbatim[Pattern][p_, Verbatim[BlankSequence][t_]] :>
Sequence[t[1], t[2]],
Verbatim[Blank][t_] :>
RuleCondition[
Pattern @@ {expr, Blank[t]},
True
],
Verbatim[BlankSequence][t_] :>
RuleCondition[
Pattern @@ {expr, Blank[t]},
True
],
Verbatim[BlankNullSequence][t_] :>
Sequence[],
Verbatim[Repeated][t_, ___] :>
t,
Verbatim[RepeatedNull][___] :>
Sequence[],
Verbatim[Pattern][_, p_] :>
Unevaluated@p,
Verbatim[PatternTest][p_,
s : StringQ | IntegerQ | AssociationQ | ListQ
] :>
RuleCondition[
Pattern @@ List[p,
Blank[
Replace[s, {
StringQ -> String,
IntegerQ -> Integer,
AssociationQ -> Association,
ListQ -> List
}]
]
],
True
],
Verbatim[PatternTest][p_, _] :>
Unevaluated@p,
Verbatim[Alternatives][a_, ___] :>
a,
Verbatim[Optional][_, v_] :>
Unevaluated@v,
Verbatim[Blank][] :>
RuleCondition[
ToExpression["expr",
StandardForm
],
True
],
Verbatim[BlankSequence][] :>
RuleCondition[
Sequence[
ToExpression["expr1",
StandardForm
],
ToExpression["expr2",
StandardForm
]
],
True
],
(Verbatim[BlankNullSequence] | Verbatim[OptionsPattern])[] :>
Sequence[],
h_[a___, Verbatim[Sequence][e__], b___] :>
h[a, e, b]
}
],
Verbatim[Sequence][],
\[Infinity]
];
And then a special generator for the options:
AutoGenerateOptionExamples[sym_Symbol, defer : True | False : False] :=
With[{
u =
FirstCase[First /@ DownValues[sym],
Verbatim[HoldPattern][
_[
___,
Verbatim[OptionsPattern][] |
Verbatim[Pattern][_, Verbatim[OptionsPattern][]],
___
]
],
HoldPattern[sym[OptionsPattern[]]]
],
ops = If[Length@OwnValues[sym] == 0, novelFunctionOptions[sym], {}]
},
With[{usages =
Map[
First@# ->
callPatternReplace[u,
Verbatim[OptionsPattern][] -> #
] &,
ops
]
},
If[Length@usages > 0,
Cell[CellGroupData[Flatten@{
Cell["Options", "ExampleSection"],
Map[
Cell@
CellGroupData[{
Cell[ToString@First@#, "ExampleSubsection"],
Cell[BoxData@
Replace[Last@#, {
Verbatim[HoldPattern][e_] :>
If[defer,
RowBox@{"Defer", "[",
toSafeBoxes[Unevaluated[e], StandardForm],
"]"},
toSafeBoxes[Unevaluated[e], StandardForm]
]
}],
"ExamplesInput"]
},
Closed] &,
usages
]
}, Closed]
],
Nothing
]
]
];
AutoGenerateOptionExamples[
e : Except[_Symbol]?(MatchQ[#, _Symbol] &),
defer : True | False : False] :=
AutoGenerateOptionExamples[System`Evaluate@e, defer];
AutoGenerateOptionExamples~SetAttributes~HoldFirst
And then the examples proper:
AutoGenerateExamples[sym_, defer : True | False : False] :=
With[
{
vals =
Map[First,
Join @@
Map[#[sym] &,
{UpValues, DownValues, SubValues}
]
]},
Flatten@{
With[{c = Context[sym]},
If[MemberQ[$ContextPath, c] &&
Not@MatchQ[c, "Global`" | "System`"],
{
Cell["Load " <> StringTrim[c, "`"] <> ":", "ExampleText"],
Cell[BoxData@
If[defer,
RowBox@{"Defer", "[", #, "]"},
#
] &@RowBox@{"Needs", "[", "\"" <> c <> "\"", "]"},
"ExamplesInput"],
Cell["\t", "ExampleDelimiter"]
},
Nothing
]
],
Flatten@Riffle[Partition[#, 2],
Cell["\t", "ExampleDelimiter"]
] &@
Riffle[
Cell[TextData@{"From ",
inlineRefBox@Replace[usagePatternReplace@#,
Verbatim[HoldPattern][e_] :> {
toSafeBoxes[Unevaluated[e], StandardForm]
}], ":"},
"ExampleText"
] & /@ vals,
Cell[
BoxData@
If[defer,
Replace[#, {
Verbatim[HoldPattern][e_] :>
RowBox@{"Defer", "[",
toSafeBoxes[Unevaluated@e, StandardForm],
"]"}
}],
Replace[#, {
Verbatim[HoldPattern][e_] :>
toSafeBoxes[Unevaluated@e, StandardForm]
}]
],
"ExamplesInput"
] & /@ callPatternReplace[vals]
],
AutoGenerateOptionExamples[sym, defer]
}
];
AutoGenerateExamples[
e : Except[_Symbol]?(MatchQ[#, _Symbol] &),
defer : True | False : False] :=
AutoGenerateExamples[System`Evaluate@e, defer];
AutoGenerateExamples~SetAttributes~HoldFirst
Now with all of this we can build a function that will generate an automatic reference page notebook:
Options[RefPageNotebook] = {
"Usage" -> Automatic,
"Details" -> Automatic,
"Examples" -> Defer,
"SeeAlso" -> Automatic,
"RelatedGuides" -> {},
"RelatedTutorials" -> {},
"RelatedLinks" -> {},
"Footer" :>
Replace[$DocFooter,
Except[_String | _TextData] :>
"Auto-Generated Documentation"
]
};
RefPageNotebook[s_Symbol, ops : OptionsPattern[]] :=
With[{
seeAlso =
Replace[OptionValue@"SeeAlso",
Automatic :>
ToExpression[relatedFunctionNames[s], StandardForm, Hold]
],
guides = OptionValue@"RelatedGuides",
tutorials = OptionValue@"RelatedTutorials",
links = OptionValue@"RelatedLinks",
details =
Replace[OptionValue@"Details",
Automatic :>
scrapeDetails@AutoGenerateDetails[s]
],
examples =
Replace[OptionValue@"Examples", {
Automatic :>
scrapeExamples@AutoGenerateExamples[s],
Defer :>
scrapeExamples@AutoGenerateExamples[s, True]
}],
usage =
Replace[OptionValue@"Usage",
Automatic :>
scrapeUsages@AutoGenerateUsage[s]
],
footer = OptionValue@"Footer"
},
iGenerateRefPages[s, usage, details, examples,
seeAlso, guides, tutorials, links, footer]
];
RefPageNotebook[s_String, ops : OptionsPattern[]] :=
With[{ns = Names[s]},
If[Length@ns === 0 || Length@ns > 1,
DocContextTemplate@s,
ToExpression[s, StandardForm,
GeneralUtilities`HoldFunction[
RefPageNotebook[#, ops]
]
]
]
]
RefPageNotebook~SetAttributes~HoldFirst;
And we can build a convenience function that also makes sure we have the right contexts and things and also provides iteration over contexts:
Options[GenerateRefPages] =
Join[
Options[RefPageNotebook],
Options[CreateDocument], {
"PostFunction" -> None
}
];
GenerateRefPages[s_Symbol, ops : OptionsPattern[]] :=
Block[{
$Context = $docGen,
$ContextPath =
Join[$ContextPath,
{Context@s, $Context}
],
makeRefOverrides =
Join[
makeRefOverrides, {
ToString@Unevaluated@s,
SymbolName@Unevaluated@s
}],
postFunc =
OptionValue["PostFunction"]
},
CheckAbort[
$DocGenLine = 1;
If[postFunc =!= None,
postFunc,
Identity
]@
CreateDocument[
RefPageNotebook[s,
FilterRules[{ops}, Options@RefPageNotebook]
],
FilterRules[{ops}, Options@CreateDocument],
WindowTitle -> (
Last@
StringSplit[
ToString@Unevaluated[s],
"`"
] <> " - Documentation"),
System`ClosingSaveDialog -> False
],
If[$Context == $docGen, End[]]
]
];
GenerateRefPages[namePattern_String, ops : OptionsPattern[]] :=
Block[{
$DocActive =
If[StringMatchQ[namePattern, "*`"],
StringTrim[namePattern, "`"],
$DocActive
],
makeRefOverrides =
Names[namePattern <> "*"]
},
Block[{docGenCounter = 0, nms = Names[namePattern <> "*"]},
If[Length@nms > 5,
Monitor[
ToExpression[
nms,
StandardForm,
GeneralUtilities`HoldFunction[
docGenCounter++;
GenerateRefPages[#, ops]
]
],
Internal`LoadingPanel@
TemplateApply[
"Generating page `` of ``",
{docGenCounter, Length@nms}
]
],
ToExpression[
nms,
StandardForm,
GeneralUtilities`HoldFunction[
GenerateRefPages[#, ops]
]
]
]
]
];
GenerateRefPages~SetAttributes~HoldFirst;
Here is an example of this in action:
It is obviously of lower quality that standard documentation, but it's more convenient that digging through down-values and it's much faster to generate.
And then we can hook into the paclet manager and standard documentation interface to provide documentation for packages like Macros` which lack it:
