Command line

The credencegraph command works on a graph file in the package's JSON format. Every write goes through the same validation as the Python API, and the file is replaced only when the write succeeds, so a rejected command leaves it exactly as it was.

credencegraph init g.json
credencegraph add-node g.json --id calibrated --base 0.9
credencegraph add-node g.json --id signal --base 0.05 --source doi:10.0000/example#p4
credencegraph add-node g.json --id claim --base 0.1 --stated 0.5
credencegraph relate g.json calibrated signal --type requires --strength 1
credencegraph relate g.json signal claim --type supports --strength beta:8,2
credencegraph check g.json
credencegraph query g.json marginal claim                          # P(claim) = 0.1324
credencegraph query g.json conditional calibrated --given claim=true
credencegraph query g.json intervene claim --set calibrated=false  # 0.1
credencegraph diagnose g.json --target claim
credencegraph diagnose g.json --target claim --given signal=true
credencegraph diagnose g.json --target signal --targets stated

Commands¶

Credences are written as a bare probability, 0.3, or as a Beta distribution, beta:ALPHA,BETA, such as beta:8,2.

Sources are DOCUMENT or DOCUMENT#LOCATOR, split at the last #, so https://example.org/paper#sec2 has the locator sec2. Repeat --source for several.

Query targets and evidence are NODE=true or NODE=false; a bare NODE target means true. A marginal query takes one target and a joint query several. Any kind of query accepts --given; only intervene accepts --set. diagnose accepts --given too, with the same form and the same rules for equivalent nodes; see Diagnosing under evidence. The relation id defaults to SOURCE-TYPE-TARGET; pass --id to add a second relation of the same type between the same nodes.

A relation type outside requires, supports, refutes, equivalent and exclusive is an annotation, which inference ignores. A type close to an inferential one, such as support, is still stored, but the command warns.

Checking against a rubric¶

A graph built from a document's own text is classified against a rubric: a TOML file that declares the node types, the form fields each type carries, and what each type must rest on. The format, the node origins and every rule are specified in Verbatim ingestion; the example rubric used throughout is docs/examples/methods-paper.toml.

credencegraph check paper.json --rubric methods.toml --level anchored
credencegraph split paper.json s-012 --at "We show that the bias vanishes for long segments" --at "the estimator is consistent"
credencegraph annotate paper.json s-012a --rubric methods.toml --type result --set quantity="the bias" --set trend=vanishes
credencegraph annotate paper.json s-012b --rubric methods.toml --type claim --set shape=universal \
    --set statement="the estimator is consistent" --rests-on s-012a
credencegraph check paper.json --rubric methods.toml --level typed
credencegraph annotate paper.json s-012b --rubric methods.toml --not-assessed "outside the scope of this review"
credencegraph coverage paper.json --rubric methods.toml

• check --rubric RUBRIC --level anchored|typed|assessed checks the graph at that level and every level below it, typed by default. Every violation is reported, in details.violations of a rubric-violation error, each a finding whose diagnostic is the violation's code and whose id is <code>:<node id>; the text message lists them all on one line. At assessed the graph must also compile, and when it does not, details.compile_error holds the details a plain check would give. --level without --rubric is an invalid-argument; without --rubric, check is unchanged.
• coverage --rubric RUBRIC never fails on a graph that loads: it reports nodes, by_origin, by_type (document nodes, with each type's share), the untyped nodes, the unexamined ones (needing an assessment and having none), and the not_assessed verdicts with their reasons.
• split NODE --at CLAUSE… takes clauses that occur exactly once in the node's text; split NODE --fields adds one field child, which gets its content from annotate --set. A node with a form, an assessment or a base cannot be split.
• annotate NODE --rubric RUBRIC validates each write: --type must be declared and allowed for the node's origin (unassigned removes the type, form, assessment and base); --set FIELD=VALUE parses the value as JSON, else as a string, and it must have the field's type and be found in the node's text; --rests-on ID adds the type's edge-rule relation from ID, which must have a type the rule names; --verdict holds|fails|undetermined or --not-assessed REASON sets the assessment.

A rubric file that cannot be read is an invalid-rubric error, or file-not-found when it does not exist.

Ingesting a LaTeX document¶

pip install 'credencegraph[ingest]'
credencegraph ingest paper/main.tex paper.json --rubric methods.toml

ingest SOURCE GRAPH [--rubric RUBRIC] [--document ID] [--force] reads the main .tex file of a document and writes a graph with one document node per chunk: a sentence of running prose, a display equation, a caption, a footnote or a bibliography entry. The written graph passes check --level anchored; every node is unassigned, unless --rubric gives its unit an initial type.

• \input and \include are followed, relative to the main file's directory. Macros defined with \newcommand, \renewcommand, \providecommand or \def are expanded; any other macro stays as written.
• Each node keeps the LaTeX source of its chunk, with comments, \label and footnotes removed, so inline mathematics, \ref and \cite stay in the text. Headings are not nodes.
• A display (\[ \], $$ $$, equation, align, …) is one equation node; its \label, if any, is attributes.label, a list when it has several.
• The bibliography is a thebibliography in the document, or <main>.bbl beside the main file when the document uses \bibliography. A reference node carries its key as attributes.key, and each citation is a cites relation from the citing node to it. Keys that name no entry are listed in unresolved_citations.
• Locators are file=<path relative to the main file>;lines=<first>-<last>.
• --document sets the anchors' document id, SOURCE as given by default.

Sentences are split at ., ? or ! followed by a space and a word not in lower case, except after common abbreviations and initials, and at paragraph breaks, headings, list items and displays. A display inside a sentence ends it, so the text after the display starts a new node. Without the ingest extra the command fails with missing-extra.

Diagnosing under evidence¶

diagnose without --given describes the graph before anything is observed. With --given NODE=true|false, repeated for several, every inference diagnostic is taken given those values, E:

• Overclaim and underclaim compare a node's stated credence with P(X | E). A node that is itself observed, or equivalent to an observed node, is skipped: its value is the observation, not something its premises deliver.
• Sensitivity is ∂P(T | E)/∂θ, and crux |∂P(T | E)/∂θ| · sd(θ), where sd(θ) is that of the parameter's own credence: parameters are not updated on the evidence. P(T | E) is a ratio, so these derivatives come from the quotient rule rather than from P(T | E) at θ = 1 minus at θ = 0; see the diagnostics.
• Single points of failure compare P(T | do(Y = false), E) with P(T | E). Observed variables are not candidates, nor is a variable whose failure the evidence rules out, P(E | do(Y = false)) = 0.
• Failure impact ranks the same candidates by P(T | do(Y = false), E) / P(T | E), lowest first, whatever the threshold; details.single_point_of_failure marks the ones below --failure-threshold. Without evidence, a premise whose only path to T is one requires of strength r keeps at least 1 − r of P(T), so it falls below the threshold only if r > 1 − threshold: never at the default 0.1 when r ≤ 0.9.
• Value of information is I(T; Y | E), over the unobserved variables Y.

The target may not be observed, directly or through an equivalent node: its diagnostics would all be trivial. That, and a --given node that is not an inference variable, is an invalid-argument or unknown-node. Evidence that the graph gives probability zero is a zero-probability error naming --given, with the values in details.given; query with the same --given values names the remedies. The JSON response adds "given" after "targets" when evidence was passed; without --given it is unchanged.

Diagnosing several targets¶

--target may be repeated, and --targets stated adds every node with a stated credence: the claims a source makes, and so the natural targets. The --target nodes come first, in the order given, then the stated nodes in the graph's order, each once. --targets stated leaves out a stated node that takes no part in inference or that is observed, directly or through an equivalent node, for the reason the overclaim check skips it; a --target node that is observed is still an error. --targets accepts no value but stated.

The graph-wide checks (missing parameters, unanchored variables, overclaims and underclaims) run once, whatever the number of targets, and each target's sensitivity, crux, single points of failure, failure impact and value of information follow, in the order of the targets. With no target only the graph-wide checks run.

With --json, credencegraph diagnose g.json --target signal --targets stated on the graph above prints, with one of its findings shown,

{
    "command": "diagnose",
    "path": "g.json",
    "targets": ["signal", "claim"],
    "findings": [
        {
            "id": "crux:strength:signal-supports-claim",
            "diagnostic": "crux",
            "nodes": ["signal", "claim"],
            "relations": ["signal-supports-claim"],
            "value": 0.004884,
            "details": { "derivative": 0.0405, "sd": 0.1206 },
            "message": "crux 0.00488 for the strength of supports relation 'signal-supports-claim': P('claim') moves 0.0405 per unit of it and its standard deviation is 0.121",
            "target": "claim"
        }
    ]
}

• targets lists the nodes diagnosed, after repeats are dropped and --targets stated is resolved; [] without a target.
• given, after targets, is present only with --given.
• findings holds the graph-wide findings, then each target's. Every finding has id, diagnostic, nodes, relations, value (a number, or null for a structural finding), details and a one-line message, as in the diagnostics, and target: null for a graph-wide finding, and the target's id for a weak point. An id is unique among the findings with the same target; the same id can recur for another target, as the crux of a premise two claims share does.

The Python form is diagnose_many(graph, targets), whose Report holds the graph-wide findings and the weak_points of each target; report.to_dicts() is the findings list above.

Obligations and discharges¶

An inferential relation may carry an obligation: what must be true for the step to hold, and, once it is met, a discharge recording how. A discharge has one of five kinds:

{
    "id": "bias-step",
    "type": "supports",
    "source": "bias",
    "target": "result",
    "strength": 0.5,
    "attributes": {},
    "obligation": {
        "statement": "the bias is below 1%",
        "discharge": {
            "kind": "numerical",
            "script": "bias.py",
            "value": 0,
            "tolerance": 0.01,
            "digest": "sha256:…",
            "status": 0
        }
    }
}

A script is a Python file, named relative to the graph file's directory. Its digest is sha256: and the hex SHA-256 of its bytes, as sha256sum prints it, and status the exit status of the run that discharged the step; record both or neither. A numerical script prints the quantity it computes as the last non-blank line of its output, and the step holds when that number lies within tolerance of value, inclusive: the bias check above prints the bias and holds it to 0 ± 0.01. A relation without an obligation is written exactly as before.

credencegraph verify g.json runs every script with the current Python, from the script's own directory, so verify only graphs whose scripts you trust. It flags a script that cannot be read (missing-script), has no recorded run (never-run), changed since that run (script-changed), has no reachable assert, raise or non-zero exit and so cannot fail (cannot-fail), or now passes or fails against its record (contradicted), or fails without a passing record to contradict (check-failed). A check under a constant condition that never holds, such as if False:, or an assert of a constant that always holds, such as assert True, is not reachable; a check behind any other condition is taken to be. When a numerical script exits 0, its printed result is held to value ± tolerance as well, and is flagged when it lies outside (out-of-tolerance), or when the last line is not a number (result-missing), whatever the script's own asserts said. A run longer than --timeout seconds, 300 by default, is stopped and counts as failed. A citation of a file beside the graph is flagged when its quote, compared after whitespace is collapsed, is not in the file (quote-not-found) or its digest is not the quote's (quote-digest-mismatch); a citation of anything else, such as a DOI, is unchecked. Each discharge gets a check with its relation, kind, outcome (passed, flagged, unchecked or judgement), the script's exit status now, the reason it was unchecked, and its findings. Any flagged discharge fails the command with a verification-failed error, whose details.checks holds every check.

credencegraph sorries g.json --target result lists the steps the target depends on: the requires, supports and refutes relations into it or into anything it depends on, and the equivalent and exclusive relations touching either. A step is discharged by a formal or citation discharge, or by a script whose recorded run exited 0 and which can fail. Every other step is a sorry, with its reason: no-obligation, undischarged, judgement, never-run, recorded-failure, missing-script or cannot-fail. The report reads the record and runs nothing, so run verify first. Credence applies only to the sorries, which are ranked by sensitivity, the exact derivative of P(target) by the step's strength, largest in size first; an equivalent or exclusive step has no strength and comes last. The JSON gives the target, the steps, the discharged relation ids and the sorries, each with its relation, type, source, target, reason, obligation statement and sensitivity.

JSON output¶

With --json, a command prints exactly one JSON object on stdout, on success and on failure. A success echoes the command and its result:

{
    "command": "query",
    "path": "g.json",
    "kind": "marginal",
    "target": { "claim": true },
    "given": {},
    "set": {},
    "seed": 0,
    "point": 0.1324,
    "band": { "q05": 0.1235, "q50": 0.1335, "q95": 0.1387 },
    "mean_over_draws": 0.1326,
    "draws": 1000
}

A failure exits with status 1 and prints an error object instead:

{
    "command": "relate",
    "error": {
        "code": "cycle",
        "message": "relation 'claim-supports-signal' would create a cycle: claim --supports[claim-supports-signal]--> signal --supports[signal-supports-claim]--> claim",
        "hint": "requires, supports and refutes relations must not form a cycle; drop or reverse one of the relations named",
        "details": {
            "cycle": ["claim", "signal", "claim"],
            "relations": ["claim-supports-signal", "signal-supports-claim"]
        }
    }
}

The code is stable and meant for programs; the message and hint are for people. The codes are file-exists, file-not-found, io-error, invalid-graph, invalid-argument, duplicate-id, unknown-node, cycle, compile-error, zero-probability, problem-too-large, invalid-rubric, rubric-violation, missing-extra and verification-failed.

A graph that cannot be compiled for inference is a compile-error, whichever command meets it: an inference variable without a base, equivalent nodes with different bases, or a cycle that appears once equivalent nodes are merged. The details name the nodes: errors holds a missing-parameter finding per node without a base, and conflicting_bases each pair of disagreeing equivalent nodes. When check fails this way, details also carries the whole verdict: the path, the nodes and relations counts and the warnings. A passing check prints {"command", "path", "nodes", "relations", "warnings"}, where the warnings are the unanchored variables, which do not fail the check.

A zero-probability error means the evidence of a query has probability zero, so the query has no answer. The command works out the cause:

• If the query fails without its --given and --set values, or none were passed, the cause is the graph: no world satisfies every exclusive relation, which needs a base or strength of exactly 0 or 1.
• Otherwise the passed values are the cause, and everything the error names is established by running the same command, of the same kind, with that one thing changed. The message names a flag when the query fails with that flag's values alone, or both flags when only their values together fail. The hint names an exclusive relation, in details.exclusive_relation, only when the passed values make both of its propositions true and removing the relation makes the command succeed; otherwise it mentions exclusive relations only when the named values succeed without them.
• The remedies name only changes that were tried and make the command succeed. Dropping any one of details.drop_any_one_of does, or, when no single value does, dropping all of details.drop_all_of: the first set of the smallest size that does, of which there may be others, and none of whose values can be kept. A drop that leaves the command ill-formed, such as an intervene query without --set, is not offered. Moving any one of details.move_any_one_of off 0 and 1 does too, or all of details.move_all_of; those parameters are listed with their values in details.extreme_parameters, as base:NODE or strength:RELATION, and a base or strength at 0 or 1 that does not bear on the query is not listed. A move is tried at 0.5, which stands for every other value: in exact arithmetic, whether a query has probability zero depends only on which bases and strengths are exactly 0 or 1. A base or strength within rounding error of 0 or 1, such as 0.9999999999999999, can round a probability to zero, and is not listed as one to move.
• The search is bounded. Drops, moves and the removal of one exclusive relation are tried separately, never combined. Every single drop and every single move is tried; when none succeeds, sets of two, three and so on are tried, up to 256 sets for the drops and 256 for the moves. Each list is empty when no such change was found. When none is, the hint says so and that combined changes were not tried. details.search_tries gives, for drop and move, the number of single changes and of sets of several tried, and, when that search stopped at 256 sets, stopped_at: the size of the first set it left untried. Such a search is named in details.search_truncated, and the hint says what it tried and that a set not tried may still make the command succeed.

A problem-too-large error means exact inference would build an intermediate factor above the limit, in table entries; details.required is the size of the first such factor and details.limit the limit. query and diagnose take the limit as --max-factor-size (default 2²²), so passing at least details.required lets that factor through, though a later one may need more. The search for remedies to a zero-probability error runs under the same limit.

Values that contradict each other outright are refused before any inference, as an invalid-argument: two equivalent nodes given different values by --given (in query or diagnose) or by --set, and a --given value that contradicts the --set value of the same proposition.

Without --json, a success prints a short summary and a failure prints error: … and hint: … on stderr. Malformed command lines, such as a missing required option, are reported by the argument parser with status 2 in both modes.