Skip to content

Evaluators

The deterministic evaluators of the checks: exact arithmetic, symbolic identity, and numeric sampling.

A model writes the expressions these evaluate, so none of them is ever passed to eval, sympify or parse_expr. Each is parsed with ast and walked against a whitelist: numerals written in plain decimal or exponent notation, + - * / **, unary signs and parentheses, and, for the symbolic evaluator only, declared symbols, a fixed set of functions and the constants pi, E and I. Anything else is refused before any value is computed.

arithmetic computes exactly, with every numeral read as a Fraction of its decimal text. identity asks sympy whether two expressions are equal for every value of their symbols, and says error when sympy cannot decide. sample and agree_at evaluate the same two expressions at points drawn from a seeded generator, so the points can be recorded and drawn again.

Arithmetic dataclass

The exact value of an arithmetic expression.

Attributes:

Name Type Description
value Fraction

The value.

numerals tuple[str, ...]

The text of every numeral in the expression, in the order written.

Source code in src/credencegraph_extract/evaluate.py
126
127
128
129
130
131
132
133
134
135
136
@dataclass(frozen=True)
class Arithmetic:
    """The exact value of an arithmetic expression.

    Attributes:
        value: The value.
        numerals: The text of every numeral in the expression, in the order written.
    """

    value: Fraction
    numerals: tuple[str, ...]

EvaluationError

Bases: ValueError

An expression is refused or cannot be evaluated.

Source code in src/credencegraph_extract/evaluate.py
90
91
class EvaluationError(ValueError):
    """An expression is refused or cannot be evaluated."""

agree_at(lhs, rhs, declared, names, points)

Compare two expressions at each point, to a relative tolerance of RTOL.

The point's values are substituted exactly, as rationals, so a pole gives no finite value rather than a large one, and each side is then evaluated to 30 significant digits. Sides agree at a point when |lhs - rhs| <= RTOL * max(|lhs|, |rhs|).

Parameters:

Name Type Description Default
lhs Expr

One side.

required
rhs Expr

The other.

required
declared Sequence[Mapping[str, Any]]

The symbols with their assumptions, which every point must satisfy.

required
names Mapping[str, Symbol]

The symbols by name.

required
points Sequence[Mapping[str, str]]

The points, as returned by sample.

required

Returns:

Type Description
str

The result and what was found: passed when the sides agree at every point, failed at

str

the first point where they do not, error when a point breaks a symbol's assumptions or a

tuple[str, str]

side has no finite value there.

Source code in src/credencegraph_extract/evaluate.py
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
def agree_at(
    lhs: sympy.Expr,
    rhs: sympy.Expr,
    declared: Sequence[Mapping[str, Any]],
    names: Mapping[str, sympy.Symbol],
    points: Sequence[Mapping[str, str]],
) -> tuple[str, str]:
    """Compare two expressions at each point, to a relative tolerance of ``RTOL``.

    The point's values are substituted exactly, as rationals, so a pole gives no finite value rather
    than a large one, and each side is then evaluated to 30 significant digits. Sides agree at a point when
    ``|lhs - rhs| <= RTOL * max(|lhs|, |rhs|)``.

    Args:
        lhs: One side.
        rhs: The other.
        declared: The symbols with their assumptions, which every point must satisfy.
        names: The symbols by name.
        points: The points, as returned by ``sample``.

    Returns:
        The result and what was found: ``passed`` when the sides agree at every point, ``failed`` at
        the first point where they do not, ``error`` when a point breaks a symbol's assumptions or a
        side has no finite value there.
    """
    for i, point in enumerate(points):
        values = {name: Fraction(Decimal(text)) for name, text in point.items()}
        for item in declared:
            for assumption in item.get("assumptions", []):
                if not ASSUMPTIONS[assumption](values[item["name"]]):
                    return ERROR, f"point {i} gives {item['name']} = {point[item['name']]}, which is not {assumption}"
        subs = {names[name]: sympy.Rational(value) for name, value in values.items()}
        sides = []
        for side in (lhs, rhs):
            value = side.xreplace(subs).evalf(_DIGITS)
            real, imaginary = value.as_real_imag()
            if not (real.is_finite and imaginary.is_finite and real.is_number and imaginary.is_number):
                return ERROR, f"at point {i} a side has no finite value: {sympy.sstr(value)}"
            sides.append((Decimal(str(real)), Decimal(str(imaginary))))
        (lr, li), (rr, ri) = sides
        difference = abs(complex(lr - rr, li - ri))
        scale = max(abs(complex(lr, li)), abs(complex(rr, ri)))
        if Decimal(repr(difference)) > RTOL * Decimal(repr(scale)):
            return FAILED, f"at point {i} the sides differ: {lr}{li:+}i against {rr}{ri:+}i"
    return PASSED, f"the sides agree to rtol {RTOL} at all {len(points)} points"

arithmetic(expression)

Evaluate an arithmetic expression exactly.

Numerals are read as the Fraction of their decimal text, so 0.1 + 0.2 is exactly 3/10. A power needs an integer exponent, since a root of a fraction is not a fraction.

Parameters:

Name Type Description Default
expression str

Numerals joined by + - * / **, unary signs and parentheses.

required

Returns:

Type Description
Arithmetic

The value and the numerals.

Raises:

Type Description
EvaluationError

If the expression holds anything else, divides by zero, or raises to a power that is not an integer or is too large.

Source code in src/credencegraph_extract/evaluate.py
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
def arithmetic(expression: str) -> Arithmetic:
    """Evaluate an arithmetic expression exactly.

    Numerals are read as the ``Fraction`` of their decimal text, so ``0.1 + 0.2`` is exactly
    ``3/10``. A power needs an integer exponent, since a root of a fraction is not a fraction.

    Args:
        expression: Numerals joined by ``+ - * / **``, unary signs and parentheses.

    Returns:
        The value and the numerals.

    Raises:
        EvaluationError: If the expression holds anything else, divides by zero, or raises to a
            power that is not an integer or is too large.
    """
    tree = _parse(expression)
    found: list[str] = []

    def walk(node: ast.expr) -> Fraction:
        if isinstance(node, ast.Constant):
            text = _numeral(expression, node)
            found.append(text)
            return Fraction(Decimal(text))
        if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub | ast.UAdd):
            value = walk(node.operand)
            return -value if isinstance(node.op, ast.USub) else value
        if isinstance(node, ast.BinOp) and type(node.op) in _OPERATORS:
            left, right = walk(node.left), walk(node.right)
            if isinstance(node.op, ast.Pow):
                if right.denominator != 1:
                    msg = f"the exponent {right} is not an integer"
                    raise EvaluationError(msg)
                _check_power(left, right)
            try:
                return _OPERATORS[type(node.op)](left, right)
            except ZeroDivisionError:
                msg = "the expression divides by zero"
                raise EvaluationError(msg) from None
        msg = f"{ast.unparse(node)!r} is not arithmetic on numerals"
        raise EvaluationError(msg)

    return Arithmetic(walk(tree), tuple(found))

identity(lhs, rhs)

Decide symbolically whether two expressions are equal for every value of their symbols.

Parameters:

Name Type Description Default
lhs Expr

One side.

required
rhs Expr

The other.

required

Returns:

Type Description
str

The result and what sympy found: passed when their difference simplifies to zero,

str

failed when sympy shows the difference is never zero, error when it can show

tuple[str, str]

neither, since a difference sympy cannot simplify is not evidence either way.

Source code in src/credencegraph_extract/evaluate.py
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
def identity(lhs: sympy.Expr, rhs: sympy.Expr) -> tuple[str, str]:
    """Decide symbolically whether two expressions are equal for every value of their symbols.

    Args:
        lhs: One side.
        rhs: The other.

    Returns:
        The result and what sympy found: ``passed`` when their difference simplifies to zero,
        ``failed`` when sympy shows the difference is never zero, ``error`` when it can show
        neither, since a difference sympy cannot simplify is not evidence either way.
    """
    difference = sympy.simplify(lhs - rhs)
    if difference == 0:
        return PASSED, "the difference simplifies to 0"
    if difference.is_zero is False:
        return FAILED, f"the difference simplifies to {sympy.sstr(difference)}, which is never 0"
    return ERROR, f"sympy cannot decide whether {sympy.sstr(difference)} is 0"

sample(declared, seed, count=POINTS)

Draw points in the symbols' domains from a generator seeded with seed.

A symbol with the integer assumption takes integers in its domain; any other takes floats, written with repr so the recorded text reads back as the same value.

Parameters:

Name Type Description Default
declared Sequence[Mapping[str, Any]]

Each symbol as {"name", "assumptions", "domain": [low, high]}.

required
seed int

The seed.

required
count int

How many points.

POINTS

Returns:

Type Description
list[dict[str, str]]

The points, each a symbol name to the text of its value.

Raises:

Type Description
EvaluationError

If a symbol has no finite domain [low, high] with low < high, or an integer symbol's domain holds no integer.

Source code in src/credencegraph_extract/evaluate.py
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
def sample(declared: Sequence[Mapping[str, Any]], seed: int, count: int = POINTS) -> list[dict[str, str]]:
    """Draw points in the symbols' domains from a generator seeded with ``seed``.

    A symbol with the ``integer`` assumption takes integers in its domain; any other takes floats,
    written with ``repr`` so the recorded text reads back as the same value.

    Args:
        declared: Each symbol as ``{"name", "assumptions", "domain": [low, high]}``.
        seed: The seed.
        count: How many points.

    Returns:
        The points, each a symbol name to the text of its value.

    Raises:
        EvaluationError: If a symbol has no finite domain ``[low, high]`` with ``low < high``, or an
            integer symbol's domain holds no integer.
    """
    rng = random.Random(seed)  # noqa: S311 - reproducible sample points, not a secret
    bounds = []
    for item in declared:
        domain = item.get("domain")
        if (
            not isinstance(domain, list)
            or len(domain) != 2  # noqa: PLR2004 - a domain is [low, high]
            or not all(isinstance(end, int | float) and not isinstance(end, bool) for end in domain)
            or not float("-inf") < domain[0] < domain[1] < float("inf")
        ):
            msg = f"symbol {item['name']!r} needs a finite domain [low, high] with low < high"
            raise EvaluationError(msg)
        bounds.append((item["name"], domain[0], domain[1], "integer" in item.get("assumptions", [])))
    points = []
    for _ in range(count):
        point = {}
        for name, low, high, integer in bounds:
            if integer:
                start, stop = int(sympy.ceiling(low)), int(sympy.floor(high))
                if start > stop:
                    msg = f"the domain of integer symbol {name!r} holds no integer"
                    raise EvaluationError(msg)
                point[name] = str(rng.randint(start, stop))
            else:
                point[name] = repr(rng.uniform(low, high))
        points.append(point)
    return points

seed_of(encoding)

Return the seed of an encoding's sample points: the first 52 bits of the SHA-256 of its canonical JSON.

52 bits keep the seed exact in any JSON reader, which reads numbers as doubles.

Parameters:

Name Type Description Default
encoding Mapping[str, Any]

The encoding.

required

Returns:

Type Description
int

The seed.

Source code in src/credencegraph_extract/evaluate.py
311
312
313
314
315
316
317
318
319
320
321
322
323
def seed_of(encoding: Mapping[str, Any]) -> int:
    """Return the seed of an encoding's sample points: the first 52 bits of the SHA-256 of its canonical JSON.

    52 bits keep the seed exact in any JSON reader, which reads numbers as doubles.

    Args:
        encoding: The encoding.

    Returns:
        The seed.
    """
    text = json.dumps(encoding, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
    return int(hashlib.sha256(text.encode("utf-8")).hexdigest()[:13], 16)

symbolic(expression, names)

Build the sympy expression of a text, without eval.

Parameters:

Name Type Description Default
expression str

Numerals, the symbols in names, the functions of FUNCTIONS called with positional arguments, the constants of CONSTANTS, + - * / **, unary signs and parentheses.

required
names Mapping[str, Symbol]

The declared symbols.

required

Returns:

Type Description
Expr

The expression; numerals are exact rationals.

Raises:

Type Description
EvaluationError

If the text holds anything else, or a power of two numbers is too large.

Source code in src/credencegraph_extract/evaluate.py
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
def symbolic(expression: str, names: Mapping[str, sympy.Symbol]) -> sympy.Expr:
    """Build the sympy expression of a text, without ``eval``.

    Args:
        expression: Numerals, the symbols in ``names``, the functions of ``FUNCTIONS`` called with
            positional arguments, the constants of ``CONSTANTS``, ``+ - * / **``, unary signs and
            parentheses.
        names: The declared symbols.

    Returns:
        The expression; numerals are exact rationals.

    Raises:
        EvaluationError: If the text holds anything else, or a power of two numbers is too large.
    """
    tree = _parse(expression)

    def walk(node: ast.expr) -> Any:
        if isinstance(node, ast.Constant):
            return sympy.Rational(Fraction(Decimal(_numeral(expression, node))))
        if isinstance(node, ast.Name) and node.id in names:
            return names[node.id]
        if isinstance(node, ast.Name) and node.id in CONSTANTS:
            return CONSTANTS[node.id]
        if isinstance(node, ast.UnaryOp) and isinstance(node.op, ast.USub | ast.UAdd):
            value = walk(node.operand)
            return -value if isinstance(node.op, ast.USub) else value
        if isinstance(node, ast.BinOp) and type(node.op) in _OPERATORS:
            left, right = walk(node.left), walk(node.right)
            if isinstance(node.op, ast.Pow):
                _check_power(left, right)
            return _OPERATORS[type(node.op)](left, right)
        if (
            isinstance(node, ast.Call)
            and isinstance(node.func, ast.Name)
            and node.func.id in FUNCTIONS
            and not node.keywords
            and node.args
        ):
            return FUNCTIONS[node.func.id](*(walk(arg) for arg in node.args))
        msg = f"{ast.unparse(node)!r} is not an allowed expression: use declared symbols, numerals and {sorted(FUNCTIONS)}"
        raise EvaluationError(msg)

    return walk(tree)

symbols(declared)

Create the sympy symbols an encoding declares.

Parameters:

Name Type Description Default
declared Sequence[Mapping[str, Any]]

Each symbol as {"name", "assumptions": [...]}.

required

Returns:

Type Description
dict[str, Symbol]

The symbols by name.

Raises:

Type Description
EvaluationError

If a name is not an identifier, is repeated, or names a function or constant; or an assumption is not one of ASSUMPTIONS, or the assumptions contradict each other.

Source code in src/credencegraph_extract/evaluate.py
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
def symbols(declared: Sequence[Mapping[str, Any]]) -> dict[str, sympy.Symbol]:
    """Create the sympy symbols an encoding declares.

    Args:
        declared: Each symbol as ``{"name", "assumptions": [...]}``.

    Returns:
        The symbols by name.

    Raises:
        EvaluationError: If a name is not an identifier, is repeated, or names a function or constant;
            or an assumption is not one of ``ASSUMPTIONS``, or the assumptions contradict each other.
    """
    made: dict[str, sympy.Symbol] = {}
    for item in declared:
        name, assumptions = item["name"], item.get("assumptions", [])
        if not name.isidentifier() or name in FUNCTIONS or name in CONSTANTS or name in made:
            msg = f"{name!r} cannot name a symbol"
            raise EvaluationError(msg)
        unknown = sorted(set(assumptions) - set(ASSUMPTIONS))
        if unknown:
            msg = f"symbol {name!r} has assumptions {unknown}, not among {list(ASSUMPTIONS)}"
            raise EvaluationError(msg)
        try:
            made[name] = sympy.Symbol(name, **dict.fromkeys(assumptions, True))
        except InconsistentAssumptions as error:
            msg = f"symbol {name!r} has contradictory assumptions: {error}"
            raise EvaluationError(msg) from None
    return made

text_numerals(text)

Return every numeral of a text with the rounding its own digits imply.

The numerals are found with the core's numeral pattern, the one its faithfulness rule uses, and each value is computed the same way: groups removed, the exponent applied, the sign kept. The rounding is half a unit of the last digit written, so 30 is 30 ± 0.5 and 1.20e-3 is 0.00120 ± 0.000005. A numeral followed by % contributes its value and its value divided by 100, each with its rounding scaled alike, as the core's numerals does.

Parameters:

Name Type Description Default
text str

The text.

required

Returns:

Type Description
list[tuple[Fraction, Fraction]]

(value, half unit) pairs, in the order the numerals occur.

Source code in src/credencegraph_extract/evaluate.py
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
def text_numerals(text: str) -> list[tuple[Fraction, Fraction]]:
    """Return every numeral of a text with the rounding its own digits imply.

    The numerals are found with the core's numeral pattern, the one its faithfulness rule uses, and
    each value is computed the same way: groups removed, the exponent applied, the sign kept. The
    rounding is half a unit of the last digit written, so ``30`` is ``30 ± 0.5`` and ``1.20e-3`` is
    ``0.00120 ± 0.000005``. A numeral followed by ``%`` contributes its value and its value divided by
    100, each with its rounding scaled alike, as the core's ``numerals`` does.

    Args:
        text: The text.

    Returns:
        ``(value, half unit)`` pairs, in the order the numerals occur.
    """
    found = []
    for match in _TEXT_NUMERAL.finditer(text):
        digits = Decimal(re.sub(_GROUP, "", match["digits"]))
        exponent = match["e"] or match["braced"] or match["bare"]
        shift = int(exponent.replace(_MINUS, "-")) if exponent is not None else 0
        value = Fraction(digits.scaleb(shift))
        half = Fraction(Decimal(1).scaleb(digits.as_tuple().exponent + shift)) / 2  # type: ignore[operator]
        if match["sign"] in ("-", _MINUS):
            value = -value
        found.append((value, half))
        if match["percent"]:
            found.append((value / 100, half / 100))
    return found