Module refinery.lib.scripts.au3.analysis.usage
The usage model: what the script does with its names. The dead-code elimination in
refinery.lib.scripts.au3.deobfuscation.eliminate asks it what a direct call names, which names
a string or a directive spells, which calls may consume a computed name, which storage every
variable occurrence binds or may bind, how often every name is read, and which expressions are
effect-free — each fact owned by the module that pinned it: the bindings by
refinery.lib.scripts.au3.analysis.scopes, the may-write summaries by
refinery.lib.scripts.au3.analysis.propagation, the builtin knowledge by
refinery.lib.scripts.au3.analysis.builtins.
Source text the parser could not read — an error node in the tree — is live source AutoIt may
run when it reaches it, so the model reads it through
read_unparsed() and counts every name it spells as a
reference: a call, a binding that may fall either side, a spelling, a read. Text that opens a
call on a computed callee or names the builtin Call makes any callee possible — the text
cannot show the literalness of Call's argument, and whether a name it spells shadows a
builtin is unpinned — an Execute it spells joins the names the census counts, and the
error-state macros it spells count as reads. The model holds the facts of every error node
keyed by its node and the variables of every error node, for the transforms that ask them. The
parser's record list is not an input: it outlives the edits that remove the nodes it names, so
a model over it would keep refusing a tree its own transforms had cleaned.
Whether a call may read a name no census sees is the question beside the write-side channel one,
and the same walk answers both. A computed callee reads — it could name Eval; so does an
Execute whose payload the tree does not show as one literal expression, and a non-literal in a
name-parameter position of a builtin the table knows: a literal there reaches a name only through
a string the spelling corpus holds, which is what the spelling answer refuses a removal over. A
literal Execute payload is parsed and its expression judged by the same shapes, recursed to a
bounded depth; a payload that fails to parse may still be one AutoIt runs, so it reads — the
empty payload AutoIt answers "" for (probe3) reads nothing — and past the depth bound the
payload is not parsed and reads. A Call invocation over a string literal resolves through
call_target(), and the target is judged as if the tree
called it directly; every refusal reads, because unread text cannot resolve what a name shadows.
A script function's reads stand in the tree, and a member callee dispatches through an object
and reads no script name. The reads of a name are counted per storage through
Au3ScopeModel.read_storages(): an occurrence reads when
it is not a write occurrence or reads what it writes, and every storage it may reach — one
function's local, or the global — counts it, a variable of unread text counting into both facts
of its node's scope. Both counts are may-answers: a read whose claim shares its statement counts
into the local and the global alike.
An expression is effect-free when evaluating it writes nothing observable and cannot terminate
the script. Termination is what the arithmetic refuses: a division by zero ends the script
(pinned by the probe corpus), an out-of-range subscript ends it the same way, and a wrong-arity
call as well, so /, ^, and array access are refused. A call is effect-free only over a pure
script function of matching parameter count that declares no ByRef parameter, its arguments
effect-free with it; no builtin call is, because a builtin's value-dependent fatals are
unpinned and the builtin name universe phase owns the table that would change that. A function
is pure when its may-write summary is empty, its body holds no Exit and no Static — a
static's value survives the call that first initializes it, so which call runs first is
observable — and every expression in it is effect-free where a call inside the body names a
pure function that cannot reach itself through the survivors' calls — the fixpoint of that
rule. Entering a function clears @error and @extended (probe9 A17/A17b), so a script that
may read the error state holds no pure function at all: every call into one writes what the
script observes. A recursion cycle may never return — AutoIt ends the script at its recursion
limit — so a call into one is not an effect-free evaluation and no member of a cycle is pure.
Expand source code Browse git
"""
The usage model: what the script does with its names. The dead-code elimination in
`refinery.lib.scripts.au3.deobfuscation.eliminate` asks it what a direct call names, which names
a string or a directive spells, which calls may consume a computed name, which storage every
variable occurrence binds or may bind, how often every name is read, and which expressions are
effect-free — each fact owned by the module that pinned it: the bindings by
`refinery.lib.scripts.au3.analysis.scopes`, the may-write summaries by
`refinery.lib.scripts.au3.analysis.propagation`, the builtin knowledge by
`refinery.lib.scripts.au3.analysis.builtins`.
Source text the parser could not read — an error node in the tree — is live source AutoIt may
run when it reaches it, so the model reads it through
`refinery.lib.scripts.au3.analysis.unparsed.read_unparsed` and counts every name it spells as a
reference: a call, a binding that may fall either side, a spelling, a read. Text that opens a
call on a computed callee or names the builtin `Call` makes any callee possible — the text
cannot show the literalness of `Call`'s argument, and whether a name it spells shadows a
builtin is unpinned — an `Execute` it spells joins the names the census counts, and the
error-state macros it spells count as reads. The model holds the facts of every error node
keyed by its node and the variables of every error node, for the transforms that ask them. The
parser's record list is not an input: it outlives the edits that remove the nodes it names, so
a model over it would keep refusing a tree its own transforms had cleaned.
Whether a call may read a name no census sees is the question beside the write-side channel one,
and the same walk answers both. A computed callee reads — it could name `Eval`; so does an
`Execute` whose payload the tree does not show as one literal expression, and a non-literal in a
name-parameter position of a builtin the table knows: a literal there reaches a name only through
a string the spelling corpus holds, which is what the spelling answer refuses a removal over. A
literal `Execute` payload is parsed and its expression judged by the same shapes, recursed to a
bounded depth; a payload that fails to parse may still be one AutoIt runs, so it reads — the
empty payload AutoIt answers `""` for (probe3) reads nothing — and past the depth bound the
payload is not parsed and reads. A `Call` invocation over a string literal resolves through
`refinery.lib.scripts.au3.analysis.existence.call_target`, and the target is judged as if the tree
called it directly; every refusal reads, because unread text cannot resolve what a name shadows.
A script function's reads stand in the tree, and a member callee dispatches through an object
and reads no script name. The reads of a name are counted per storage through
`refinery.lib.scripts.au3.analysis.scopes.Au3ScopeModel.read_storages`: an occurrence reads when
it is not a write occurrence or reads what it writes, and every storage it may reach — one
function's local, or the global — counts it, a variable of unread text counting into both facts
of its node's scope. Both counts are may-answers: a read whose claim shares its statement counts
into the local and the global alike.
An expression is effect-free when evaluating it writes nothing observable and cannot terminate
the script. Termination is what the arithmetic refuses: a division by zero ends the script
(pinned by the probe corpus), an out-of-range subscript ends it the same way, and a wrong-arity
call as well, so `/`, `^`, and array access are refused. A call is effect-free only over a pure
script function of matching parameter count that declares no `ByRef` parameter, its arguments
effect-free with it; no builtin call is, because a builtin's value-dependent fatals are
unpinned and the builtin name universe phase owns the table that would change that. A function
is pure when its may-write summary is empty, its body holds no `Exit` and no `Static` — a
static's value survives the call that first initializes it, so which call runs first is
observable — and every expression in it is effect-free where a call inside the body names a
pure function that cannot reach itself through the survivors' calls — the fixpoint of that
rule. Entering a function clears `@error` and `@extended` (probe9 A17/A17b), so a script that
may read the error state holds no pure function at all: every call into one writes what the
script observes. A recursion cycle may never return — AutoIt ends the script at its recursion
limit — so a call into one is not an effect-free evaluation and no member of a cycle is pure.
"""
from __future__ import annotations
import itertools
from collections import Counter
from collections.abc import Iterator, Mapping, Sequence
from dataclasses import dataclass, field
from refinery.lib.scripts import Node
from refinery.lib.scripts.au3 import ast
from refinery.lib.scripts.au3.analysis.builtins import NAME_PARAMETERS
from refinery.lib.scripts.au3.analysis.existence import call_target, execute_kills, parse_payload
from refinery.lib.scripts.au3.analysis.scopes import Au3ScopeModel, Binding, INCLUDE_SPELLING
from refinery.lib.scripts.au3.analysis.unparsed import Au3UnparsedText, read_unparsed
from refinery.lib.scripts.au3.model import (
Au3ArrayLiteral,
Au3BinaryExpression,
Au3BooleanLiteral,
Au3Call,
Au3DefaultLiteral,
Au3Directive,
Au3EnumDeclarator,
Au3ErrorNode,
Au3ExitStatement,
Au3FloatLiteral,
Au3FunctionDeclaration,
Au3Identifier,
Au3IntLiteral,
Au3Macro,
Au3MemberAccess,
Au3NullLiteral,
Au3Script,
Au3StringLiteral,
Au3Ternary,
Au3UnaryExpression,
Au3Variable,
Au3VariableDeclaration,
Au3VariableDeclarator,
Expression,
)
_EFFECT_FREE_READS = (
Au3BooleanLiteral,
Au3DefaultLiteral,
Au3FloatLiteral,
Au3Identifier,
Au3IntLiteral,
Au3Macro,
Au3NullLiteral,
Au3StringLiteral,
Au3Variable,
)
@dataclass(frozen=True)
class Au3UsageModel:
"""
What one tree version does with its names, built once per tree version by
`build_usage_model`: the script's functions, the direct-identifier call count of each, how
often each storage is read — one function's local, or the global per scope, the variables of
unread text among its readers — the functions that are pure, the calls and error nodes that
may consume a computed name, the calls and error nodes that may read one that no census
sees, the lowercased text of every string literal and directive, parsed or unread, the
local storage an occurrence in a scope may bind, per scope, and the storage every occurrence
provably binds globally, per name, the script level among the readers as `None`. The facts
of every error node are held keyed by their node, and the variables of every error node
answer the transforms that ask which names unread text may reference.
"""
functions: dict[str, Au3FunctionDeclaration]
call_counts: dict[str, int]
local_read_counts: dict[Au3FunctionDeclaration, dict[str, int]]
global_read_counts: dict[Au3FunctionDeclaration | None, dict[str, int]]
pure_functions: frozenset[str]
channels: frozenset[Au3Call | Au3ErrorNode]
reading_channels: frozenset[Au3Call | Au3ErrorNode]
spellings: str
local_bindings: dict[Au3FunctionDeclaration, frozenset[str]]
global_bindings: dict[str, frozenset[Au3FunctionDeclaration | None]]
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText]
unparsed_variables: frozenset[str]
_spelled: dict[str, bool] = field(default_factory=dict, compare=False, repr=False)
def spells(self, name: str) -> bool:
"""
Whether *name* — a function name, or a variable name without its `$` — is a substring of
any string literal or directive the tree holds, parsed or unread: a computed name has to
be built from string data, so a name some string spells is one no removal may take away.
The answer is memoized per name — the scan is linear in the corpus — and the model is
rebuilt per tree version, so a memo never outlives the facts it answered from.
"""
answer = self._spelled.get(name)
if answer is None:
answer = name.lstrip('$') in self.spellings
self._spelled[name] = answer
return answer
def holds_block_keyword_text(self, root: Node) -> bool:
"""
Whether *root*'s subtree holds error text the reader marks with an AutoIt block keyword:
deleting it could move where the real parser's block boundaries fall, and with them which
code is live at all.
"""
return any(
self.unparsed[node].block_keyword
for node in root.walk()
if isinstance(node, Au3ErrorNode)
)
def holds_unread_include(self) -> bool:
"""
Whether unread source text holds an include directive: the file it names is outside every
fact the tree holds, and may call anything a removal would take away.
"""
return any(
INCLUDE_SPELLING.match(directive)
for facts in self.unparsed.values()
for directive in facts.directives
)
def effect_free(self, expression: Expression | None) -> bool:
"""
Whether evaluating *expression* writes nothing observable and cannot terminate the
script.
"""
return _effect_free(expression, self.functions, self.pure_functions)
#: The interpreter's error-state macros: entering a user function clears both, so a read of
#: either observes what every call wrote.
ERROR_MACROS = frozenset({'error', 'extended'})
@dataclass(frozen=True)
class Au3CalleeCensus:
"""
What the calls of one tree name, gathered once for every question that asks it: the names a
direct identifier or a `Call` invocation over a string literal spells, whether any callee is
computed at run time — a value the tree does not show, which could name anything — and
whether the tree reads the interpreter's `@error` or `@extended`.
"""
callees: frozenset[str]
computed_callee: bool
reads_error_macros: bool
def observes_error_state(self) -> bool:
"""
Whether the script may read the interpreter's `@error` and `@extended` at run time: a
read of either macro anywhere, or a call to `Execute` by any spelling the census sees —
whose payload text is a reader no walk reads.
"""
return (
self.reads_error_macros
or self.computed_callee
or 'execute' in self.callees
)
def callee_census(
root: Au3Script,
functions: Mapping[str, Au3FunctionDeclaration],
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText],
) -> Au3CalleeCensus:
"""
Count what the calls of *root* name. A member callee is the one exception to every reader: it
dispatches through an object and reaches no builtin and no script function. A script function
named `Call` shadows the builtin, so its first argument is an ordinary argument rather than a
name; otherwise a `Call` names the callee `call_target` resolves its string literal to, and
one whose name is no literal or whose resolution refuses makes any callee possible. Source
text the parser could not read is read through the facts *unparsed* holds: a
computed callee it opens, or a `Call` it spells — whose argument's literalness the text cannot
show — makes any callee possible, an `Execute` it spells joins the names, and the error-state
macros it spells count as reads.
"""
callees: set[str] = set()
computed_callee = False
reads_error_macros = False
function_names = frozenset(functions)
for node in root.walk():
if isinstance(node, Au3Macro):
reads_error_macros = reads_error_macros or node.name in ERROR_MACROS
elif isinstance(node, Au3Call):
callee = ast.direct_callee(node)
if callee is not None:
callees.add(callee.name)
if callee.name == 'call' and callee.name not in functions:
literal = (
ast.unwrap_parens(node.arguments[0])
if node.arguments else None
)
target = (
call_target(literal.value, node.arguments[1:], function_names)
if isinstance(literal, Au3StringLiteral) else None
)
if target is None:
computed_callee = True
else:
callees.add(target)
elif not isinstance(ast.unwrap_parens(node.callee), Au3MemberAccess):
computed_callee = True
for facts in unparsed.values():
computed_callee = (
computed_callee
or facts.computed_callee
or 'call' in facts.identifiers
)
reads_error_macros = (
reads_error_macros
or not ERROR_MACROS.isdisjoint(facts.macros)
)
if 'execute' in facts.identifiers:
callees.add('execute')
return Au3CalleeCensus(frozenset(callees), computed_callee, reads_error_macros)
def _effect_free(
expression: Expression | None,
functions: dict[str, Au3FunctionDeclaration],
pure: frozenset[str],
) -> bool:
node = ast.unwrap_parens(expression)
if node is None or isinstance(node, _EFFECT_FREE_READS):
return True
if isinstance(node, Au3BinaryExpression):
if node.operator in ('/', '^'):
return False
return (
_effect_free(node.left, functions, pure)
and _effect_free(node.right, functions, pure)
)
if isinstance(node, Au3UnaryExpression):
return _effect_free(node.operand, functions, pure)
if isinstance(node, Au3Ternary):
return (
_effect_free(node.condition, functions, pure)
and _effect_free(node.consequent, functions, pure)
and _effect_free(node.alternate, functions, pure)
)
if isinstance(node, Au3ArrayLiteral):
return all(_effect_free(element, functions, pure) for element in node.elements)
if isinstance(node, Au3Call):
return _call_is_effect_free(node, functions, pure)
return False
def _call_is_effect_free(
call: Au3Call,
functions: dict[str, Au3FunctionDeclaration],
pure: frozenset[str],
) -> bool:
callee = ast.direct_callee(call)
if callee is None:
return False
if callee.name not in functions:
if callee.name == 'execute':
return _execute_is_effect_free(call, functions, pure)
return False
if callee.name not in pure:
return False
function = functions[callee.name]
if len(call.arguments) != len(function.params):
return False
if any(parameter.by_ref for parameter in function.params):
return False
return all(_effect_free(argument, functions, pure) for argument in call.arguments)
def _execute_is_effect_free(
call: Au3Call,
functions: dict[str, Au3FunctionDeclaration],
pure: frozenset[str],
) -> bool:
"""
An `Execute` call over one string literal: the payload must parse, write nothing — the one
question `execute_kills` answers — and its parse must be effect-free.
"""
if execute_kills(call) is None:
return False
literal = ast.unwrap_parens(call.arguments[0])
if not isinstance(literal, Au3StringLiteral):
return False
expression = parse_payload(literal.value)
if expression is None:
return False
return _effect_free(expression, functions, pure)
def _holds_exit(function: Au3FunctionDeclaration) -> bool:
return any(isinstance(node, Au3ExitStatement) for node in ast.scope_nodes(function.body))
def _holds_static(function: Au3FunctionDeclaration) -> bool:
return any(
isinstance(node, Au3VariableDeclaration) and node.is_static
for node in ast.scope_nodes(function.body)
)
def _called_names(function: Au3FunctionDeclaration) -> set[str]:
"""
The names the calls directly in *function*'s body name — a nested function's body is a scope
of its own and its calls are not this function's.
"""
names: set[str] = set()
for node in ast.scope_nodes(function.body):
if not isinstance(node, Au3Call):
continue
callee = ast.direct_callee(node)
if callee is not None:
names.add(callee.name)
return names
def _cyclic_within(
pure: set[str],
functions: dict[str, Au3FunctionDeclaration],
) -> set[str]:
"""
The functions of *pure* that can reach themselves through calls into *pure*: a call into one
of them may never return — AutoIt ends the script at its recursion limit — so no evaluation
of such a call is effect-free.
"""
graph = {name: _called_names(functions[name]) & pure for name in pure}
cyclic: set[str] = set()
for name in pure:
reachable: set[str] = set()
pending = list(graph[name])
while pending:
current = pending.pop()
if current in reachable:
continue
reachable.add(current)
pending.extend(graph[current])
if name in reachable:
cyclic.add(name)
return cyclic
def _holds_only_effect_free_expressions(
function: Au3FunctionDeclaration,
functions: dict[str, Au3FunctionDeclaration],
pure: frozenset[str],
) -> bool:
for node in ast.scope_nodes(function.body):
if isinstance(node, Expression) and not isinstance(node.parent, Expression):
if not _effect_free(node, functions, pure):
return False
return True
def _build_purity(
root: Au3Script,
scope_model: Au3ScopeModel,
summaries: dict[str, frozenset[str] | None],
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText],
) -> frozenset[str]:
"""
The pure functions of the script: every function whose may-write summary is empty, whose body
holds no `Exit` and no `Static` — a static's value survives the call that first initializes
it, so which call runs first is observable — shrunk until every survivor's body also holds no
expression that is not effect-free under the survivors themselves and no survivor can reach
itself through the survivors' calls, which casts its callers out with it: a recursion cycle
may never return, and AutoIt ends the script at its recursion limit.
Entering a function clears `@error` and `@extended` (probe9 A17/A17b), so a script that may
read the error state holds no pure function at all: every call into one writes what the
script observes.
"""
if callee_census(root, scope_model.functions, unparsed).observes_error_state():
return frozenset()
functions = scope_model.functions
pure = {
name
for name, function in functions.items()
if summaries.get(name) == frozenset()
and not _holds_exit(function)
and not _holds_static(function)
}
while True:
survivors = {
name
for name in pure
if _holds_only_effect_free_expressions(functions[name], functions, frozenset(pure))
and name not in _cyclic_within(pure, functions)
}
if survivors == pure:
return frozenset(pure)
pure = survivors
@dataclass(frozen=True)
class Au3StorageCensus:
"""
The storage every variable occurrence binds or may bind, and how often each storage is read:
the names a function's occurrences may bind locally, the scopes every name provably binds
globally in, the reads that may reach one function's local, and the reads standing in each
scope that may reach the global — script level keyed under `None`. Both read counts are
may-answers: a read that may reach either storage counts into both.
"""
local_bindings: dict[Au3FunctionDeclaration, frozenset[str]]
global_bindings: dict[str, frozenset[Au3FunctionDeclaration | None]]
local_read_counts: dict[Au3FunctionDeclaration, dict[str, int]]
global_read_counts: dict[Au3FunctionDeclaration | None, dict[str, int]]
def _collect_bindings(
root: Au3Script,
scope_model: Au3ScopeModel,
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText],
) -> Au3StorageCensus:
"""
The storage every variable occurrence binds or may bind, over every scope of *root*, and how
often each storage is read: a write through the scope model's write binding and a read
through its read binding, an occurrence that may reach a local of its scope through
`may_bind_local`, and the variables of a function's parameter defaults among the occurrences
— a default evaluates in the function when a call leaves its argument out. The declared
variable of a declarator is excluded — the declaration is what the answers are asked about
— while a bare-variable initializer reads like any other occurrence: it is not the
declarator's own variable. A write that proves nothing binds no storage: its value reaches a
global only through a read somewhere, and that read is the occurrence that protects the
name. A read counts once per storage it may reach, through `read_storages`; a variable of
unread text counts into both facts of its node's scope — unread text may bind either side,
and a script-level error node has no local.
"""
local: dict[Au3FunctionDeclaration, set[str]] = {}
global_names: dict[str, set[Au3FunctionDeclaration | None]] = {}
local_reads: dict[Au3FunctionDeclaration, dict[str, int]] = {}
global_reads: dict[Au3FunctionDeclaration | None, dict[str, int]] = {}
for scope, body in ast.scope_bodies(root):
occurrences: Iterator[Node] = ast.scope_nodes(body)
if scope is not None:
occurrences = itertools.chain(occurrences, *(
parameter.default.walk()
for parameter in scope.params
if parameter.default is not None
))
for node in occurrences:
if not isinstance(node, Au3Variable):
continue
parent = node.parent
if (
isinstance(parent, (Au3VariableDeclarator, Au3EnumDeclarator))
and parent.variable is node
):
continue
statement = ast.owning_statement(node)
if statement is None:
continue
if not ast.is_write_occurrence(node) or ast.reads_what_it_writes(node):
for storage in scope_model.read_storages(scope, node, node.name):
if storage is Binding.GLOBAL:
counts = global_reads.setdefault(scope, {})
else:
assert scope is not None
counts = local_reads.setdefault(scope, {})
counts[node.name] = counts.get(node.name, 0) + 1
if ast.is_write_occurrence(node):
binding = scope_model.binding(scope, statement, node.name)
else:
binding = scope_model.read_binding(scope, node, node.name)
if binding is Binding.GLOBAL:
global_names.setdefault(node.name, set()).add(scope)
if (
binding is Binding.LOCAL
or scope_model.may_bind_local(scope, node, node.name)
):
assert scope is not None
local.setdefault(scope, set()).add(node.name)
for node, facts in unparsed.items():
scope = ast.enclosing_function(node)
for name in facts.variables:
counts = global_reads.setdefault(scope, {})
counts[name] = counts.get(name, 0) + 1
if scope is not None:
counts = local_reads.setdefault(scope, {})
counts[name] = counts.get(name, 0) + 1
local.setdefault(scope, set()).add(name)
global_names.setdefault(name, set()).add(scope)
return Au3StorageCensus(
{scope: frozenset(names) for scope, names in local.items()},
{name: frozenset(scopes) for name, scopes in global_names.items()},
local_reads,
global_reads,
)
#: The depth a literal `Execute` payload's read verdict recurses to: each level is one payload
#: parse, and past the bound the payload is not parsed and counts as reading. Literal payloads
#: are short and rare once the constant fold has run.
_PAYLOAD_READ_DEPTH = 8
def _name_parameter_is_computed(name: str, arguments: Sequence[Expression]) -> bool:
"""
Whether a call to *name* holds a non-literal in a name-parameter position the builtin table
knows: a value the tree does not show reaches a name through it.
"""
for index in NAME_PARAMETERS.get(name, frozenset()):
if index >= len(arguments):
continue
if not isinstance(ast.unwrap_parens(arguments[index]), Au3StringLiteral):
return True
return False
def _collect_channels(
root: Au3Script,
scope_model: Au3ScopeModel,
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText],
) -> tuple[frozenset[Au3Call | Au3ErrorNode], frozenset[Au3Call | Au3ErrorNode]]:
"""
Every call and error node that may consume a computed name, and every call and error node
that may read one that no census sees — the write-side question and the read-side one,
answered by the same walk. A write channel: a callee that is neither an identifier nor a
member access, an `Execute` whose payload `execute_kills` refuses, and a non-literal in a
name-parameter position of a builtin the table knows. A read channel: the shapes
`_named_callee_reads` names. A member callee dispatches through an object and reaches no
script function or variable, so it is the one non-identifier callee that is neither. A call
naming a script function is answered by the call-count and spelling facts instead. An error
node joins both under one rule: its text opens a call on a computed callee or spells
`Execute` or a name-parameter builtin — whether a name it spells shadows the builtin is
unresolvable in unread text, so a name it spells always counts.
"""
channels: set[Au3Call | Au3ErrorNode] = set()
reading: set[Au3Call | Au3ErrorNode] = set()
for node in root.walk():
if not isinstance(node, Au3Call):
continue
callee = ast.direct_callee(node)
if callee is None:
if not isinstance(ast.unwrap_parens(node.callee), Au3MemberAccess):
channels.add(node)
reading.add(node)
continue
if callee.name in scope_model.functions:
continue
if _named_callee_reads(callee.name, node.arguments, scope_model, _PAYLOAD_READ_DEPTH):
reading.add(node)
if callee.name == 'execute':
if execute_kills(node) is None:
channels.add(node)
continue
if _name_parameter_is_computed(callee.name, node.arguments):
channels.add(node)
for node, facts in unparsed.items():
if (
facts.computed_callee
or 'execute' in facts.identifiers
or not NAME_PARAMETERS.keys().isdisjoint(facts.identifiers)
):
channels.add(node)
reading.add(node)
return frozenset(channels), frozenset(reading)
def _named_callee_reads(
name: str,
arguments: Sequence[Expression],
scope_model: Au3ScopeModel,
depth: int,
) -> bool:
"""
Whether a call to *name* over *arguments* — as the tree spells it, or as a `Call` invocation
over a string literal resolves it — may read a name no census sees: a computed callee could
name `Eval`, an `Execute` whose payload is not one literal expression can execute arbitrary
code, and a non-literal in a name-parameter position reaches a name through a value the tree
does not show. A script function's reads stand in the tree, and a member callee is answered
by the caller. Every resolution refusal reads: unread text cannot resolve what a name
shadows.
"""
if name in scope_model.functions:
return False
if name == 'execute':
return _execute_reads(arguments, scope_model, depth)
if name == 'call':
literal = ast.unwrap_parens(arguments[0]) if arguments else None
if not isinstance(literal, Au3StringLiteral):
return True
target = call_target(literal.value, arguments[1:], scope_model.function_names)
return target is None or _named_callee_reads(target, arguments[1:], scope_model, depth)
return _name_parameter_is_computed(name, arguments)
def _execute_reads(
arguments: Sequence[Expression],
scope_model: Au3ScopeModel,
depth: int,
) -> bool:
"""
Whether an `Execute` over *arguments* may read a name no census sees: a payload the tree does
not show as one literal expression can execute arbitrary code, and a literal one reads only
what its parsed expression holds of the shapes `_named_callee_reads` names, judged one depth
level down. A payload that fails to parse may still be one AutoIt runs, so it reads; the
empty payload AutoIt answers `""` for (probe3) reads nothing.
"""
if depth <= 0 or len(arguments) != 1:
return True
literal = ast.unwrap_parens(arguments[0])
if not isinstance(literal, Au3StringLiteral):
return True
if not literal.value.strip():
return False
expression = parse_payload(literal.value)
if expression is None:
return True
for node in expression.walk():
if not isinstance(node, Au3Call):
continue
callee = ast.direct_callee(node)
if callee is None:
if not isinstance(ast.unwrap_parens(node.callee), Au3MemberAccess):
return True
elif _named_callee_reads(callee.name, node.arguments, scope_model, depth - 1):
return True
return False
def _collect_call_counts(
root: Au3Script,
scope_model: Au3ScopeModel,
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText],
) -> dict[str, int]:
counts: Counter[str] = Counter()
for node in root.walk():
if not isinstance(node, Au3Call):
continue
callee = ast.direct_callee(node)
if callee is not None and callee.name in scope_model.functions:
counts[callee.name] += 1
for facts in unparsed.values():
for identifier in facts.identifiers:
if identifier in scope_model.functions:
counts[identifier] += 1
return dict(counts)
def _collect_spellings(
root: Au3Script,
unparsed: Mapping[Au3ErrorNode, Au3UnparsedText],
) -> str:
pieces = []
for node in root.walk():
if isinstance(node, Au3StringLiteral):
pieces.append(node.value)
elif isinstance(node, Au3Directive):
pieces.append(node.text)
for facts in unparsed.values():
pieces.extend(facts.strings)
pieces.extend(facts.directives)
return '\n'.join(pieces).lower()
def _collect_unparsed(scope_model: Au3ScopeModel) -> dict[Au3ErrorNode, Au3UnparsedText]:
"""
The facts of every error node the scope model holds, keyed by the node.
"""
return {node: read_unparsed(node.text) for node in scope_model.error_nodes}
def build_usage_model(
root: Au3Script,
scope_model: Au3ScopeModel,
summaries: dict[str, frozenset[str] | None],
) -> Au3UsageModel:
"""
Collect what *root* does with its names over the scope model's bindings and the propagation
walk's may-write summaries.
"""
unparsed = _collect_unparsed(scope_model)
unparsed_variables = {
name
for facts in unparsed.values()
for name in facts.variables
}
census = _collect_bindings(root, scope_model, unparsed)
channels, reading_channels = _collect_channels(root, scope_model, unparsed)
return Au3UsageModel(
functions=scope_model.functions,
call_counts=_collect_call_counts(root, scope_model, unparsed),
local_read_counts=census.local_read_counts,
global_read_counts=census.global_read_counts,
pure_functions=_build_purity(root, scope_model, summaries, unparsed),
channels=channels,
reading_channels=reading_channels,
spellings=_collect_spellings(root, unparsed),
local_bindings=census.local_bindings,
global_bindings=census.global_bindings,
unparsed=unparsed,
unparsed_variables=frozenset(unparsed_variables),
)
Global variables
var ERROR_MACROS-
The interpreter's error-state macros: entering a user function clears both, so a read of either observes what every call wrote.
Functions
def callee_census(root, functions, unparsed)-
Count what the calls of root name. A member callee is the one exception to every reader: it dispatches through an object and reaches no builtin and no script function. A script function named
Callshadows the builtin, so its first argument is an ordinary argument rather than a name; otherwise aCallnames the calleecall_targetresolves its string literal to, and one whose name is no literal or whose resolution refuses makes any callee possible. Source text the parser could not read is read through the facts unparsed holds: a computed callee it opens, or aCallit spells — whose argument's literalness the text cannot show — makes any callee possible, anExecuteit spells joins the names, and the error-state macros it spells count as reads.Expand source code Browse git
def callee_census( root: Au3Script, functions: Mapping[str, Au3FunctionDeclaration], unparsed: Mapping[Au3ErrorNode, Au3UnparsedText], ) -> Au3CalleeCensus: """ Count what the calls of *root* name. A member callee is the one exception to every reader: it dispatches through an object and reaches no builtin and no script function. A script function named `Call` shadows the builtin, so its first argument is an ordinary argument rather than a name; otherwise a `Call` names the callee `call_target` resolves its string literal to, and one whose name is no literal or whose resolution refuses makes any callee possible. Source text the parser could not read is read through the facts *unparsed* holds: a computed callee it opens, or a `Call` it spells — whose argument's literalness the text cannot show — makes any callee possible, an `Execute` it spells joins the names, and the error-state macros it spells count as reads. """ callees: set[str] = set() computed_callee = False reads_error_macros = False function_names = frozenset(functions) for node in root.walk(): if isinstance(node, Au3Macro): reads_error_macros = reads_error_macros or node.name in ERROR_MACROS elif isinstance(node, Au3Call): callee = ast.direct_callee(node) if callee is not None: callees.add(callee.name) if callee.name == 'call' and callee.name not in functions: literal = ( ast.unwrap_parens(node.arguments[0]) if node.arguments else None ) target = ( call_target(literal.value, node.arguments[1:], function_names) if isinstance(literal, Au3StringLiteral) else None ) if target is None: computed_callee = True else: callees.add(target) elif not isinstance(ast.unwrap_parens(node.callee), Au3MemberAccess): computed_callee = True for facts in unparsed.values(): computed_callee = ( computed_callee or facts.computed_callee or 'call' in facts.identifiers ) reads_error_macros = ( reads_error_macros or not ERROR_MACROS.isdisjoint(facts.macros) ) if 'execute' in facts.identifiers: callees.add('execute') return Au3CalleeCensus(frozenset(callees), computed_callee, reads_error_macros) def build_usage_model(root, scope_model, summaries)-
Collect what root does with its names over the scope model's bindings and the propagation walk's may-write summaries.
Expand source code Browse git
def build_usage_model( root: Au3Script, scope_model: Au3ScopeModel, summaries: dict[str, frozenset[str] | None], ) -> Au3UsageModel: """ Collect what *root* does with its names over the scope model's bindings and the propagation walk's may-write summaries. """ unparsed = _collect_unparsed(scope_model) unparsed_variables = { name for facts in unparsed.values() for name in facts.variables } census = _collect_bindings(root, scope_model, unparsed) channels, reading_channels = _collect_channels(root, scope_model, unparsed) return Au3UsageModel( functions=scope_model.functions, call_counts=_collect_call_counts(root, scope_model, unparsed), local_read_counts=census.local_read_counts, global_read_counts=census.global_read_counts, pure_functions=_build_purity(root, scope_model, summaries, unparsed), channels=channels, reading_channels=reading_channels, spellings=_collect_spellings(root, unparsed), local_bindings=census.local_bindings, global_bindings=census.global_bindings, unparsed=unparsed, unparsed_variables=frozenset(unparsed_variables), )
Classes
class Au3UsageModel (functions, call_counts, local_read_counts, global_read_counts, pure_functions, channels, reading_channels, spellings, local_bindings, global_bindings, unparsed, unparsed_variables)-
What one tree version does with its names, built once per tree version by
build_usage_model(): the script's functions, the direct-identifier call count of each, how often each storage is read — one function's local, or the global per scope, the variables of unread text among its readers — the functions that are pure, the calls and error nodes that may consume a computed name, the calls and error nodes that may read one that no census sees, the lowercased text of every string literal and directive, parsed or unread, the local storage an occurrence in a scope may bind, per scope, and the storage every occurrence provably binds globally, per name, the script level among the readers asNone. The facts of every error node are held keyed by their node, and the variables of every error node answer the transforms that ask which names unread text may reference.Expand source code Browse git
@dataclass(frozen=True) class Au3UsageModel: """ What one tree version does with its names, built once per tree version by `build_usage_model`: the script's functions, the direct-identifier call count of each, how often each storage is read — one function's local, or the global per scope, the variables of unread text among its readers — the functions that are pure, the calls and error nodes that may consume a computed name, the calls and error nodes that may read one that no census sees, the lowercased text of every string literal and directive, parsed or unread, the local storage an occurrence in a scope may bind, per scope, and the storage every occurrence provably binds globally, per name, the script level among the readers as `None`. The facts of every error node are held keyed by their node, and the variables of every error node answer the transforms that ask which names unread text may reference. """ functions: dict[str, Au3FunctionDeclaration] call_counts: dict[str, int] local_read_counts: dict[Au3FunctionDeclaration, dict[str, int]] global_read_counts: dict[Au3FunctionDeclaration | None, dict[str, int]] pure_functions: frozenset[str] channels: frozenset[Au3Call | Au3ErrorNode] reading_channels: frozenset[Au3Call | Au3ErrorNode] spellings: str local_bindings: dict[Au3FunctionDeclaration, frozenset[str]] global_bindings: dict[str, frozenset[Au3FunctionDeclaration | None]] unparsed: Mapping[Au3ErrorNode, Au3UnparsedText] unparsed_variables: frozenset[str] _spelled: dict[str, bool] = field(default_factory=dict, compare=False, repr=False) def spells(self, name: str) -> bool: """ Whether *name* — a function name, or a variable name without its `$` — is a substring of any string literal or directive the tree holds, parsed or unread: a computed name has to be built from string data, so a name some string spells is one no removal may take away. The answer is memoized per name — the scan is linear in the corpus — and the model is rebuilt per tree version, so a memo never outlives the facts it answered from. """ answer = self._spelled.get(name) if answer is None: answer = name.lstrip('$') in self.spellings self._spelled[name] = answer return answer def holds_block_keyword_text(self, root: Node) -> bool: """ Whether *root*'s subtree holds error text the reader marks with an AutoIt block keyword: deleting it could move where the real parser's block boundaries fall, and with them which code is live at all. """ return any( self.unparsed[node].block_keyword for node in root.walk() if isinstance(node, Au3ErrorNode) ) def holds_unread_include(self) -> bool: """ Whether unread source text holds an include directive: the file it names is outside every fact the tree holds, and may call anything a removal would take away. """ return any( INCLUDE_SPELLING.match(directive) for facts in self.unparsed.values() for directive in facts.directives ) def effect_free(self, expression: Expression | None) -> bool: """ Whether evaluating *expression* writes nothing observable and cannot terminate the script. """ return _effect_free(expression, self.functions, self.pure_functions)Instance variables
var functions-
The type of the None singleton.
var call_counts-
The type of the None singleton.
var local_read_counts-
The type of the None singleton.
var global_read_counts-
The type of the None singleton.
var pure_functions-
The type of the None singleton.
var channels-
The type of the None singleton.
var reading_channels-
The type of the None singleton.
var spellings-
The type of the None singleton.
var local_bindings-
The type of the None singleton.
var global_bindings-
The type of the None singleton.
var unparsed-
The type of the None singleton.
var unparsed_variables-
The type of the None singleton.
Methods
def spells(self, name)-
Whether name — a function name, or a variable name without its
$— is a substring of any string literal or directive the tree holds, parsed or unread: a computed name has to be built from string data, so a name some string spells is one no removal may take away. The answer is memoized per name — the scan is linear in the corpus — and the model is rebuilt per tree version, so a memo never outlives the facts it answered from.Expand source code Browse git
def spells(self, name: str) -> bool: """ Whether *name* — a function name, or a variable name without its `$` — is a substring of any string literal or directive the tree holds, parsed or unread: a computed name has to be built from string data, so a name some string spells is one no removal may take away. The answer is memoized per name — the scan is linear in the corpus — and the model is rebuilt per tree version, so a memo never outlives the facts it answered from. """ answer = self._spelled.get(name) if answer is None: answer = name.lstrip('$') in self.spellings self._spelled[name] = answer return answer def holds_block_keyword_text(self, root)-
Whether root's subtree holds error text the reader marks with an AutoIt block keyword: deleting it could move where the real parser's block boundaries fall, and with them which code is live at all.
Expand source code Browse git
def holds_block_keyword_text(self, root: Node) -> bool: """ Whether *root*'s subtree holds error text the reader marks with an AutoIt block keyword: deleting it could move where the real parser's block boundaries fall, and with them which code is live at all. """ return any( self.unparsed[node].block_keyword for node in root.walk() if isinstance(node, Au3ErrorNode) ) def holds_unread_include(self)-
Whether unread source text holds an include directive: the file it names is outside every fact the tree holds, and may call anything a removal would take away.
Expand source code Browse git
def holds_unread_include(self) -> bool: """ Whether unread source text holds an include directive: the file it names is outside every fact the tree holds, and may call anything a removal would take away. """ return any( INCLUDE_SPELLING.match(directive) for facts in self.unparsed.values() for directive in facts.directives ) def effect_free(self, expression)-
Whether evaluating expression writes nothing observable and cannot terminate the script.
Expand source code Browse git
def effect_free(self, expression: Expression | None) -> bool: """ Whether evaluating *expression* writes nothing observable and cannot terminate the script. """ return _effect_free(expression, self.functions, self.pure_functions)
class Au3CalleeCensus (callees, computed_callee, reads_error_macros)-
What the calls of one tree name, gathered once for every question that asks it: the names a direct identifier or a
Callinvocation over a string literal spells, whether any callee is computed at run time — a value the tree does not show, which could name anything — and whether the tree reads the interpreter's@erroror@extended.Expand source code Browse git
@dataclass(frozen=True) class Au3CalleeCensus: """ What the calls of one tree name, gathered once for every question that asks it: the names a direct identifier or a `Call` invocation over a string literal spells, whether any callee is computed at run time — a value the tree does not show, which could name anything — and whether the tree reads the interpreter's `@error` or `@extended`. """ callees: frozenset[str] computed_callee: bool reads_error_macros: bool def observes_error_state(self) -> bool: """ Whether the script may read the interpreter's `@error` and `@extended` at run time: a read of either macro anywhere, or a call to `Execute` by any spelling the census sees — whose payload text is a reader no walk reads. """ return ( self.reads_error_macros or self.computed_callee or 'execute' in self.callees )Instance variables
var callees-
The type of the None singleton.
var computed_callee-
The type of the None singleton.
var reads_error_macros-
The type of the None singleton.
Methods
def observes_error_state(self)-
Whether the script may read the interpreter's
@errorand@extendedat run time: a read of either macro anywhere, or a call toExecuteby any spelling the census sees — whose payload text is a reader no walk reads.Expand source code Browse git
def observes_error_state(self) -> bool: """ Whether the script may read the interpreter's `@error` and `@extended` at run time: a read of either macro anywhere, or a call to `Execute` by any spelling the census sees — whose payload text is a reader no walk reads. """ return ( self.reads_error_macros or self.computed_callee or 'execute' in self.callees )
class Au3StorageCensus (local_bindings, global_bindings, local_read_counts, global_read_counts)-
The storage every variable occurrence binds or may bind, and how often each storage is read: the names a function's occurrences may bind locally, the scopes every name provably binds globally in, the reads that may reach one function's local, and the reads standing in each scope that may reach the global — script level keyed under
None. Both read counts are may-answers: a read that may reach either storage counts into both.Expand source code Browse git
@dataclass(frozen=True) class Au3StorageCensus: """ The storage every variable occurrence binds or may bind, and how often each storage is read: the names a function's occurrences may bind locally, the scopes every name provably binds globally in, the reads that may reach one function's local, and the reads standing in each scope that may reach the global — script level keyed under `None`. Both read counts are may-answers: a read that may reach either storage counts into both. """ local_bindings: dict[Au3FunctionDeclaration, frozenset[str]] global_bindings: dict[str, frozenset[Au3FunctionDeclaration | None]] local_read_counts: dict[Au3FunctionDeclaration, dict[str, int]] global_read_counts: dict[Au3FunctionDeclaration | None, dict[str, int]]Instance variables
var local_bindings-
The type of the None singleton.
var global_bindings-
The type of the None singleton.
var local_read_counts-
The type of the None singleton.
var global_read_counts-
The type of the None singleton.