aiken/fuzz/scenario
A testing framework for running valid and invalid scenarii on a validator. The framework instruments the generation and execution of arbitrarily generated test scenarii working at the transaction-level for maximum reproducibility.
A full tutorial is coming soon. For now, see examples:
Types
Alias
Label = String
Alias
MintHandler = fn(Data, PolicyId, Transaction) -> Bool
Constructors
-
Ok
-
Ko
Alias
PublishHandler = fn(Data, Certificate, Transaction) -> Bool
Constructors
-
Done
-
Step(List<Label>, st, Transaction)
Alias
SpendHandler = fn(Option<Data>, Data, OutputReference, Transaction) -> Bool
Alias
VoteHandler = fn(Data, Voter, Transaction) -> Bool
Alias
WithdrawHandler = fn(Data, Credential, Transaction) -> Bool
Functions
Running
run(
scenario: List<Transaction>,
script: ScriptHash,
mint: MintHandler,
spend: SpendHandler,
withdraw: WithdrawHandler,
publish: PublishHandler,
vote: VoteHandler,
) -> Void
ko(
initial_state: st,
step: fn(st) -> Fuzzer<Scenario<st>>,
) -> Fuzzer<(List<Label>, List<Transaction>)>
Generate only K.O. scenarii from an initial state and a stepping function.
test prop_end_to_end_ko((labels, scenario) via scenario.ko(initial_state, step)) {
list.for_each(labels, fuzz.label)
scenario.run(
scenario,
my_script,
handlers.mint,
handlers.spend,
scenario.into_withdraw_handler(handlers.else),
scenario.into_publish_handler(handlers.else),
scenario.into_vote_handler(handlers.else),
)
}
Generate only O.K. scenarii from an initial state and a stepping function.
test prop_end_to_end_ok(scenario via scenario.ok(initial_state, step)) {
scenario.run(
scenario,
my_script,
handlers.mint,
handlers.spend,
scenario.into_withdraw_handler(handlers.else),
scenario.into_publish_handler(handlers.else),
scenario.into_vote_handler(handlers.else),
)
}
Inspecting
A generator meant to analyze the scenario generator, to control that it is
relatively well-balanced between O.K. and K.O. scenarios.
test prop_scenario_coverage(
outcome via scenario.report_coverage(initial_state, step),
) {
fuzz.label_when(outcome == scenario.Ok, @"O.K.", @"K.O.")
}
Repeatedly run a K.O. scenario fuzzer to ensure that all labels eventually
appear in scenarii. Fails after a while if it takes too long.
const all_failures: List<String> = [ @"...", @"..." ]
test ko_coverage() {
scenario.check_coverage(
all_failures,
scenario.ko(default_state, step)
)
}
A helper for classifying scenarios based on some given predicates.
[]
|> classify(
@"✓ more authorizations than necessary",
spent_badges_count + ref_badges_count > list.length(policies),
)
|> classify(
@"✓ at least one spent authorization",
spent_badges_count > 0,
)
|> classify(
@"✓ at least one reference authorization",
ref_badges_count > 0,
)
|> classify(
@"✓ more than one badge checked",
list.length(policies) > 1,
)
|> for_each(label)
Plumbing
Turn a else
handler into a MintHandler
.
Turn a else
handler into a SpendHandler
.
Turn a else
handler into a WithdrawHandler
.
Turn a else
handler into a PublishHandler
.
Turn a else
handler into a VoteHandler
.
Fuzzing
Fork between two scenarios using with a frequency proportional to the given
pivot.
The pivot
must be value between and which represents the
probability of NOT forking the baseline scenario into a branch.
It needs to be high enough to have a high probability chance of
generating valid scenarios despite many forks, but small enough that forks
still occur. Use report_coverage
to analyze your scenarios
and tweak the weights accordingly.
Ideally, if is the total number of forks one wants:
fork_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork
, but take a continuation immediately. This is mostly a
convenient way to avoid wrapping all calls to .fork
inside and_then(...)
.
let (st, inputs) <- fork_and_then(
weights.scenario_input,
scenario_baseline(st),
scenario_inject_fault(st),
)
// equivalent to
let (st, inputs) <- fuzz.and_then(fork(
weights.scenario_input,
scenario_baseline(st),
scenario_inject_fault(st),
))
fork_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but only forks if the predicate
is True
. Runs the
baseline
scenario otherwise.
Useful to selectively fork into a scenario only when specific conditions are
verified.
fork_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if
but takes a continuation.
See also fork_and_then
.
fork2(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but allows multiple alternative scenarios with an equal
probability.
fork2_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_and_then
, but allows multiple alternative
scenarios with an equal probability.
fork2_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork_if
, but allows multiple alternative scenarios with an equal
probability.
fork2_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if_and_then
, but allows multiple
alternative scenarios with an equal probability.
fork3(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but allows multiple alternative scenarios with an equal
probability.
fork3_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_and_then
, but allows multiple alternative
scenarios with an equal probability.
fork3_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork_if
, but allows multiple alternative scenarios with an equal
probability.
fork3_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if_and_then
, but allows multiple
alternative scenarios with an equal probability.
fork4(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but allows multiple alternative scenarios with an equal
probability.
fork4_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_and_then
, but allows multiple alternative
scenarios with an equal probability.
fork4_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork_if
, but allows multiple alternative scenarios with an equal
probability.
fork4_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if_and_then
, but allows multiple
alternative scenarios with an equal probability.
run(
scenario: List<Transaction>,
script: ScriptHash,
mint: MintHandler,
spend: SpendHandler,
withdraw: WithdrawHandler,
publish: PublishHandler,
vote: VoteHandler,
) -> Void
ko(
initial_state: st,
step: fn(st) -> Fuzzer<Scenario<st>>,
) -> Fuzzer<(List<Label>, List<Transaction>)>
Generate only K.O. scenarii from an initial state and a stepping function.
test prop_end_to_end_ko((labels, scenario) via scenario.ko(initial_state, step)) {
list.for_each(labels, fuzz.label)
scenario.run(
scenario,
my_script,
handlers.mint,
handlers.spend,
scenario.into_withdraw_handler(handlers.else),
scenario.into_publish_handler(handlers.else),
scenario.into_vote_handler(handlers.else),
)
}
Generate only O.K. scenarii from an initial state and a stepping function.
test prop_end_to_end_ok(scenario via scenario.ok(initial_state, step)) {
scenario.run(
scenario,
my_script,
handlers.mint,
handlers.spend,
scenario.into_withdraw_handler(handlers.else),
scenario.into_publish_handler(handlers.else),
scenario.into_vote_handler(handlers.else),
)
}
A generator meant to analyze the scenario generator, to control that it is relatively well-balanced between O.K. and K.O. scenarios.
test prop_scenario_coverage(
outcome via scenario.report_coverage(initial_state, step),
) {
fuzz.label_when(outcome == scenario.Ok, @"O.K.", @"K.O.")
}
Repeatedly run a K.O. scenario fuzzer to ensure that all labels eventually appear in scenarii. Fails after a while if it takes too long.
const all_failures: List<String> = [ @"...", @"..." ]
test ko_coverage() {
scenario.check_coverage(
all_failures,
scenario.ko(default_state, step)
)
}
A helper for classifying scenarios based on some given predicates.
[]
|> classify(
@"✓ more authorizations than necessary",
spent_badges_count + ref_badges_count > list.length(policies),
)
|> classify(
@"✓ at least one spent authorization",
spent_badges_count > 0,
)
|> classify(
@"✓ at least one reference authorization",
ref_badges_count > 0,
)
|> classify(
@"✓ more than one badge checked",
list.length(policies) > 1,
)
|> for_each(label)
Plumbing
Turn a else
handler into a MintHandler
.
Turn a else
handler into a SpendHandler
.
Turn a else
handler into a WithdrawHandler
.
Turn a else
handler into a PublishHandler
.
Turn a else
handler into a VoteHandler
.
Fuzzing
Turn a else
handler into a MintHandler
.
Turn a else
handler into a SpendHandler
.
Turn a else
handler into a WithdrawHandler
.
Turn a else
handler into a PublishHandler
.
Turn a else
handler into a VoteHandler
.
Fork between two scenarios using with a frequency proportional to the given pivot.
The pivot
must be value between and which represents the
probability of NOT forking the baseline scenario into a branch.
It needs to be high enough to have a high probability chance of
generating valid scenarios despite many forks, but small enough that forks
still occur. Use report_coverage
to analyze your scenarios
and tweak the weights accordingly.
Ideally, if is the total number of forks one wants:
fork_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork
, but take a continuation immediately. This is mostly a
convenient way to avoid wrapping all calls to .fork
inside and_then(...)
.
let (st, inputs) <- fork_and_then(
weights.scenario_input,
scenario_baseline(st),
scenario_inject_fault(st),
)
// equivalent to
let (st, inputs) <- fuzz.and_then(fork(
weights.scenario_input,
scenario_baseline(st),
scenario_inject_fault(st),
))
fork_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but only forks if the predicate
is True
. Runs the
baseline
scenario otherwise.
Useful to selectively fork into a scenario only when specific conditions are verified.
fork_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if
but takes a continuation.
See also fork_and_then
.
fork2(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but allows multiple alternative scenarios with an equal
probability.
fork2_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_and_then
, but allows multiple alternative
scenarios with an equal probability.
fork2_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork_if
, but allows multiple alternative scenarios with an equal
probability.
fork2_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if_and_then
, but allows multiple
alternative scenarios with an equal probability.
fork3(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but allows multiple alternative scenarios with an equal
probability.
fork3_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_and_then
, but allows multiple alternative
scenarios with an equal probability.
fork3_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork_if
, but allows multiple alternative scenarios with an equal
probability.
fork3_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if_and_then
, but allows multiple
alternative scenarios with an equal probability.
fork4(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork
, but allows multiple alternative scenarios with an equal
probability.
fork4_and_then(
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_and_then
, but allows multiple alternative
scenarios with an equal probability.
fork4_if(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
) -> Fuzzer<a>
Like fork_if
, but allows multiple alternative scenarios with an equal
probability.
fork4_if_and_then(
predicate: Bool,
pivot: Int,
baseline: fn() -> Fuzzer<a>,
branch1: fn() -> Fuzzer<a>,
branch2: fn() -> Fuzzer<a>,
branch3: fn() -> Fuzzer<a>,
branch4: fn() -> Fuzzer<a>,
continue: fn(a) -> Fuzzer<b>,
) -> Fuzzer<b>
Like fork_if_and_then
, but allows multiple
alternative scenarios with an equal probability.