Usable research
Include every assigned ETF–week in the denominator. Judge missing material facts, unsupported claims and incorrect product interpretations.
/
ASCEND AIOS · WORKFLOW EVALUATION AND PUBLICATION PROTOCOL
How to evaluatefinancial-research AI.
Evaluate accuracy, evidence, client relevance and review effort through an actual ETF deliverable workflow. We separate the artifact audit, the earlier question-answering pilot and the next validation study.
WORKFLOW CASE / ETF RESEARCH DELIVERABLES
Evaluate the work,
not just the answer.
We reviewed the NH ETF deliverable workflow: turning product structure and market comparisons into client-facing cards, beyond a price summary. The audit links code to archived outputs from a batch dated 28 August 2026 and recorded as executed on 3 September.
Numbers and explanations
Check product, currency, period, price-versus-total-return basis and evidence timing across tiles, prose and charts.
Relevance to the client
Independent practitioners assess whether the selected insight improves product understanding. Fluent prose alone is not success.
Review effort and updates
Measure human checking and correction time alongside retained quality, and test whether new inputs update the right explanations.
| Audit target | Observed result | Interpretation |
|---|---|---|
| Deliverables | 100 HTML · 100 PNG | Files found; not certification of delivery or all rendered layouts |
| Headline numeric tiles | 265 / 300 | All 265 recalculated tiles match; 35 were outside recalculation coverage |
| Sentence rules | 1,806 PASS | Rule non-detection reproduced; not factual or regulatory certification |
IMPLEMENTATION AND EVALUATION BOUNDARYUse this batch as a development case. Compare a same-input template baseline, a single research stage and the full workflow on fresh weeks and product groups. Shared index or event exposure creates dependence; re-rendering time is not AI research-generation time.
Market cutoffs and metadata retrieval dates must remain distinct. Event evidence needs a binding to actual claims, and numerical checks must cover prose and charts. Replaying a calculation is different from independently validating the research.
See the deliverable workflowFive financial-research strata
Three no-search arms, search, protocol
Equal-weight arithmetic mean
One run per condition
Of 250 stored responses, 245 remain eligible after one item is retired. Original scores are preserved; metric definitions and populations are corrected.
Until these steps are completed, the result must not be described as journal-validated or externally certified.
01 / RESEARCH QUESTIONS
What the evaluation is meant to test
The target is not a generic model ranking. It is the contribution of data, evidence, calculation, and safe failure to current financial research.
Current facts and temporal validity
Does full AIOS deliver more complete, factually correct answers—with no unverified material additions—than the same base model with web search? Temporal validity is assessed separately.
Evidence quality
With model, retrieved evidence, and budget fixed, does verification ground more required facts without an unacceptable loss in fully supported responses?
Numerical reasoning
Can reported and derived values be reproduced with the correct unit, currency, period, and formula?
Safe failure
When a premise is false or evidence is insufficient, does the system abstain or qualify instead of inventing an answer?
02 / PILOT DESIGN
Same questions.
Same rules.
The original 50-item set was reduced to 49 scored items after responses were collected: nine company-results, 15 ETF/product, ten market-event, ten regulation/index, and five trap questions. Only 44 contribute to the numeric/abstention score; five nonnumeric keys are excluded.
The package records five subagent-context conditions: three no-search model arms, Opus with search, and Opus with search plus the ASCEND protocol. The last is not production AIOS. Questions and scoring rules are shared, but tool permissions, instructions, and procedures differ. Model names are record labels, not independently authenticated provider executions.
UNIT OF ANALYSISThe current report presents condition-level averages. The publication protocol averages paired question effects while preserving document/event dependence; repeated runs remain nested within questions.
03 / OPERATIONAL DEFINITIONS
Define the metric before reading the score.
Reported pilot metrics are separated from measures proposed for publication. Proposed measures are not presented as completed results.
Numeric / abstention score
REPRODUCEDThe historical ‘factual accuracy’ metric averages numeric-token recall for 40 questions and an abstention-pattern score for four. Matching ignores sign and uses a 2% relative tolerance; it does not verify the entity, period, direction, or additional assertions. Five nonnumeric keys are excluded.
Abstention-pattern match
REPRODUCEDA refusal-expression regex matches in all four designated questions. It does not establish that the explanation is correct or free of invented detail. The five-question trap category is a different population.
Source-list presence
REPRODUCEDA nonempty sources field earns credit, regardless of whether entries are URLs, primary sources, or support the answer. The protocol arm scores 49/49. Calling this ‘primary-source attachment’ overstates what the code checks.
As-of-marker presence
REPRODUCEDA populated asof_claimed field or a date-like substring earns credit. No validation establishes that the date is valid or governs the relevant claim. The protocol arm scores 49/49.
Banned-pattern non-detection
REPRODUCEDThe scorer detects a fixed set of prohibited Korean expressions with contextual exemptions. Two meta-questions are excluded; the protocol arm scores 47/47. This is neither a legal-compliance assessment nor proof of safe financial advice.
Composite
REPRODUCEDS is the mean of the five rounded axis scores, each weighted 0.2 despite different populations. The same four abstention questions enter two axes. A score of 97.3 is not a 97.3% rate of correct or safe answers.
RQ1 · Complete factual answer
PROPOSEDAll required facts must be correct and every material assertion—including additions—must be verified correct. Report factual precision and gold-fact recall separately; repeating a fact adds no credit.
RQ2 · Grounded required facts
PROPOSEDCount unique required facts that are correct, fully supported by cited evidence, and time-eligible. Require noninferiority of the fully supported-response rate; citations alone earn no credit.
RQ3 · Required numerical facts
PROPOSEDReproduce the numbers required by the frozen answer key, including unit, currency, period, formula, and rounding. Omitting a difficult calculation does not remove it from the denominator.
RQ4 · Explained safe failure
PROPOSEDA nonempty response must correctly explain an invalid premise or insufficient evidence without unverified material assertions. A crash, timeout, or empty refusal does not count as safe abstention.
Zero denominators and partial credit
PROPOSEDNo claims or citations means undefined conditional precision, not 100%. Required-fact omissions score zero. Primary scoring awards no partial credit; all raw counts remain available.
04 / PILOT RESULTS
Descriptive results—not inferential claims
Scores use a 0–100 scale; the composite equally weights five rounded axes. Scoring has been reproduced from stored responses. Confirmatory intervals and p-values are withheld because sampling, dependence, and measurement validity have not been established.
| Scoring axis | Haiku 4.5 No search | Sonnet 5 No search | Opus 5 No search | Opus 5 + Search | ASCEND Protocol |
|---|---|---|---|---|---|
| Numeric / abstention score · n = 44 | 23.0 | 39.1 | 51.0 | 88.6 | 86.6 |
| Abstention-pattern match · n = 4 | 100 | 100 | 100 | 100 | 100 |
| Source-list presence · n = 49 | 0 | 0 | 0 | 100 | 100 |
| As-of-marker presence · n = 49 | 49.0 | 49.0 | 57.1 | 95.9 | 100.0 |
| Banned-pattern non-detection · n = 47 | 100 | 100 | 100 | 97.9 | 100 |
| Composite score | 54.4 | 57.6 | 61.6 | 96.5 | 97.3 |
| Task category | Haiku 4.5 No search | Sonnet 5 No search | Opus 5 No search | Opus 5 + Search | ASCEND Protocol |
|---|---|---|---|---|---|
| Latest company results · n = 9 | 5.2 | 5.9 | 10.4 | 100.0 | 97.2 |
| Latest market events · n = 9 | 10.9 | 10.5 | 21.2 | 70.0 | 72.2 |
| ETF and product knowledge · n = 12 | 15.1 | 58.8 | 70.4 | 85.0 | 85.0 |
| Regulation and index methodology · n = 9 | 31.5 | 49.0 | 76.5 | 94.3 | 85.3 |
| Trap questions · n = 5 | 80 | 85 | 85 | 100 | 100 |
| Reported measure | Web search | + Protocol | Observed change |
|---|---|---|---|
| Composite score | 96.5 | 97.3 | +0.8 |
| Numeric / abstention score | 88.6 | 86.6 | −2.0 |
| Keyword Tier 1 · original n = 50 | 22/50 · 44% | 35/50 · 70% | +26 pp |
| Keyword Tier 1 · eligible n = 49 | 21/49 · 42.9% | 34/49 · 69.4% | +26.5 pp |
| Keyword Tier 3 · original n = 50 | 18/50 · 36% | 12/50 · 24% | −12 pp |
REPRODUCTION AND POPULATION CORRECTIONThe 3 September 2026 offline audit reproduces all 245 score records and stored summaries. Scored category sizes are 9 company-results, 12 ETF, 9 event, 9 regulation, and 5 trap items: 44 total. Tiers classify each response by its highest source-string keyword match. The original 70% is 35/50, including the retired item; eligibility-aligned Tier 1 is 34/49 = 69.4%. The summary's unscored count is also corrected from one to five.
05 / DESCRIPTIVE ABLATION
Read the observed gap
by system component.
The arms labeled Opus 5 scored 61.6 without search, 96.5 with search, and 97.3 with the ASCEND protocol. The +34.9 and +0.8 contrasts do not measure production AIOS's data network, causal graph, permission separation, or deterministic gates.
These are not causal effects: there is one run per arm without matched retrieval, response length, or batch context. Source-keyword tiers shifted upward, while the numeric/abstention score fell from 88.6 to 86.6. Independent claim adjudication must distinguish safer abstention, omitted correct facts, and scoring artifacts before explaining that decline.
Δ +34.9
Δ +0.8
CASE / ORIGINAL-RESPONSE AUDIT
Inspect the answer
behind the score change.
Across 49 paired items, numeric scores are higher on 3, equal on 38, lower on 3, and unscored on 5. All six changed pairs were examined. These are score directions, not adjudicated wins or overall error rates.
Formatting changes the score
In T3-09, replacing only the section label ‘2)’ with a Roman numeral changes 0.4 to 0.2. Financial content is unchanged. This is a scorer-sensitivity probe, not a new model response.
The question was incomplete
T4-02 asks about ‘that BIS rule’ without a rule identifier or antecedent in the supplied batch. Answer errors and question defects require separate adjudication.
Absence asserted, not abstention
The protocol uses zero search hits to assert that no BIS rule names an AMD product. The official rule's preamble supplies a counterexample. Evaluation must check both source location and claim scope.
VERIFIED PRIMARY SOURCEBIS final rule 2026-00789, published 15 January 2026, names AMD MI325X in its preamble: printed page 1685, Sections I and II. This does not certify a product-name mention in the operative CFR amendment or the entire answer. Official GovInfo source
Historical scores and responses are unchanged. This disclosed development case is not held-out evidence of generalization in a future study.
See the architecture requirementsMETHOD VALIDATION · SYNTHETIC DATA
Statistical inference
under stress
Correct arithmetic does not establish a valid test for a particular sample. We cross-checked candidate CR1-t inference against a statistical package and ran 10,000 simulations in each of 11 settings. These are not AIOS performance trials.
Across 30 synthetic contrasts, calculations matched statsmodels 0.14.6 within an absolute tolerance of 10⁻¹¹. Yet false rejections rose from 4.80% under a balanced null to 21.90% when one cluster dominated. More questions from the same document or event do not supply the same amount of independent evidence.
The candidate is not approved as a universal primary test. Calibration must reflect the actual study’s clusters, task mix, and rare errors. If inadequate, revise the design or method before viewing confirmatory outcomes. No confirmatory p-value or confidence interval has been added to the legacy 97.3 score.
All simulation results and failure regimes
A cluster groups questions that may depend on the same document or event. CR1-t is a candidate cluster-aware mean-difference test. The 95% MC intervals quantify simulation-rate uncertainty, not AIOS effects. They are per-setting Wilson intervals, not simultaneous bounds.
| Synthetic setting | False rejection / power | 95% MC interval | Interval coverage¹ | Inconclusive |
|---|---|---|---|---|
| Balanced null | 4.80% | 4.40%–5.24% | 94.85% | 0.00% |
| Only six independent clusters | 5.26% | 4.84%–5.71% | 93.71% | 0.00% |
| 100 of 195 questions in one cluster | 21.90% | 21.10%–22.72% | 71.43% | 0.00% |
| Unmodeled common shock | 42.20% | 41.24%–43.17% | 18.15% | 0.00% |
| Rare cluster discordance | 0.08% | 0.04%–0.16% | 100.00% | 66.92% |
| Noninferiority boundary −0.03 | 4.94% | 4.53%–5.38% | 94.85% | 0.00% |
| Synthetic effect +0.12 · power | 99.22% | 99.03%–99.37% | 94.81% | 0.00% |
¹ Coverage uses only replications with a computable interval. The rare-discordance 100% covers 3,308 replications; the other 6,692 are inconclusive. All replications stay in the rejection denominator. Only the final row is power at a hypothesized positive effect; other rows measure rejection at a null boundary. The common-shock setting intentionally violates cluster independence.
| True nulls | Any false RQ rejection / replications | Familywise error | 95% MC interval |
|---|---|---|---|
| All four RQs null | 336 / 10,000 | 3.36% | 3.02%–3.73% |
| Only RQ2 support null | 439 / 10,000 | 4.39% | 4.01%–4.81% |
| Only RQ2 recall null | 479 / 10,000 | 4.79% | 4.39%–5.23% |
| Only RQ3 null | 482 / 10,000 | 4.82% | 4.42%–5.26% |
Development screens required a null-rejection MC upper bound ≤6%, a conditional-coverage MC lower bound ≥94%, and ≤1% inconclusive results. Six-cluster, dominant-cluster, common-shock, and rare-discordance scalar settings failed. All four family screens passed only in the specified generator. Multiplicity adjustment does not repair invalid component tests.
This is a disclosed development experiment using seed 2026090301 and NumPy PCG64. The local reproducibility package retains the plan, code, environment, counts, denominators, and failures. Covariance implementations differ, but both use SciPy’s t distribution; this is not independent validation of every numerical primitive. Preregistration and independent review of the actual study remain outstanding.
06 / PUBLICATION PROTOCOL
What a publication-grade study must add
These are prospective requirements, not completed claims. They define the minimum evidence package before journal submission.
Primary endpoints and contrasts
Freeze each endpoint, analysis population, comparison, and precision target on a 0–1 scale. RQ2 jointly requires grounded-fact coverage superiority and supported-response noninferiority. The pilot composite remains descriptive.
Sampling
Freeze the sampling frame, task mix, precision target, and dependent document/event families before splitting data. Report questions and independent clusters separately. The 49-item set remains a pilot, not a population-representative sample.
Repeated runs
This protocol proposes at least five runs per stochastic configuration, with the final count justified by pilot variance and precision—not a universal sufficiency rule. Freeze model, prompt, tool, retrieval, and date versions.
Blind adjudication
Use two independent finance-domain raters blinded to system identity, followed by a third adjudicator. Preserve raw counts, complete a HEDS disclosure, and report agreement with Krippendorff's α or an appropriate alternative.
Dependence-aware inference
Average repeats within questions, then preserve dependent document/event clusters. Validate the mean-difference test before use. RQ2 combines superiority and noninferiority with the larger component p-value, followed by Holm across four questions. Nominal intervals are not simultaneous confidence bounds.
Leakage control
Audit exact source vintages, not just publication-date labels. Keep dependent question families out of training and test overlap. Historical evidence replay does not remove later knowledge learned by the model; prospective validation is a separate design.
Efficiency
Report end-to-end latency, cost, token use, retrieval volume, tool failures, and abstention rate beside quality metrics.
Error taxonomy
Publish errors by unsupported claim, wrong period, unit or currency error, stale evidence, bad premise acceptance, retrieval miss, and policy false positive or false negative.
Open artifacts
Release item IDs or a rights-safe surrogate set, prompts, model and tool versions, raw completions, retrieved evidence, scoring code, rater guide, and an executable environment manifest.
RQ1 · RQ4 CONTRASTFull AIOS vs same base model + search
RQ2 · RQ3 CONTRASTVerification on vs off with frozen retrieval
ESTIMATION · TESTINGCluster-aware effects · joint testing · Holm
ROBUSTNESSRepeated runs · strata · length control · error taxonomy
07 / LIMITATIONS AND VALIDITY THREATS
What the current result does not establish
We disclose threats to internal, external, construct, and statistical-conclusion validity instead of treating the pilot as certification.
Pilot sample
The 49 questions provide pilot observations. Adequacy for subgroup or population claims depends on task selection, independent clusters, variability, and required precision—not question count alone.
Internal ownership
ASCEND designed the questions, answer key, scoring code, and protocol. The study has not undergone independent replication or external adjudication.
One run per condition
The reported study contains one run per condition. It does not estimate run-to-run variance or rank stability.
Scoring reproduction, not runtime authentication
A local audit reproduced all 245 eligible score records and stored summaries from the original package. Self-described model labels and response files do not independently authenticate provider execution, gold-access isolation, or complete tool traces.
Length confounding
Average outputs ranged from 130 to 1,205 Korean characters. Longer answers may expose more candidate values to rule-based matching.
Limited task boundary
The benchmark tests research retrieval and verification. It does not evaluate portfolio construction, risk, suitability, trade execution, or realized returns.
Dependent questions
Several questions may share a source, event, or template. Repeated calls and correlated questions do not increase independent evidence one-for-one. Generic item-level significance tests are not confirmatory without a justified sampling design.
Historical hindsight
A present-day model may know later outcomes even when every citation predates the cutoff. Date-filtered retrieval is not certified unlearning, and clean citations do not establish historically attainable predictions.
Protocol arm, not production AIOS
The original package explicitly describes d_aios as a protocol implementation in a subagent context. Its score does not evaluate the deployed data network, causal graph, multi-agent permission boundaries, or deterministic controls.
Retrospective corrections
One question was retired after inspecting responses, and the package records five scorer corrections. Symmetric repairs can be reasonable but do not turn a revised exploratory pilot into a preregistered confirmatory test.
Metric validity
Constructed scorer probes award full numeric credit for opposite signs and wrong entities. These demonstrate blind spots in the rule, not measured error rates of the models. Independent semantic adjudication remains necessary.
08 / RELATED WORK AND STANDARDS
Research foundations
The revised protocol is grounded in primary work on financial QA, citation evaluation, multi-metric benchmarking, statistical inference, and generative-AI risk management.
- 01Can Prompts Rewind Time for LLMs? Evaluating the Effectiveness of Prompted Knowledge Cutoffs
Gao et al. · EMNLP 2025
Motivates separate tests for indirect use of learned future knowledge; an earlier-date prompt is not reliable unlearning. - 02Temporal Leakage in Search-Engine Date-Filtered Web Retrieval: A Retrospective Forecasting Case Study
El Lahib et al. · ACL 2026 · Short Papers
Shows why publication-date filtering is not payload-vintage verification; motivates frozen source versions. - 03ExAnte: A Benchmark for Ex-Ante Inference in Large Language Models
Liu et al. · EACL 2026
Separately evaluates temporal leakage and task quality, including finance-related prediction tasks. - 04A Practitioner's Guide to Cluster-Robust Inference
Cameron and Miller · Journal of Human Resources, 2015
Motivates preserving document/event dependence rather than treating every question or repeated call as independent. - 05FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Min et al. · EMNLP 2023
Motivates judging every atomic assertion rather than only matching expected facts. Our separate frozen-gold coverage and strict-response endpoints are protocol choices, not FActScore results. - 06Enabling Large Language Models to Generate Text with Citations
Gao, Yen, Yu, and Chen · EMNLP 2023
Separates citation correctness and completeness from answer quality; informs claim-level evidence precision and recall. - 07FinanceBench: A New Benchmark for Financial Question Answering
Islam et al. · arXiv 2023
Provides evidence-linked, ecologically valid financial questions and manual review of system configurations. - 08Verifiable Financial Enterprise Question Answering via Inference-Time Grounding and Traceability
Kabra et al. · Financial Narrative Processing Workshop, 2026
Provides a directly comparable deployment-aware evaluation of inference-time grounding, hallucination control, and document-span traceability in enterprise finance. - 09FinanceReasoning: Benchmarking Financial Numerical Reasoning More Credible, Comprehensive and Challenging
Tang et al. · ACL 2025
Supports executable financial calculations, explicit solution programs, corrected questions, and more demanding coverage of financial concepts. - 10FinQA: A Dataset of Numerical Reasoning over Financial Data
Chen et al. · EMNLP 2021
Treats numerical reasoning programs and financial evidence as first-class evaluation objects. - 11TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance
Zhu et al. · ACL-IJCNLP 2021
Motivates explicit evaluation of reasoning across tables, prose, and symbolic calculations. - 12Holistic Evaluation of Language Models
Liang et al. · TMLR 2023
Supports standardized conditions, multi-metric reporting, and release of prompts and completions. - 13Statistical Significance Tests for Machine Translation Evaluation
Koehn · EMNLP 2004
Provides paired-resampling foundations; the sampling unit and dependence assumptions must be adapted to this study. - 14Better than Average: Paired Evaluation of NLP Systems
Peyrard, Zhao, Eger, and West · ACL-IJCNLP 2021
Shows why comparisons on shared test instances must preserve pairing instead of ranking systems only by independent averages. - 15HEDS 3.0: The Human Evaluation Data Sheet Version 3.0
Belz and Thomson · GEM² 2025
Supplies a standardized disclosure template for human-evaluation design, raters, instruments, quality control, and reproducibility. - 16Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics
Godbole and Jia · Findings of ACL 2025
Shows that automated factuality metrics can disagree and misestimate quality, supporting domain validation and blinded human adjudication rather than scorer-only claims. - 17Artificial Intelligence Risk Management Framework: Generative AI Profile
NIST · NIST AI 600-1, 2024
Frames measurement, documentation, human oversight, and residual-risk disclosure for generative AI.
ASSESSMENT
The pilot is informative.
Publication validation remains.
The historical arithmetic reproduces, but construct validity and production-system efficacy remain separate questions. The next study needs a frozen production implementation, a fresh sample, independent adjudication, and dependence-aware inference. The current 97.3 is not a certification of production AIOS.
View architecture and conformance protocol