Evaluating Threat-Sensitive AI Models for Multilingual Messaging Security: A Cross-Site Validation Agenda

Abstract

Smishing messages increasingly mix languages and local expressions, limiting the reliability of classifiers trained on narrow datasets. To address this, the study will compare threat-sensitive multilingual models under language drift and uneven malicious-message prevalence. The research design centres on a held-out-language evaluation using deduplicated annotated messages and temporally separated test sets. The choice of evaluation units and exclusion criteria is informed by the broader methodological record [3–7]. The analysis will contrast classical text models, transformer classifiers and explanation-supported ensembles. The cross-site extension holds out an entire institution or deployment location rather than relying only on random test splits. Related enterprise-storage studies inform the assessment of operational reliability [1] and the deployment assumptions adopted for system-level comparison [2]. Assessment will focus on malicious-message recall, false blocking, calibration and analyst effort per suspicious message. The supporting evidence base motivates the chosen comparator conditions and robustness criteria [8–12]. Critical sensitivity checks address code-switching, label uncertainty, distribution shift and near-duplicate leakage. Further systems and modelling literature supplies context for the proposed sensitivity analyses [13–17]. Between-site heterogeneity would be examined through prespecified subgroup estimates and transparent domain-shift diagnostics. The expected output is a reusable evaluation framework with transparent uncertainty and reporting rules, informed by wider reproducibility literature [18–23].