Evaluating Threat-Sensitive AI Models for Multilingual Messaging Security

Abstract

Smishing messages increasingly mix languages and local expressions, limiting the reliability of classifiers trained on narrow datasets. This gap has consequences for interpreting performance outside carefully curated evaluation settings. The research question is whether it is possible to compare threat-sensitive multilingual models under language drift and uneven malicious-message prevalence. Previous work provides context for sampling, input validation and study boundaries [3–7]. The planned analysis uses a held-out-language evaluation using deduplicated annotated messages and temporally separated test sets. Prior work on storage operations provides technical context for validation [1], while complementary systems evidence motivates service-continuity checks [2]. Under matched evaluation conditions, the protocol compares classical text models, transformer classifiers and explanation-supported ensembles. Methodological sources are used to specify transparent comparison and reporting practices [8–12]. The design prioritises precise operational definitions and a straightforward baseline before introducing more complex alternatives. Performance will be judged using malicious-message recall, false blocking, calibration and analyst effort per suspicious message. Interpretation will account for code-switching, label uncertainty, distribution shift and near-duplicate leakage. Primary analyses would be reported by the main deployment or participant subgroup so that averaged results do not conceal systematic failures. The supplementary technical literature is used to identify plausible failure mechanisms [13–17]. No study result is assumed in advance; the analysis is designed to show both when the method helps and when it fails, with reporting context from the remaining sources [18–23].