Automating cyber risk assessment with public LLMs: an expert-validated framework and comparative analysis

Basit Öğe Kaydı
dc.authorid0009-0009-4710-2569
dc.authorid0000-0003-2865-6370
dc.contributor.authorÜnal, Nezih Mahmuten_US
dc.contributor.authorÇeliktaş, Barışen_US
dc.date.accessioned2026-04-20T12:12:59Z
dc.date.available2026-04-20T12:12:59Z
dc.date.issued2026-03-26
dc.departmentIşık Üniversitesi, Lisansüstü Eğitim Enstitüsü, Siber Güvenlik Yüksek Lisans Programıen_US
dc.departmentIşık University, School of Graduate Studies, Master’s Program in Cybersecurityen_US
dc.departmentIşık Üniversitesi, Mühendislik ve Doğa Bilimleri Fakültesi, Bilgisayar Mühendisliği Bölümüen_US
dc.departmentIşık University, Faculty of Engineering and Natural Sciences, Department of Computer Engineeringen_US
dc.description.abstractTraditional cyber risk assessment methodologies face a critical dilemma: they are either quantitative yet static and context-agnostic (e.g., CVSS), or context-aware yet highly labor-intensive and subjective (e.g., NIST SP 800-30). Consequently, organizations struggle to scale risk assessment to match the pace of evolving threats. This paper presents an automated, context-aware risk assessment framework that leverages the reasoning capabilities of publicly available Large Language Models (LLMs) to operationalize expert knowledge. Rather than positioning the LLM as the final decision-maker, the framework decouples semantic interpretation from risk scoring authority through a transparent, deterministic Dynamic Metric Engine. Unlike complex closed box machine learning models, our approach anchors the AI's reasoning to this expert-validated metric schema, with weights derived using the Rank Order Centroid (ROC) method from a survey of 101 cybersecurity professionals. We evaluated the framework through a comparative study involving 15 diverse real-world vulnerability scenarios (C1-C15) and three supplementary sensitivity stress tests (C16-C18). The validation scenarios were independently assessed by a cohort of ten senior human experts and two state-of-the-art LLM agents (GPT-4o and Gemini 2.0 Flash). The results show that the LLM-driven agents achieve scoring consistency closely aligned with the human median (Pearson r ranging from 0.9390 to 0.9717, Spearman ρ from 0.8472 to 0.9276) against a highly reliable expert baseline (Cronbach's α =0.996), while reducing the assessment cycle time by more than 100× (averaging under 4 seconds per case vs. a human average of 6 minutes). Furthermore, a dedicated context sensitivity analysis (C13-C15) indicates that the framework adapts risk scores based on organizational context (e.g., SME vs. Critical Infrastructure) for identical technical vulnerabilities. Importantly, the system is designed not merely to replicate expert intuition, but to enforce bounded, policy-consistent risk evaluation under predefined governance constraints. Overall, these findings suggest that commercially available LLMs, when constrained by expert-validated metric schemas, can support reproducible, transparent, and real-time risk assessments.en_US
dc.description.versionPublisher's Versionen_US
dc.identifier.citationÜnal, N. M. & Çeliktaş, B. (2026). Automating cyber risk assessment with public LLMs: an expert-validated framework and comparative analysis. IEEE Access, 14, 47754-47778. doi:https://doi.org/10.1109/ACCESS.2026.3678044en_US
dc.identifier.doi10.1109/ACCESS.2026.3678044
dc.identifier.endpage47778
dc.identifier.issn2169-3536
dc.identifier.scopus2-s2.0-105035002785
dc.identifier.scopusqualityQ1
dc.identifier.startpage47754
dc.identifier.urihttps://hdl.handle.net/11729/7323
dc.identifier.urihttps://doi.org/10.1109/ACCESS.2026.3678044
dc.identifier.volume14
dc.identifier.wosWOS:001732696000028
dc.identifier.wosqualityQ2
dc.indekslendigikaynakScopusen_US
dc.indekslendigikaynakWeb of Scienceen_US
dc.indekslendigikaynakScience Citation Index Expanded (SCI-EXPANDED)en_US
dc.institutionauthorÜnal, Nezih Mahmuten_US
dc.institutionauthorÇeliktaş, Barışen_US
dc.institutionauthorid0009-0009-4710-2569
dc.institutionauthorid0000-0003-2865-6370
dc.language.isoenen_US
dc.peerreviewedYesen_US
dc.publicationstatusPublisheden_US
dc.publisherInstitute of Electrical and Electronics Engineers Inc.en_US
dc.relation.ispartofIEEE Accessen_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Öğrencien_US
dc.relation.publicationcategoryMakale - Uluslararası Hakemli Dergi - Kurum Öğretim Elemanıen_US
dc.rightsinfo:eu-repo/semantics/openAccessen_US
dc.subjectAutomated risk scoringen_US
dc.subjectCyber risk assessmenten_US
dc.subjectGenerative AIen_US
dc.subjectHuman-AI comparisonen_US
dc.subjectLarge Language Models (LLMs)en_US
dc.subjectRank Order Centroid (ROC)en_US
dc.subjectArtificial intelligenceen_US
dc.subjectAutomationen_US
dc.subjectCritical infrastructuresen_US
dc.subjectCybersecurityen_US
dc.subjectDecision makingen_US
dc.subjectLearning systemsen_US
dc.subjectRisk analysisen_US
dc.subjectRisk assessmenten_US
dc.subjectRisk managementen_US
dc.subjectSemanticsen_US
dc.subjectCybe risk assessmenten_US
dc.subjectLanguage modelen_US
dc.subjectLarge language modelen_US
dc.subjectRank order centroiden_US
dc.subjectRank orderingen_US
dc.subjectRisk scoringen_US
dc.subjectRisks assessmentsen_US
dc.subjectSensitivity analysisen_US
dc.subjectInterneten_US
dc.titleAutomating cyber risk assessment with public LLMs: an expert-validated framework and comparative analysisen_US
dc.typeArticleen_US
dspace.entity.typePublicationen_US

Dosyalar

Orijinal paket
Listeleniyor 1 - 1 / 1
Küçük Resim
İsim:
Automating_Cyber_Risk_Assessment_With_Public_LLMs_An_Expert_Validated_Framework_and_Comparative_Analysis.pdf
Boyut:
4.36 MB
Biçim:
Adobe Portable Document Format
İndir
Lisans paketi
Listeleniyor 1 - 1 / 1
Küçük Resim Yok
İsim:
license.txt
Boyut:
1.17 KB
Biçim:
Item-specific license agreed upon to submission
Açıklama:
İndir

Koleksiyon

Öğrenci Yayınları Makale Koleksiyonu
Lisansüstü Eğitim Enstitüsü Diğer Yayınlar Koleksiyonu
Makale Koleksiyonu | Bilgisayar Mühendisliği Bölümü
Scopus İndeksli Yayınlar Koleksiyonu
WoS İndeksli Yayınlar Koleksiyonu