Development of Smart Assessment System For Evaluating Maritime English Competence Using Machine Learning

Abstract

Maritime English proficiency assessment is essential for cadets and maritime professionals, yet manual scoring can be time-consuming and prone to inter-rater variability. This study proposes a web-based smart assessment system that integrates machine learning to classify Maritime English proficiency into Beginner/Intermediate/Advanced using four feature scores: listening, reading, writing, and speaking. The dataset used in this work is simulated (1,000 records) for proof-of-concept evaluation because access to large, standardized real examination data was limited and required institutional clearance; simulation enables controlled class balance and repeatable experimentation. Class labels are generated using rubric-based threshold rules, and the labeling scheme is validated by two Maritime English examiners who review the thresholds and independently rate a random subset of 200 samples; agreement is quantified using Cohen’s kappa (κ) to ensure reliability. We adopt an 80/20 hold-out split and apply stratified 5-fold cross-validation on the training set for model selection, using grid-search hyperparameter tuning. We compare Support Vector Machine (SVM) and Random Forest and report accuracy, precision, recall, macro-F1, and brief per-class performance for Beginner/Intermediate/Advanced. SVM achieves 92% accuracy with macro-F1 = 0.905, outperforming Random Forest (89%, macro-F1 = 0.875). Future work will validate the system using real assessment datasets in operational training settings.