ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports - ENAC - École nationale de l'aviation civile Accéder directement au contenu
Article Dans Une Revue Aerospace Année : 2022

ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports

Samuel Kierszbaum
Laurent Lapasset

Résumé

In this article, the authors built and used a custom transformer-based model, based on a compact version of RoBERTa, named ASRS-CMFS, to classify aviation incident reports. The classification is applied to fourteen distinct sets of specific aviation incident-related anomalies, such as Aircraft Equipment problems or Altitude Deviation problems. The authors extracted the incident reports and the associated fourteen sets of categories from the Aviation Safety Reporting System. After discussing the choice of evaluation metric, the authors evaluated the model using the Matthews Correlation Coefficient metric. To measure the precision of the scores obtained on the different text classification problems, the authors provided the results with confidence intervals. They also used statistical hypothesis testing to evaluate the impact of the document length on the performance of the custom model. The authors provided a mathematical demonstration for the use of confidence intervals and hypotheses testing on MCC values. Finally, the authors discussed whether the model was fit for use in a professional environment. The authors found that while the model showed promising results, this question could only remain unanswered at this stage, but the steps to take are clear. The authors also proposed that hypothesis testing could 1 be valuable in any situation where one wanted to study the impact of a particular document feature on the performance of a document classifier.

Mots clés

Fichier principal
Vignette du fichier
Manuscript.pdf (1.05 Mo) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)

Dates et versions

hal-03487944 , version 1 (17-12-2021)

Identifiants

Citer

Samuel Kierszbaum, Thierry Klein, Laurent Lapasset. ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports. Aerospace, 2022, 9 (591), ⟨10.3390/aerospace9100591⟩. ⟨hal-03487944⟩
141 Consultations
116 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More