Accéder directement au contenu Accéder directement à la navigation
Pré-publication, Document de travail

ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports

Abstract : In this article, the authors built and used a custom transformer-based model, based on a compact version of RoBERTa, named ASRS-CMFS, to classify aviation incident reports. The classification is applied to fourteen distinct sets of specific aviation incident-related anomalies, such as Aircraft Equipment problems or Altitude Deviation problems. The authors extracted the incident reports and the associated fourteen sets of categories from the Aviation Safety Reporting System. After discussing the choice of evaluation metric, the authors evaluated the model using the Matthews Correlation Coefficient metric. To measure the precision of the scores obtained on the different text classification problems, the authors provided the results with confidence intervals. They also used statistical hypothesis testing to evaluate the impact of the document length on the performance of the custom model. The authors provided a mathematical demonstration for the use of confidence intervals and hypotheses testing on MCC values. Finally, the authors discussed whether the model was fit for use in a professional environment. The authors found that while the model showed promising results, this question could only remain unanswered at this stage, but the steps to take are clear. The authors also proposed that hypothesis testing could 1 be valuable in any situation where one wanted to study the impact of a particular document feature on the performance of a document classifier.
Keywords : ASRS BERT MCC NLP aviation
Type de document :
Pré-publication, Document de travail
Liste complète des métadonnées

https://hal.archives-ouvertes.fr/hal-03487944
Contributeur : Samuel Kierszbaum Connectez-vous pour contacter le contributeur
Soumis le : vendredi 17 décembre 2021 - 15:58:35
Dernière modification le : lundi 4 juillet 2022 - 09:58:47
Archivage à long terme le : : vendredi 18 mars 2022 - 19:31:47

Fichier

Manuscript.pdf
Fichiers produits par l'(les) auteur(s)

Identifiants

  • HAL Id : hal-03487944, version 1

Citation

Samuel Kierszbaum, Thierry Klein, Laurent Lapasset. ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports. 2021. ⟨hal-03487944⟩

Partager

Métriques

Consultations de la notice

78

Téléchargements de fichiers

26