Skip to Main content Skip to Navigation
Preprints, Working Papers, ...

ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports

Abstract : In this article, the authors built and used a custom transformer-based model, based on a compact version of RoBERTa, named ASRS-CMFS, to classify aviation incident reports. The classification is applied to fourteen distinct sets of specific aviation incident-related anomalies, such as Aircraft Equipment problems or Altitude Deviation problems. The authors extracted the incident reports and the associated fourteen sets of categories from the Aviation Safety Reporting System. After discussing the choice of evaluation metric, the authors evaluated the model using the Matthews Correlation Coefficient metric. To measure the precision of the scores obtained on the different text classification problems, the authors provided the results with confidence intervals. They also used statistical hypothesis testing to evaluate the impact of the document length on the performance of the custom model. The authors provided a mathematical demonstration for the use of confidence intervals and hypotheses testing on MCC values. Finally, the authors discussed whether the model was fit for use in a professional environment. The authors found that while the model showed promising results, this question could only remain unanswered at this stage, but the steps to take are clear. The authors also proposed that hypothesis testing could 1 be valuable in any situation where one wanted to study the impact of a particular document feature on the performance of a document classifier.
Keywords : ASRS BERT MCC NLP aviation
Document type :
Preprints, Working Papers, ...
Complete list of metadata

https://hal.archives-ouvertes.fr/hal-03487944
Contributor : Samuel Kierszbaum Connect in order to contact the contributor
Submitted on : Friday, December 17, 2021 - 3:58:35 PM
Last modification on : Tuesday, January 4, 2022 - 5:58:34 AM

File

Manuscript.pdf
Files produced by the author(s)

Identifiers

  • HAL Id : hal-03487944, version 1

Citation

Samuel Kierszbaum, Thierry Klein, Laurent Lapasset. ASRS-CMFS: Using a custom Transformer-based model to predict anomalies in aviation incident reports. 2021. ⟨hal-03487944⟩

Share

Metrics

Les métriques sont temporairement indisponibles