How good can machine generated texts be identified and can language models be trained to avoid identification?

Simple item page
dc.contributor.authorSchneider, Sinclair
dc.contributor.authorSteuber, Florian
dc.contributor.authorSchneider, João A. G.
dc.contributor.authorDreo Rodosek, Gabi
dc.date.accessioned2023-12-26T18:39:02Z
dc.date.available2023-12-26T18:39:02Z
dc.date.issued2024-01-03
dc.identifier.doihttps://doi.org/10.24251/HICSS.2024.328
dc.identifier.isbn978-0-9981331-7-1
dc.identifier.other6dcf2dea-daf2-4966-a2bb-240e6cf85bd7
dc.identifier.urihttps://hdl.handle.net/10125/106711
dc.language.isoeng
dc.relation.ispartofProceedings of the 57th Hawaii International Conference on System Sciences
dc.rightsAttribution-NonCommercial-NoDerivatives 4.0 International
dc.rights.urihttps://creativecommons.org/licenses/by-nc-nd/4.0/
dc.subjectGenerative AI and AI-generated Contents on Social Media
dc.subjectlanguage model detection
dc.subjectlanguage models
dc.subjecttransformer reinforcement learning
dc.titleHow good can machine generated texts be identified and can language models be trained to avoid identification?
dc.typeConference Paper
dc.type.dcmiText
dcterms.abstractWith the rise of generative pre-trained transformer models such as GPT-3, GPT-NeoX, or OPT, distinguishing human-generated texts from machine-generated ones has become important. We refined five separate language models to generate synthetic tweets, uncovering that shallow learning classification algorithms, like Naive Bayes, achieve detection accuracy between 0.6 and 0.8. Shallow learning classifiers differ from human-based detection, especially when using higher temperature values during text generation, resulting in a lower detection rate. Humans prioritize linguistic acceptability, which tends to be higher at lower temperature values. In contrast, transformer-based classifiers have an accuracy of 0.9 and above. We found that using a reinforcement learning approach to refine our generative models can successfully evade BERT-based classifiers with a detection accuracy of 0.15 or less.
dcterms.extent10 pages
prism.startingpage2716

Files

Original bundle

Now showing 1 - 1 of 1
Thumbnail Image
Name:
0265.pdf
Size:
2.61 MB
Format:
Adobe Portable Document Format
Download

Collections

Generative AI and AI-generated Contents on Social Media