Analisis Performa Ekstraksi Konten GPT-3 Dengan Matrik Bertscore Dan Rouge
DOI:
https://doi.org/10.25126/jtiik.1168088Kata Kunci:
GPT-3, Model Bahasa , Ekstraksi Konten, Pembuatan Teks , NLPAbstrak
Integrasi model bahasa canggih dalam tugas-tugas pembangkitan teks telah menampilkan beberapa aplikasi yang luas di berbagai bidang, termasuk ekstraksi konten. Penelitian ini memanfaatkan model bahasa OpenAI GPT-3 untuk mengembangkan aplikasi yang membantu dalam proses persiapan konten penulisan kreatif dengan menerapkan fitur ekstraksi konten. Fitur-fitur ini mencakup ekstraksi informasi, meringkas paragraf, mengidentifikasi topik utama, dan menafsirkan teks untuk presentasi terminologi yang optimal. Penelitian ini menggunakan pendekatan 'few-shot learning' yang melekat pada model GPT-3. Kinerja aplikasi ini dievaluasi secara ketat melalui uji coba, membandingkan efektivitasnya dengan mesin pembangkitan teks komersial yang banyak digunakan saat ini. Tujuannya adalah menganalisis tingkat kelayakan sistem yang telah kami bangun terhadap aplikasi lain yang populer. Metrik evaluasi termasuk BERTscore dan ROUGE digunakan sebagai pengujian. Aplikasi ini mencapai BERTscore sebesar 86% untuk precision, 88% untuk recall, dan 87% untuk F1-Score. Selain itu, evaluasi ROUGE menghasilkan skor ROUGE-L sebesar 55% pada precision, 60% pada recall, dan 57% pada F1-Score, hasil tersebut menunjukkan kekuatan model dalam tugas ekstraksi konten. Hasil ini memberikan gambaran bahwa model GPT-3 berpotensi baik dalam meningkatkan efisiensi dan akurasi untuk tugas persiapan konten tulisan dalam industri penulisan kreatif.
Abstract
The integration of advanced language models in text generation tasks has featured some extensive applications in various fields, including content extraction. This research utilises the OpenAI GPT-3 language model to develop an application that assists in the content preparation process of creative writing by implementing content extraction features. These features include information extraction, summarising paragraphs, identifying main topics, and interpreting text for optimal terminology presentation. This research utilises the ‘few-shot learning’ approach inherent to the GPT-3 model. The performance of this application was rigorously evaluated through trials, comparing its effectiveness with commercial text generation engines widely used today. The aim is to analyse the feasibility of the system we have built against other popular applications. Evaluation metrics including BERTscore and ROUGE were used as tests. The application achieved a BERTscore of 86% for precision, 88% for recall, and 87% for F1-Score. In addition, the ROUGE evaluation resulted in ROUGE-L scores of 55% in precision, 60% in recall, and 57% in F1-Score, these results show the strength of the model in the content extraction task. These results illustrate that the GPT-3 model has good potential in improving efficiency and accuracy for the task of writing content preparation in the creative writing industry.
Downloads
Referensi
ALADINI, Dr. A. 2023. AI applications impact on improving EFL University Academic writing skills and their logical thinking. https://doi.org/10.21608/ssj.2023.320166
ALAYRAC, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., & Reynolds, M. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35, 23716–23736.
BAHJA, M. 2020. Natural Language Processing Applications in Business. In R. M. X. Wu & M. Mircea (Eds.), E-Business (p. Ch. 4). IntechOpen. https://doi.org/10.5772/intechopen.92203
BROWN, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., … Amodei, D. 2020. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, & H. Lin (Eds.), Advances in Neural Information Processing Systems (Vol. 33, pp. 1877–1901). Curran Associates, Inc.
CHAN, A. 2023. GPT-3 and InstructGPT: technological dystopianism, utopianism, and “Contextual” perspectives in AI ethics and industry. AI and Ethics, 3(1), 53–64. https://doi.org/10.1007/s43681-022-00148-6
CHEN, S. F., & Goodman, J. 1999. An empirical study of smoothing techniques for language modeling. Computer Speech & Language, 13(4), 359–394. https://doi.org/https://doi.org/10.1006/csla.1999.0128
CHIU, K.-L., Collins, A., & Alexander, R. 2021. Detecting Hate Speech with GPT-3. March, 1–29. http://arxiv.org/abs/2103.12407
DALE, R. 2021. GPT-3: What’s it good for? Natural Language Engineering, 27(1), 113–118. https://doi.org/DOI: 10.1017/S1351324920000601
DING, B., Qin, C., Liu, L., Bing, L., Joty, S., & Li, B. 2022. Is GPT-3 a Good Data Annotator? http://arxiv.org/abs/2212.10450
GEVAERT, C. M., Carman, M., Rosman, B., Georgiadou, Y., & Soden, R. 2021. Fairness and accountability of AI in disaster risk management: Opportunities and challenges. Patterns, 2(11), 100363. https://doi.org/https://doi.org/10.1016/j.patter.2021.100363
HALUZA, D., & Jungwirth, D. 2023. Artificial Intelligence and Ten Societal Megatrends: An Exploratory Study Using GPT-3. Systems, 11(3). https://doi.org/10.3390/systems11030120
HOWARD, J., & Ruder, S. 2018. Universal Language Model Fine-tuning for Text Classification. Annual Meeting of the Association for Computational Linguistics. https://api.semanticscholar.org/CorpusID:40100965
KURNIATI, E. Y., & Fithriani, R. 2022. Post-Graduate Students’ Perceptions of Quillbot Utilization in English Academic Writing Class. Journal of English Language Teaching and Linguistics, 7(3), 437. https://doi.org/10.21462/jeltl.v7i3.852
LAMPLE, G., & Conneau, A. 2019. Cross-lingual language model pretraining. ArXiv Preprint ArXiv:1901.07291.
LATINOVIC, Z., & Chatterjee, S. C. 2022. Achieving the promise of AI and ML in delivering economic and relational customer value in B2B. Journal of Business Research, 144, 966–974. https://doi.org/https://doi.org/10.1016/j.jbusres.2022.01.052
LESTER, B., AL-RFOU, R., & CONSTANT, N. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning. https://doi.org/10.18653/v1/2021.emnlp-main.243
LOHR, L. L. 2000. Designing the instructional interface. Computers in Human Behavior, 16(2), 161–182. https://doi.org/https://doi.org/10.1016/S0747-5632(99)00057-6
MIKOLOV, T., KARAFIÁT, M., BURGET, L., ERNOCKÝ, J. H., & KHUDANPUR, S. 2010. Recurrent neural network based language model. Interspeech. https://api.semanticscholar.org/CorpusID:17048224
RADFORD, A., NARASHIMAN, K., SALIMANS, T., & SUTSKEVER, I. 2018. Improving Language Understanding by Generative Pre-Training. OpenAI. https://openai.com/blog/language-unsupervised/
RAMACHANDRAN, R., RAMASUBRAMANIAN, M., KOIRALA, P., GURUNG, I., & MASKEY, M. 2022. Language Model for Earth Science: Exploring Potential Downstream Applications as well as Current Challenges. IGARSS 2022 - 2022 IEEE International Geoscience and Remote Sensing Symposium, 4015–4018. https://api.semanticscholar.org/CorpusID:252590791
SCAO, T. Le, Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., & Gallé, M. (2022). Bloom: A 176b-parameter open-access multilingual language model. ArXiv Preprint ArXiv:2211.05100.
SINGH, R., Garg, V., & GPT-3. 2021. Human Factors in NDE 4.0 Development Decisions. Journal of Nondestructive Evaluation, 40(3), 71. https://doi.org/10.1007/s10921-021-00808-3
TAYLOR, R., Kardas, M., Cucurull, G., Scialom, T., Hartshorn, A., Saravia, E., Poulton, A., Kerkez, V., & Stojnic, R. 2022. Galactica: A large language model for science. ArXiv Preprint ArXiv:2211.09085.
THOMAS J Ackermann. (2020, November 29). GPT-3: a robot wrote this entire article. Are you scared yet, human? Artificial Intelligence: ANI, LogicGate Computing, AGI, ASI.
VASWANI, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. 2017. Attention is All you Need, Advances in neural information processing systems. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in Neural Information Processing Systems (Vol. 30). Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
WANG, Y., YAO, Q., KWOK, J. T., & NI, L. M. 2020. Generalizing from a Few Examples: A Survey on Few-Shot Learning. ACM Comput. Surv., 53(3). https://doi.org/10.1145/3386252
YADIN, D. L. 2001. Creative Marketing Communications: A Practical Guide to Planning, Skills and Techniques (3rd ed.). Kogan Page Publishers.
ZHANG, M., & LI, J. 2021. A commentary of GPT-3 in MIT Technology Review 2021. Fundamental Research, 1(6), 831–833. https://doi.org/https://doi.org/10.1016/j.fmre.2021.11.011
Unduhan
Diterbitkan
Terbitan
Bagian
Lisensi
Hak Cipta (c) 2024 Jurnal Teknologi Informasi dan Ilmu Komputer
Artikel ini berlisensiCreative Commons Attribution-ShareAlike 4.0 International License.
Artikel ini berlisensi Creative Common Attribution-ShareAlike 4.0 International (CC BY-SA 4.0)
Penulis yang menerbitkan di jurnal ini menyetujui ketentuan berikut:
- Penulis menyimpan hak cipta dan memberikan jurnal hak penerbitan pertama naskah secara simultan dengan lisensi di bawah Creative Common Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) yang mengizinkan orang lain untuk berbagi pekerjaan dengan sebuah pernyataan kepenulisan pekerjaan dan penerbitan awal di jurnal ini.
- Penulis bisa memasukkan ke dalam penyusunan kontraktual tambahan terpisah untuk distribusi non ekslusif versi kaya terbitan jurnal (contoh: mempostingnya ke repositori institusional atau menerbitkannya dalam sebuah buku), dengan pengakuan penerbitan awalnya di jurnal ini.
- Penulis diizinkan dan didorong untuk mem-posting karya mereka online (contoh: di repositori institusional atau di website mereka) sebelum dan selama proses penyerahan, karena dapat mengarahkan ke pertukaran produktif, seperti halnya sitiran yang lebih awal dan lebih hebat dari karya yang diterbitkan. (Lihat Efek Akses Terbuka).
