We obtained the dataset from ConvAI2 repository on the Facebook Parlai platform. Here, we omit the information on persona profiles since the current version of D-score framework only takes into consideration the dialogue contexts. (D-score 논문)

Evaluation dataset

For correlation analysis, we rely on the Persona-Chatlog human evaluation dataset collected by See et al. (A. See, S. Roller, D. Kiela, and J. Weston, “What makes a good conversation? how controllable attributes affect human judgments,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Minneapolis, Minnesota: Association for Computational Linguistics, June 2019, pp. 1702–1723.

human_eval.jsonl

{"assignment_id_hashed":"0345101360a5f71ffe33504eac80f78d2b488376","dialog":[{"speaker":"model","text":"Hello there! My name is Nina, what's yours?"},{"speaker":"human_evaluator","text":"I am George! How are you doing?"},{"speaker":"model","text":"I'm doing well, it's nice to meet you George. Do you have any hobbies?"}.............

"evaluation_results":{"enjoy":3,"interest":3,"listen":3,"turing":4,"avoid_rep":3,"make_sense":4,"fluency":4,"persona_guess":1,"inquisitive":2},"evaluator_id_hashed":"bbac4c5bcaaa125635812f63be06458f1ba5c700","human_persona":"i enjoy american sports.\ni have a children and a dogs.\ni own a house in florida.\ni work in it and have been at the same company for 15 years.","model_bucket":"","model_name":"human_eval","model_persona":"i am a huge coffee lover.\ni do not drink alcohol.\ni enjoy mountain biking as a hobby.\ni hate the color pink.\nmy father was a construction worker.","model_type":"human","oz_id_hashed":"3f03823334bf91fbe9873a340dc47e6806f335f8","start_time":0.0,"total_time":652.0225076675}

In our evaluation, a crowdworker chats with a model (or in the human-human case, another crowdworker) for six conversational turns, then answers eight multiple-choice questions which each capture different aspects of conversational quality: avoiding repetition, interestingness, making sense, fluency, listening, inquisitiveness, humanness and engagingness. The eight questions are Likert questions on a 1-4 scale, where higher is better.

To match the ConvAI2 Challenge, we also add a persona retrieval question, in which the crowdworker is asked to select which of two possible personas was the model’s persona.

→D-score에서는 persona 필요없어서 지워야함

each of our 28 model configurations was evaluated by over 100 crowdworkers, and the results were adjusted for annotator variance via a Bayesian calibration