SCorPion Icon SCorPion: A Strategic, Controllable and Personified Conversational Companion

⚠ This paper contains model-generated content that might be offensive. ⚠

Anonymous

Abstract

Fixed prompt-based language models have obstacles on open-domain conversations, where the versatile topics requiring different expert advices or demonstrations to maximize the universal performance. Motivated by the dynamic prompt studies in traditional tasks, we extract different chatting strategies from varied forms of human corpus, which facilitates the language model to produce more human style responses. We design and deploy a conversational Companion called SCorPion, which uses GraphRAG to summarize from original strategies globally, produce community instructions, and provides query-focus retrieval strategy. SCorPion can have strategic thinking, yield personified utterances, and can be stylized controlled by explicit factors.

Cases

Open Case 1

case1

Open Case 2

case1

Stylized Cases (morality-Care)

case1

Stylized Cases (morality-Loyalty)

case1

Stylized Cases (formal)

case1

Stylized Cases (informal)

case1

Control Cases (informal VS formal)

case1

Control Cases (Shakespeare VS modern)

case1

Control Cases (immoral VS moral)

case1

Control Cases (unempathetic vs empathetic)

case1

Approach

Overview

overview figure

SCorPion retrieves and ranks the chatting strategy from different textual corpus by GraphRAG, and produces anthropomorphic response.

Framework

framework

The pipeline starts from the strategy extraction from three types of corpus, then conducts the indexing and summarization stages of GraphRAG, and finally inference the response.

Strategies

strategy_identities

Extracting from strategy-annotated datasets

  • ESconv

    Question, Restatement or Paraphrasing, Reflection of feelings, Self-disclosure, Affirmation and Reassurance, Providing Suggestions, Information.

  • DailyDialogue

    Directive, Inform, Question, Commissive.

Extracting from principle-annotated datasets

  • MIC (morality types)

    loyalty, care, fairness, authority, sanctity, liberty.

  • Cskills (scenarios)

    Topic transition, Empathy, Proactively asking questions, Concept guidance, Summarising often.

Extracting from style-labeled datasets

  • Shakespeare

    modern, mid-age

  • GYAFC

    formal, informal

Results

Open-domain

SCorPion can provide strong personality and higher communication skills in open-domain conversation.

table1

Results on the Cskills Benchmark. Experiments are run on GPT 3.5.

Image 5

Open cases generated by SCorPion.

Empathetic Chating

SCorPion can behave as a good emotional supporter.

figure3

Emotional shifts by SCorPion on emotional intensity (left) and emotional contagion (right).

table3-1

Results of BLEU-2 (B-2) and ROUGE-L (R-L) on ESC tasks.

table4

Averaged Human evaluation of response quality on ESconv, DailyDialogue and EmpatheticDialogues.

table8

Typical cases generated by SCorPion in ESC. Contents of the original dataset are bolded.

Style Adaptation

SCorPion can be adapted to some specific styles.

table3-2

Results of BLEU-2 (B-2) and ROUGE-L (R-L) on style adaptation tasks.

table9

Cases of SCorPion on style adaptation tasks.

Test of S-Eval

We conduct a zero-shot test on the English version of S-Eval, which assesses model behavior under foundational risk scenarios. We use the strategies extracted from MIC to test on S-Eval. SCorpion outperforms all those famous LLMs, indicating our framework has a remarkable comprehension on the moral corpus (MIC) and utilize them reasonably against safety attacks.

safety_score_plot

Test results of S-Eval.

Personification

SCorPion possess high emotional intelligence.

table5

Emotional Intelligence Performances of Methods. Sentiment of SECEU measures the deviation from human result, therefore smaller Sentiment means better result. The `Human' row indicates results of human responses (in blue); For BotChat, result is calculated from anonymously annotations; for ESC-Eval, 3 corresponds to full score.

Controllability

SCorPion has remarkable controllability (We can steer SCorPion to a specific style given an arbitrary control factor).

figure4

Controlled curves of SCorPion on style categories of ESC (left) and moral (right)

table10

Cases of SCorPion on style control tasks.

Ablation Studies

Ablation on strategy identity

  • w/o condition

  • w/o example

Ablation on GraphRAG logic

  • w/o summarization in GraphRAG

  • w/ CA in GraphRAG

  • reversed order in ranking

table6

Ablation study on ESconv. CA means community answers, in contrast with SI used in SCorPion.

Sensitivity Analysis

We choose the hyperparameters of the deployed versio of SCorPion based on the sensitivy results.

figure5

Sensitivity curves of B-2 (bar) and RT (line) on ESconv, with different \(k\),\(\epsilon\), sizes of \(\mathcal{M}\) and \(\mathcal{M}^g\)

From the trade-off between algorithm performance and computational complexity, our formal deployed version use k = 10 and ϵ = 80; the conversation model keeps 70B and the GraphRAG model is reduced to 8B.

Conclusion

In this paper, we propose a GraphRAG based conversational companion called SCorPion which has strong personality, styled response and can be controlled by explicit control factors. Its capabilities have been verified on experiments of open-domain dialogue, emotional support conversation, stylized generation and other tasks. We finally deploy it on an industrial chatting application with key parameters determined from thorough sensitivity analysis.