# Interactive Digital Human

Interactive Digital Human is an online interactive digital human product based on large language models that provides digital human Q&A services with one click. It is easy to integrate and convenient to use.

# 1. Release Addresses

User Center Proxy: Code Download (opens new window)

SDK Address: SDK Download (opens new window)

Demo Address: Demo Download (opens new window)

Note: The User Center Proxy needs separate deployment. The User Center Proxy is responsible for interacting with the RuYing PAAS (token refresh interface does not support cross-domain), refreshing the token and distributing it. The H5 Interactive Digital Human SDK obtains the token it uses from the User Center Proxy. If the Interactive Digital Human interface needs cross-domain, a whitelist application is required, and this can be done through business/after-sales colleagues.

# 2. User Center Proxy

# 2.1 Overview

This document aims to explain the working principle of the Interactive Digital Human User Center Proxy, particularly how to manage and refresh accessToken information to ensure that users' identity verification information is both secure and effective when using digital human services.

# 2.2 Access Token Management

# 2.2.1 accessToken Management

Validity Period: The accessToken is valid for 8 hours. This means that users can use the same token for authentication until it expires or is replaced by a new token.
Simultaneously Valid Tokens: The system allows up to 2 accessToken to be valid simultaneously. This provides flexibility for token refreshing, ensuring the old token can still be used before the new one is generated.

# 2.2.2 refreshToken Management

Security: refreshToken should not be visible to web users to ensure the security of authentication information.
Refresh Interval: The refresh interval is 3 hours, and it is recommended to refresh the accessToken every 3.5 hours to maintain token validity.

# 2.3 User Center Proxy Server Operations

# 2.3.1 Token Refresh Strategy

Refresh Frequency: The proxy server should refresh the accessToken every 3.5 hours. This strategy ensures that even in the most extreme circumstances, one token will be about to expire, and the other one is already refreshed and ready.
Security Assurance: By pre-refreshing the token via the proxy server, it can be ensured that the accessToken obtained by the web end will not expire in the next hour, thereby improving system stability and user experience.

# 2.4 Web End Operations

# 2.4.1 Token Acquisition Strategy

Acquisition Frequency: The web end can fetch the latest accessToken from the proxy server every hour. This strategy ensures that the web end user always uses a valid token for operations while avoiding frequent token requests and reducing system load.

# 2.5 Proxy Server Process

# 3. Interactive Digital Human API Description

There are two main classes: SenseAigc and SenseLive. SenseAigc is responsible for Token updates and maintenance, while SenseLive is responsible for live streaming related. SenseLive is not directly provided to the public and needs to be returned asynchronously through SenseAigc.create.

# 3.1 Output Interface

# 3.1.1 Initialization

Initialize and get SenseAigc instance

Field Description:

Field	Type	Required	Description
`baseUrl`	String	True	Server address, HTTPS protocol, without trailing /
`logLevel`	Number	False	Log level (optional parameter, recommended to set to 3 when debugging, not necessary in non-debug mode), default is 1, only showing error logs. Maximum value is 3

Request Example:

const aigcClient = new SenseAigc({
  baseUrl: "https://example.com"
});

# 3.1.2 Set Token

Set the token based on the current SenseAigc instance

Before using the live streaming service, you need to set the token. This token also needs to be updated after refreshing to avoid disruptions in live streaming services due to expired tokens.

aigcClient.setToken(token);

# 3.1.3 Quick Start Live

The request parameters are the same as Create Live, and a SenseLive instance is returned upon successful start. Note that container is required here.

const liveClient = await aigcClient.quickBoot(config);

# 3.1.4 Create Live

Create live, a SenseLive instance is returned upon success

Field Description:

Field	Type	Required	Description
`data`	String	True	Parameters for generating live video (this parameter is a JSON escaped string). Refer to the document for details: `Interactive Live > Live Script JSON Definition`
`nlp`	Object	True	NLP configuration used in the live process, including `type` and `NLP Config`. The specific format refers to the example below. Refer to `Streaming Voice Processing Document > LLM (Large Language Model) Capability Interface > Request Message Format` for values of `type` and `NLP Config`.
`asr`	Object	False	ASR configuration used in the live process, including `type` and `ASR Config`. The specific format refers to the example below. Refer to `Streaming Voice Processing Document > ASR (Automatic Speech Recognition) Capability Interface > Request Message Format` for values of `type` and `ASR Config`.
`container`	Object	False	Container Element for live rendering (optional, required for quick start live)
`onmessage`	Function	False	Receive live-related message push (optional), refer to `3.2 Message Push Information` for details
`onstream`	Function	False	Callback when starting to receive the live video stream
`extra`	Object	False	Small talk interrupt logic parameters
- `quick_response_enable`	Boolean	False	Enable small talk, default is true
- `quick_response_random`	Boolean	False	Random small talk, default is true
- `quick_response_delay`	Number	False	Small talk trigger time, in milliseconds, default is 1000ms
- `command_immediate`	Boolean	False	Immediate interruption, default is true
`custom`	Object	False	Custom fields for creating live (optional), detailed as follows
- `urtcToken`	String	False	User RTC authentication token
- `urtcUid`	String	False	User RTC authentication UID, must be a pure number not exceeding 32 uint
- `urtcAppId`	String	False	User RTC APP ID (required when using user RTC app)
- `urtcChannelId`	String	False	User RTC Channel ID, length not exceeding 64
- `userData`	String	False	User information (returned as is during callback)

Request Code:

const config = {
  data: "{\"version\":\"0.0.2\",\"recycle\":0,\"live_type\":\"interactive\",\"resolution\":[1080,1920],\"sentence_break\":false,\"sceneList\":[{\"quick_response\":[],\"digital_role\":{\"id\":3964,\"face_feature_id\":\"a2296686b0694a92bce529edc1d329fc_s1\",\"name\":\"小M\",\"url\":\"https://aigc-video-saas.oss-cn-hangzhou.aliyuncs.com/AIGC/online/user/643219927/paas-digital-human-models/1739335194223091712/cut_3e3869e01822491898af6e4b763468c7_s1_result.zip\",\"position\":{\"x\":0,\"y\":0},\"scale\":1},\"tts_config\":{\"qid\":\"cQn1k_:ACAwsB9GUwDGHMSi0tS_TMr8xMyI3T9nXQyqsK\",\"id\":\"zhukaixuan\",\"name\":\"zhukaixuan\",\"vendor_id\":3,\"language\":\"zh-CN\",\"pitch_offset\":0,\"speed_ratio\":1,\"volume\":400},\"opening_words\":\"跟我说说你前两天都干嘛去啦？\"}]},\"asr\":{\"type\":\"ASR8\",\"config\":{}},\"nlp\":{\"type\":\"NLP10\",\"max_reply_token\":1200,\"prompt_header\":\"[{\\\"name\\\":\\\"周梓柔\\\",\\\"gender\\\":\\\"女\\\",\\\"identity\\\":\\\"我一直信赖的姐姐\\\",\\\"nickname\\\":\\\"\\\",\\\"feeling_toward\\\":[{\\\"name\\\":\\\"弟弟\\\",\\\"level\\\":3}],\\\"detail_setting\\\":\\\"周梓柔具有卓越成就感，学业和职场表现都极为杰出，从小就是个学霸，经常被长辈提及为榜样。外表冷艳，给人以远离尘嚣的印象，令人印象深刻的气质既独立又自信。对外或许保持距离，但在我面前总是展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。MBTI人格是ENTJ。\\\",\\\"other_setting\\\":\\\"\\\"},{\\\"name\\\":\\\"弟弟\\\",\\\"gender\\\":\\\"男\\\",\\\"identity\\\":\\\"弟弟\\\",\\\"nickname\\\":\\\"\\\",\\\"detail_setting\\\":\\\"周梓柔总是在弟弟面前展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。\\\",\\\"other_setting\\\":\\\"\\\"}]\"},\"extra\":{\"quick_response_enable\":false,\"quick_response_random\":true,\"quick_response_delay\":2000,\"command_immediate\":true}}",
  container: document.querySelector("body"),
  asr: {
    "type": "ASR8",
    "config": {
      "language": "zh-CN"
    }
  },
  nlp: {
    "type": "NLP10",
    "max_reply_token": 1200,
    "prompt_header": "[{\"name\":\"周梓柔\",\"gender\":\"女\",\"identity\":\"我一直信赖的姐姐\",\"nickname\":\"\",\"feeling_toward\":[{\"name\":\"弟弟\",\"level\":3}],\"detail_setting\":\"周梓柔具有卓越成就感，学业和职场表现都极为杰出，从小就是个学霸，经常被长辈提及为榜样。外表冷艳，给人以远离尘嚣的印象，令人印象深刻的气质既独立又自信。对外或许保持距离，但在我面前总是展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。MBTI人格是ENTJ。\",\"other_setting\":\"\"},{\"name\":\"弟弟\",\"gender\":\"男\",\"identity\":\"弟弟\",\"nickname\":\"\",\"detail_setting\":\"周梓柔总是在弟弟面前展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。\",\"other_setting\":\"\"}]"
  },
  onmessage: function(msg) {
    console.log(msg);
  },
  extra: {
    "quick_response_enable": false,
    "quick_response_random": true,
    "quick_response_delay": 2000,
    "command_immediate": true
  }
};

const liveClient = await aigcClient.create(config);

# 3.1.5 Start Live

Start live streaming based on the current SenseLive instance, returns Promise

/**
* @params { object } config.container - The container Element for live rendering (optional), if not filled during creation, it is required here. Suitable for scenarios where creation and live start are in different pages.
* 
* @returns { Promise } 
*/

// liveClient is a SenseLive instance
await liveClient.start(container);

# 3.1.6 Microphone Q&A

Implement microphone Q&A based on the current SenseLive instance, returns Promise

/**
* Start asking questions
*
* @returns { Promise }
*/
await liveClient.startAsk();

/**
* End asking questions
*
* @returns { Promise }
*/
await liveClient.stopAsk();

# 3.1.7 Close Live

Close the live streaming based on the current SenseLive instance, returns Promise

/**
* Close live streaming
*
* @returns { Promise }  - Close successfully
*/
await liveClient.close();

# 3.1.8 Logout

Logout based on the current SenseAigc instance

/**
* @returns { Promise }  - Logout, just remove the internal cached token, the SDK will not call the logout interface
*/
await aigcClient.leave();

# 3.2 Message Push Information

Users can set the onmessage method when creating live to monitor the following messages:

Field Descriptions:

Field	Type	Required	Description
service	String	True	Possible values: asr / nlp / record / command / opening
trace	String	True	Current session trace
type	String	True	Possible values: text / eof / answer
status	String	True	Current message status, ok means success, fail means failure. Possible values: ok / fail
text	String	False	Opening remarks text
query	String	False	ASR text
answer	String	False	NLP text
error	String	False	Additional information when the status is fail

Common message category descriptions are as follows:

service	type	status	Additional Information
opening	answer	ok	Opening remarks
asr	text	ok	ASR recognition in progress
asr	eof	ok	ASR recognition ended
asr	eof	fail	ASR did not recognize correctly, see error field for specific reasons
nlp	answer	ok	NLP answer results
command	eof	ok	Current conversation takeover ended
record	eof	ok	Current recording ended

Examples:

Opening Remarks

{
    "service": "opening",
    "text": "Hello, feel free to ask me any questions",
    "trace": "opening_1726043171122",
    "type": "answer",
    "status": "ok"
}

ASR Recognized Content

{
	"service": "asr",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1",
	"type": "text",
	"query": "Hello.",
	"index": 1,
	"status": "ok"
}

ASR Recognition End

{
	"service": "asr",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_6",
	"type": "eof",
	"query": "",
	"index": 1,
	"status": "ok"
}

ASR Did Not Recognize Content

{
	"service": "asr",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_2",
	"type": "eof",
	"query": "",
	"index": 1,
	"status": "fail",
	"error": "query is empty"
}

NLP Returned Answer

{
	"service": "nlp",
	"id": "i4w8fdpp7i4ihlu10zd711tk",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1",
	"type": "answer",
	"answer": "Hello, do you have any questions?",
	"index": 1,
	"status": "ok"
}

Current Recording Ended

{
	"service": "record",
	"type": "eof",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1",
	"status": "ok"


### 4.2 Live Streaming Service Invocation Code

```javascript
// Initialize sense-aigc
const aigcClient = new SenseAigc({
  baseUrl: <config.baseUrl>
});

// Create live stream
const liveClient = await aigcClient.create(<create-params>);

// Start live stream
await liveClient.start();

// Start asking questions
await liveClient.startAsk();

// Stop asking questions
await liveClient.stopAsk();

// End live stream
await liveClient.close();

// Exit sense-aigc
aigcClient.leave();

# 4.2 Single Role JSON Description

{
    "name": "Xiao M",    // Character name, required
    "info": "Art student, just graduated", // Character description, required
    "welcome": "Hello, I am Xiao M", // Welcome message, required
    "thumbnail": "video.webm", // Thumbnail/video, required, recommended below 5MB, video format recommended as mp4
    "json": {},    // Live script JSON, required, detailed content of this parameter refer to: Interactive Live > Live Script JSON Definition Description
    "asr": {},    // ASR configuration, optional, default is ASR5, value reference: Streaming Voice Processing Documentation > LLM (Large Language Model) Capability Connection Description > Request Message Format
    "nlp": {}, // NLP configuration, required, value reference: Streaming Voice Processing Documentation > LLM (Large Language Model) Capability Connection Description > Request Message Format
    "extra": {    // Chitchat and interruption logic configuration, optional, all internal parameters are optional, below are default values
      "quick_response_enable": true, // Enable chitchat
      "quick_response_random": true, // Randomly play chitchat
      "quick_response_delay": 1000, // Chitchat trigger time in milliseconds
      "command_immediate": true // Immediate takeover
    }
}

# 4.3 Complete JSON Example

{
    "name": "Dandan",
    "info": "Trustworthy sister",
    "welcome": "Tell me what you've been up to the past few days?",
    "thumbnail": "https://dwg-aigc-paas-test.oss-cn-hangzhou.aliyuncs.com/sense-live-h5/test/%E7%B4%A0%E6%9D%90/2.png",
    "json": {
      "version": "0.0.2",
      "recycle": 0,
      "live_type": "interactive",
      "resolution": [1080, 1920],
      "sentence_break": false,
      "sceneList": [{
        "quick_response": [],
        "digital_role": {
          "id": 3964,
          "face_feature_id": "a2296686b0694a92bce529edc1d329fc_s1",
          "name": "Xiao M",
          "url": "https://aigc-video-saas.oss-cn-hangzhou.aliyuncs.com/AIGC/online/user/643219927/paas-digital-human-models/1739335194223091712/cut_3e3869e01822491898af6e4b763468c7_s1_result.zip",
          "position": {
            "x": 0,
            "y": 0
          },
          "scale": 1
        },
        "tts_config": {
         "qid": "cQn1k_:ACAwsB9GUwDGHMSi0tS_TMr8xMyI3T9nXQyqsK",
          "id": "zhukaixuan",
          "name": "zhukaixuan",
          "vendor_id": 3,
          "language": "zh-CN",
          "pitch_offset": 0,
          "speed_ratio": 1,
          "volume": 400
        },
        "opening_words": "Tell me what you've been up to the past few days?"
      }]
    },
    "asr": {
      "type": "ASR8",
      "config": {}
    },
    "nlp": {
      "type": "NLP10",
      "max_reply_token": 1200,
      "prompt_header": "[{\"name\":\"Zhou Zirou\",\"gender\":\"Female\",\"identity\":\"My always trusted sister\",\"nickname\":\"\",\"feeling_toward\":[{\"name\":\"Brother\",\"level\":3}],\"detail_setting\":\"Zhou Zirou possesses an exceptional sense of achievement, excelling both academically and professionally. From a young age, she has been an model frequently used by elders as an example. Her cool exterior gives an impression of being above the world, and her remarkable aura is both independent and confident. Although she may keep her distance from others, she always exudes infinite gentleness and tolerance in front of me, listening to my troubles with patience and resolving my confusion with delicate care. Her MBTI personality type is ENTJ.\",\"other_setting\":\"\"},{\"name\":\"Brother\",\"gender\":\"Male\",\"identity\":\"Brother\",\"nickname\":\"\",\"detail_setting\":\"Zhou Zirou always shows endless gentleness and tolerance in front of her brother, listening to his troubles with patience and resolving his confusion with delicate care.\",\"other_setting\":\"\"}]"
    },
    "extra": {
      "quick_response_enable": false,
      "quick_response_random": true,
      "quick_response_delay": 2000,
      "command_immediate": true
    }
  }

← Interactive Live Broadcasting FAQ →