# Interactive Digital Human

Interactive Digital Human is an online interactive digital human product based on large language models, providing one-click digital human Q&A services with simple integration and ease of use.

# 1. Release URLs

用户中心代理：代码下载 (opens new window)

SDK地址：SDK下载 (opens new window)

demo地址：Demo下载 (opens new window)

Note: The User Center Proxy needs to be deployed separately. It is responsible for interacting with the RuYing PaaS (the token refresh API does not support cross-origin requests), refreshing tokens and distributing them. The H5 page Interactive Digital Human SDK obtains the token from the User Center Proxy. If cross-origin access is needed for the Interactive Digital Human API, a whitelist application is required through business/after-sales contacts.

# 2. User Center Proxy

# 2.1 Overview

This document explains the working principles of the Interactive Digital Human User Center Proxy, particularly how to manage and refresh accessToken information to ensure that authentication information is both secure and valid when users access digital human services.

# 2.2 Access Token Management

# 2.2.1 accessToken Management

有效期限：accessToken的有效期为8小时。这意味着用户在获取令牌后可以使用相同的令牌进行身份验证，直到令牌过期或被刷新新令牌时替换。
同时有效的令牌：系统允许最多有2个accessToken同时有效。这为令牌刷新提供了灵活性，确保在新令牌生成之前，旧令牌依然可以使用。

# 2.2.2 refreshToken Management

安全性：refreshToken不应对web端用户可见，以确保认证信息的安全性。
刷新间隔：刷新间隔为3小时，建议每3.5小时刷新一次accessToken，以保持令牌的有效性。

# 2.3 User Center Proxy Server-Side Operations

# 2.3.1 Token Refresh Strategy

刷新频率：代理服务端应每间隔3.5小时刷新一次accessToken。这一策略旨在确保即使在最极端的情况下，当一个令牌即将过期时，另一个令牌已经被刷新并准备就绪。
安全保障：通过代理服务端提前刷新令牌，可以保证web端获取的accessToken在接下来的1小时内不会过期，从而提高系统的稳定性和用户体验。

# 2.4 Web-Side Operations

# 2.4.1 Token Retrieval Strategy

获取频率：Web端可每隔1个小时从代理服务端获取一次最新的accessToken。这一策略确保了web端用户总是使用有效的令牌进行操作，同时避免了频繁的令牌请求，减少了系统负载。

# 2.5 Proxy Server-Side Flow

# 3. Interactive Digital Human API Reference

Overall, there are two classes: SenseAigc and SenseLive. SenseAigc is responsible for token updates and maintenance, while SenseLive handles live streaming. SenseLive is not directly exposed; it needs to be returned asynchronously through SenseAigc.create.

# 3.1 Output Interfaces

# 3.1.1 Initialization

Initialize to get a SenseAigc instance

Field Description:

Field	Type	Required	Description
`baseUrl`	String	True	Server URL, HTTPS protocol, without trailing /
`logLevel`	Number	False	Log level (optional; recommended to set to 3 during debugging, not required in non-debug mode). Default is 1, showing only error logs. Maximum value is 3

Request Example:

const aigcClient = new SenseAigc({
  baseUrl: "https://example.com"
});

# 3.1.2 Set Token

Set token based on the current SenseAigc instance

You need to set the token before using live streaming services. After token refresh, this method must also be used to update the token to avoid service exceptions caused by token expiration.

aigcClient.setToken(token);

# 3.1.3 Quick Boot

Request parameters are the same as Create Live Stream. Returns a SenseLive instance after successful broadcast start. Note that container is required here.

const liveClient = await aigcClient.quickBoot(config);

# 3.1.4 Create Live Stream

Creates a live stream and returns a SenseLive instance upon success

Field Description:

Field	Type	Required	Description
`data`	String	True	Live video generation parameters (this parameter is a JSON-escaped string). For detailed parameter content, refer to: `Interactive Live > Live Script JSON Definition`
`nlp`	Object	True	NLP configuration used during live streaming, 包含 `type` 和 `NLP Config` 的配置, 具体格式参考下方示例, `type` 和 `NLP Config` 的取值参考 `流式语音处理文档 > LLM（大语言模型）能力对接Description > 请求报文格式`
`asr`	Object	False	ASR configuration used during live streaming, 包含 `type` 和 `ASR Config` 的配置, 具体格式参考下方示例, `type` 和 `ASR Config` 的取值参考 `流式语音处理文档 > ASR（语音识别）能力对接Description > 请求报文格式`
`container`	Object	False	Container Element for live stream rendering (optional; required for Quick Boot)
`onmessage`	Function	False	Receives live stream-related message push notifications (optional)，详参考后面 `3.2 推送消息信息`
`onstream`	Function	False	开始收到直播视频流的回调
`extra`	Object	False	Filler speech and interruption logic parameters
- `quick_response_enable`	Boolean	False	Whether to enable filler speech, default is true
- `quick_response_random`	Boolean	False	Whether to randomly broadcast filler speech, default is true
- `quick_response_delay`	Number	False	Filler speech trigger time in milliseconds, default is 1000ms
- `command_immediate`	Boolean	False	Whether to interrupt immediately, default is true
`custom`	Object	False	User-defined fields for creating live stream (optional), for custom broadcast use, details below
- `urtcToken`	String	False	User RTC authentication token
- `urtcUid`	String	False	User RTC authentication uid, must be a pure number not exceeding 32-bit uint
- `urtcAppId`	String	False	User RTC APP ID (required when using user RTC app)
- `urtcChannelId`	String	False	User RTC Channel ID, length must not exceed 64 characters
- `userData`	String	False	User information (returned as-is during callback)

Request Code:

  const config = {
    data: "{\"version\":\"0.0.2\",\"recycle\":0,\"live_type\":\"interactive\",\"resolution\":[1080,1920],\"sentence_break\":false,\"sceneList\":[{\"quick_response\":[],\"digital_role\":{\"id\":3964,\"face_feature_id\":\"a2296686b0694a92bce529edc1d329fc_s1\",\"name\":\"小M\",\"url\":\"https://aigc-video-saas.oss-cn-hangzhou.aliyuncs.com/AIGC/online/user/643219927/paas-digital-human-models/1739335194223091712/cut_3e3869e01822491898af6e4b763468c7_s1_result.zip\",\"position\":{\"x\":0,\"y\":0},\"scale\":1},\"tts_config\":{\"qid\":\"cQn1k_:ACAwsB9GUwDGHMSi0tS_TMr8xMyI3T9nXQyqsK\",\"id\":\"zhukaixuan\",\"name\":\"zhukaixuan\",\"vendor_id\":3,\"language\":\"zh-CN\",\"pitch_offset\":0,\"speed_ratio\":1,\"volume\":400},\"opening_words\":\"跟我说说你前两天都干嘛去啦？\"}]},\"asr\":{\"type\":\"ASR8\",\"config\":{}},\"nlp\":{\"type\":\"NLP10\",\"max_reply_token\":1200,\"prompt_header\":\"[{\\\"name\\\":\\\"周梓柔\\\",\\\"gender\\\":\\\"女\\\",\\\"identity\\\":\\\"我一直信赖的姐姐\\\",\\\"nickname\\\":\\\"\\\",\\\"feeling_toward\\\":[{\\\"name\\\":\\\"弟弟\\\",\\\"level\\\":3}],\\\"detail_setting\\\":\\\"周梓柔具有卓越成就感，学业和职场表现都极为杰出，从小就是个学霸，经常被长辈提及为榜样。外表冷艳，给人以远离尘嚣的印象，令人印象深刻的气质既独立又自信。对外或许保持距离，但在我面前总是展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。MBTI人格是ENTJ。\\\",\\\"other_setting\\\":\\\"\\\"},{\\\"name\\\":\\\"弟弟\\\",\\\"gender\\\":\\\"男\\\",\\\"identity\\\":\\\"弟弟\\\",\\\"nickname\\\":\\\"\\\",\\\"detail_setting\\\":\\\"周梓柔总是在弟弟面前展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。\\\",\\\"other_setting\\\":\\\"\\\"}]\"},\"extra\":{\"quick_response_enable\":false,\"quick_response_random\":true,\"quick_response_delay\":2000,\"command_immediate\":true}}",
    container: document.querySelector("body"),
    asr: {
      "type": "ASR8",
      "config": {
        "language": "zh-CN"
      }
    },
    nlp: {
      "type": "NLP10",
      "max_reply_token": 1200,
      "prompt_header": "[{\"name\":\"周梓柔\",\"gender\":\"女\",\"identity\":\"我一直信赖的姐姐\",\"nickname\":\"\",\"feeling_toward\":[{\"name\":\"弟弟\",\"level\":3}],\"detail_setting\":\"周梓柔具有卓越成就感，学业和职场表现都极为杰出，从小就是个学霸，经常被长辈提及为榜样。外表冷艳，给人以远离尘嚣的印象，令人印象深刻的气质既独立又自信。对外或许保持距离，但在我面前总是展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。MBTI人格是ENTJ。\",\"other_setting\":\"\"},{\"name\":\"弟弟\",\"gender\":\"男\",\"identity\":\"弟弟\",\"nickname\":\"\",\"detail_setting\":\"周梓柔总是在弟弟面前展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。\",\"other_setting\":\"\"}]"
    },
    onmessage: function(msg) {
      console.log(msg);
    },
    extra: {
      "quick_response_enable": false,
      "quick_response_random": true,
      "quick_response_delay": 2000,
      "command_immediate": true
    };

  const liveClient = await aigcClient.create(config);

# 3.1.5 Start Live Stream

Starts the live stream based on the current SenseLive instance, returns a Promise

/**
* @params { object } config.container - 直播渲染的容器 Element（可选参数），如果用户在创建时未填，则此处为必填。适用于创建和开播在不同页面的场景
* 
* @returns { Promise } 
*/

// liveClient 为 SenseLive 实例
await liveClient.start(container);

# 3.1.6 Microphone Q&A

Implements microphone Q&A based on the current SenseLive instance, returns a Promise

/**
* 开始提问
*
* @returns { Promise }
*/
await liveClient.startAsk();

/**
* 结束提问
*
* @returns { Promise }
*/
await liveClient.stopAsk();

# 3.1.7 Close Live Stream

Closes the live stream based on the current SenseLive instance, returns a Promise

/**
* 关闭直播
*
* @returns { Promise }  - 关闭成功
*/
await liveClient.close();

# 3.1.8 Logout

Logs out based on the current SenseAigc instance

/**
* @returns { Promise }  - 登出，只是去掉内部缓存token，sdk不会调用登出接口
*/
await aigcClient.leave();

# 3.2 Push Message Information

Users can set the onmessage method when creating a live stream to listen for the following messages:

Field descriptions:

Field	Type	Required	Description
service	String	True	可选值: asr / nlp / record / command / opening
trace	String	True	Current conversation trace
type	String	True	可选值: text / eof / answer
status	String	True	Current message status. ok means success, fail means failure. Values: ok / fail
text	String	False	Opening speech文本
query	String	False	ASR text
answer	String	False	NLP text
error	String	False	status 为 fail 时的Additional notes

Common message categories are described below:

service	type	status	Additional notes
opening	answer	ok	Opening speech
asr	text	ok	ASR recognition in progress
asr	eof	ok	ASR recognition ended
asr	eof	fail	ASR did not recognize properly; see error field for specific reason
nlp	answer	ok	NLP answer result
command	eof	ok	Current round conversation takeover ended
record	eof	ok	Current round recording ended

Examples:

Opening speech

{
    "service": "opening",
    "text": "你好，有什么疑问都可以向我咨询",
    "trace": "opening_1726043171122",
    "type": "answer",
    "status": "ok"
}

asr识别到内容

{
	"service": "asr",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1",
	"type": "text",
	"query": "你好。",
	"index": 1,
	"status": "ok"
}

ASR recognition ended

{
	"service": "asr",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_6",
	"type": "eof",
	"query": "",
	"index": 1,
	"status": "ok"
}

asr未识别到内容

{
	"service": "asr",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_2",
	"type": "eof",
	"query": "",
	"index": 1,
	"status": "fail",
	"error": "query is empty"
}

nlp返回答案

{
	"service": "nlp",
	"id": "i4w8fdpp7i4ihlu10zd711tk",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1",
	"type": "answer",
	"answer": "你好，有什么问题需要解答吗？",
	"index": 1,
	"status": "ok"
}

Current round recording ended

{
	"service": "record",
	"type": "eof",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1",
	"status": "ok"
}

本轮接管结束

{
	"service": "command",
	"type": "eof",
	"status": "ok",
	"trace": "sense-live-nlp_d001a3ed-f53f-4d3a-a984-eac9f2939b1d_1"
}

# 4. Interactive Digital Human H5 Demo

Note: Since interfaces such as microphone are used, the protocol must be a valid HTTPS protocol.

After login, periodically refresh the token based on actual usage needs to prevent service expiration.

# 4.1 Project Description

The project is built with Vite + Vue 3.

Project directory description:

src
 ┣ assets                  // 静态资源
 ┃ ┗ images                // 图片资源
 ┣ components              // 组件
 ┃ ┣ Chat.vue              // 聊天组件
 ┃ ┣ ChatList.vue          // 聊天列表组件
 ┃ ┣ MediaBg.vue           // 媒体组件：图片/视频
 ┣ requests                // http请求模块
 ┃ ┗ login.ts              // 登录请求 + 刷新token请求
 ┣ utils                   // 拓展工具
 ┃ ┣ common.ts             // 公用方法
 ┃ ┣ http.ts               // axios封装
 ┃ ┗ md5.js                // md5算法
 ┣ views                   // 视图 / 页面
 ┃ ┣ Home.vue              // 首页
 ┃ ┣ Live.vue              // 直播模块，内嵌首页
 ┃ ┣ Setting.vue           // 设置页
 ┃ ┗ Video.vue             // 教学视频页
 ┣ App.vue                 // 全局框架
 ┣ main.ts                 // 项目入口
 ┣ router.ts               // 全局路由
 ┣ store.ts                // 全局状态
 ┣ style.css               // 全局样式
 ┗ vite-env.d.ts           // 全局类型定义
config                     // 数据配置
 ┣ account.json            // 账号及全局数据配置
 ┗ roles.json              // 角色列表配置

# 4.2 Live Stream Service Call Code

// 初始化sense-aigc
const aigcClient = new SenseAigc({
  baseUrl: <config.baseUrl>
});

// 创建直播
const liveClient = await aigcClient.create(<create-params>);

// 开始直播
await liveClient.start();

// 开始提问
await liveClient.startAsk();

// 结束提问
await liveClient.stopAsk();
                       
// 结束直播
await liveClient.close();

// 退出 sense-aigc
aigcClient.leave()

# 4.2 Single Role JSON Description

{
    "name": "小M",	// 人物名称，必填
    "info": "艺术生，刚毕业", // 人物介绍，必填
    "welcome": "你好，我是小M", // 加载欢迎语，必填
    "thumbnail": "video.webm", // 预览图/视频，必填，建议5M以下，视频建议mp4格式
    "json": {},	// 直播脚本json，必填，该参数详细内容参考文档：互动直播 > 直播脚本json定义说明
    "asr": {},	// asr配置，可选，默认是ASR5，取值参考:流式语音处理文档 > LLM（大语言模型）能力对接说明 > 请求报文格式
    "nlp": {}, // nlp配置，必填， 取值参考:流式语音处理文档 > LLM（大语言模型）能力对接说明 > 请求报文格式
    "extra": {	// 口水话和打断逻辑配置，可选，内部所有参数都可选，下面是默认值
      "quick_response_enable": true, // 是否开启口水话
      "quick_response_random": true, // 随机播报口水话
      "quick_response_delay": 1000, // 播报口水话触发时间，单位毫秒
      "command_immediate": true // 是否立即接管
    }
}

# 4.3 Complete JSON Example

 {
    "name": "丹丹",
    "info": "信赖的姐姐",
    "welcome": "跟我说说你前两天都干嘛去啦？",
    "thumbnail": "https://dwg-aigc-paas-test.oss-cn-hangzhou.aliyuncs.com/sense-live-h5/test/%E7%B4%A0%E6%9D%90/2.png",
    "json": {
      "version": "0.0.2",
      "recycle": 0,
      "live_type": "interactive",
      "resolution": [1080,1920],
      "sentence_break": false,
      "sceneList": [{
        "quick_response": [],
        "digital_role": {
          "id": 3964,
          "face_feature_id": "a2296686b0694a92bce529edc1d329fc_s1",
          "name": "小M",
          "url": "https://aigc-video-saas.oss-cn-hangzhou.aliyuncs.com/AIGC/online/user/643219927/paas-digital-human-models/1739335194223091712/cut_3e3869e01822491898af6e4b763468c7_s1_result.zip",
          "position": {
            "x": 0,
            "y": 0
          },
          "scale": 1
        },
        "tts_config": {
			"qid": "cQn1k_:ACAwsB9GUwDGHMSi0tS_TMr8xMyI3T9nXQyqsK",
          "id": "zhukaixuan",
          "name": "zhukaixuan",
          "vendor_id": 3,
          "language": "zh-CN",
          "pitch_offset": 0,
          "speed_ratio": 1,
          "volume": 400
        },
        "opening_words": "跟我说说你前两天都干嘛去啦？"
      }]
    },
    "asr": {
      "type": "ASR8",
      "config": {
      }
    },
    "nlp": {
      "type": "NLP10",
      "max_reply_token": 1200,
      "prompt_header": "[{\"name\":\"周梓柔\",\"gender\":\"女\",\"identity\":\"我一直信赖的姐姐\",\"nickname\":\"\",\"feeling_toward\":[{\"name\":\"弟弟\",\"level\":3}],\"detail_setting\":\"周梓柔具有卓越成就感，学业和职场表现都极为杰出，从小就是个学霸，经常被长辈提及为榜样。外表冷艳，给人以远离尘嚣的印象，令人印象深刻的气质既独立又自信。对外或许保持距离，但在我面前总是展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。MBTI人格是ENTJ。\",\"other_setting\":\"\"},{\"name\":\"弟弟\",\"gender\":\"男\",\"identity\":\"弟弟\",\"nickname\":\"\",\"detail_setting\":\"周梓柔总是在弟弟面前展现出无限的温柔与包容，耐心倾听我的烦恼，用细腻的关怀化解我的困惑。\",\"other_setting\":\"\"}]"
    },
    "extra": {
      "quick_response_enable": false,
      "quick_response_random": true,
      "quick_response_delay": 2000,
      "command_immediate": true
    }
  },

← Video Live Streaming (Deprecated) Voice Processing →