Alibaba Cloud Integrates Tongyi Qianwen into AI Assistant to Enhance Productivity

Alibaba Cloud, the digital technology and intelligence backbone of Alibaba Group, announced today that its large language model (LLM), Tongyi Qianwen, has been integrated into its intelligent assistant named Tingwu. The assistant excels at converting speech and videos to text in real-time, aiming to enhance both personal and workplace productivity.

The recently launched LLM enables Tingwu to comprehend and analyze multimedia content with high levels of accuracy and efficiency, such as generating summary text from video and audio files, capturing key talking points for each speaker, and creating a timeline of multimedia files with a summary of each section.

The LLM-powered Tingwu, called “Tongyi Tingwu”, is now available for public beta testing. Tongyi Tingwu will also be integrated into DingTalk, Alibaba’s digital collaboration workplace and application development platform, supporting users’ AI demands at work. In addition to improving workplace efficiency, Tongyi Tingwu can also be used across various multimedia platforms, responding to the growing needs for faster and easier knowledge sharing in online education, training, interviews, livestreaming, podcasts, and short-form videos.

“We live at a time when a growing amount of video and audio content is being consumed in various formats every day. In line with this, Tongyi Tingwu aims to use the large language model to facilitate faster and better comprehension and easier sharing of multimedia content,” said Jingren Zhou, CTO of Alibaba Cloud Intelligence. “As we gradually integrate the Tongyi Qianwen model into our products and services, we hope users can reap the benefits from these compelling AI innovations for their work, study, play and interaction with each other.”

By leveraging proprietary audio and video models developed by Alibaba’s research institute DAMO – including the self-developed speech recognition model Paraformer and speaker verification model CAM++, along with the newly-released LLM Tongyi Qianwen – Tingwu can transcribe video and audio files with higher accuracy while enabling numerous AI-powered features. Additional AI features offered by Tongyi Tingwu will be available later this year. These features include automatically compiling text answers to address user queries across audio/video files, generating a summary based on PowerPoint slides extracted from videos, and providing real-time translation between English and Chinese for multimedia content with Tingwu as a Chrome plugin.

General public can access the upgraded AI-powered assistant online (tingwu.aliyun.com) starting from today to experience its capabilities through their Alibaba Cloud accounts and receive free transcription services as part of an open trial.

Alibaba Cloud unveiled Tongyi Qianwen on April 11, which is scheduled to be integrated across Alibaba’s various businesses to improve the user experience in the near future. The company’s customers and developers will have access to the model to create customized AI features in a cost-effective manner, too.

The cloud pioneer also launched the “Tongyi Qianwen Partnership Program,” aiming to co-create large language models tailored for different industries with partners across sectors, including petrochemicals, electricity, transportation, hospitality, enterprise services, telecommunications and finance.