Rich Media with Generative AI

Large-scale generative visual models, such as DALL·E2 and Stable Diffusion, have made content creation as little effort as writing a short text description. However, these models are typically trained on an enormous amount of Internet data, often containing copyrighted material, licensed images, and personal photos. How can we remove these images if creators decide to opt-out? How can we properly compensate them if they choose to opt in?

In this talk, I will first describe an efficient method for removing copyrighted materials, artistic styles of living artists, and memorized images from pretrained text-to-image diffusion models. I will then discuss our data attribution algorithm for assessing the influence of each training image for a generated sample. Collectively, we aim to enable creators to retain control over the ownership of training images.

3D digital content has been in high demand for a variety of applications, including gaming, entertainment, architecture, and robotics simulation. However, creating such 3D content requires professional 3D modeling expertise with a significant amount of time and effort. In this talk, I will talk about recent advances on automating high-quality 3D digital content creation from text prompts. I will also cover Magic3D, which can generate high-resolution 3D mesh models from input text descriptions, as well as our recent efforts on 3D generative AI with NVIDIA Picasso. With these text-to-3D approaches, we aim to democratize and turbocharge 3D content creation for all, from novices to expert 3D artists.

In this talk, I will first present DragGAN, an interactive point-dragging image manipulation method. Unlike previous works that gain controllability of GANs via manually annotated training data or a prior 3D model, DragGAN allows users to "drag" any points of the image to precisely reach target points, thus manipulating the pose, shape, expression, and layout of diverse categories such as animals, cars, humans, landscapes, etc. While DragGAN produces continuous animations, such effects are hard to obtain on diffusion models. So I will then introduce how we address this limitation with DiffMorpher, a method enabling smooth and natural image morphing based on diffusion models.

Mobile and embedded computing devices have become key carriers of deep learning to facilitate the widespread use of machine intelligence. However, there is a widely recognized challenge to achieve real-time DNN inference on edge devices, due to the limited computation/storage resources on such devices. Model compression of DNNs, including weight pruning and weight quantization, has been investigated to overcome this challenge. However, current work on DNN compression suffers from the limitation that accuracy and hardware performance are somewhat conflicting goals difficult to satisfy simultaneously.

We present our recent work Compression-Compilation Codesign, to overcome this limitation towards the best possible DNN acceleration on edge devices. The neural network model is optimized in a hand-in-hand manner with compiler-level code generation, achieving the best possible hardware performance while maintaining zero accuracy loss, which is beyond the capability of prior work. We are able to achieve real-time on-device execution of a number of DNN tasks, including object detection, pose estimation, activity detection, speech recognition, just using an off-the-shelf mobile device, with up to 180 X speedup compared with prior work. Recently, for the first time, we enable large-scale language and AIGC models such as GPT and Stable Diffusion on mobile devices. We will also introduce our breakthrough in digital avatar, stable diffusion for video generation, and interaction systems. Last we will introduce our recent breakthrough of superconducting logic based neural network acceleration that achieves 10^6 times energy efficiency gain compared with state-of-the-art solutions, achieving the quantum limit in computing.