Experiments with AI Video for Song Lyrics, Poetry, and Book Passages

This video is a rough demonstration of how a video might be generated from Music Lyric prompts using various techniques of AI coercion. It shows some of the promise, but also some of the challenges in attempting such a feat.

Obvious benefits

  • Quickly generate thousands of original works of art from mere text prompts
Extremely rough AI interpretation at 4 frames-per-second of Macklemore’s “Downtown”. I chose this song because it had lots of words that I thought could be visually represented in pictures. Some songs are about abstract things like… “love” and “relationships”… this song has lots of visual words about… mopeds… mostly.

Some challenges..

  • undesirable images… human moderation may always be required
  • dirty lyrics will generate pictures that are not always family friendly
  • sometimes lyrics are interpreted too literally
  • camera motion is simulated
  • rate of frame change if often unnatural, but this is something I’m working on balancing through algorithms

Now here’s a AI interpretation of a book passage that is considerably cleaner. This video has a higher framerate and constant motion and doesn’t suffer from as many visual glitches as the previous video. A goal of this project is to achieve smooth dreamy motion influenced by mere words and music.

23 thoughts on “Experiments with AI Video for Song Lyrics, Poetry, and Book Passages”

  1. You should have a QR code linking to your site for drunk asses like me that forget the URL on my wan back from the DJ stand to the bar.

Leave a Comment