Experiments with AI Video for Song Lyrics, Poetry, and Book Passages

This video is a rough demonstration of how a video might be generated from Music Lyric prompts using various techniques of AI coercion. It shows some of the promise, but also some of the challenges in attempting such a feat.

Obvious benefits

  • Quickly generate thousands of original works of art from mere text prompts
Extremely rough AI interpretation at 4 frames-per-second of Macklemore’s “Downtown”. I chose this song because it had lots of words that I thought could be visually represented in pictures. Some songs are about abstract things like… “love” and “relationships”… this song has lots of visual words about… mopeds… mostly.

Some challenges..

  • undesirable images… human moderation may always be required
  • dirty lyrics will generate pictures that are not always family friendly
  • sometimes lyrics are interpreted too literally
  • camera motion is simulated
  • rate of frame change if often unnatural, but this is something I’m working on balancing through algorithms

Now here’s a AI interpretation of a book passage that is considerably cleaner. This video has a higher framerate and constant motion and doesn’t suffer from as many visual glitches as the previous video. A goal of this project is to achieve smooth dreamy motion influenced by mere words and music.

1 thought on “Experiments with AI Video for Song Lyrics, Poetry, and Book Passages”

  1. You should have a QR code linking to your site for drunk asses like me that forget the URL on my wan back from the DJ stand to the bar.

Leave a Comment