14 Dec, 2023 @ m-onz web page

Command line audio visual art

This is a little recipe for linux command line driven audio visual art.

Firstly I assume your on a debian based linux distro and you don't have ffmpeg installed.

sudo apt update
sudo apt install ffmpeg
Step 1 - generate video

Lets create some .mp4 video's from a text prompt using /deforum/deforum_stable_diffusion replicate.com Set the max frame length to less than 500 to reduce the likelihood it might crash on long sequence length.

I use a prompt like: "digital glitchy geometric particle system" but you can experiment with the prompt to get the result your looking for.

Step 2 - concatenate videos

Lets say you put a number of these videos into a folder, instead of using a fully fledged video editing application we can concatenate these video's together using ffmpeg.

Firstly we need a text file the name of each video... from the terminal change directory into the folder and run:

Step 3 - create a filelist.txt
ls *.mp4 | awk '{print "file \x27"$0"\x27"}' > filelist.txt

Having created a filelist.txt with the video names we can use ffmpeg to concatenate the videos together.

Step 4 - concatenate videos
ffmpeg -f concat -safe 0 -i filelist.txt -c copy output.mp4

Lets say we have a very long video and we want to speed it up we can use ffmpeg for this...

Step 5 - speed up or slow down videos
ffmpeg -i input.mp4 -filter:v "setpts=PTS/speed_factor" -filter:a "atempo=speed_factor" -preset fast output.mp4

Remember to change speed_factor: The factor by which you want to speed up the video. For example, 2.0 will double the speed, 1.5 for 1.5 times the speed, etc.

Step 6 - add audio to the video

Finally we can add some audio to the video to finish it off.

I used a musicGen model to generate the music.

Within musicGen the model was set to "large" i used a prompt:

"develop a cyberpunk breakbeat track that fuses the edgy dystopian sounds of cyberpunk with the raw energy of breakbeat aim for a bpm range of 130 150 with aggressive breakbeats-and-a-heav emphasis on bass integrate electro"

Add audio to the video we can also use ffmpeg.
ffmpeg -i input.mp4 -i audio.wav -c:v copy -c:a aac -strict experimental output.mp4
In conclusion

So there you have it, command line driven audio visual art with no video editing software or DAW in sight.

You can play around with even more re-processing, I also used vdcrpt to add datamoshing but this isn't a command line tool.

The next step for me is to add automation and combine models using the replicate API. This is something I am going to cover next time as I attempt to create a hallucinated ai video using the last video frame of the output to generate the next video, and so on.

The result
Other cool models to play around with.
image-to-video https://replicate.com/stability-ai/stable-video-diffusion