Losing time to dependencies
I have a goal of turning a video into a kind of anime style. I've seen some clips of this online and think it looks really, really cool. I also think it'll be a great way for me to learn more about diffusion, control net, and the various techniques involved in achieving a specific style.
Yesterday I got to work on it, but as it turns out, I ended up spending the whole day tracking down missing dependencies to build / install software instead of actually using any of the software.
In the end, futzing about with dependencies took the better part of the whole day. I did end up playing around with the software by the evening, but not without sacrificing unrecoverable hours at the alter of the Dependency Gods. I blogged about the experience as it happened, because I wanted to be able to retrace my steps later on. They aren't the steps that I was hoping to take, but they're here all the same, so I figure I'll publish this post anyway.
Maybe there's something to learn from days like this. But I haven't figured out what exactly that is.
The article I wanted to write
I've seen some seriously impressive videos (like this one and this one) that have been created with the help of generative AI.
My goal today is to transform a short video clip of myself into some kind of anime style.
I'll be following this tutorial to get started.
Installing the Temporal Kit Extension
While this stuff work should work on my local machine, I recently got access to a computer with eight (!) 40GB A100's, which is a total of 320GB of video ram. That's a lot of power. I'll use that for this project.
The first step is getting AUTOMATIC1111/stable-diffusion-webui running on the machine and making sure I can access it locally through my web browser. I like to run these kinds of applications in docker containers so that later I don't have to manage dependency conflicts, so I just cloned my osai-apps repo and used the setup scripts I had in there.
I added my remote machine's IP to my host file so that I could refer to it by name in my web browser, made sure its ports were exposed to the public internet, ran the docker container, and voilà : it's running!
After that I moved on to step 1 from the guide, which is installing the TemporalKit extension. And immediately I ran into some dependency issues, which has been reported several times in the source repo, and which required different steps to debug on different operating systems. Yikes.
After sacrificing an hour at the alter of the Dependency Gods, I finally had things working.
I'll still need to automate the fix/workaround if I want a repeatable docker set up (in osai-apps). But I'll just put that on the TODO list for now and move on.
Oh and in case you were wondering what critical code tqdm holds that justifies all the headache -- it's a f*cking status bar. I'm so annoyed.
Anyway. Moving on.
Installing ffmpeg
The next step is to install ffmpeg. I suspect I can just get away with running it inside a separate docker container, so that's what I'll do for now. It looks like the docker-ffmpeg project exists for this reason, but it seems convenient to use the same base nvidia / cuda containers I've been using so far and modify them to add ffmpeg, rather than using their Dockerfile. Besides, NVIDIA provides documentation on building ffmpeg with hardware acceleration for Linux and it looks pretty straight-forward. I'll write my own config for osai-apps and see how it goes.
As it turns out, I lost another couple hours to the dependency gods. I couldn't find any mention of the dependency on something called pkg-config. Maybe it's such a basic or well-known package that everyone assumes you'll either have it installed or know to install it. I did not know, and it took a couple hours of thinking I had the wrong version of nv-codec-headers installed, or that maybe my PKG_CONFIG_PATH was misconfigured, based on this important-looking box in the official ffmpeg docs:
After all, it seemed to match the error I was getting:
As it turns out, pkg-config isn't required in the base installation. I only see this error once I tried compiling with --enable-ffnvcodec. So pkg-config would have failed to find any packages (on account of not being installed on the system).
The error message comes from this part of the source code.
First,
# Mark specifically enabled, but normally autodetected libraries as requested.
for lib in $AUTODETECT_LIBS; do
enabled $lib && request $lib
done
Second,
# Check if requested libraries were found.
for lib in $AUTODETECT_LIBS; do
requested $lib && ! enabled $lib && die "ERROR: $lib requested but not found";
done
So in the line where the error message is being created:
requested $libis true because I explicitly requested that ffmpeg enable ffnvcodec (via the--enable-ffnvcodecflag).! enabled $libis true because the library did not get enabled.
It seems like the configure script could easily check whether the default or provided pkg-config utility was available at all and print an error message specifically related to that.
--pkg-config=PKGCONFIG use pkg-config tool PKGCONFIG [$pkg_config_default]
However, the contributing guide for ffmpeg is way too complicated for me to get involved, especially because I have no idea whether anyone in the world has ever had this problem that I just ran into or if it's just me.
Anyway, by now it's been many hours since I started my humble quest to turn video into anime, and all I've done so far is try to install two programs, both of which were supposed to be auto-installing, and both of which had some obscure dependency issue. The joy of programming. (Am I just approaching everything incorrectly??)
In any case, apt-get install -y pkg-config got me over this hurdle, but actually trying to run a job with my newly built ffmpeg fails:
docker run --rm -it \
--gpus=all \
-v $(pwd):/config \
ffmpeg-nvidia \
ffmpeg \
-hwaccel nvdec \
-i /config/input.mkv \
-c:v h264_nvenc \
-b:v 4M \
-vf scale=1280:720 \
-c:a copy \
/config/output.mkv
In my naivite, I thought that I had configured my NVIDIA driver and dependencies correctly. Ha ha. ha. I did not.
In looking around for more information in the official docs, I found another project that supposedly builds ffmpeg in docker with NVIDIA hardware acceleration support. Let's try it!
docker run --rm -it \
--runtime=nvidia \
-v $(pwd):/config \
jrottenberg/ffmpeg:4.1-nvidia \
-hwaccel cuvid \
-c:v h264_cuvid \
-i /config/input.mkv \
-vf \
scale_npp=-1:720 \
-c:v h264_nvenc \
-preset slow \
/config/output.mkv
Hmm, on the big powerful computer that I'm running on, this still fails:
However, the same command on the AI machine at home works just fine:
It seems I need to track down where these libraries are coming from and install them to the big beefy box in the cloud.
After some sleuthing about, I figured out that the base system was maybe missing some of the key libraries / dependencies to begin with. Lambda lab's setup script perhaps installs a specific subset of the nvidia server packages rather than the big meta package that includes lots of little pieces. When I installed the meta package, I at least get to a new error, which is:
Unsupported hardware. Oops! I guess this should have been the first thing I checked. Sure enough, the hardware I have at home (RTX 4090) is supported:
But the hardware I have in the cloud is not:
I guess when it comes time to actually use ffmpeg for something, I can decide where and how to run it. The cloud machine does have ffmpeg already installed, and I should have just accepted this one to begin with. But given the power of the GPUs, I just wanted to make sure whatever hardware acceleration was available was actually enabled.
Update: It turns out, I don't just need ffmpeg available in some docker container. The extension code needs to access it, so I want ffmpeg in the same container as the extension.
I feel like an idiot for wasting many hours on this, especially when I wasn't enjoying myself while doing it. This was an unnecessary / self-inflicted problem. Sad. Oh well. Moving on.
Actually doing the thing
To be continued...