Coding Koans

When switching LLMs to a GPTQ model, my program suddenly started to fail because it required some new modules (the joys of Python dependency management): importlib.metadata.PackageNotFoundError: No package metadata was found for auto-gptq So, I tried installing auto-gptq but that was having problems: Building cuda extension requires PyTorch (>=1.13.0) being installed, please install PyTorch first: No module named 'torch' Even though, I definitely had torch installed. Well, I had it installed in my venv using PDM , perhaps that's the problem. I recommend trying to install it globally, first deactivate your venv: deactivate Then install torch: pip install torch --upgrade Then try the auto-gptq install again. Another approach, if you don't need CUDA support, is to disable the building of the CUDA extension: export BUILD_CUDA_EXT=0 Then try installing auto-gptq again, that worked for me.

One time setup Click on Theme in your blog settings, click the down arrow on the Customize button and choose Edit HTML. Find the </head> tag and paste in the following code just above it, so you end up with: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/default.min.css" /> <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script> </head> Now find the </body> tag and paste in the following code just above it, so you end up with: <script> //<![CDATA[ let checkHljsInterval; checkHljsInterval = setInterval(() => { const numCodeSnippets = document.querySelectorAll("pre > code")?.length; const numHighlighted = document.querySelectorAll("pre > code.hljs")?.length; if (numCodeSnippets > 0) { if (numCodeSnippets !== numHig...

I wanted to share a problem I ran into whilst trying to run a Python app with the torch library on Heroku. If you see a message like this in your Heroku deployment logs: Compiled slug size: 504MB is too large (max is 500MB). Its probably because the torch library is being included in its entirety including all the CUDA libraries. At time of writing, none of the Heroku Dyno instances support CUDA, so there is no need to include all those libraries. You need to specify the cpu only version to fix it. In your requirements.txt, add the following at the top: -f https://download.pytorch.org/whl/cpu/torch_stable.html And then update your torch dependency to add the "+cpu" suffix: torch==2.3.1+cpu Note, the latest version of torch may not have a CPU-only build, so you may need to check the CPU build releases here to find the latest version: https://download.pytorch.org/whl/cpu/torch_stable.html