Installing the auto-gptq Python module

-

When switching LLMs to a GPTQ model, my program suddenly started to fail because it required some new modules (the joys of Python dependency management):

importlib.metadata.PackageNotFoundError: No package metadata was found for auto-gptq

So, I tried installing auto-gptq but that was having problems:

Building cuda extension requires PyTorch (>=1.13.0) being installed, please install PyTorch 
first: No module named 'torch'

Even though, I definitely had torch installed. Well, I had it installed in my venv using PDM, perhaps that's the problem. I recommend trying to install it globally, first deactivate your venv:

deactivate

Then install torch:

pip install torch --upgrade

Then try the auto-gptq install again. Another approach, if you don't need CUDA support, is to disable the building of the CUDA extension:

export BUILD_CUDA_EXT=0

Then try installing auto-gptq again, that worked for me.


Comments

Post a Comment

Popular posts from this blog

Getting code highlighting working on Blogger

-
One time setup Click on Theme in your blog settings, click the down arrow on the Customize button and choose Edit HTML. Find the </head> tag and paste in the following code just above it, so you end up with: <link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/styles/default.min.css" /> <script src="https://cdnjs.cloudflare.com/ajax/libs/highlight.js/11.9.0/highlight.min.js"></script> </head> Now find the </body> tag and paste in the following code just above it, so you end up with: <script> //<![CDATA[ let checkHljsInterval; checkHljsInterval = setInterval(() => { const numCodeSnippets = document.querySelectorAll("pre > code")?.length; const numHighlighted = document.querySelectorAll("pre > code.hljs")?.length; if (numCodeSnippets > 0) { if (numCodeSnippets !== numHig...
Read more

Using PyTorch on Heroku

-
I wanted to share a problem I ran into whilst trying to run a Python app with the torch library on Heroku. If you see a message like this in your Heroku deployment logs: Compiled slug size: 504MB is too large (max is 500MB). Its probably because the torch library is being included in its entirety including all the CUDA libraries. At time of writing, none of the Heroku Dyno instances support CUDA, so there is no need to include all those libraries. You need to specify the cpu only version to fix it. In your requirements.txt, add the following at the top: -f https://download.pytorch.org/whl/cpu/torch_stable.html And then update your torch dependency to add the "+cpu" suffix: torch==2.3.1+cpu Note, the latest version of torch may not have a CPU-only build, so you may need to check the CPU build releases here to find the latest version: https://download.pytorch.org/whl/cpu/torch_stable.html
Read more

Problem deploying app to heroku using GitHub Actions

-
If you get the following error when trying to use GitHub actions to deploy a Docker-based app to Heroku:  > Error: This command is for Docker apps only. Switch stacks by running > heroku stack:set container. Or, to deploy *** with heroku-24, Its telling you how to solve it in the error. I don't know why this happened only when I created a new app, it didn't seem to occur for the first app I created. Fixing using the heroku-cli seems like the only solution, I tried for a way to do it using the heroku GUI but it didn't seem possible. First, install heroku-cli: brew install heroku Firstly, you have to login: heroku login This should launch a webpage where you can login. After that, try to run a command which will list the current stack of your app: heroku stack If you get an error message such as: > Error: The following error occurred: > Missing required flag app > See more help with --help It means your app isn't hoo...
Read more