Tried that new local LLM tool last night and it crashed my entire PC

I downloaded Llama 3.1 70B quantized to run on my gaming rig with 32GB RAM. Thought I could handle it but the thing maxed out my memory in like 2 minutes and locked everything up hard. Had to hard reset and lost about an hour of unsaved work on a spreadsheet. Anyone else run into memory issues trying to run big models locally?

3 comments

3 Comments

faith_thomas1mo ago

Hang on, isn't that kind of on you though? 70B models are huge and 32GB of RAM is honestly not enough to run one smoothly. You gotta check the recommended specs before you hit download. Most people running those use dual GPUs or like 64GB of system memory. It's not the tool's fault if your rig can't handle the workload. Maybe try a smaller model like 8B or 13B first to see what local LLMs are really about. You could even run them on your CPU with less memory pressure if you tweak the settings right.

blair_webb1mo ago

Wait, you're telling me 32GB of RAM isn't enough for a 70B model?

susan_wright341mo ago

Wow, seriously! @faith_thomas is right though, 32GB just won't cut it for 70B.