also at beehaw

  • 3 Posts
  • 26 Comments
Joined 1 year ago
cake
Cake day: June 15th, 2023

help-circle
  • This is it right here, at least for me personally. I’m a huge Dragon Age fan (played through DAO and DA2 before Inquisition’s release) who has always been vaguely interested in Larian’s Divinity Original Sin games but never made them a priority in my backlog. Seeing the cinematic cutscenes and the 3rd-person voice acted dialog for BG3 made me immediately interested and now I’m 10-ish hours deep into Baldur’s Gate and loving it!

    Also slowly resigning myself to DA4 not even coming close to matching BG3 in quality given the circumstances of its development.














  • So I’m no expert at running local LLMs, but I did download one (the 7B vicuña model recommended by the LocalLLM subreddit wiki) and try my hand at training a LoRA on some structured data I have.

    Based on my experience, the VRAM available to you is going to be way more of a bottleneck than PCIe speeds.

    I could barely hold a 7B model in 10 GB of VRAM on my 3080, so 8 GB might be impossible or very tight. IMO to get good results with local models you really have large quantities of VRAM and be using 13B or above models.

    Additionally, when you’re training a LoRA the model + training data gets loaded into VRAM. My training dataset wasn’t very large, and even so, I kept running into VRAM constraints with training.

    In the end I concluded that in the current state, running a local LLM is an interesting exercise but only great on enthusiast level hardware with loads of VRAM (4090s etc).