Running AI models without floating point matrix math could mean far less power consumption.

Running AI models without matrix math means far less power consumption—and fewer GPUs?

  • bitfucker@programming.devEnglish
    61·
    5 months ago

    Good

    Edit: Oh shit nvm. It still requires dedicated HW (FPGA). This is no different than say, an NPU. But to be fair, they also said the researcher tested the model on traditional GPU too and reduce memory consumption.

    • Pennomi@lemmy.worldEnglish
      2·
      5 months ago

      Only for maximum efficiency. LLMs already run tolerably well on normal CPUs and this technique would make it much more efficient there as well.

    • FaceDeer@fedia.io
      1·
      5 months ago

      I don’t think that making LLMs cheaper and easier to run is going to “pop that bubble”, if bubble it even is. If anything this will boost AI applications tremendously.

  • blindsight@beehaw.orgEnglish
    3·
    5 months ago

    This could be huge, but we’ll need to wait and see. The economic and ecological footprint of LLMs is problematic.

    That said, will this actually help, or will they just use 3T parameter models to outcompete competitors 1T parameter models using GPUs? Really, this is more about small-scale models competing with midsize models. Like, this could bring a model as big as GPT 3.5 down to be something you could run on affordable hardware, right?

    That would be really compelling for my sector (education) where there’s a lot of concern about student data privacy. I could definitely pitch building a local $5K-cost LLM server that could handle a dozen or so simultaneous users. That would be enough for a small school district.