Is there anyway to make it use less at it gets more advanced or will there be huge power plants just dedicated to AI all over the world soon?

  • vrighter@discuss.tchncs.de
    2·
    2 months ago

    that’s why they need huge datacenters and thousands of GPUs. And, pretty soon, dedicated power plants. It is insane just how wasteful this all is.

    • hisao@ani.socialEnglish
      1·
      2 months ago

      So do they load all those matrices (totalling to 175b params in this case) to available GPUs for every token of every user?

      • vrighter@discuss.tchncs.de
        1·
        2 months ago

        yep. you could of course swap weights in and out, but that would slow things down to a crawl. So they get lots of vram (edit: for example, an H100 has 80gb of vram)