Obviously there’s not a lot of love for OpenAI and other corporate API generative AI here, but how does the community feel about self hosted models? Especially stuff like the Linux Foundation’s Open Model Initiative?

I feel like a lot of people just don’t know there are Apache/CC-BY-NC licensed “AI” they can run on sane desktops, right now, that are incredible. I’m thinking of the most recent Command-R, specifically. I can run it on one GPU, and it blows expensive API models away, and it’s mine to use.

And there are efforts to kill the power cost of inference and training with stuff like matrix-multiplication free models, open source and legally licensed datasets, cheap training… and OpenAI and such want to shut down all of this because it breaks their monopoly, where they can just outspend everyone scaling , stealiing data and destroying the planet. And it’s actually a threat to them.

Again, I feel like corporate social media vs fediverse is a good anology, where one is kinda destroying the planet and the other, while still niche, problematic and a WIP, kills a lot of the downsides.

  • tkw8@lemm.eeEnglish
    1·
    2 months ago

    I think it’s amazing. I’m running Ollama with a bunch of open-source llms. You’re right. It’s so good. The problem is keeping up to date on what the newest development is.

    The pace of progress is so fast and it’s really difficult to know what the cool kids are experimenting with this moment.

    • brucethemoose@lemmy.worldOP
      1·
      2 months ago

      Oh, and if your hardware is AMD or Nvidia, you should really give exllama a shot.

      If it’s Apple, you should investigate kobold.cpp and more “nitty gritty” llama.cpp backends.

      I have largely negative feelings towards ollama for a lot of reasons, but one of them is that it hides a lot of the knobs to get the absolute best out of LLMs, and understand how they work.

    • brucethemoose@lemmy.worldOP
      0·
      2 months ago

      Honestly a big problem is that the community for filtering the news has “collapsed.”

      The only reasonable congregation was basically /r/localllama, and due to a number of factors (including, apparently, a Reddit bug that was driving away traffic according to a mod), and its shrunken a ton.

      Twitter, linkedin, youtube and such are awful and full of straight up lies. Huggingface is just impossible to navigate and filter. There are a few niche aggregators, but they come and go.

      Hence I was hoping lemmy would grow its existing ML communities, but most of lemmy seems broadly anti AI, even anti open source AI, hence this post to get a feel if that’s true.

      • tkw8@lemm.eeEnglish
        0·
        2 months ago

        I read localllama through redlib but I don’t contribute. I am not technical enough to contribute and I don’t understand the math.

        I have been looking at YouTube for some videos to try to explain it, but I haven’t found anything that is in the sweet spot between “video for non-technical people” and “video for people with PhD and quantum physics”

        • brucethemoose@lemmy.worldOP
          0·
          2 months ago

          It’s a giant mess. Even the technical vidoes tend to be theoretical, and are either obsolete or do nothing to help you actually run them.

          I would know nothing if I hadn’t been following the community since the Pygmalion/ESRGAN days

          • Bob Robertson IX@lemmy.world
            0·
            2 months ago

            I’ve spent the past 2 years looking for the open source AI community, but haven’t really found it. I’ve tinkered with Stable Diffusion and Ollama and I want to learn more, but haven’t found the right places online yet.