I’ve tried coding and every one I’ve tried fails unless really, really basic small functions like what you learn as a newbie compared to say 4o mini that can spit out more sensible stuff that works.
I’ve tried explanations and they just regurgitate sentences that can be irrelevant, wrong, or get stuck in a loop.
So. what can I actually use a small LLM for? Which ones? I ask because I have an old laptop and the GPU can’t really handle anything above 4B in a timely manner. 8B is about 1 t/s!
You must log in or register to comment.
I installed Llama. I’ve not found any use for it. I mean, I’ve asked it for a recipe because recipe websites suck, but that’s about it.
you can do a lot with it.
I heated my office with it this past winter.
Sorry, I am just gonne dump you some links from my bookmarks that were related and interesting to read, cause I am traveling and have to get up in a minute, but I’ve been interested in this topic for a while. All of the links discuss at least some usecases. For some reason microsoft is really into tiny models and made big breakthroughs there.
https://reddit.com/r/LocalLLaMA/comments/1cdrw7p/what_are_the_potential_uses_of_small_less_than_3b/
https://github.com/microsoft/BitNet
https://www.microsoft.com/en-us/research/blog/phi-2-the-surprising-power-of-small-language-models/
https://news.microsoft.com/source/features/ai/the-phi-3-small-language-models-with-big-potential/
I’ve used smollm2:135m for projects in DBeaver building larger queries. The box it runs on is Intel HD 530 graphics with an old i5-6500T processor. Doesn’t seem to really stress the CPU.
UPDATE: I apologize to the downvoter for not masochistically wanting to build a 1000 line bulk insert statement by hand.
Have you tried RAG? I believe that they are actually pretty good for searching and compiling content from RAG.
So in theory you could have it connect to all of you local documents and use it for quick questions. Or maybe connected to your signal/whatsapp/sms chat history to ask questions about past conversations
No, what is it? How do I try it?
RAG is basically like telling an LLM “look here for more info before you answer” so it can check out local documents to give an answer that is more relevant to you.
You just search “open web ui rag” and find plenty kf explanations and tutorials
I think RAG will be surpassed by LLMs in a loop with tool calling (aka agents), with search being one of the tools.
LLMs that train LoRas on the fly then query themselves with the LoRa applied
Converting free text to standardized forms such as json
Oh—do you happen to have any recommendations for that?
DeepSeek-R1-Distill-Qwen-1.5B
It’ll work for quick bash scripts and one-off things like that. But there’s not usually enough context window unless you’re using a 24G GPU or such.
Snippets are a great use.
I use StableCode on my phone as a programming tutor for learning Python. It is outstanding in both speed and in accuracy for this task. I have it generate definitions which I copy and paste into Anki the flashcard app. Whenever I’m on a bus or airplane I just start studying. Wish that it could also quiz me interactively.
I’ve integrated mine into Home Assistant, which makes it easier to use their voice commands.
I haven’t done a ton with it yet besides set it up, though, since I’m still getting proxmox configured on my gaming rig.
What are you using for voice integration? I really don’t want to buy and assemble their solution if I don’t have to
I just use the companion app for now. But I am designing a HAL9000 system for my home.
I’ve run a few models that I could on my GPU. I don’t think the smaller models are really good enough. They can do stuff, sure, but to get anything out of it, I think you need the larger models.
They can be used for basic things, though. There are coder specific models you can look at. Deepseek and qwen coder are some popular ones
Been coming to similar conclusions with some local adventures. It’s decent but not as able to process larger contexts.
Currently I’ve been using a local AI (a couple different kinds) to first - take the audio from a Twitch stream; so that I have context about the conversation, convert it to text, and then use a second AI; an LLM fed the first AIs translation + twitch chat and store ‘facts’ about specific users so that they can be referenced quickly for a streamer who has ADHD in order to be more personable.
That way, the guy can ask User X how their mothers surgery went. Or he can remember that User K has a birthday coming up. Or remember that User G’s son just got a PS5 for Christmas, and wants a specific game.
It allows him to be more personable because he has issues remembering details about his users. It’s still kind of a big alpha test at the moment, because we don’t know the best way to display the ‘data’, but it functions as an aid.
Hey, you’re treating that data with the respect it demands, right? And you definitely collected consent from those chat participants before you Hoover’d up their [re-reads example] extremely Personal Identification Information AND Personal Health Information, right? Because if you didn’t, you’re in violation of a bunch of laws and the Twitch TOS.
If I say my name is Doo doo head, in a public park, and someone happens to overhear it - they can do with that information whatever they want. Same thing. If you wanna spew your personal life on Twitch, there are bots that listen to all of the channels everywhere on twitch. They aren’t violating any laws, or Twitch TOS. So, *buzzer* WRONG.
Right now, the same thing is being done to you on Lemmy. And Reddit. And Facebook. And everywhere else.
Look at a bot called “FrostyTools” for Twitch. Reads Twitch chat, Uses an AI to provide summaries of chat every 30 minutes or so. If that’s not violating TOS, then neither am I. And thousands upon thousands of people use FrostyTools.
I have the consent of the streamer, I have the consent of Twitch (through their developer API), and upon using Twitch, you give the right to them to collect, distribute, and use that data at their whim.
So, buzzer WRONG.
Quite arrogant after you just constructed a faulty comparison.
If I say my name is Doo doo head, in a public park, and someone happens to overhear it - they can do with that information whatever they want. Same thing.
That’s absolutely not the same thing. Overhearing something that is in the background is fundamentally different from actively recording everything going on in a public space. You film yourself or some performance in a park and someone happens to be in the background? No problem. You build a system to identify everyone in the park and collect recordings of their conversations? Absolutely a problem, depending on the jurisdiction. The intent of the recording(s) and the reasonable expectations of the people recorded are factored in in many jurisdictions, and being in public doesn’t automatically entail consent to being recorded.
See for example https://www.freedomforum.org/recording-in-public/
(And just to clarify: I am not arguing against your explanation of Twitch’s TOS, only against the bad comparison you brought.)
You’re both getting side-tracked by this discussion of recording. The recording is likely legal in most places.
It’s the processing of that unstructured data to extract and store personal information that is problematic. At that point you go from simply recording a conversation of which you are a part, to processing and storing people’s personal data without their knowledge, consent, or expectation.
@kattfisk That seems to imply that you cannot personally listen to or watch recordings that you have made in public. In doing so, you are abstracting personal details that you might have missed before, refreshing your memory, and so on. What is the material difference between you doing this without machine help versus with automation that makes it ethically problematic? What if a friend helped you, not a machine?
You build a system to identify everyone in the park and collect recordings of their conversations? Absolutely a problem, depending on the jurisdiction.
Literally not. The police use this right now to record your location and time seen using license plates all over the nation - with private corporations providing the service.
and being in public doesn’t automatically entail consent to being recorded.
And yes, it’s called ‘expectation to the right of privacy’. Public venues are not ‘private’ locations, and thus do not need consent. You can, quite literally, record anyone in public.
Even the link you provided agrees.
In the US maybe but not in Germany, Austria and probably most countries in Europe.
Doesn’t Twitch own all data that is written and their TOS will state something like you can’t store data yourself locally.
I’m not storing their data. I’m feeding it to an LLM which infers things and storing that data. Other Twitch bots store twitch data too. Everything from birthdays to imaginary internet points.
lol. Way to contradict yourself.
Was this system vibe coded? I get the feeling it was…
Most US states are single party consent. https://recordinglaw.com/united-states-recording-laws/one-party-consent-states/
There is no expectation of privacy in public spaces. Participants to these streams which are open to all do not have a prohibition on repeating what they have heard.
Repeating what they heard is very different from automatically processing the chat to harvest personal information about the participants.
Just because some data is publicly available doesn’t mean all processing of that data is legal and moral.
It is qualitatively equivalent. Any single piece of information could have been copied, it is safe to assume it has all been copied.
Although I would be onboard for supporting an expectation of pruvacy in public spaces and making private cctv recording illegal.
sounds like salesforce for a twitch setting. cool use case, must make fun moments when he mentions such things.
Esp. if the LLM just hallucinates 50% of the “facts” a about the users 👌
That hasn’t been a problem at all for the 200+ users it’s tracking so far for about 4 months.
I don’t know a human that could ever keep up with this kind of thing. People just think he’s super personable, but in reality he’s not. He’s just got a really cool tool to use.
He’s managed some really good numbers because being that personal with people brings them back and keeps them chatting. He’ll be pushing for partner after streaming for only a year and he’s just some guy I found playing Wild Hearts with 0 viewers one day… :P
Surely none of that uses a small LLM <= 3B?
Yes. The small LLM isn’t retrieving data, it’s just understanding context of text enough to know what “Facts” need to be written to a file. I’m using the publicly released Deepseek models from a couple of months ago.
I have it roleplay scenarios with me and sometimes I verbally abuse it for fun.
7b is the smallest I’ve found useful. I’d try a smaller quant before going lower, if I had super small vram.
for coding tasks you need web search and RAG. It’s not the size of the model that matters, since even the largest models find solutions online.
Any suggestions for solutions?
Not on top of my head, but there must be something. llama.cpp and vllm have basically solved the inference problem for LLMs. What you need is a RAG solution on top that also combines it with web search.
As cool and neato as I find AI to be, I haven’t really found a good use case for it in the selfhosting/homelabbing arena. Most of my equipment is ancient and lacking the GPU necessary to drive that bus.
Learning/practice, and any use that feeds in sensitive data you want to keep on-prem.
Unless you’re set to retire within the next 5 years, the best reason is to keep your resume up to date with some hands-on experience. With the way they’re trying to shove AI into every possible application, there will be few (if any) industries untouched. If you don’t start now, you’re going to be playing catch up in a few years.
Nothing.
