• 0 Posts
  • 45 Comments
Joined 2 years ago
cake
Cake day: July 4th, 2023

help-circle

  • If this is your first time trying to selfhost I highly recommend Cosmos Cloud, I’ve been using it for 6 months and it’s made every step of the way so much easier for me. It manages docker containers and has included reverse proxy and security features, with paid option for personal VPN like tailscale.

    Most services work perfectly from a catalog of pre-built docker compose files, but Jellyfin I remember I did have to go to the internal docker IP on the actual host machine to set the server up and working properly to be visible from other machines







  • So I googled it and if you have a Pi 5 with 8gb or 16gb of ram it is technically possible to run Ollama, but the speeds will be excruciatingly slow. My Nvidia 3060 12gb will run 14b (billion parameter) models typically around 11 tokens per second, this website shows a Pi 5 only runs an 8b model at 2 tokens per second - each query will literally take 5-10 minutes at that rate:
    Pi 5 Deepseek
    It also shows you can get a reasonable pace out of the 1.5b model but those are whittled down so much I don’t believe they’re really useful.

    There are lots of lighter weight services you can host on a Pi though, I highly recommend an app called Cosmos Cloud, it’s really an all-in-one solution to building your own self-hosted services - it has its own reverse proxy like Nginx or Traefik including Let’s Encrypt security certificates, URL management, and incoming traffic security features; it has an excellent UI for managing docker containers and a large catalog of prepared docker compose files to spin up services with the click of a button; it has more advanced features you can grow into using like OpenID SSO manager, your own VPN, and disk management/backups.
    It’s still very important to read the documentation thoroughly and expect occasional troubleshooting will be necessary, but I found it far, far easier to get working than a previous Nginx/Docker/Portainer setup I used.



  • mierdabird@lemmy.dbzer0.comtoSelfhosted@lemmy.world1U mini PC for AI?
    link
    fedilink
    English
    arrow-up
    3
    ·
    edit-2
    2 months ago

    I’m actually right there with you, I have a 3060 12gb and tbh I think it’s the absolute most cost effective GPU option for home use right now. You can run 14B models at a very reasonable pace.
    Doubling or tripling the cost and power draw just to get 16-24gb doesn’t seem worth it to me. If you really want an AI-optimized box I think something with the new Ryzen Max chips would be the way to go - like an ASUS ROG Z-Flow, Framework Desktop or the GMKtek option whatever it’s called. Apple’s new Mac Minis are also great options. Both Ryzen Max and Apple make use of shared CPU/GPU memory so you can go up 96GB+ at much much lower power draws.




  • SSO is single sign on, so you don’t need individual username and password for every service. It’s a bit more advanced so don’t worry about it until you have what you want working properly for a while.

    DNS is like the yellow pages of the internet - when you type www.google.com your computer uses a DNS server to look up what actual IP address corresponds to the website name. The point of Adguard or pihole is that when a website tries to load an ad your custom DNS server just says it doesn’t recognize the address