The three ingredients to my setup for agentic coding with local models for Flutter development are unsurprisingly:
With the availability of very powerful cloud models, the question I naturally get is why use local models? Well to start with I would like to point out that right now its not a either/or decision and I continue to use cloud models like GPT and Gemini on a daily basis. For me the choice to use local models comes down to control.
And control has many aspects:
Firstly its control of which models, rather than having them changed, replaced or modified at the whim of some VP of some lab somewhere in the valley.
Secondly its control of costs. While its very true that having a capable enough computer to run local models is not cheap (though if done carefully its not horrendously expensive either) running local models means I am completely in control of the cost rather than again being at the mercy of some VC funded company that has suddenly realised their investors want to see some sort of actual revenue and its time to start squeezing their customers for more money in the form of higher prices, lower quotas or both.
Thirdly its control over my tools of production, that I don't need to live in constant fear that again at a whim a company can decide to cancel my account at the drop of a hat
So how to actually run local models and which models do I use? Well to start with I went the "for dummies" route of using ollama to download and run models and honestly I still recommend it for beginners just starting out as being the easiest way to get local LLMs running on your own machine.
Having said that, I have since switched to using llama.cpp which while it required a (very little) more effort to download and build it myself has opened up far more options in terms of running not to mention better performance straight off the bat and then even better performance still just days ago when llama.cpp merged MTP support (more on that in a sec).
On the model front, I have been trying different models pretty much on a weekly basis as over the last few months thats how quickly new open weights models have been released! Currently the model I use daily is Qwen 3.6, both the MOE 35B version which runs at a very usable ~50 t/s on my strix halo desktop and in the last few days the 27B dense model which has gone from a unusable ~7 t/s to a just usable ~12 t/s
Since pi-agent is always running in "YOLO" mode and by default has access to all the same shell tools that you do and importantly with all the same filesystem permissions, it is probably a good idea to put some guard rails around this. I will however first admit that I spent the first couple of weeks using pi-agent with no such guard rails in place, but hey just because I threw the dice and the never came up 6-6 doesn't mean that its a good idea!
So how do you put these guard rails in place? Well I've seen a number of applications pop up over the last few months using Linux cgroups, full docker containers and even complete VMs but I'll be honest that all seemed to much to work to setup for me on the spot so I put them on my to-read list and kept putting it off. That is until I spotted in [an article about running local LLMs] (https://willemvandenende.com/blog/engineering/my-local-agentic-dev-setup-today) reference to a sandbox tool called nono.
If this was an LLM generated article, this would be the point where you would see the inevitable "...the concerns are real", "... the limitations are real" or "... has real trade-off". Since this article was actually written by me you won't get those stupid phrase here and I will simply point out what I've so far been able to achieve using the combination of pi & local models and how well it works.
To the point of how well it works: its definitely good enough to get serious work done! I have already previously published an article outlining how well this very setup did in migrating my website from its old Dart based SSG to a new Dart SSG, adding required new functionality to the new Dart code and updating the websites visual design and structure. I have since then started using this setup to add features to an existing, fairly complex FFI-based Dart package and while its had a harder time there in places this setup has also been able to produce good results there too.
Of course using local LLMs in the size range of tens of billion of parameters is not going to match the performance of cloud ones that are in the hundreds to thousands of billion parameters and they still have a place in being used for the very hardest technical coding and debugging I'm doing, but local LLMs with a good agentic harness are certainly capable of getting useful work done today. With MOE local LLMs it also makes a very important difference how you make use of them as this very good video illustrates and that I highly recommend watching if you are using local MOE models for coding tasks.
Finally the area of speed is another place where you will be compromising on in choosing to use local vs cloud models as unless you have very expensive dedicated high VRAM and very high ram bandwidth GPUs like NVidia 5090 or the like the input prompt parsing and token generation speeds of local models, especially as the context grows is going to be signficantly slower than of cloud based LLMs.
Despite that, the speeds I get of 40-60 t/s using MOE models like Qwen 3.6 35B on my strix halo Framework desktop are more than fast enough for me when working interactively and when leaving the agent to do long running tasks, having it to need to work longer while I am doing other tasks is not a show stopper for me especially given all the benefits I outlined above. My workflow requirements however are not your requirements, so this is an issue that is worth closely considering when deciding on if and how you will use local LLMs.
One optional thing in the setup that I have found increasingly useful is being able to access local LLMs when I am out and about and this has proved to be very easy to setup in a secure way because I was already using Tailscale to have secure connectivity back into my local host network without having to expose any device ports on my home networks public IP address. One thing I have yet to find is a good Android app agent to use with my local models so if you have any recommendations please reach out to me via the usual ways or leave a comment below if you are reading this via Medium.
Hopefully the above run through of my current setup is helpful to someone wanting to try using local LLMs with agentic harnesses in their work on Flutter apps or even any other type of software development work.
I'll have more articles coming up in the near future documenting my further use and exploration of using local LLMs with agentic harnesses so do subscribe if you want to be notified.
Until then, I wish you all a happy and safe today and tomorrow in these very uncertain times.
Get new posts delivered straight to your inbox. No spam, unsubscribe anytime.