AI Assistant With Hermes

 

Over the past week or maybe two I have been setting up an assistant using Hermes.  I'm using a locally running model, currently Qwen-3.6-35b-a3b.  This model seems to run fast enough on my laptop.  I have an NVidia 5080 with 8GB of VRAM.  I also have 32 GB of system ram.  I run Hermes in WSL on Windows 11.

I did the initial setup through the Hermes cli, but I have mostly been communicating with it through email.  I setup an email address for the agent and have it checking the inbox every 5 minutes.  The agent does what I ask in email, and then sends me back a response, sometimes including attachments.

I also gave the agent ftp access to a web server, so I can make requests like "Take this data and make an html chart, then upload it and send me a link".  I haven't used this too much yet, but I think it will have some interesting uses.

I also have ComfyUI running with a Qwen image generation model setup.  I created a skill for the agent so that it can generate images.  So I can email and ask for an image and it will email it back to me.

So far I have mostly been using it to expand its own abilities, but I plan to continue exploring what I can get done with it.  I also hoping that they will release a Qwen 3.8 35b model, but 3.6 works well for what I have done with it so far.

Recent AI Work and Updates

 I have put my side scroller game on hold to work more generally on getting local models to work when trying to build larger projects.  Early on I was running into issue with local agents (Qwen3.6, Gemma4 etc.) and file edits.  They seemed to always fail.  I installed a plugin called snap-edits that has 2 tools called quick_edit and target_edit.  This seemed to help.  It allowed the agents to make more progress using smaller edits.

After a while, though, they seemed to get stuck in loops trying to edit files.  Those tools require matching exact text in the file to make an edit.  This seems to be an issue, maybe because of hallucination, or because the file content has changed and the agent was not re-reading before trying to make an edit.

After watching this for several days and trying several different setups and agents and still not making much progress I decided to try something else.  I had the agent build an extension for pi that has 2 new tools named delete_lines and insert_code.  The agent picked those names after I described what I wanted.  I don't like them especially because insert_code could work for anything, not just code, but I decided to keep them.  

I built them to be simple so the agent could use them without to much thinking.  Delete_lines just takes a file path, a start line and a number of lines and it deletes them.  Insert_code takes a file path, a start line and a block of code.  The other editing tools seemed to be too restrictive and too focused on not making mistakes.

These new tools have worked well.  The agents are able to get work done with them, but I had to give them a few more directions to get it working smoothly, or at least as smoothly as I could for now.  Here is what I put in the AGENTS.md file.

***Large File Writing Workflow***

insert_code (20-30 lines at most)
read file
if correct
  start back at insert_code
if incorrect
  delete_lines that were inserted
read file to make sure delete is correct
try again or rethink how to break up the insert
start back at insert_code

This seems to work well when the agents follow this workflow.  They seem to forget after a while and fall back to inserting code and not checking the file until they have made a mess.  I'm still working on trying to keep them focused and following directions.

I'll post again when I have tried this on a larger project.

Still working on Side-scroller game

 I have switched to using the PI agent to work on the game.  I'm still using the Qwen3.6 model through Ollama.  It seems to work a little better through Pi.  I have still been trying to get it to work continuously.  I tried a few extensions and finally ended up making my own that seems to do what I want.

With no extensions I was running into two problems keeping the agent working continuously.  The first issue is that the agent would just stop every so often and I would have to restart it by just saying something like "Continue".  I found an extension for the pi agent that would auto continue when this happens and that seemed to work fine.

That extension didn't solve the second problem that I was running into, however.  Sometimes the agent would just get stuck either going through the same thoughts over and over, or just outputting something very simple, like a br tag, over and over again.  To solve this I ended up making my own extension that solves both problems.  I used pi to make the extension and then modified it slightly by hand.

The way my extension works is, anytime the agent stops it just sends the message "Continue".  It also has a timeout that can be set to some number of seconds.  I set it to 1800 sec. or 30 minutes.  This means every 30 minutes the agent is stopped, and then the extension sends "Continue" and it starts working again.

This seems to keep the agent working well, and stops it often enough that it doesn't get lost thinking about something unimportant.  It seems to be making progress on the game again.  The game is here https://github.com/mattdyer/side-scroller.  I have it writing unit tests and acceptance tests for all parts of the game.  It has written a lot of code, but the game isn't playable yet.

Opencode side scroller game

I have the Qwen 3.6 coder model working in Opencode on a side scrolling game.  The code is here https://github.com/mattdyer/side-scroller.  We'll see how far it gets.  I installed a goal extension in opencode, but it doesn't seem to be working.  It is supposed to keep running the model until the goal is complete, but so far I have to manually keep re-running it.  In the repo you can see a PLAN.md.  I have a goal set to implement all the steps in the plan.  The model sees the goal and keeps working on it, but it just stops every hour or so and I have to start it again.  Progress has been pretty slow so far.  I might try playing with some of the settings for the goal plugin and see if I can get it working.  Here is the plugin that I installed. https://github.com/prevalentWare/opencode-goal-plugin  Maybe there is a better plugin that I could try.  I'll look around.

Combining an Opencode coding model and a ComfyUI image generation model

 

I setup ComfyUI as I posted about previously.  I also have Opencode setup with some local models.  For this I used Qwen 3.6 coder.

I had the model help me write a python script that can call the ComfyUI image generation api endpoint.  After you have a workflow setup in ComfyUI you can download json that can be posted to ComfyUI to run that workflow.  The python script allows passing the image prompt and the dimensions as command line arguments.

Then I asked the model to use the python script and generate an html image gallery with six pictures of nature scenes.  The model was able to come up with 6 prompts, run them through the script and save the images.  It then built a simple html page that shows the images.  You can see this in gallery.html in the repo.

I then setup another workflow in ComfyUI to remove backgrounds from images so you can make pngs with transparent backgrounds.  I took that script and the original one and moved them into a skills folder with a SKILL.md explaining how to use them.

Then I asked the coding agent to make a simple solar system simulation with a star and 4 planets.  I told it to use the new skill to generate images with transparent backgrounds.  This took a few tries and some rearranging of the scripts and the skill before it worked, but the end result looks good.  You can see it in simulation.html in the repo.

Here is the repo with all the code I talked about in this article https://github.com/mattdyer/comfyuiapi

ComfyUI

 Over the last few days I have been playing with ComfyUI.  I've never tried it before, but I have seen it mentioned many times.  I have mostly been generating images with Qwen-Image 2512.  It works well and I'm able to run it on my laptop.  It is slow.  It takes about 20 min. to generate an image, but it has been fun to mess around with.  Here are a couple of examples of things I generated.

Prompt: dragon: red and black scales, wings, left side of image, facing towards center

female warrior: black hair, messy hair, blue eyes, holding sword right side of image, facing towards center, black armor

background: castle with fires burning on top of towers

Prompt: a space scene with a sun and planets, asteroids shooting across

You can do different resolutions and aspect ratios.  It seems to work fine with simple prompts, but it will follow specific instructions if you provide them.

EDIT: I got this to start using my video card and it is much much faster.  What was taking about 20 minutes now takes 30-40 sec.  I have a GeForce 5060 laptop version.

Choosing a Direction with AI

 Recently I feel blocked.  There is so much that can be done with AI that I don't know where to start.  I am playing with a local image model currently (https://github.com/ideogram-oss/ideogram4), but it is really slow on my computer.  It is fun, but there isn't much to explore there as far as coding.  I have seen some things recently where people took old hardware, an mp3 player in one example, and had AI write new firmware for it.  I have had some interest in trying to write a basic operating system for a computer.  Maybe I could get AI to help me with that.  I think I may have tried that once already, but AI wasn't quite good enough yet.  It probably is now if I use a paid model.  I need to decide what to do there also.  I'm not really making any money on any side projects that I have worked on, so I don't know how many AI accounts I can pay for.  I'm currently paying for Github copilot, but they just changed the billing, so I don't know if it is worth it anymore.  I might switch to something else.

I have run some local coding models using opencode and had decent results. I've also been playing with paperclip AI.  It doesn't seem to work very well with local models, but it works really well with Anthropic models.  I have built a few different projects.  None of them are production ready yet, but I think I could get them there using that interface.

The image at the top was generated with edeogram4 on my laptop.