Thanks for sharing nicholas! I'd like to I'd like to know what model or models the system uses, if it can be can be run locally and whether or not it is open source.

After doing some more testing, as one would expect I'm running into a few issues and inconsistencies, indicating more work is needed. Moreover, I'm struck by how much you can learn about a model by employing more advanced prompting on it. I'm using Llama 3 8B Instruct, and one thing I've discovered is that it likes to think out loud. At first I tried to constrain it to only generating the final answer, then I switched gears to telling it explicitly to express its thinking process but now I'm just giving it instructions and letting it output whatever it wants; as long as the output I need is in there, it will be up to me to extract it. Also, I've been imposing my preference for using the pound sign (#) to indicate specific sections of the prompt and to create the bullets for hierarchical lists, but I've noticed that Llama 3 will natively choose to use asterisks for these purposes. In fact, it seems to be employing markup to some extent, so I think I'll re-tool my prompts to comply with how it likes these things and see if it's more consistent.

I re-tooled the prompts to use asterisks as I mentioned and sure enough, it really helped! The responses now are more focused and less wandering, and there's much less tendency to exhibit inconsistent output formats. I decided I would step back a bit from providing the model with a type of analysis to perform and giving it the steps needed and instead ask the model to make those decisions. In this way, it would become truly autonomous, essentially just being told "look at this input and try to get really smart about it". It was easy to generate simple prompts telling it to decide the best type of analysis and to create steps, then I plug those steps in and away we go! It worked really well, except for one small snag I hit. For no obvious reason, at one point instead of performing the next step in the process it insisted on performing step 2 over again. The results clearly already had the data from steps 1 through 5, but instead of going on to step 6, it simply got stuck going back to step 2. Looking at the few shot examples I'd given it (which had always worked fine before) I realized that the steps in those examples were numbered using values likely to occur in the real step list. In this case, it was supposed to perform steps 1 through 7 in the **STEPS** list, while the example steps were numbered 2, 5 and 3, respectively. Notice that the first example is given as step 2, and the model kept repeating step 2... I changed the numbers used for example steps to double digit numbers (not likely to appear in the actual step list) and "POW!" the problem went away! Finally, I decided to compare the results of these multi-step analyses to what I would get just by putting the input through directly; if the complex analysis wasn't better than a simple one-pass answer, the whole exercise would be pointless. What I found was that given the input "Teaching Music Theory to Children", the multi-step analysis output was focused on the inner concepts of the topic while the one-pass answer gave a more practical answer consisting mostly of suggestions for how to go about teaching music theory to kids. The difference was striking. Clearly it's better to use one-pass for responses to the user (more direct and practical) but use multi-step when engaging in inner thought-type rumination. This was my intent; use multi-step to re-assess memories and concepts already stored in order to develop a deeper understanding and create more intelligent content in memory that can be accessed later as needed. Seems to work!

Saturday at 2 sounds great. I'm in the Pacific zone - bear in mind that we are currently on Daylight Saving Time, so do you mean to use Pacific Standard Time instead of Pacific Daylight Time? Either is good for me - just want to be clear.

I will put aside some time soon to review your videos. Happy to assist your hard work in any way I can! 🙂

Checked out some of the vids today - this is looking really cool! I'll watch more and see if I can come up with some suggestions. Great work, Evan!