How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.

This is a good overview of how this is done: https://www.anthropic.com/engineering/demystifying-evals-for...

The actual tokens might be non-deterministic, but you could look for proxy measures that are supposed to be invariant. Eg. correctness/performance on benchmarks, "thinking level" on complex problems, etc

That's like the entire field of statistics.

Those simple prompts produce nearly the exact same layout in the 2 different models?

My harness expands the prompt into a json representation that specifies layout much more rigorously, which is why you see such that amount of alignment between the two.

That internal json backing helps significantly when you want to maintain consistent design system components/patterns across multiple pages. The aligned layout is it working as intended.

That's so cool!

Private harness?

It is, yes. This is for diffui.ai, which for now is closed source.

Totally normal for modern models due to training on the same datasets supplied by third parties, dataset contamination, and mode collapse, especially for simple prompts that don't have enough semantic capacity. -isms are often very similar even without distillation, and tend to come and go in waves along with model generations.

Equally confused with this. They must be using a lot more guidance than just the provided prompt.

Qwen is trained off of gpt’s outputs. This is both a positive and negative

Agreed. There's also a lot of bad tinging/yellow saturation that very much reminds me of early gpt-image outputs on a lot of the non-cherry picked stuff I've been seeing on Twitter/Reddit.

A lot of people were putting ZiT as a refiner downstream in early Qwen-Image 1.0 workflows, so I'm wondering if we're going to see something similar with 2.1.

Pretty much doing everything that takes more time doing manually on computers

It's painfully obvious these decisions were not made by people who play Magic

They are also beholden to enterprise pricing and can't use the subsidized consumer max plans.

Tried it for a few weeks, was really buggy and unpredictable.

Kudos for the concept though, I ended up rolling my own agentic system with Claude Code from scratch that works much more reliably for my use cases.

Yep. I had posted a comment earlier detailing my usecases. But I too replaced that with my own system that does those same things.

It's way too bloaty, felt like operating windows start menu search.

But you might have missed so far some of the ideas they have. So it's useful to try it out, see what combination of features you use in particular and then just set those up for yourself with claude code or whatever as the LLM harness. Telegram integration is dead easy.

Definitely worth following for ideas.

How did you solve memory in your system?

I ended up using QMD + custom skills to keep it up to date as new data comes in. Essentially a cross-referenced set of markdown files forming a knowledge base: https://github.com/tobi/qmd

Didn't know about qmd.

I use a mix of markdown notes, an sqlite database, and my image store searchable by text. I use immich.

For now I do it manually by giving it skills for each data store I wanna access.

My usecases are all ad-hoc I am not a "pro" user by any means. So I don't mind some manual work.

You should open the issue with your IT department and see if they can optimize the performance. You can gather proof that GoGuardian is causing slow downs by looking at it's memory usage in the Task Manager (right click empty area in Chrome tab bar).

The knowledge graph is well done. I think what's missing from all coworking apps is the UX.

Prompting is a very specialized skill, average users just don't know what to ask for to get the most out of the LLMs.

Ideally the UX should organize and surface information to the user that is important automatically, without needing to be prompted.

Thanks, completely agree. UX is probably the hardest part here. Prompting should not be a prerequisite for getting value. We have been thinking about making the system more proactive, for example surfacing relevant notes ahead of meetings or highlighting changes that need attention. Would love to hear how you think this should ideally work.

I'd be happy to meet and discuss. I will reach out.

Great, looking forward to it.

/sandbox https://code.claude.com/docs/en/sandboxing

> The sandboxed bash tool uses OS-level primitives to enforce both filesystem and network isolation.

As I can't trust Claude Code to use a correct shell, I don't know why I would trust this feature.

Note that there are reports that it can disable sandbox, so personally I wouldn't trust this.

We use https://devin.ai for this and it works very well. Devin has it's own virtual environment, IDE, terminal and browser. You can configure it to run your application and connect to whatever it needs. Devin can modify the app, test changes in the browser and send you a screen recording of the working feature with a PR.

Interestingly, Devin lists Ramp (the OP) as a customer on their front page.

Surprised they need both.