how do you define quality broke? for me it broke when nothing broke. score from llm moved, >= 4 filter after it meant something else. gating on exit code would have passed the whole time.
Yeah thats the real failure mode and it took me a while to name it too. A threshold isnt a test, it is a coordinate system that assumes a fixed origin. When the judge moves the filter keeps passing but it is selecting a different population and nothing throws because no invariant anywhere names that population.
So i define it as: quality broke when an observable moved without a change that authorizes it.
Thanks for trying it out i am really happy for feed back :D
Do you really need residential ip for this? for my news scraping i have choose static datacenter proxy first, rotating only as fallback, direct as last. flaresolverr covered cloudflare js challenge, but it wedge on memory time to time, so i add healthcheck for it. on my own site aws waf meet datacenter ua with captcha, so i understand other side too. residential never was necessary for me, but maybe my targets are just easier, what do you scrape?
Do you really need to provide AI full access for postgres? I rather created a small set of fixed tools, rather then raw sql. Each tool has bunch of predefined params and limited context. Each has own purpose: query explain and analize, db monitoring, slow log queries, db healthcheck, backups, etc.
I have choose pgbouncer for my small db, because it does one thing and does it good - transaction pooling, other solutions seemed too complicated for me. All that features which should keep you allow to use listen/notify and set was unnecessary for me, i solved it on code level
i see blocked status for my website, which is expected, because aws waf meet any suspicious(datacenter or non regular ua) with a captcha challenge. How to properly add you bot to exceptions? could you provide a list of ip publicly maybe, just an idea
i hit the same 400 querying the algolia, a raw > in the query string is what does it, encode it as %3E and points>100 comes back fine for me right now. curl -G with --data-urlencode handles the escaping if youd rather not do it by hand
never got a name, best we managed was network attribution. ours came mostly from tencent cloud plus a second hetzner asn, all spoofed chrome on windows user agents, and it rotated from digitalocean to tencent within a day once we started blocking asns. need to tell that it executed the AWS WAF js challenge and replayed the token, but never solved an interactive captcha, so flipping the action from challenge to captcha is what finally stopped it
we moved our django app behind pgbouncer transaction pooling a few days ago and the surprise wasn't SET so much as queryset.iterator(). it relies on server side cursors, which don't survive being pooled, so we had to disable it everywhere and let it fall back to client side. also had to move statement_timeout out of the app's connection options into the pooler's own connect query, since libpq startup params just get silently ignored behind it.
Using server-side cursors is a sign of bad design. You use the PostgreSQL server's resources as a cache when you're fetching and processing stuff row by row, which is very bad. Just get the data you requested in the first place in one go and do your stuff in your app. Or process all data on the server and then get the processed data in one batch as well.
As said above, SQL is a set based language and one shouldn’t write a cursor to deal with individual rows unless no other way.
And if writing a server side cursor, probably better to write a stored procedure /function and put the cursor and its logic in it, and then call that rather than handle in application
two different things are getting called "cursor" here. queryset.iterator() isn't server-side row-by-row processing, it's one set-based query streamed to the client in chunks so a big export doesn't materialize in memory all at once, "get it in one go" is exactly the OOM it avoids.
the honest knock on the streaming kind isn't "bad design", it's that it holds a portal open server-side, which is why it can't survive transaction pooling. so the pooler-friendly fix isn't one giant fetch, it's keyset pagination (where id > last limit n), stateless and constant memory, which is what we moved those paths to.
Handling cursors is tough - they are very much session-level objects, so even if we, say, pinned your client while it uses that cursor, which would work, that would decrease the performance of connection pooling overall.
So, what's better, breaking your app initially so you know to remove that feature, or letting it work silently while the connection pool isn't 100% in transaction mode? Tough call.
Maybe don't reassign the sesssion to a different client so long as there's a cursor open (the way that most poolers have a mode that won't reassign a session that has a transaction open)?
