Introduction Today we’re open-sourcing a research project from Labs, ZipPy, a very fast LLM text detection tool. Unless you’ve been living under a rock (without cellphone coverage), you…
I made this and thought you all might enjoy it, happy hacking!
Super cool approach. I wouldn’t have guessed it would be that effective if someone had explained it to me without the data.
I’m curious how easy it is to “defeat”. If you take an AI generated text that is successfully identified with high confidence and superficially edit it to include something an LLM wouldn’t usually generate (like a few spelling errors), is that enough to push the text out of high confidence?
I ask because I work in higher ed, and have been sitting on the sidelines watching the chaos. My understanding is that there’s probably no way to automate LLM detection to a high enough certainty for it to be used in an academic setting as cheat detection, the false positives are way too high.
ZipPy is much less robust to defeat attempts than larger model-based detectors. Earlier I asked ChatGPT to write in the voice of a highschool student and it fooled the detectors. The web-UI let’s you add LLM-generated text in the style that you’re looking at to improve the accuracy of those types of content.
I don’t think we’ll ever be able to detect it reliably enough to fail students, if they co-write with a LLM.
Super cool approach. I wouldn’t have guessed it would be that effective if someone had explained it to me without the data.
I’m curious how easy it is to “defeat”. If you take an AI generated text that is successfully identified with high confidence and superficially edit it to include something an LLM wouldn’t usually generate (like a few spelling errors), is that enough to push the text out of high confidence?
I ask because I work in higher ed, and have been sitting on the sidelines watching the chaos. My understanding is that there’s probably no way to automate LLM detection to a high enough certainty for it to be used in an academic setting as cheat detection, the false positives are way too high.
ZipPy is much less robust to defeat attempts than larger model-based detectors. Earlier I asked ChatGPT to write in the voice of a highschool student and it fooled the detectors. The web-UI let’s you add LLM-generated text in the style that you’re looking at to improve the accuracy of those types of content.
I don’t think we’ll ever be able to detect it reliably enough to fail students, if they co-write with a LLM.