shish_mish@lemmy.world to Technology@lemmy.worldEnglish · 5 months agoGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coexternal-linkmessage-square34fedilinkarrow-up1191arrow-down18cross-posted to: searchengines@lemmy.worldplanetdyne@fed.dyne.orgtechnology@hexbear.nettechnology@beehaw.orghackernews@lemmy.smeargle.fansreddit@lemmy.world
arrow-up1183arrow-down1external-linkGoogle Is the Only Search Engine That Works on Reddit Now Thanks to AI Dealwww.404media.coshish_mish@lemmy.world to Technology@lemmy.worldEnglish · 5 months agomessage-square34fedilinkcross-posted to: searchengines@lemmy.worldplanetdyne@fed.dyne.orgtechnology@hexbear.nettechnology@beehaw.orghackernews@lemmy.smeargle.fansreddit@lemmy.world
minus-squarereddig33@lemmy.worldlinkfedilinkEnglisharrow-up10arrow-down1·5 months agoWhat isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.
What isn’t the norm is to serve one robots.txt to one company, and a different robots.txt to everyone else. Which is what Reddit is doing here.