Maybe someone smarter than me can explain things, but It’s been about a month since I’ve started the process of creating a magazine to support the reddit/discord community I’ve helped mod for the past 4 years… but I’ve noticed that zero posts show up in google search.
Lot’s of communities are indexed, some tags are, but not really any posts (unless I don’t know how to search, I assumed site://kbin.social was enough to skim for this). Compared to something like: site://lemmy.world where individual posts are indexed.
Is it just because it’s new(er)? Is there something technically wrong with Kbin that it’s taking so long to be crawled? (I thought maybe some noindex was setup but that doesn’t seem it)
Until posts start getting indexed at a relatively decent rate, even slowly, I can’t antipate any progress in adoption since that typically is a large driving factor, at least for my niche. I know that for the subreddit part that was always a large source of traffic.
While I’m being social - can someone explain what Tags and Badges are?
Thanks.
@raphael Why is that? To prevent the site and posts becoming ad focused? I mean I can see that as a benefit. But on the other side, content that is not indexed by search machines is not much different from Discord. I always criticized Discord content for not being indexed, when arguing with my bro, and tell why Reddit (or any other open platform) is much better to post. Honestly, kbin disallowing crawlers to index the site is a big blow to me! I don’t like that at all.
🤷♂️
It is just a decision that every instance owner can make for themselves (if they are aware of it).
It will be a huge headache for search engines anyways, all posts are basically replicated across all instances and look local to a search engine. So for a single post it will have hundreds of copies in its database and probably outputting all of them as results (for now).
Is it possible/reasonable to have some sort of a fediverse-encompassing api for search engines that would help index only the original threads? A separate instance maybe? Or is it going to stay as is?
The search engines are going to have to deal with that. However you can provide context in the instance in the form of a canonical URL, to tell a search engine where content originated.
@fearout It just occurred to me that all you need is your own server and you just need to index that server only. It basically gets data from all other instances through the standard activityPub protocol. It works differently than traditional crawlers, but the outcome is the same.
@raphael I didn’t know the instances would copy the messages. Interesting! I think search engines need to be redesigned to respect robots of the origin instance then. If they are not designed for this, it surely looks local. That’s kind of a mess then, from search engine perspective.
Strange enough, if I search with my search engine based on SearXNG the terms “final fantasy site:kbin.social”, then it finds a few links. They are only based on tags or person, not the actual content. So maybe use tags, if you want to get indexed anyway.