As somebody that mods other communities on ani.social, lemmy has a sizeable portion of its userbase that browses by All rather than by Subscribed. Lemmy also has a sizeable portion of users that are extremely hostile to anything anime or anime-adjacent (just see the lemmy.ml defederation drama from last year for example). So, random downvotes on active posts happen a lot since each new comment pushes that post to the top of the feed when sorted by Active. The important part is the engagement from your community members. One person commenting is worth >100 random up/down votes for keeping a community going.
We celebrated a one year birthday for both !anime@ani.social and !manga@ani.social. Over the past year we have grown from nothing to being (in my biased opinion), the best place on the fediverse to discuss anime and manga. It has been a lot of work, but they are both active enough now that I can do something like go on vacation for a couple days and they will keep going in my absence (including other mods that will manage things).
I made a series of birthday-themed anime clips to celebrate. If you are interested, here they are: