If the joke can be explained well using alt text I don’t see the harm in doing so even if very few people will read it. If there isn’t a clear or good way to explain the joke in the alt text then I don’t think it makes a lot of sense.

I personally try to use alt text where it makes sense (see this post of mine).

Markdown also allows for hover text:

[link name](url "hover text")
![alt text](url "hover text")

This can be used to provide extra info when hovering over the image (also used in the post linked above). Hover text may be more appropriate for explaining the actual joke (I don’t know how hover text plays with screen readers though) or serve as a nice mirror of the image’s alt text.

tl;dr: you only should use alt text for memes where the joke can be translated into text format and use hover text if you want to give extra information.