Sure, you may have hundreds, even thousands of fans or followers, but your fans on Facebook aren't really your fans, and your followers on Twitter aren't really your followers. It's just too easy to become a fan or to follow online. It they're on your roster, you can rest assured that they're ignoring you unless you have evidence to the contrary. And it won't change till you do something different, engaging and interesting. So drop the fantastical notion that you're a celebrity on Facebook; you're not. And lose the positive paranoia that tells you all your followers on Twitter hang on your every tweet; they don't.
Power Twitter users, with several tens of thousands, or even millions, of followers are probably succeeding at Twitter. But this isn't about them. Say you have 2,500-ish followers. Does it matter? Where did they come from? Did most of them follow you after meeting you? Or did they read just one tweet about a blog post and decide to click on the follow button?
It's too easy to click on that button, and in and of itself, it means nothing. A small, focused following on Twitter, one that's paying attention, is much better than a large following. Each follower is a potential emissary telling others about you. Nurture this following. Make it your core. Then, the organic growth of your followership will be strong, and their connection to you tight.
What about your Facebook fans? Sure, they've been "likes" for a while, but we still like to call them fans. Are they really your fans? Or were they on autopilot one day and just clicked on your like button almost out of habit? I hate to break it to you, but again, it's too easy to click on that button, and your fans aren't your fans, because fans hang on every word and activity of the object of their fandom.
Followers and fans on social media are nothing but static numbers until you do something to prompt those followers and fans to start acting like actual followers and fans. Be interesting and make frequent visits to your profiles a rewarding experience. This means volume of content. It means dedicating a team to the effort. And it means having the patience and stick-to-it-ness to see that effort through.