DropTracker.io
The DropTracker is an all-in-one loot and achievement tracking system, with a few components:
- A Java-based client side plugin, written for RuneLite, an approved third-party Old School RuneScape client.
- A Pythonic back-end, which includes six different applications that all work hand-in-hand to provide the services our users have come to appreciate.
- Those include:
- The Primary Discord Bot (@DropTracker.io) - Handles player commands, sends notifications and updates loot leaderboard messages with their most recent copies.
- The API - Processes incoming requests from players using the RuneLite plugin with the API enabled, including submissions and search results.
- Hall of Fame - A separate Discord bot responsible for processing and updating groups' hall of fames—a listing of the highest-ranking loot and personal bests at selected bosses.
- Webhook Processor - Processes incoming submissions from players using the RuneLite plugin without the API enabled--does not carry the ability to respond; only takes/stores incoming requests.
- Loot Leaderboard Updater - Runs an infinite loop, constantly updating group lootboards every ~2-3 minutes, so that each iteration of the primary discord bot's update cycle sends the most up-to-date board possible. Also allows for calls to update a specific group instantly on demand.
- Player Updater (deprecated) - In an effort to ensure accurate player data stored in redis (a memory caching software), we run a loop that invalidates players every 30 days, ensuring that their data is re-written directly from the database again.
- Those include:
- A PHP-based front-end, which leverages XenForo, a powerful forums-style website, and provides easy access to clan configurations, subscriptions, leaderboards and more.
- As previously mentioned, a robust and nuanced Redis database exists, which we use to store nearly all player drop statistics in memory rather than needing to query the database each loot leaderboard update iteration or leaderboard query.
The Challenges
Building a reliable, high-throughput tracking platform for Old School RuneScape groups comes with some interesting and relatively unique constraints:
- External Connection limitations — RuneLite developers have an expectation that plugins do not need to send requests externally to servers that do not belong to RuneLite, or are not verified by them. This leads to some trouble when considering that we need to communicate directly with our endpoint API in order to process players' submissions automatically; otherwise our plugin serves little purpose.
- Real-time updates — with almost 10,000 users (and growing), and an average of 380 requests per minute (peaking around 1200), it is incredibly important that incoming data can be processed and stored in an easily-retrievable location quickly. Otherwise, the risk exists for a loot leaderboard or search query to return results that do not yet include submissions that just happened.
- Performance — similar to the real-time updates issue, it's important that all of this processing does not lead to hung processes, lost requests or failed submissions.
Architectural Decisions
The Backend
Our backend ecosystem is designed for both scale and reliability. The core logic is almost purely Pythonic, allowing us to handle complex data processing with ease—leveraging the vast network of available packages Python has to offer.
To bridge the gap between the game and our servers, we utilize a client-side plugin written in Java that players use to transmit data seamlessly.
Players who install the plugin initially have their data transmitted to our server through Discord Webhooks. Due to the inherent security issues associated with webhook URLs, this added an additional layer of complexity to our implementation.
The web interface remains lightweight and responsive, powered by a PHP-based website. This multi-language approach ensures we use the best tool for each specific job—Python for the heavy lifting, Java for client compatibility, and PHP for the front-end delivery.
Database Design
MySQL handles the heavy lifting for persistent storage. We've invested heavily in proper indexing and query optimization—some of our tables have hundreds of millions of rows. Redis sits in front for caching frequently accessed data and managing rate limits.
The API
The API primarily serves as the bridge between our Python backend and the Java client-side plugin. It is optimized to handle high-frequency data transmission and retrieval, ensuring that player events are processed and synced with minimal latency.
Lessons Learned
* Start with good logging — You can't fix what you can't see
* Optimize queries early — Database bottlenecks sneak up on you
* Cache aggressively — But have a clear invalidation strategy
* Monitor everything — Set up alerts before you need them
What's Next
We're constantly improving and adding new features. The roadmap includes better analytics, more integrations, and some exciting community features I can't talk about yet.
If you're interested in the technical details, feel free to reach out — I love talking shop.