Chief Software Architect, ScienceLogic
ScienceLogic is a market-leading Network Monitoring company. Since its founding in 2003, ScienceLogic's appliances have received numerous industry accolades, including being named "2013 Best Network Monitoring System on Earth" by InfoWorld. In addition, it recently received Best Interop for Monitoring and Management, Red Herring's Global Top 100, Deloitte's Fast 500 Technology, and the MSP World Cup awards. ScienceLogic's over 15,000 customers range from Service Providers, to large and small enterprises, and federal agencies. They include organizations such as CloudOps, Opus Interactive, Cisco Systems, Gannett Media, and the Social Security Administration.
ScienceLogic's physical or virtual appliances provide a single, integrated IT operations management solution, which in the words of an InfoWorld reviewer, "bring ultraflexible, ultrascalable, carrier-grade network monitoring to the enterprise." Unlike other solutions with multiple tools and modules that can require months to integrate and add layers of administrative complexity, their solution, EM7, is a single unified system that can be managed easily and deployed in less than one day. They support a broad array of APIs and datatypes, enabling them to monitor and manage virtually any device. ScienceLogic uses solid-state drives (SSDs) to achieve the highest performance and scalability required by the largest enterprises and Cloud providers. In step with its customers' needs, EM7 also offers TrueTM Multi-Tenancy for distributed, Cloud-based monitoring and management. And, ScienceLogic's product relies on MySQL for its ever-growing data storage and management needs.
Figure 1: ScienceLogic Dashboard with SLA Compliance (click image to enlarge)
The Database Challenge
ScienceLogic requires a database to store its configuration data and all of the data that is collected from EM7's monitoring agents. For ScienceLogic's larger customers, that's two terabytes (2TBs) of data on average. Their smallest customers may put the entire solution onto a single server while the largest can use up to 100 or more servers. When ScienceLogic first developed their product ten years ago, they wanted a database that could deliver on their primary value propositions of easy deployment and administration, coupled with extreme scalability, performance and low cost. They found that in MySQL and have been using it ever since.
Figure 2: Architecture of the ScienceLogic Platform
Why MySQL After 10 Years (and Why Not NoSQL-Only)
Ease of use, high performance and scalability, and low cost remain the primary reasons for ScienceLogic's continued reliance on MySQL. "MySQL has provided us with solutions for our growing data needs. We've never found a competitor that can match it," said Matt Luebke, ScienceLogic Chief Software Architect. ScienceLogic has been approached by a number of Big Data / NoSQL vendors, but decided against using a NoSQL-only solution. Said Matt, "There are times when you really need SQL. NoSQL can't show me the top 10 users of CPU, or show me the bottom ten consumer of hard disk. That's why we weren't interested in changing and why we are very interested in MySQL 5.6. It's great that it can do relational and key-value using memcached."
Performance and Scalability
ScienceLogic's product and its MySQL database must scale and perform. Their larger customers use EM7 to monitor and manage up to 20,000 devices and beyond, each of which generates a steady stream of data, and a workload that is 85% write and only 15% read. On a large system, the MySQL database:
- Averages 8,000 queries every second or about 1 billion queries a day
- Can reach 175,000 tables and up to 20 million rows in a single table.
- Is 2 terabytes on average and up to 6 terabytes
Not surprisingly, the product team's primary focus has been on improving scalability. It was obvious to them that MySQL had the same goal with MySQL 5.5 and 5.6. By simply updating MySQL from 5.1 to 5.5, and without making any application-specific tuning adjustments, they saw tremendous performance gains. Because EM7 is so heavily write-intensive, moving log-flushing from a main process to a background process in MySQL 5.5 had a big impact on their performance.
"We told our customers they could add more and more devices with the result that some of the largest use us with over 20,000 devices. With MySQL 5.5, we haven't had any problems. When our customers have problems, we get calls. Not getting calls is a huge benefit, " said Matt.
The ScienceLogic team is very cautious about putting only very stable technology into their product, and according to Matt, MySQL has been very stable: "We've been using MySQL for 10 years and we have never had any reliability problems. Ever."
Key to EM7's ease of use and lower TCO is the fact that MySQL can be "silently" installed, and administered entirely by the embedding application. "We try to provide no-hands administration, or administer through our tuning and coding rather than through a DBA. That MySQL can do this is one of the biggest values it provides to its customers," said Matt.
Huge, Active Install Base
Due to MySQL's popularity and active community, the ScienceLogic team has found that whatever they've wanted to do with MySQL, there is a high likelihood that someone else has already done it, and that they can simply, for example, reuse the same query. This has been a tremendous cost- and time-savings benefit.
MySQL has and continues to provide the database solution that ScienceLogic and its customers require. Matt summarized as follows: "MySQL is integral to our product. It pushes configuration data from one server to another, providing the glue that keeps everything together. There is a major MySQL release every two years, and frequent patch releases. There's a huge install base and a great support team. And, MySQL scales exactly to what the customer requires. We've never found a competitor that can match it."
ScienceLogic Purpose-built Appliance
- OS: CentOS 5.8
- Largest Hardware: Intel S3700 SSD, 32 Total Core, 256 GB RAM, 2 Terabytes SSD storage, 1600 Mhz / CPU
- Smallest Hardware: 2 Core, 6 GB RAM; or on VM
- Database: MySQL 5.5, with plans to move to 5.6