Anatomy of MySQL on the GRID
January 19th, 2007 at 10:16 amApril 5, 2007 - UPDATE:
The new MySQL technologies referenced in this post have been released from beta and are now available to all (gs) Grid-Service customers. For more information, please see our two new products: MySQL SmartPool v.2 and MySQL GridContainers. Thank you.
An in-depth look at the MySQL issues on (mt) Media Temple’s Grid platform.
Despite early problems with the system, (mt) Media Temple’s recently launched GRID hosting system has been an overall success for our customers and the company as a whole. The system has helped a new generation of site owners successfully accomplish hosting projects that were previously impossible with normal hosting. With scalability now at a price point unobtainable in the general hosting industry, users are getting more results for less money. In just 3 months the system has become home to over 40,000 domains and a plethora of amazing applications that simply were not sustainable with old Shared Hosting and competing hosting solutions. The GRID however has not been without its trials and has produced several reported bugs and issues of temporary downtime.
Only a few weeks after launch, (mt) Media Temple moved quickly toward release of (GMR v.1.1) an upgrade which alleviated a majority of issues, fixed hundreds of bugs and stabilized most aspects of the system. As the system, and the customers who use it, grow and evolve, newer and more esoteric anomalies are found which are cause for further instability. These new conditions are diagnosed and engineers are quickly responding, literally every 24 hours, with micro-patches that are enhancing the system. One segment of the system continues to be a nuisance for its customers and the architects of the system however; the MySQL databases. Hence, this article was created to provide insight as to why the MySQL issue is the longest outstanding matter on the GRID system and ultimately what is being done to address the situation.
Bluntly, MySQL has been a continuous headache for (mt) and many database users on the GRID. While we have made numerous tuning and configuration changes and hardware revisions, which have resulted in dramatic improvements, we still have fallen short of our own internal quality standards. We’ve furthered our efforts by adding hundreds of thousands of dollars worth of new hardware, and we’ve also hired some of the world’s most highly regarded MySQL experts - all with the goal of trying to stabilize this segment of the GRID. Unfortunately our remedies have still proved to be imperfect. We’ve been forced to go back to the whiteboard and completely reevaluate our approach. The good news is that we have the re-architected our approach to MySQL, this time with improved intelligence and more actual experience, which has resulted in a more robust design which we are confident will quickly solve the ongoing issues.
What follows is a recap of how our system was originally designed, disclosure of some of the attempted remedies, and what ultimately resulted in the permanent fix. To those who continue reading we want to thank you in advance for your time.
The original MySQL design.
Fundamentally it is important to understand that the MySQL segments are not clustered or redundant at an application level within the GRID. The machines, storage, power, networking, web, email, ftp and related applications are n+1 redundant – but the MySQL application itself is not. Although it is possible using commercial and open-source software to cluster MySQL at an application level, such technologies have shown to be counter-effective in multi-tenant hosting environments such as the (mt) GRID. Further to this point, MySQL is not inherently designed to scale and deal with the random usage behaviors of thousands of different users all running different applications and query types. In layman’s terms, there are ways to cluster MySQL if you’re just running one application, even a very large one, if you know how it’s going to behave. However, in a hosting environment where you cannot know ahead of time what users will try and do, there is not an acceptable solution using existing clustering technologies that allows us to “just add more hardware” and scale up automatically.
The MySQL shortcomings being a known issue when planning our implementation, we approached the scalability problem by designing a load-leveling system combined with a server-farm model inside the GRID. The load-leveler system named Tahiti, a special technology created by (mt), analyzes database usage patterns, load, query times, etc. and balances databases into healthy resources within the segment (in this case a server farm). Tahiti works automatically in near real-time and performs its operations with little to no material interruption to database applications. The end result is a server farm of MySQL servers which can move around databases intelligently and automatically keep the system as a whole functional and stable. It’s a great system and we are proud of its technical achievement.
The load requirement plan for the initial MySQL segment at launch was developed by analyzing the current database usage of our previous (ss) Shared-Server system; which we believed was a great data-set to use as a benchmark. This analysis took into account the database behavior patterns of approximately 75,000 active sites. Based on our findings, (mt) Media Temple constructed a system sufficient to handle twice the load of our findings as well as the load from an additional 10,000 new sites (number of anticipated early adopters of the GRID). We also validated our measurements during a 4 month beta test of the GRID which consisted of over 100 users so that we could further develop our baseline performance metrics. We over-built the hardware, based on our internal metrics, so that we could accommodate nearly any load condition until Tahiti was launched and the second or third Cluster went online. We believed this plan was more than adequate to handle our launch as we knew the rate of new customers and migrating customers was lower than this system’s limitations. It was a great plan – on the whiteboard – but it had shortcomings.
What went wrong?
We conclude there were two primary factors which caused the issues experienced with the first generation MySQL segment. First, not having Tahiti ready at launch proved to be a mistake. The 24/7 “human orchestrated balancing system” was not planned for, nor did it scale well. Our DBA team was not able to respond quickly enough to mitigate the unforeseen load issues as indicated in the remainder of this report.
Secondly, our initial database segment became hundreds of times more overloaded than we ever planned for, and did so in an incredibly short period of time. This problem mainly derived from another mistake; which is that we didn’t anticipate attracting a new breed of user - the hosting industry “orphan.” Having an 8-year history of catering to high-demand websites we thought we had seen it all but this new level of load “requirement” was blindsiding. Our new offering quickly became a refuge for sites that were kicked off their old hosting company; a common industry practice. Because of their high database load “requirements” and need of resources, these site owners were shut down immediately and told to leave other hosts. Many of these “orphaned” users had applications, code, and query instructions that were grossly inefficient for even a massive dedicated server. A number of these users came running to (mt) Media Temple with the promise that their applications, despite all of their deficiencies, would be accepted and not turned off. These users are radically different, by orders of magnitude, from anything we had previously analyzed or benchmarked.
So, in keeping with our original promise to sustain our customer’s ability to keep sites up regardless, we needlessly taxed our system, asked no one to leave and relied on our GPU billing model to account for unimaginable database usage. We continued to add hardware, analyze data, add more hardware, analyze data, and add more hardware yet again. In some situations were still unable to sustain these users. No matter what load-leveling we did, these systematically pathological applications overloaded every database we put them on – no matter how big or resourceful.
What is the current status?
Tahiti, which was previously discussed, is currently keeping a mass majority of the MySQL segments stable and operating at speeds that continue to outperform older shared hosting environments. Unfortunately, we have been forced to manually balance a small number of databases onto quarantine servers, and they’re still having issues, until their code is optimized to scale up effectively. Also, there is an infrequent occurrence of a “Too many connections” error message, which is a remaining fragment of the combined issues discussed here within. We’ve updated and installed Tahiti (v.0.6) into the GRID with improved MySQL load-leveling capabilities compared to previous versions. The system is automatically keeping the MySQL segments intelligently and properly loaded which will ensure that 99% of all MySQL dependent sites are functioning at exceptional levels.
THE PERMANENT FIX.
With a near obsession to figure out how to offer reliable, high-performing, low-cost MySQL services, the (mt) Media Temple GRID architects went back to the whiteboard to completely redevelop the database strategy. What resulted is a new container system which will give each GRID customer their own dedicated MySQL server. This is great news for our customers because they will now have their own unshared copy of MySQL with their own dedicated RAM and CPU space which allows for a more stable, predictable, understandable MySQL application environment. The technology is based on the same proven approach that is successfully powering the RoR container system. It eliminates the ambiguous and disruptive nature of the shared MySQL model and allows customers the confidence that any performance issues derive only from the RAM and CPU limitations of the container itself, not from competing customers.
Below is a summary of the benefits of the new MySQL container system:
- On demand upgrades. At any time a customer is able to upgrade to a higher level container. Although the base container given to each customer will be sufficient for the vast majority of customers - there are clearly some who require more. If a customer is experiencing poor MySQL performance they can easily upgrade the container (by clicking a button) to a level which meets their application requirements.
- Reboot capable. In the event a customer exceeds the RAM/CPU limitations of their container a customer can reboot on-the-fly and reset a crashed MySQL environment. This allows for quick site restoration and true application troubleshooting and realistic performance benchmarking.
- Single instance of MySQL for each customer. It is unshared, more reliable, and the only limitation is the container resources themselves (RAM, CPU). No bad neighbor effect.
- Better disk I/O management. Container system is optimized to handle disk I/O for the single container, not an entire server. Results in an improvement in speed and reliability.
- Repair MySQL tool for a clean repair of corrupted databases should you experience out of resource issues.
- Increased insight into application performance. Because you have guaranteed resources that are not shared with others, the use of existing MySQL analysis tools to gain insight into your database activity is much more meaningful.
- Does not count against GPUs. Use as much as you want within the limits of your container resources. No usage fees or confusing billing procedures.
Despite the high costs associated with giving every customer their own container for MySQL, (mt) Media Temple sees no other choice as we have exhausted and ruled out all other possibilities. The container system has proven to be highly effective in our testing and has worked near-flawlessly with other applications. We’re confident that our obsession to correct this deficiency will result in the GRID becoming what (mt) set out to build; a platform that will exceed your expectations.
The current plan is to roll out the new MySQL container system on March 1st. We will also be involving some customers in a private beta from now until this time. The roll-out plan does not require customers to do anything special in order to begin using the newly improved system. Customer databases will start out in the current server farm system (non-container) and will be monitored by a new version of Tahiti (v.0.7). When Tahiti detects that the resource needs of a database resource expand, it will move the databases into the dedicated container allocated as part of the normal (gs) Grid-Server hosting plan. This framework will help conserve resources within the GRID as only a very small number of databases technically require a dedicated MySQL container. As soon as a customer does require the resources of a MySQL container, our system will automatically provision a container and will handle all of aspects of the one-time move. As previously noted, the Tahiti system is extremely fast and sophisticated allowing it to handle database moves with little or no visible disruption to customer site operations.
Other things we are doing.
Although we believe the MySQL container design is the best of all worlds when it comes to multi-tenant MySQL hosting, we will remain committed to providing resources and continued improvement of the system. Here we have outlined some of the additional things we will be doing to help you keep your MySQL databases running clean and performing as best as possible.
- Container +PLUS. With the new MySQL container system, (mt) Media Temple has already begun developing an enhanced version of the container which will allow for cool new features such as; Version switching between MySQL 4 and 5, my.cnf editing for ultimate configuration control, and also future MySQL clustering capability between multiple containers.
- Container Auto-Upgrade. In our quest to provide true scalability (mt) Media Temple is looking to provide you the option of an automatic upgrade to your MySQL container. If you need more resources, they’ll be automatically provided 24/7. After the burst has subsided, your container would then be taken back to your original setting. A micro-billing system would keep track of your usage and notify your email account when it has occurred.
- MySQL coding resources. As we have witnessed the growing popularity of MySQL usage in user applications, we have also seen the spectrum of good coding technique widen (for the worse). In an effort to help our users write better code, (mt) Media Temple will begin a MySQL resource center where our DBAs will provide database tips, links to articles, and write-ups regarding code optimization and MySQL best practices. This will help our users understand better how to write efficient code and define what we mean by “good” and “bad”.
Conclusion
We are emphatically sorry to all of our customers impacted by this aspect of the new GRID system. Your disappointment and frustration has been heard and noted by us. We have never accepted the status-quo in ourselves and what we provide to our customers, which is why we’ve never been remotely satisfied with this situation and have acted as swiftly as we could to correct it. Since 1998 our company’s success has come from providing a great product and great service; not slick ads, referral payoffs and ultra low pricing. (mt) Media Temple is fiercely proud of the high standing we’ve earned in the hosting industry and we’ll continue to do so by working hard to exceed what you demand from your provider. Whether you are someone who has been with us for 9 years or 1 day, it is our hope that your patience will be rewarded by our efforts in this matter and in our pursuit to offer the most advanced hosting platforms in the industry.
» 
January 19th, 2007 at 10:42 am
Thank you for this explanation; it goes a long way. One clarification please: Will all (gs) users eventually be moved to their own MySQL container, or only those who are determined to need the extra resources (perhaps due to using poorly desgined applications).
January 19th, 2007 at 10:51 am
Basically once your database requires the needs of the container (which comes as part of your plan now) it will be given to you automatically by Tahiti. All of the moving, rebuilding, etc will be handled automatically as well. The process is extremely fast, safe, and non-disruptive. The reason it works this way is that a very large number of customers don’t need a container at all. They are working extremely well in our server-farm and are very happy there.
January 19th, 2007 at 10:54 am
Bravo. I’m glad to hear that you’re taking the responsibility to fix this. I also appreciate the explanation on what’s going on. I have faith that this new system will work. You’ve always been a great hosting company, and I’m sure you’ll continue to be after these changes.
BTW, due to the massive problems with Grid as of late, I’ve recently jumped hosts. I planned this to be a temporary thing, so I will be back on (mt) once you get this update rolled out.
January 19th, 2007 at 11:20 am
Thanks for a update. Wonderful to have such transparency. I just wish it was done a little earlier in order to ease our pain.
January 19th, 2007 at 11:25 am
Understandable explanation, makes sense to the layperson and now all we have to do is get popular to drive enough traffic to warrant the auto-move
This is why you guys are the best.
January 19th, 2007 at 11:30 am
Thanks for the post and explanation. The team has been great dealing with all of our complaints and problems, and throughout the frustration, everyone remained professional and got the job done. I’ve been using (mt) for years, and even after this recent frakas, I can’t image using any other hosting company. Your disclosure speaks volumes to those of us that are serious about our sites, and we appreciate the hard work the (mt) Team does day in and day out.
January 19th, 2007 at 11:39 am
Interesting approach. Count me among those who have had many issues with MySQL on the grid. It has been more stable as of late, for sure. The hundreds of “cannot connect” errors reported each day has dwindled to a dozen or less, which is acceptable for any shared hosting joint. But there are still speed issues, and the MySQL container is an interesting idea.
And really great for the folks at MT who have to support it…”Hey, you’re messing up your container man, don’t look at us!”
Wish we would have had such a system in place at the hosts I’ve worked for. It will be infinitely easier to weed out those who are causing the majority of problems.
January 19th, 2007 at 11:40 am
Agree 100% with previous poster. I feel the first 1/2 of this post could have been made weeks ago.
January 19th, 2007 at 11:45 am
We’ve experienced these MySQL problems with the Grid - it’s good to see that you are being open and honest about the problems you’ve faced. I hope that the issues get rectified - at the end of the day, it’s all about the user experience.
January 19th, 2007 at 11:49 am
DC and Andrew… you guys are totally right. We are working to improve the speed of communications. This was a tricky issue however. We wanted to make sure we indeed had the silver bullet to put this issue to rest. Thank you for your comments and your patience.
January 19th, 2007 at 11:54 am
Thanks for this update! It’s been a stressful wait at times but we were always confident you’d pull through. Well done!
January 19th, 2007 at 12:06 pm
I just wanted to throw in a quick (mt) admin/engineering note here mentioning the fact that with the advent of tahiti, the primary mysql farms have been leveled to operating at an average of 1/4 of their hardware capacity and exactly %0.0157 of database users thus far have necessitated Tahiti intervention.
Please also keep in mind while looking at these numbers that the total number of users accessing mysql is still a fraction of the total number of (gs) Grid-Server customers.
This is to say that an extremely small number of mysql users, primarily unintentionally, are a vast majority of the party crashers, and we are dedicated to expanding the framework of systems and tools in place to ensure a consistent experience while simultaneously not leaving anyone “hung out to dry”.
January 19th, 2007 at 12:17 pm
[…] Turns out the demand for this service was so unprecedented they weren’t able to handle the traffic, and optimizating MySQL for the grid environment was not working out. It will be a few months before i throw the switch so fortunately i’m not worried about this too much yet, in hopes that they will fix things by the time we are ready - but if there’s a lesson here it’s wait for v1.1 with new technology. Check out this public apology from the Media Temple team. iPhone anyone? […]
January 19th, 2007 at 12:20 pm
Good to hear a clear description of the current and past status, that kind of transparancy I like…just wait till you get all the details you need to know as a customer. Keep up the good work.
But I also hope that you could integrate the “MySQL coding resources”-idea further onto the GRID system (RoR modules, best-of, tutorials), so developers can start more easily with building there applications onto the GRID-system using the best of the platform (MySQL, RoR, etc…).
I am curious about the coming extensions and updates you have in mind!
January 19th, 2007 at 12:33 pm
Help me understand this… MySQL will, on March 1, be dedicated to each (gs) hosting account and therefore any domains hosted within that account and (gs) will communicate localhost as opposed to the shared MySQL storage system? Also, once the conversion is made will we be able to finally modify the my.cnf to increase variables such as packet size from the current 16MB?
Otherwise, what you guys are doing is great, thank you!!!
January 19th, 2007 at 1:21 pm
Brian K:
On March 1st the first phase will be rolled out. The database connection string will not need any modification on your part.
This first phase will not include the ability to edit you my.cnf but this feature, along with many others, will be included in a future release we are calling Container +PLUS
January 19th, 2007 at 1:24 pm
Thanks for the update. I’ve been working on a couple sites on the grid server, and have also ran into mysql issues every now and then. I’m looking forward to seeing how well this new approach works out!
January 19th, 2007 at 4:26 pm
Apart from automatically quarantining accounts that may be abusing the current shared DBs, will there be any sort of notification to the account owners of the statistics that made the move necessary?
Such information could be very useful in identifying and correcting any bottlenecks in custom software the we owners might be running.
January 19th, 2007 at 4:34 pm
[…] Anatomy of MySQL on the MediaTemple Grid By David Evans on Jan 19th, 2007 My host, Media Temple, has had trouble moving MySQL to their new grid platform. This incredibly lucid explanation of the trouble they have had re-architecting the database system several times over showed up in my inbox this morning. Kudos to them for the transparency (if not a bit a delay explaining all of this to us laypeople.) mediatemple […]
January 19th, 2007 at 4:59 pm
Bravo. Thank You. This does go a long way in customer service, and MediaTemple just scored major points on that front.
January 19th, 2007 at 5:03 pm
This post is really why Media Temple is absolutely awesome. After nearly having my website down for 20% or so of the time some months ago, I’ve noticed a serious increase in speed myself with the new changes.
It’s highly evident that you guys place some of the craziest dedication upon customer service possible, and I’m really not sure how you guys are so patient. I would’ve just given the orphans a goodbye noticed and cut them from the Grid Server as well, because they seem like they’ve costed the company more money than you’ll ever make from them anyway. It’s kind of funny how a few users kill the experience for everyone, and in doing so it only makes you (the host) look bad, since we never really know who the problem is.
That said, this also serves as a reminder for us as developers to start watching the efficiency of our own applications. When I first started hearing about what issues were causing the MySQL service losses, I changed up some of my own site’s database / GPU load use by installing caching software and trying to rebuild some MySQL routines to make my site more efficient. Now, I’ve ended up with a site that could probably handle hundreds of thousands of more hits hourly without fail, and the end result is good for both the grid and my own site.
January 19th, 2007 at 6:07 pm
Would all customers be eventually moved to the dedicated MySQL containers? It sounds like only the customers that crash MySQL all day would get moved. Even though I’m not one of those people, I still get quite a few MySQL errors every day. My concern is that if the number of errors is not significant enough, I won’t get moved to the dedicated containers at all.
(It sounds like I wish I had more MySQL errors, but I don’t.)
Either way, great work (mt)!
January 19th, 2007 at 6:43 pm
Great work guys. I am exceedingly pleased with the transparency you are bringing to the issues and excited about the containers. I would not object to being moved to a container immediately if you want some more beta testers! I am running many db’s and a content management framework that does make tons of calls for data. I am certain that my interactions with MySQL are best practice and “good” as you say and rather optimized so I am unsure that you want the good citizens moved at this time. So all I can say really is, Please!
Amazing work (mt), you guys rock as always. Years with you and it just gets better by leaps and bounds. Thank you.
January 19th, 2007 at 7:18 pm
Marco,
You raise a very important question.
“… will there be any sort of notification to the account owners of the statistics that made the move necessary?”
Yes there will! At the time of move, we will gather up all of the statistical information that we have. This may include ultra frequent queries as well as poor performing queries. Whether the problem is super frequent queries or simply queries which are poorly indexed, we will let you know when it happens and provide a way to look at that report.
Thanks for asking!
January 19th, 2007 at 7:32 pm
Thanks MT
I look forward to browsing the MySQL coding resources.
January 19th, 2007 at 10:52 pm
David B., While the post is generally good news, I’m really pleased that users will be notified of inefficient queries. That’s the exceptional support. I’m looking forward to my email
Thanks, MT.
January 20th, 2007 at 3:05 am
[…] Anatomy of MySQL on the GRID […]
January 20th, 2007 at 6:15 am
[…] Media Temple and Its New MySQL Containers Tagged in mediatemple | mysql | vps Posted on January 20, 2007 - 11:34pm Media Temple, home of the Grid Server which I wrote about here, has just posted an excellent post on their blog, titled Anatomy of MySQL on the Grid Server. It […]
January 20th, 2007 at 6:46 am
Is there any way we can tell if a feature we install is unnecessarily taxing MySql so we can switch to a better feature? I’m talking about forums, web forms, stats, etc, etc that make database calls?
cheers big ears
January 20th, 2007 at 12:33 pm
Yes, thank you for the explanation. It does make a big difference to my thoughts about migrating the rest of my sites to MT.
January 20th, 2007 at 3:06 pm
great! Keep up the good work.
January 20th, 2007 at 10:28 pm
[…] MySQL on the Media Temple GRID Posted in Blogging, Computers & Technology, Scalability, Performance by Elliott Back on January 21st, 2007. [Del.icio.us] Media Temple has an excellent article on their blog about the abuse of their poor MySQL servers on the “gridâ€?. […]
January 21st, 2007 at 2:08 am
Extremely impressed with the detailed explanation. I’m sure (mt) will get more clients because of this
January 21st, 2007 at 6:21 am
I cannot wait! Bravo!
January 21st, 2007 at 9:03 am
Thanks guys!
This entry was amazing. I really appreciate the transparency of it. It makes me want to give each and every one of you a big hug!
January 21st, 2007 at 11:49 am
[…] While taking in my daily RSS feeds, I came across a post from Matthew Mullenweg linking over to another Web post describing one of the better forms of communicating technical information to end users. A few months ago I read (also from a post on Matt’s site) about a hosting company called Media Temple that was developing a new way of large scale Web hosting based on GRID computing. […]
January 21st, 2007 at 4:44 pm
I am a new GRID customer as of 1/7/06, and although my requirement for MySQL has been nearly non-existent thus far, I have to say that I’m impressed by the level of commitment to address ongoing issues with a new product. There will always be growing pains with anything new, and because we’re human, we are not perfect — yet, through trial and error, we learn and adapt to our challenges to prevent repeating the past.
I left GoDaddy, where I paid $3.99/month, for (mt). I needed a host which reflected my open and creative personality and lifestyle, and so, I’ve arrived. Even after more than two weeks of transferring sites, overlooking the obvious because of excitement (or a lack of rest), I still feel like a kid in a candy store. I am looking forward to a long relationship with (mt), as I have many ideas and plans to leverage the Internet for great success!
Thank you, (mt).
Regards,
Ronald Lewis
Social Media Producer
Founder and CTA
DirecPod / Riverscape
January 21st, 2007 at 6:02 pm
[…] MT’s blog has a great post about the original design, the new design, and how they fixed the problems. While I originally loved the idea of the Grid, and then soured on it due to poor performance, I am not only back in love with it, but i have become a big MediaTemple fan due to the company’s openness, integrity, and commitment to performance. […]
January 21st, 2007 at 9:26 pm
[…] (mt) Media Temple - (mt) weblog » Blog Archive » Anatomy of MySQL on the GRID i’m going to have to talk to the Media Temple guys about their GRID offering eventually, but this is an interesting case study of running MySQL in a cutting edge environment (tags: GRID MediaTemple MySQL grid hosting virtualization) […]
January 22nd, 2007 at 9:28 am
Thanks for this informative update. I’ve had horrendous MySQL and connectivity issues in the past but have noticed things have calmed down a bit recently. I’ve held on hoping and wishing the (mt) service to work. I’m looking forward to the protected mysql containers.
One question - since we will have our own MySQL containers will we be able to assign different passwords to different databases? I fear losing all database data in the unlikely event of a hack.
January 22nd, 2007 at 1:01 pm
[…] The article goes in great detail about what the problem is, what didn’t turn out to be the solution and what solution gives the best results for the scaling problem at hand. Interesting read if you are developing an architecture that needs to scale […]
January 22nd, 2007 at 1:09 pm
[…] The Permanent Fix Monday, January 22, 2007 around 2 pm eastern Anatomy of MySQL on the GRID […]
January 25th, 2007 at 4:17 am
[…] Am I completely sold? Nope. MediaTemple is certainly still working through the bugs. To their credit, they are publicly reporting their issues as they occur through their Current Incident Trackers. We also have not been satisfied with their MySQL performance, and MediaTemple admits that these issues are a continued problem. […]
January 25th, 2007 at 7:25 am
Poor scalability with Mysql? Hardly a surprise for anyone who does more performance testing than “select count(*) from foo” in a single thread. In a read-only environment, sure, Mysql may be sufficient, but for highly concurrent OLTP load - forget it. Btw, I found the following articles that pretty much sums it up:
http://pda.tweakers.net/?reviews/661
http://pda.tweakers.net/?reviews/657
http://pda.tweakers.net/?reviews/649
January 25th, 2007 at 10:16 am
I’d love to be able to look at efficiency on my end of the databases, but I cannot (and frankly, will not) look to revise my db until such problems on the server side are worked out. In the two weeks I have been with mt, my site has been reliably quick at times, sometimes extremely slow, and other times it has been completely down. While transparency is a good thing, such issues were not posed to me when I entered into a contract with mt. So, after the fact, I’m left with a site that I cannot present to the public, and I have no idea of the reliability that future changes will bring to my site’s speed. Once again, I’m impressed by the grid build, but very unimpressed at the fact that you are serving a product that is incomplete. Granted I could wait a few months, and see what occurs (and I’d like to), but can we assume the best on March 1?
January 27th, 2007 at 12:01 am
And now the non-technical summary: a nicely put together architecture for handling the assorted nasty things that end users do to hosting sites and likely to get significantly more robust and capable once the improvements are in place. Lots of good work went on there.
January 28th, 2007 at 3:13 pm
[…] That’s not to say the Grid is flawless, however. I’ve seen an insane amount of downtime during periods of very little activity, which is partially why I moved my projects over to DreamHost. Despite the flaws, I’ve decided to stick it out for at least a few more months to see how they pull through with the fix to their biggest problem — MySQL. (Plus, I racked up a few complementary months between referrals and complaining to the helpdesk about the downtime.) No Comments Leave a CommentThere was an error with your comment, please try again. name (required)email (will not be published) (required)url […]
February 22nd, 2007 at 8:04 pm
You’re 1.2 Grid announcement mentions to visit this post to become a part of the beta. I’m interested in the beta, where can I sign up?
February 25th, 2007 at 11:13 pm
I’ve never personally had any problems using mysql on the grid, but it’s good to see such honesty and dedication coming from a hosting company. I was “kicked off” my old host, it wasn’t for inefficient software (I just use wordpress), but from a few traffic spikes, and MT was exactly what I was looking for. Keep up the good work!
May 4th, 2007 at 3:54 pm
[…] Ok the site was just down again for a few hours with exactly the same problem I reported before… In a fit of rage a friend of mine pointed me to this blog post from the (mt) team explaining why the MySQL outtages occur so often (abusive users). It sounds like when you become abusive with your usage, you are automatically moved onto a quarentined MySQL Container instance… that’s pretty slick… I hope they move all the crap-bags off my shared server so I stop going off line. […]
August 15th, 2007 at 6:15 am
And now the non-technical summary: a nicely put together architecture for handling the assorted nasty things that end users do to hosting sites and likely to get significantly more robust and capable once the improvements are in place. Lots of good work went on there.
September 4th, 2007 at 4:41 pm
I’m no tech guy and really only understood less than 20% of the first sentence. After that I just scrolled to the end interested in the conclusion. But the point here is, strategically on a corporate level, you guys did the right thing. Instead of keeping it in the labs, you launched it as best you knew how at that time. You DID something, instead of endless test and wondering how to get it out.
After that you took the punches like a champ, rolled in the mud and continued to FIX it. Launching a first mover product with very few competitors is always tough as you have no one to learn from. In the end, this will be a success as you did something about it first and learnt along the way. Even this blog is a good thing.
MT as a brand is very funny, the associations with the cool sites is one factor, but the badge is almost worn like a “purple heart” in a been there done that kinda of way. That should mean something.
I just signed up, with lots of problems which i figured out, so where can i get my badge?
September 4th, 2007 at 5:08 pm
@DT,
Thanks for the kind words and brutal honesty. I see your problem is with moving a large DB. Were the KB articles we pointed to helpful or confusing? I’ll email you as well.