http://www.linux-mag.com/cache/7379/1.html https://launchpad.net/drizzle https://code.launchpad.net/drizzle/trunk Para los que no puedan acceder al articulo de Linux Magazine, les pego el texto (no se habras algun voluntario para traducirlo): Drizzle is a re-thought and re-worked version of the MySQL kernel designed specifically for high-performance, high-concurrency environments. In this exclusive article, MySQL guru Jeremy Zawodny takes an inside look at the goals and state of Drizzle development. Drizzle is building a database optimized for cloud and [network] applications. [Drizzle] is being designed for massive concurrency on modern multi-cpu/core architectures. The code is originally derived from MySQL. In other words, Drizzle is a re-thought and re-worked version of the MySQL kernel designed specifically for high-performance, high-concurrency environments. The goal is to make Drizzle an extensible database kernel that runs extremely fast on emerging hardware. Much like Linux is a fast kernel that can be extended with loadable modules (notably filesystems and device drivers, among others), Drizzle explicitly says “no” to building in some features, instead pushing them out of the core and into plug-ins. This separation allows Drizzle to focus on only the most essential features of a modern database kernel. In fact, it’s almost a become a joke on the Drizzle mailing list that someone will say “Make it a plugin!” when anyone questions removing functionality from the server. At this point you may be wondering who would be crazy enough to think that MySQL, known for its speed and agility, needs to be radically re-worked. After all, it’s been the dominant open source relational database for the better part of a decade now. The Drizzle project is headed by Brian Aker, the former director of architecture for MySQL. Aker now works in the Office of the Chief Technology Officer at Sun (soon to be Oracle) and heads up a team of developers working on Drizzle and a few other projects (see one of Aker’s early posts on Drizzle). Some of those developers are Sun employees (Eric Day, Jay Pipes, Monty Taylor, Mark Atwood, and others), while others are either volunteers or contributing to Drizzle on behalf of their employer. Toru Maesaka of Mixi is one such developer. A lot of the Drizzle work is driven by significant changes in hardware that have occurred since widespread adoption of MySQL began. Before discussing what Drizzle is and is not, it’s important to understand a bit about the last 5-7 years of evolution in hardware and systems and the challenges those presented to MySQL users, especially those who were constantly pushing the envelope of what’s possible. Moore’s Law Turned Sideways The most significant change in recent hardware history is that Moore’s Law took a surprising turn. For years, developers have been able to depend on the clock speed of CPUs doubling roughly every 18 months. In other words, the code you ran in early 2001 would be more than twice as fast if you simply upgraded the CPU in your server in late 2002. Why “more than” twice as fast? Not only did clock speeds increase, changes in the architecture of CPUs made them more efficient, too: larger caches, more registers, better instruction pipelines, and so on. This doubling trend went on for so long that Moore’s Law was practically law and something destined to continue for the foreseeable future. But then something strange happened a couple years ago. Instead of seeing 4 GHz CPUs appear to replace 3.2 GHz CPUs, Intel and AMD started pushing multi-core CPUs that ran at slower clock speeds. Effectively, the megahertz and gigahertz game had come to an end. Instead of pure “vertical” scaling, Moore’s Law had been redirected along a different axis: horizontal scaling. At the 2009 MySQL Conference, Sun co-founder Andy Bechtolsheim delivered a keynote presentation called The Solid State Storage Revolution, which contained a chart plotting CPU clock speeds over the last 20 years or so. It illustrated this sudden change very clearly and removed any doubt about where things are headed. (Sadly, his slides are not available online). In a sense this isn’t new. Symmetric multiprocessing (SMP) has been around for years now in the form of motherboards with two or four physically separate CPUs. But having multiple CPU cores on the same physical chip was a new approach. Intel previewed the future with the introduction of Hyper-Threadingon some Pentium 4 CPUs. The technology allowed a single CPU to run two execution threads in parallel. It’s not the same as having two physical cores on the same chip, but it was a nice boost at the time. With Moore’s Law overturned, the new road to efficiency and performance is to design for concurrency and parallelism. Any problem that can be split into sub-problems and executed in parallel fits this new world very well. As you’ll see momentarily, there are places where MySQL is able to exploit multi-core well, but others in which it was of no help at all. RAID and RAM As with all things hardware, the cost of running substantial RAID arrays (6-12 disks) with battery-backed write caches has dropped substantially. That doesn’t mean that disk I/O isn’t still the single biggest bottleneck in large databases—it is. Any time your data size grows substantially beyond what can fit in RAM, I/O bottlenecks appear. But today’s “commodity” hardware lets you push that barrier out quite a long way. Thanks to commodity 64-bit CPUs, it’s not hard to find a server than can be loaded with 64 GB or even 128 GB of memory. That alone is sufficient to keep a very large percentage of MySQL deployments from having to worry about going to disk that often. But what about those really big deployments? The transfer rates of modern disks and their controllers is measured hundreds of megabytes per second. But that’s assuming a best case scenario: linear (sequential) reads or writes. The real killer is random disk I/O. Any time the disk head needs to move (seek), you incur a delay of several milliseconds. Event the fastest hard disks available claim average seek times in the 4 ms range. Expecting more than about 200 seeks per second is unrealistic. In fact, the rule of thumb for years has been not to expect more than about 100 I/O operations (iops) from a single spindle. Having larger disk arrays means being able to perform a greater number of iops per second: typically something like (N/2)*100 where N is the number of disks. You must divide by two because of mirroring. In a typical setup of 12 disks, there are 6 mirrored pairs (RAID-1) and stripe (RAID-0) data across them. This is commonly referred to as RAID 1+0. Using this model, the only way to get another 200 iops out of your array (without sacrificing data integrity) is to add two or four more disks. Like Moore’s Law, this had been true for a very long time. But just like Moore’s Law, the storage game has recently changed in a most unexpected way. Fast Networks and Memcached The classic LAMP architecture has quietly been evolving to incorporate new pieces, largely thanks to gigabit Ethernet (which provides disk-like transfer speeds), 64-bit CPUs, and dropping RAM prices. Many high-volume web sites have added a “caching tier” to their architecture. This caching tier is often based on memcached and sits between the application layer and the database. Refactoring, Removing, and Plugins As you can see, many of the fundamental features of systems have evolved and changed since MySQL began it’s hockey stick growth nearly a decade ago. When you combine that with the organic growth of MySQL’s code base over the years and all the features that have been crammed into it, you end up with a code base that, to put it politely, needs work. So Aker and team have set out to modernize the code, removing old abstractions that are no longer relevant (mysys), removing custom code and replacing it with modern C++ data structures and algorithms, and fixing up various inconsistencies. Along the way the team has been ruthlessly removing needless locks (mutexes) that would greatly reduce concurrency and overall throughput. The SHOW PROCESSLIST command in MySQL is a perfect example. Every time you run it on a server, it grabs a global lock that is not released until it has finished reading all the information about the state of threads in the server, including the text of currently executing queries. Now that’s something that happens very quickly on most servers, but what about a 64- or 128-core server (available today) running thousands of queries per second? Even a stall of a few milliseconds can really back things up. Worse, you’re more likely to run SHOW PROCESSLIST when trying to debug why a server has slowed down. In doing so, you’d further degrade the situation. Hence, the Drizzle implementation of SHOW PROCESSLIST no longer performs any locking. It’s effectively performing a dirty read of the thread status. And that’s just fine for 99.999% of uses. As the team has combs through the code, it occasionally stumbles upon a feature that adds complexity to the system (or reduces concurrency) and asks if it’s worth keeping. Often times the answer is no. The result is that many features have been dropped entirely and others are being moved to plugins. What’s Gone Here’s a short list of some major features that have been removed from Drizzle. With the popularity of memcached and other application level caching, the need for a query cache has decreased over time. It was of limited utility in high-volume applications because every update to a table would invalidate all related entries in the cache. Moreover, locking in the query cache was not nearly granular enough. Hence, it’s been removed from Drizzle. Triggers have been removed with an eye toward re-adding them once a proper plug-in interface can be defined. The MyISAM storage engine that served as the bedrock of MySQL for many years does not exist in Drizzle. It suffered from very poor concurrency and maintaining support for it “contaminated” other parts of the server code. InnoDB is the default storage engine in Drizzle. Both the Event Scheduler and Stored Procedures have been elided while developers debate the merits of having them in the kernel and/or developing a reasonable API to support different schedulers and stored procedures in multiple languages. Views are gone but are expected to reappear with a better implementation at some point. Prepared Statements are gone as well. The implementation in MySQL had numerous problems to begin with, plus Drizzle has a new protocol which allows for a better implementation in the future. Replication has been removed in favor of a more flexible, modular logging system upon which its replacement can be built. There are numerous other changes in Drizzle as well. Character sets are gone. Everything in Drizzle is UTF-8. And the variety of column types has been greatly reduced as well. A list of the differences between Drizzle and MySQL is available. What’s New? The old MySQL protocol and the client libraries have been re-thought and re-implemented. The protocol work itself is now part of the libdrizzle sub-project that’s used by both the server and client utilities, as well as any other projects that wish to talk to Drizzle servers. Aside from simply being more efficient, the protocol allows for checksums on data, has built-in sharding support, and support for connecting via TCP, UDP, and Unix Domain Sockets. When used over TCP or UDS, the protocol is “full-duplex”, which means that the client and server can preempt otherwise long operations to cancel them or provide additional information (errors, warnings, and the like). To ease migration, libdrizzle completely supports the MySQL protocol as well, so you can build applications using libdrizzle that talk to either Drizzle or MySQL servers. If you ever move from MySQL to Drizzle, this feature will be invaluable. If you’d like to know more, have a look at the Drizzle protocol draft. The on-disk format for table definitions (.frm files) has been replaced with a representation based on Google Protocol Buffers. This removes a lot of custom code from the server and makes it easier for other tools to understand table definitions as well. While not an end-user feature, the development model around Drizzle is new, too. It’s completely open (unlike MySQL) and is very easy to get involved with. Testing is a big focus for the Drizzle team. When the team compiles Drizzle, every compiler warning is treated as an error, and each new build is checked against previous builds for any performance regressions. When a regression occurs, no new code is committed to the trunk until the source of the regression has been identified and fixed. A recent example involved the tcmalloc library. Brian Aker discusses some of the testing philosophy in Drizzle, State of Testing. Getting Involved As noted earlier, Drizzle has a very open and approachable development model. The first thing you’ll probably want to do is check out and build a copy of the source code. That means building and installing both libdrizzle and Drizzle itself. You’ll need a few prerequisites, many of which are probably already on your system, but the configure script will notify you of any omissions. The most problematic tends to be Google Protocol Buffers. You may need to build and install the package from source if the package repository for your distribution doesn’t have an up-to-date version available. Once you have the prerequisities, the builds are simple enough. $ bzr branch lp:libdrizzle $ cd libdrizzle $ ./config/autorun.sh $ ./configure $ make $ make test $ sudo make install $ bzr branch lp:drizzle $ cd drizzle $ ./config/autorun.sh $ ./configure $ make $ make test The commands yield an installed copy of libdrizzle and a locally-built Drizzle server that you can start (without installing) by simply running drizzled/drizzled --datadir=/tmp. And then you’re off and running. You can connect to the Drizzle server and use it in much the same way you use MySQL today. To get move involved, here are some links to visit: * Drizzle Client and Protocol Library (libdrizzle) * Drizzle Project on Launchpad * Drizzle Wiki * drizzle-discuss mailing list * drizzle-jdbc (a JDBC driver for Drizzle) * Drizzle PHP Extension * DBD::Drizzle (Perl) Keep in mind that the Drizzle team does not believe that Drizzle is ready for prime-time yet. However, that hasn’t stopped some people from using it on a daily basis. Nonetheless, the project is still evolving at a rapid pace, so if you do plan to use Drizzle soon, it’s best to stay informed about what’s going on in the project. And, by all means, report any bugs you find! Jeremy Zawodny is a software engineer at Craigslist where he works on MySQL, Search, and various back-end infrastructure. He's also the co-author of "High Performance MySQL" and blogs at http://jeremy.zawodny.com/blog/ -- USA LINUX OPENSUSE QUE ES SOFTWARE LIBRE, NO NECESITAS PIRATEAR NADA Y TAMPOCO TE VAS A PREOCUPAR MAS POR LOS VIRUS Y SPYWARES: http://www.opensuse.org/es/ -- Para dar de baja la suscripción, mande un mensaje a: opensuse-es+unsubscribe@opensuse.org Para obtener el resto de direcciones-comando, mande un mensaje a: opensuse-es+help@opensuse.org