Testing??? PLEASE!!!!
A few days ago I came across an article about a test comparing Oracle to PostgreSQL. First of all, where was MySQL? The only figure I saw in the article was that PostgreSQL was 12% slower than Oracle.
Then the article talks about how the hardware was “similar” but not the same.
HMMMMMMMMMMMMMMMMMMMMMMMM
Now I am not a testing expert. I didn’t stay in a Holiday Inn Express last night. However, this I know. This test is worthless. It doesn’t prove anything. If you can’t run the test on the same hardware than you can’t rely on the results. I won’t even get it into if one database was tuned and the other wasn’t. I don’t know.
Anyways, just my little rant for the day.
5 Comments so far
Leave a reply
Postre seems to be taking a lot of effort nowadays to convince people that it is a good free alternative to Oracle. They ware always ware and still are lacking in performance. MySQL has always been good in that department, but has been lacking in features. Now that MySQL is gaining, the reasons to switch to PostgeSQL instead of MySQL are slimming by the day. So I can imagine that a posgre fan isn’t keep in mentioning MySQL.
[…] Mechanism, not Policy Filed under: Postgres, MySQL — Mark Dennehy @ 22:42 Over on Diamond Notes, a somewhat skeptical take on the recent Postgres benchmarks news. Can’t say I agree with it, and I did comment on it, but something did go click for me while thinking about it, hence this entry. See, the problem isn’t just performance. MySQL just doesn’t do the right thing. […]
The problem with the “similar” hardware is a real one but it invalidates Oracle’s claim to first place, not Postgres’ (Oracle was running on their own custom hardware, not Postgres).
As to MySQL vs Postgres, refer to the earlier tests on Josh Berkus’ blog. There’s a bit more to the comparison than “just” performance though - MySQL just doesn’t do the right thing.
Firstly a disclaimer: I work and like to work with both MySQL and PostgreSQL, I’m not particularly biased towards either, but post this as I fell (and have for a long time) that a significant proportion of the MySQL cheerleading team don’t get why PostgreSQL is a good thing, and additionally seem to not accept that MySQL also has shortcomings (honestly I’m not blaspheming).
Arnold: The test shows that when compared side by side with Oracle on similar hardware (admittedly not the same) an open source database PostgreSQL, (PostgreSQL is still considered to be the closest OSS DB to Oracle in terms of feature set) can compete in the same league as the mighty Oracle. For all the reasons why, Machanism, Not Policy said it best in the blog entry reference in this thread. The browser analogy is particularly good and one which I’ve been making for many years to deaf ears. There has to be some rules, that’s why Indy cars, NASCAR, Formula 1, all have a set of rules, otherwise there’d be no level playing field.
The reason this benchmark got a lot of press is because it was the first official benchmark performed on PostgreSQL, due to the costs etc involved. It means the world now have a performance benchmark, which gives an objective comparison point (Spec benchmark). All previous tests have been unofficial based on random testing criteria, especially those performed in the late 90’s, when PostgreSQL was repeatedly compared against MySQL when MySQL was considerably lighter in features and with no real objective yard stick each time.
I think MySQL is a great DB, as is PostgreSQL, but if I want a DB that I’m going to entrust my most important data to, which I know has had a robust feature set for a significant amount of time, I would have gone with PostgreSQL (personal choice). But with an official benchmark under its belt, it’s a really viable option for organizations.
The most important point to come from this; for small and medium sized organizations $110,000 of Oracle software (this was the cost of the Oracle software required to perform the benchmark) would put a real hurting on their IT budget, they now have somewhere else to go.
I would love to see PostgreSQL and MySQL compared on a level playing field with a Spec benchmark. Sadly the PostgreSQL team almost certainly doesn’t have the funds for this (hence the long wait for the first benchmark) but MySQL Corp certainly does have the funds to sponsor this, so how about it?? Surely now that MySQL has so much more functionality and features, it would be far easier to be tested side-by-side on a level playing field??
Hey everyone. I think I have hit a nerve. Wow. Really, I promise, my only point to all this is that the testing was unfair. Let’s take MySQL out of the equation for now. The test was still unfair. I don’t care who the participants are. Since, according to people who read the article better than I, Oracle was running on “custom” hardware then it tilted the balance towards them. So, Postgresql really ran better than the “12% slower than Oracle” that the test indicated.
I am going to stay out of the MySQL vs Postgresql features religious wars. I have never used Postgresql so I can not speak intelligently on what features it has or does not have.
Aside from a feature/feature comparison I would say this. It is my opinion that — rightly or wrongly — many enterprises go with MySQL because of the ability to buy a support contract with MySQL AB (the company “behind” MySQL). Just like many companies buy Oracle because they can get support from Oracle. It gives CIOs/CTOs C-whoever-Os a sense of security. Postgresql doesn’t have a company in the same sense behind it. I am sure you can buy support contracts from people in the Postgresql world but from the executives level they wouldn’t view it the same.