[Bacula-users] Fwd: Spooling attrs takes forever

[Charles Douglass]

This appears to have never posted. My apologies if I'm wrong and this
is duplicate.


-------- Original Message --------
Subject: Spooling attrs takes forever
Date: Mon, 11 Nov 2013 08:30:59 -0800
From: Charles Douglass <***@gmail.com>
To: bacula-***@lists.sourceforge.net



I am running :
Linux Mint 15 (based on Ubuntu 13.04)
Bacula "Version: 5.2.6 (21 February 2012)"
MySQL version "14.14 Distrib 5.5.32, for debian-linux-gnu (x86_64) using
readline 6.2"

It's a network installation with several clients.

When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.

There is no recent mention of this problem that I could find. One
suggestion from 2012 was to reinitialize the DB so I did that, with no
change to the problem.

Job 1, which is a local backup of about 20G shows this on the console:
11-Nov 00:35 maple-sd JobId 1: Sending spooled attrs to the Director.
Despooling 9,801,461 bytes ...
11-Nov 01:11 maple-dir JobId 1: Bacula maple-dir 5.2.6 (21Feb12):

Which seems long, but is way faster than Job 2.

Chas Douglass

[John Drescher]

Post by Charles Douglass
Linux Mint 15 (based on Ubuntu 13.04)
Bacula "Version: 5.2.6 (21 February 2012)"
MySQL version "14.14 Distrib 5.5.32, for debian-linux-gnu (x86_64) using
readline 6.2"
It's a network installation with several clients.
When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.
There is no recent mention of this problem that I could find. One
suggestion from 2012 was to reinitialize the DB so I did that, with no
change to the problem.
11-Nov 00:35 maple-sd JobId 1: Sending spooled attrs to the Director.
Despooling 9,801,461 bytes ...
Which seems long, but is way faster than Job 2.

Did you optimize your my.cnf for your system? In many distributions
the default parameters are for a machine with a few MB of rm. Yes I
mean MBs of ram.

John

[Charles Douglass]

Thanks for the reply.

I know nothing about MySQL optimization. I downloaded the
"mysqltuner.pl" script and I will run it after the next full backup to
see if it has any recommendations.

Chas Douglass

Post by John Drescher

Post by Charles Douglass
Linux Mint 15 (based on Ubuntu 13.04)
Bacula "Version: 5.2.6 (21 February 2012)"
MySQL version "14.14 Distrib 5.5.32, for debian-linux-gnu (x86_64) using
readline 6.2"
It's a network installation with several clients.
When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.
There is no recent mention of this problem that I could find. One
suggestion from 2012 was to reinitialize the DB so I did that, with no
change to the problem.
11-Nov 00:35 maple-sd JobId 1: Sending spooled attrs to the Director.
Despooling 9,801,461 bytes ...
Which seems long, but is way faster than Job 2.

Did you optimize your my.cnf for your system? In many distributions
the default parameters are for a machine with a few MB of rm. Yes I
mean MBs of ram.
John

[Phil Stracchino]

Post by Charles Douglass
Thanks for the reply.
I know nothing about MySQL optimization. I downloaded the
"mysqltuner.pl" script and I will run it after the next full backup to
see if it has any recommendations.

The mysqltuner script is a pretty decent basic tool. The thing you need
to keep in mind regarding the "mysql-large", "mysql-huge" sample
configurations that are STILL widely distributed in MySQL packages (and
still widely used by the unsuspecting) is that most of those sample
configurations were originally written back when a "large" server was
one that might have as much as a whole 32MB of RAM. These days, that is
less than the compiled-in default size of some individual MySQL
*buffers*. So the unwary install the "suggested" configurations, and
can't understand why MySQL is performing like a geriatric tortoise.

[Charles Douglass]

I've been running Bacula on MySQL for more than five years on Ubuntu. I
don't know why it changed. I suppose there were big changes that came
in the MySQL 5 line.

Chas Douglass

Post by Phil Stracchino

Post by Charles Douglass
Thanks for the reply.
I know nothing about MySQL optimization. I downloaded the
"mysqltuner.pl" script and I will run it after the next full backup to
see if it has any recommendations.

The mysqltuner script is a pretty decent basic tool. The thing you need
to keep in mind regarding the "mysql-large", "mysql-huge" sample
configurations that are STILL widely distributed in MySQL packages (and
still widely used by the unsuspecting) is that most of those sample
configurations were originally written back when a "large" server was
one that might have as much as a whole 32MB of RAM. These days, that is
less than the compiled-in default size of some individual MySQL
*buffers*. So the unwary install the "suggested" configurations, and
can't understand why MySQL is performing like a geriatric tortoise.

[Charles Douglass]

Post by Phil Stracchino

Post by Charles Douglass
Thanks for the reply.
I know nothing about MySQL optimization. I downloaded the
"mysqltuner.pl" script and I will run it after the next full backup to
see if it has any recommendations.

The mysqltuner script is a pretty decent basic tool. The thing you need
to keep in mind regarding the "mysql-large", "mysql-huge" sample
configurations that are STILL widely distributed in MySQL packages (and
still widely used by the unsuspecting) is that most of those sample
configurations were originally written back when a "large" server was
one that might have as much as a whole 32MB of RAM. These days, that is
less than the compiled-in default size of some individual MySQL
*buffers*. So the unwary install the "suggested" configurations, and
can't understand why MySQL is performing like a geriatric tortoise.

I ran mysqltuner before the backup and increased a couple of buffer
sizes as well as taking this suggestion:

innodb_flush_log_at_trx_commit=0


Then I ran a full backup and see this:

18-Nov 12:45 maple-sd JobId 46: Sending spooled attrs to the Director. Despooling 153,470,245 bytes ...
18-Nov 21:41 maple-dir JobId 46: Bacula maple-dir 5.2.6 (21Feb12):
Build OS: x86_64-pc-linux-gnu ubuntu 13.04

So spooling/despooling took just about 9 hours.

And running mysqltuner after shows:

-------- Performance Metrics
-------------------------------------------------
[--] Up for: 2d 0h 15m 14s (765K q [4.407 qps], 203 conn, TX: 16M, RX: 158M)
[--] Reads / Writes: 1% / 99%
[--] Total buffers: 560.0M global + 2.7M per thread (100 max threads)
[OK] Maximum possible memory usage: 828.8M (13% of installed RAM)
[OK] Slow queries: 0% (3/765K)
[OK] Highest usage of available connections: 4% (4/100)
[OK] Key buffer size / total MyISAM indexes: 16.0M/6.0M
[OK] Key buffer hit rate: 100.0% (5M cached / 3 reads)
[OK] Query cache efficiency: 47.9% (11K cached / 23K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 32 sorts)
[OK] Temporary tables created on disk: 17% (175 on disk / 976 total)
[OK] Thread cache hit rate: 98% (4 created / 203 connections)
[OK] Table cache hit rate: 24% (443 open / 1K opened)
[OK] Open file limit used: 23% (261/1K)
[OK] Table locks acquired immediately: 100% (742K immediate / 742K locks)
[OK] InnoDB data size / buffer pool: 246.3M/256.0M

-------- Recommendations
-----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance

I believe this only refers to other tables, since I just previously
deleted and recreated the bacula database.

There was one other suggestion to make the spool file on a different
file system, but does this apply to me since I'm writing to tape?

Thanks for any further help.

Chas Douglass

[l***@kwsoft.de]

Post by Charles Douglass

Post by Phil Stracchino

Post by Charles Douglass
Thanks for the reply.
I know nothing about MySQL optimization. I downloaded the
"mysqltuner.pl" script and I will run it after the next full backup to
see if it has any recommendations.

The mysqltuner script is a pretty decent basic tool. The thing you need
to keep in mind regarding the "mysql-large", "mysql-huge" sample
configurations that are STILL widely distributed in MySQL packages (and
still widely used by the unsuspecting) is that most of those sample
configurations were originally written back when a "large" server was
one that might have as much as a whole 32MB of RAM. These days, that is
less than the compiled-in default size of some individual MySQL
*buffers*. So the unwary install the "suggested" configurations, and
can't understand why MySQL is performing like a geriatric tortoise.

I ran mysqltuner before the backup and increased a couple of buffer
innodb_flush_log_at_trx_commit=0
18-Nov 12:45 maple-sd JobId 46: Sending spooled attrs to the
Director. Despooling 153,470,245 bytes ...
Build OS: x86_64-pc-linux-gnu ubuntu 13.04
So spooling/despooling took just about 9 hours.

Even for MySQL on a not too fast machine this is ridiculus slow. We
despool around 1.2GB attributes within around 2 minutes to our
Postgres server. Something is still wrong with your setup. Have you
checked with tools like iotop,top,vmstat what the server is actually
doing during the 9 hours.

Regards

Andreas

[Josh Fisher]

Post by l***@kwsoft.de

Post by Charles Douglass
I ran mysqltuner before the backup and increased a couple of buffer
innodb_flush_log_at_trx_commit=0 Then I ran a full backup and see
this: 18-Nov 12:45 maple-sd JobId 46: Sending spooled attrs to the
Director. Despooling 153,470,245 bytes ... 18-Nov 21:41 maple-dir
x86_64-pc-linux-gnu ubuntu 13.04 So spooling/despooling took just
about 9 hours.

Even for MySQL on a not too fast machine this is ridiculus slow. We
despool around 1.2GB attributes within around 2 minutes to our
Postgres server. Something is still wrong with your setup. Have you
checked with tools like iotop,top,vmstat what the server is actually
doing during the 9 hours.

Yes. Something is wrong. Is there a script configured to run after the
job? Is there a broken disk drive? Is the machine low on RAM and using
swap space constantly? Are other processes slow on this machine while
the despooling is in progress? Is there a messed up or missing index?
Have you tried rebuilding the database from a dump file?

[Charles Douglass]

Post by l***@kwsoft.de

Post by Charles Douglass

Post by Phil Stracchino

Post by Charles Douglass
Thanks for the reply.
I know nothing about MySQL optimization. I downloaded the
"mysqltuner.pl" script and I will run it after the next full backup to
see if it has any recommendations.

The mysqltuner script is a pretty decent basic tool. The thing you need
to keep in mind regarding the "mysql-large", "mysql-huge" sample
configurations that are STILL widely distributed in MySQL packages (and
still widely used by the unsuspecting) is that most of those sample
configurations were originally written back when a "large" server was
one that might have as much as a whole 32MB of RAM. These days, that is
less than the compiled-in default size of some individual MySQL
*buffers*. So the unwary install the "suggested" configurations, and
can't understand why MySQL is performing like a geriatric tortoise.

I ran mysqltuner before the backup and increased a couple of buffer
innodb_flush_log_at_trx_commit=0
18-Nov 12:45 maple-sd JobId 46: Sending spooled attrs to the
Director. Despooling 153,470,245 bytes ...
Build OS: x86_64-pc-linux-gnu ubuntu 13.04
So spooling/despooling took just about 9 hours.

Even for MySQL on a not too fast machine this is ridiculus slow. We
despool around 1.2GB attributes within around 2 minutes to our
Postgres server. Something is still wrong with your setup. Have you
checked with tools like iotop,top,vmstat what the server is actually
doing during the 9 hours.
Regards
Andreas

iostat during the attrs despooling shows:

334 be/3 root 0.00 B/s 0.00 B/s 0.00 % 99.99 % [jbd2/sda1-8]
24997 be/4 mysql 0.00 B/s 64.98 K/s 0.00 % 0.12 % mysqld

Doing some research this appears to be a problem between MySQL and ext4
with file system journaling (jbd2/sda1-8).

Thanks, everyone, for helping me get this far.

My next step will be to create an ext3 partition and move all my MySQL
Dbs there.

Chas Douglass

[Josh Fisher]

<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
<div class="moz-cite-prefix">On 11/12/2013 7:09 PM, Charles Douglass
wrote:<br>
</div>
<blockquote cite="mid:***@gmail.com" type="cite">
<meta http-equiv="content-type" content="text/html;
charset=ISO-8859-1">
This appears to have never posted.&nbsp; My apologies if I'm wrong and
this is duplicate.<br>
<div class="moz-forward-container"><br>
<br>
-------- Original Message --------
<table class="moz-email-headers-table" cellpadding="0"
cellspacing="0" border="0">
<tbody>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">Subject:

</th>
<td>Spooling attrs takes forever</td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">Date:
</th>
<td>Mon, 11 Nov 2013 08:30:59 -0800</td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">From:
</th>
<td>Charles Douglass <a moz-do-not-send="true"
class="moz-txt-link-rfc2396E"
href="mailto:***@gmail.com">&lt;***@gmail.com&gt;</a></td>
</tr>
<tr>
<th align="RIGHT" nowrap="nowrap" valign="BASELINE">To: </th>
<td><a moz-do-not-send="true"
class="moz-txt-link-abbreviated"
href="mailto:bacula-***@lists.sourceforge.net">bacula-***@lists.sourceforge.net</a></td>
</tr>
</tbody>
</table>
<br>
<br>
<pre>I am running :
Linux Mint 15 (based on Ubuntu 13.04)
Bacula "Version: 5.2.6 (21 February 2012)"
MySQL version "14.14 Distrib 5.5.32, for debian-linux-gnu (x86_64) using
readline 6.2"

It's a network installation with several clients.

When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.</pre>
</div>
</blockquote>
<br>
If backups are being written to fast local disk storage, then you
would likely be better off with no spooling ( SpoolData=no and
SpoolAttributes=no in bacula-dir.conf ).<br>
<br>
In any case, when spooling attributes, Bacula writes the attribute
info to a temporary file in the Bacula work directory, then despools
when the job is finished receiving data from the client by reading
the temp file and updating the DB as a batch. My guess is that the
problem you are having is the result of having the Bacula work
directory and the DB storage on the same physical drive(s). Unless
you have SSD drives, the constant movement of the read/write heads
is killing performance (ie. disk thrashing).<br>
<br>
<blockquote cite="mid:***@gmail.com" type="cite">
<div class="moz-forward-container">
<pre>

There is no recent mention of this problem that I could find. One
suggestion from 2012 was to reinitialize the DB so I did that, with no
change to the problem.

Job 1, which is a local backup of about 20G shows this on the console:
11-Nov 00:35 maple-sd JobId 1: Sending spooled attrs to the Director.
Despooling 9,801,461 bytes ...
11-Nov 01:11 maple-dir JobId 1: Bacula maple-dir 5.2.6 (21Feb12):

Which seems long, but is way faster than Job 2.

Chas Douglass

</pre>
<br>
</div>
<br>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">------------------------------------------------------------------------------
DreamFactory - Open Source REST &amp; JSON Services for HTML5 &amp; Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
<a class="moz-txt-link-freetext" href="http://pubads.g.doubleclick.net/gampad/clk?id=63469471&amp;iu=/4140/ostg.clktrk">http://pubads.g.doubleclick.net/gampad/clk?id=63469471&amp;iu=/4140/ostg.clktrk</a></pre>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Bacula-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Bacula-***@lists.sourceforge.net">Bacula-***@lists.sourceforge.net</a>
<a class="moz-txt-link-freetext" href="https://lists.sourceforge.net/lists/listinfo/bacula-users">https://lists.sourceforge.net/lists/listinfo/bacula-users</a>
</pre>
</blockquote>
<br>
</body>
</html>

[Kern Sibbald]

Hello,

I would suggest one minor modification to what you
wrote. See below ...

Post by Josh Fisher

Post by Charles Douglass
This appears to have never posted. My apologies if I'm wrong and
this is duplicate.
-------- Original Message --------
Subject: Spooling attrs takes forever
Date: Mon, 11 Nov 2013 08:30:59 -0800
Linux Mint 15 (based on Ubuntu 13.04)
Bacula "Version: 5.2.6 (21 February 2012)"
MySQL version "14.14 Distrib 5.5.32, for debian-linux-gnu (x86_64) using
readline 6.2"
It's a network installation with several clients.
When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.

If backups are being written to fast local disk storage, then you
would likely be better off with no spooling ( SpoolData=no and
SpoolAttributes=no in bacula-dir.conf ).

Yes, for disk storage, it does not make much sense to have data spooling
turned off.
I would suggest to always turn attribute spooling on (default off) so
that attributes
will be inserted in batch mode (much faster), and if possible ensure
that the
working directory, where attributes are spooled is on a different drive
from the
Archive Directory. Of course this last suggestion is most often not
possible.

Kern

Post by Josh Fisher
In any case, when spooling attributes, Bacula writes the attribute
info to a temporary file in the Bacula work directory, then despools
when the job is finished receiving data from the client by reading the
temp file and updating the DB as a batch. My guess is that the problem
you are having is the result of having the Bacula work directory and
the DB storage on the same physical drive(s). Unless you have SSD
drives, the constant movement of the read/write heads is killing
performance (ie. disk thrashing).

Post by Charles Douglass
There is no recent mention of this problem that I could find. One
suggestion from 2012 was to reinitialize the DB so I did that, with no
change to the problem.
11-Nov 00:35 maple-sd JobId 1: Sending spooled attrs to the Director.
Despooling 9,801,461 bytes ...
Which seems long, but is way faster than Job 2.
Chas Douglass
------------------------------------------------------------------------------
DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
https://lists.sourceforge.net/lists/listinfo/bacula-users

------------------------------------------------------------------------------
DreamFactory - Open Source REST & JSON Services for HTML5 & Native Apps
OAuth, Users, Roles, SQL, NoSQL, BLOB Storage and External API Access
Free app hosting. Or install the open source package on any LAMP server.
Sign up and see examples for AngularJS, jQuery, Sencha Touch and Native!
http://pubads.g.doubleclick.net/gampad/clk?id=63469471&iu=/4140/ostg.clktrk
_______________________________________________
Bacula-users mailing list
https://lists.sourceforge.net/lists/listinfo/bacula-users

[Thomas Lohman]

Post by Kern Sibbald
Yes, for disk storage, it does not make much sense to have data spooling
turned off.
I would suggest to always turn attribute spooling on (default off) so
that attributes
will be inserted in batch mode (much faster), and if possible ensure
that the
working directory, where attributes are spooled is on a different drive
from the
Archive Directory. Of course this last suggestion is most often not
possible.

One reason to turn on spooling even if you use disk storage for your
volumes if you tend to have hosts that abruptly get pulled off the
network during backups or otherwise have hiccups that cause backups to
fail. With spooling, you shouldn't get volumes filling up with backup
data from the partially completed failed backups.


--tom

[Charles Douglass]

Post by Kern Sibbald
Hello,
I would suggest one minor modification to what you
wrote. See below ...

Post by Josh Fisher

Post by Charles Douglass
This appears to have never posted. My apologies if I'm wrong and
this is duplicate.
-------- Original Message --------
Subject: Spooling attrs takes forever
Date: Mon, 11 Nov 2013 08:30:59 -0800
Linux Mint 15 (based on Ubuntu 13.04)
Bacula "Version: 5.2.6 (21 February 2012)"
MySQL version "14.14 Distrib 5.5.32, for debian-linux-gnu (x86_64) using
readline 6.2"
It's a network installation with several clients.
When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.

If backups are being written to fast local disk storage, then you
would likely be better off with no spooling ( SpoolData=no and
SpoolAttributes=no in bacula-dir.conf ).

Yes, for disk storage, it does not make much sense to have data
spooling turned off.
I would suggest to always turn attribute spooling on (default off) so
that attributes
will be inserted in batch mode (much faster), and if possible ensure
that the
working directory, where attributes are spooled is on a different
drive from the
Archive Directory. Of course this last suggestion is most often not
possible.
Kern

Post by Josh Fisher
In any case, when spooling attributes, Bacula writes the attribute
info to a temporary file in the Bacula work directory, then despools
when the job is finished receiving data from the client by reading
the temp file and updating the DB as a batch. My guess is that the
problem you are having is the result of having the Bacula work
directory and the DB storage on the same physical drive(s). Unless
you have SSD drives, the constant movement of the read/write heads is
killing performance (ie. disk thrashing).

Post by Charles Douglass
There is no recent mention of this problem that I could find. One
suggestion from 2012 was to reinitialize the DB so I did that, with no
change to the problem.
11-Nov 00:35 maple-sd JobId 1: Sending spooled attrs to the Director.
Despooling 9,801,461 bytes ...
Which seems long, but is way faster than Job 2.
Chas Douglass

Sorry, I should have mentioned: I'm backing up to a 80/160 DAT drive.

As a side note, spooling has never done what I think spooling should
do. It says "spooling" and reads the disk for awhile, then says
"writing to tape" and writes to tape for awhile. There doesn't seem to
be any parallel processing going on.

Chas Douglass

[James Harper]

Post by Charles Douglass
When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.

I had this spring up suddenly after a kernel upgrade. strace showed that mysql was constantly doing 512 byte writes + fsync's, and I guess there was a kernel update that handled fsync differently. I have a RAID1 of SATA disks, and a 7200RPM SATA disk gets something like 75 IOPS, and if each op is 512 bytes that's a very low throughput.

I think I added:

innodb_flush_log_at_trx_commit=0

to my config file and the performance went back up to what I was used to, at the expense of a slight exposure to data loss in the event of a power failure.

James

[Phil Stracchino]

Post by James Harper

Post by Charles Douglass
When it does a full backup of one client (also running Mint 15) of
around 40G it gets stuck on the "Sending spooled attrs to the Director.
Despooling 151,437,267 bytes ...". This one step takes over eight hours
to complete. For instance it started at 03:10 my time today and it's
now 08:20 and it's still running.

I had this spring up suddenly after a kernel upgrade. strace showed that mysql was constantly doing 512 byte writes + fsync's, and I guess there was a kernel update that handled fsync differently. I have a RAID1 of SATA disks, and a 7200RPM SATA disk gets something like 75 IOPS, and if each op is 512 bytes that's a very low throughput.
innodb_flush_log_at_trx_commit=0
to my config file and the performance went back up to what I was used to, at the expense of a slight exposure to data loss in the event of a power failure.

Actually, innodb_flush_log_at_trx_commit = 2 is the recommended setting
for most cases, except where it is critical that no data be lost, ever.
With that value, MySQL will attempt to flush the log once per second,
which is a very good compromise between performance and risk of data loss.