{"id":4715,"date":"2013-08-18T15:00:39","date_gmt":"2013-08-18T22:00:39","guid":{"rendered":"http:\/\/crimeandtheforcesofevil.com\/blog\/?p=4715"},"modified":"2013-08-18T15:00:39","modified_gmt":"2013-08-18T22:00:39","slug":"linux-filesystem-performance-help","status":"publish","type":"post","link":"https:\/\/crimeandtheforcesofevil.com\/blog\/2013\/08\/18\/linux-filesystem-performance-help\/","title":{"rendered":"linux filesystem performance help"},"content":{"rendered":"<p><b>NEW READERS: IT&#8217;S NOT ABOUT THE FILESYSTEM ANYMORE BUT IT&#8217;S STILL BROKEN: SEE UPDATES AT BOTTOM OF POST. Addressing filesystem performance only partly fixed it. Thanks!<\/b><\/p>\n<p>Since always, I&#8217;ve had latency issues on my digital audio workstation, which is running Ubuntu Linux (currently 12.04 LTS) against a Gigabyte motherboard with 4G of RAM and a suitably symmetric four-core processor. CPUs run 20%-ish in use most of the time (and all the time for these purposes), and I never have to swap.<\/p>\n<p>In this configuration, I should be able to get down to around 7ms of buffer time and not get XRUNs (data loss due to buffer overrun) in my audio chain. 14ms if I want to be safe.<\/p>\n<p>In reality, I can&#8217;t make it reliably at 74ms, and that has hitches I just have to live with. To get no XRUNs or close to it I have to go up to like 260ms, which is insane. I even tried getting a dedicated root-device USB card &#8211; I&#8217;ve long assumed it was some sort of USB issue. But no.<\/p>\n<p>With some new tools (latencytop in particular) I have <em>found it<\/em>. It&#8217;s the file system. Specifically, it&#8217;s in the ext3&#8217;s internal transaction logging. To wit:<\/p>\n<blockquote><p><code><\/p>\n<pre>EXT3: committing transaction     302.9ms\nlog_wait_commit                  120.3ms<\/pre>\n<p><\/code><\/p><\/blockquote>\n<p>If I turn off read-time updating, which I tried last night, I get rid of 90% of the XRUNs, because the file system does about 90% less transaction logging to update all those inodes with new times.<\/p>\n<p>But any attempt to write &#8211; well, you can guess. Even the pure realtime kernel doesn&#8217;t help; I compiled and installed a custom build of one <em>today<\/em>, but apparently this is still atomic: I get exactly the same behaviour. I may be able to live with that to some degree, because it&#8217;s a start-and-stop-of-writes thing, and as long as it doesn&#8217;t trigger <em>during<\/em> writes, I can get by.<\/p>\n<p>But it&#8217;s bullshit, and it pisses me off.<\/p>\n<p>I&#8217;m currently in progress of updating ext3 to ext4. I&#8217;d like to think that would solve it, given ext4&#8217;s dramatically better performance, <em>but I have no such assurances at this point<\/em>. I genuinely thought the realtime kernel might do it.<\/p>\n<p>DO YOU HAVE ANYTHING YOU CAN TELL ME, DEAR INTERNETS? Particularly about filesystem tuning. Because this shouldn&#8217;t be happening; it just shouldn&#8217;t. Honestly, <eM>three tenths of a second to commit a transaction?<\/eM> I&#8217;ve been places where that kind of number was reasonable; it was called <em>1983<\/eM>, and I <em>don&#8217;t live there anymore<\/em>.<\/p>\n<p>Anybody?<\/p>\n<p>THINGS IT IS NOT:<\/p>\n<ul>\n<li>Shared interrupt\n<li>This particular hard drive (the previous drive did it too; this one is faster)\n<li>ondemand CPU scheduling (i&#8217;m running in performance)\n<li>this particular USB port or a USB hub or extension cord or any of the sort\n<li>bluetooth or other random services (including search)\n<li>Corrupt HD\n<li>Old technology (it&#8217;s SATA; the drive is like six months old)\n<li>lack of RT kernel. I built this RT kernel today.\n<li>Going to be solved by installing a different operating system. Please don&#8217;t.<\/ul>\n<p><b>ETA:<\/b> I got the ext3 filesystem upgraded to ext4, which made all those above numbers get dramatically smaller, but no further XRUN improvement. So I then disabled journaling, a configuration which outperforms raw ext2 in benchmarks I saw, and the machine is screamingly fast despite the RT kernel&#8230;<\/p>\n<p>&#8230;and it hasn&#8217;t made one goddamn whit of difference in the remaining XRUNs. WTF, computer? WTF.<\/p>\n<p><b>ETA2 (23:51 18 August):<\/b> Okay, while screwing with the filesystem did solve many XRUN problems, there are still other XRUNs which are apparently unrelated, most notably, the master-record-enable XRUN. Even moving the project to a tmpfs RAM disk and running from there produced identical results, so I&#8217;m concluding this is an entirely separate problem.<\/p>\n<p>I&#8217;ve already done pretty much everything there is to do <a href=\"http:\/\/wiki.linuxaudio.org\/wiki\/system_configuration\">the LinuxMusicians configuration consultation page<\/a> and my setup actually passes their evaluation script. I should be golden, but I&#8217;m not. Help?<\/p>\n<p><b>ETA3 (0:26 19 August)<\/b>: Every two minutes, right now, with the system mostly idle, I&#8217;m getting a burst of XRUNs. On an <em>idle machine<\/em>. But it is <em>exactly<\/em> every two minutes. And while Ardour remains on top of Top even when idle (at 10% of CPU and 13.5% of RAM), Xorg pops up just underneath it, and its CPU use spikes.<\/p>\n<p>What does Xorg do every two minutes? Anybody? Seriously I have no idea.<\/p>\n<p><b>ETA4 (13:19 19 August)<\/b>: ARDOUR 3 TRIGGERS SESSION SAVE EVERY TWO MINUTES BY DEFAULT. Disabling that STOPS the two-minute failures entirely. We&#8217;re back to file system adventures. Holy hell. THIS HAPPENS EVEN ON RAMDISK so it&#8217;s not filesystem or media specific. What the <em>hell<\/em> is going on here?<\/p>\n","protected":false},"excerpt":{"rendered":"<p>NEW READERS: IT&#8217;S NOT ABOUT THE FILESYSTEM ANYMORE BUT IT&#8217;S STILL BROKEN: SEE UPDATES AT BOTTOM OF POST. Addressing filesystem performance only partly fixed it. Thanks! Since always, I&#8217;ve had latency issues on my digital audio workstation, which is running Ubuntu Linux (currently 12.04 LTS) against a Gigabyte motherboard with 4G of RAM and a [&#038;hellip<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4,11],"tags":[],"class_list":["post-4715","post","type-post","status-publish","format-standard","hentry","category-diy","category-recording-gear"],"_links":{"self":[{"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/posts\/4715","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/comments?post=4715"}],"version-history":[{"count":0,"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/posts\/4715\/revisions"}],"wp:attachment":[{"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/media?parent=4715"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/categories?post=4715"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/crimeandtheforcesofevil.com\/blog\/wp-json\/wp\/v2\/tags?post=4715"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}