Solaris and Linux
Solaris and Linux
Posted Apr 8, 2015 21:36 UTC (Wed) by kloczek (guest, #6391)In reply to: Solaris and Linux by vonbrand
Parent article: Kernel prepatch 4.0-rc7
> I remember some discussions here about Solaris, IllumnOS, and related software. None for quite some time now (except for trolls around ZFS). In the same vein, I've seen nothing about Oracle Linux either here. Looks to me that all that is dead as a doornail.
ZFS has been introduced at November 2005. Whatever you been doing 5 years earlier you not been able to evaluate ZFS on Sun hardware.
In 2005 all Sun USparc hardware on which Linux been working was on EOS or very close to EOS.
If you are talking about compare Linux vs Solaris on very old hardware which was not supported by Solaris 10 it is really pure bollocks.
Just try to take 5 years (or more) old disks and try to complete any x86 hardware to run few benchmarks.
What is the sense of doing this? I have completely no idea.
Posted Apr 8, 2015 22:09 UTC (Wed)
by vonbrand (subscriber, #4458)
[Link] (13 responses)
Just that I won't believe that Solaris suddenly got performant after that little affair.
Posted Apr 8, 2015 22:33 UTC (Wed)
by kloczek (guest, #6391)
[Link] (12 responses)
From Solaris 10 express development cycle (pre GA 10) every microbench showing that something is slower on Solaris than on Linux is treated by Solaris support as *critical bug* and nothing has changed up to now.
First Solaris 10 GA has been released at January 2005.
In last 10 years of using Solaris I had many examples where Solaris running on the same x86 commodity hardware with paid support (even on non-Oracle/Sun HW) was cheaper than free Linux only because was possible to stay longer on the same hardware or in case Linux was necessary to buy more powerfull hardware.
If you are thinking that for example ZFS is worthless I can only tell you that few months age I've migrated on the same hardware some MySQL DB and only by switching from ext4 to ZFS was possible to observe drop down of physical IOs/s by factor up to 3 (not 3% but up to three times less).
Try to have look on:
https://www.youtube.com/watch?v=HRnGZYEBpFg&list=PLH8...
You can start from:
https://www.youtube.com/watch?v=TrfD3pC0VSs&index=6&...
FYI: at the moment OpenSolaris/IlumOS on many areas is behind latest Oracle Solaris.
Posted Apr 8, 2015 22:50 UTC (Wed)
by Cyberax (✭ supporter ✭, #52523)
[Link] (11 responses)
And of course, let's not forget the historical: http://cryptnet.net/mirrors/texts/kissedagirl.html which sums it up perfectly.
Posted Apr 9, 2015 0:04 UTC (Thu)
by kloczek (guest, #6391)
[Link] (10 responses)
Sorry do you really want to say that IO scheduling can beat using free list instead allocation structures or using COW semantics?
Maybe some quotes:
https://storagegaga.wordpress.com/tag/copy-on-write/
"btrfs is going to be the new generation of file systems for Linux and even Ted T’so, the CTO of Linux Foundation and principal developer admitted that he believed btrfs is the better direction because “it offers improvements in scalability, reliability, and ease of management”."
If you don't know btrfs is using COW semantics by default.
Just below above is next paragraph:
"For those who has studied computer science, B-Tree is a data structure that is used in databases and file systems. B-Tree is an excellent data structure to store billions and billions of objects/data and is able to provide fast data retrieval in logarithmic time. And the B-Tree implementation is already present in some of the file systems such as JFS, XFS and ReiserFS. However, these file systems are not shadow-paging filesystems (popularly known as copy-on-write file systems).
You see, B-Tree, in its native form of implementation, is very incompatible with COW file systems. In fact, the implementation was thought of impossible, until someone by the name of Ohad Rodeh came along. He presented a paper in Usenix FAST ’07 which described the challenges of combining the B-Tree concept with shadow paging file systems. The solution, as he described it, was to introduce insert/remove key into the tree structure, and removing the dependency of intra-leaf linking"
And second one few years earlier:
https://lkml.org/lkml/2008/9/27/217
"What do you mean by "copy on write", precisely? Do you mean at the
How it was possible that Ted Tso changed his mind about COW in last few years????
In real scenarios LVM snaphoting is not enough because every new LVM snapshot slows down interaction with snapshoted block device. In case ZFS you can create as many snapshots as you can ans still performance will be the same. Such effect is combination of using free lists and COW.
BTW: try to read https://rudd-o.com/linux-and-free-software/ways-in-which-...
If you want to continue this discuss please .. tell more about what you been really testing. I'm really interested what exactly you done and what kind of results you had :)
> And of course, let's not forget the historical: http://cryptnet.net/mirrors/texts/kissedagirl.html which sums it up perfectly
Of course you did't notice that above link points to text which has line:
Date: 1996/10/29
Do want to say that you been testing ZFS or Solaris 10 in 1996?
Posted Apr 9, 2015 0:18 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (8 responses)
We had a project doing lots and lots of IO interspersed with heavy multithreading. Lot's of this IO was O_DIRECT, so it didn't care a fig about COW.
And most certainly, ZFS is not the _fastest_ FS. Ext4 or XFS are usually faster in many benchmarks than either ZFS or btrfs simply because they need to do much less work for a typical IO request, doubly so for many metadata-heavy workloads.
> Do want to say that you been testing ZFS or Solaris 10 in 1996?
Posted Apr 9, 2015 1:09 UTC (Thu)
by kloczek (guest, #6391)
[Link] (4 responses)
Seems you don't understand COW. It needs to be plugged below VFS layer on block allocation stage.
Using COW is causing that random VFS layer reads will cause random reads on block layer as well.
> We had a project doing lots and lots of IO interspersed with heavy multithreading. Lot's of this IO was O_DIRECT, so it didn't care a fig about COW.
ZFS on lower layers does not care about MTs sources of IOs. From this point of view ZFS is multithread agnostics (from app point of view). In the same time ZFS as in kernel space application internally is multithreaded and able much better balance or spread even single stream of IOs across pooled storage using threads.
O_DIRECT was designed for in-place filesystems to allow IO to bypass the filesystem layer and caching. Generally bypassing ZFS caching is probably most stupidest thing which may happen if someone don't understand ZFS or don't understand what exact application is doing.
However if you really understand your application and really know what you are doing and want to obey zfs cashing you can do this without magic .. by change per volume primarycache=none or primarycache=metadata only. Everything OOTB :)
Posted Apr 9, 2015 1:30 UTC (Thu)
by kloczek (guest, #6391)
[Link] (2 responses)
Above feature was implemented by my friend when we been working together in the same company.
BTW claiming that other FSes are the same fast as ZFS as none of them can concatenate write IOs is really not true.
Posted Apr 9, 2015 1:53 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link] (1 responses)
If you mean "IO request coalescing" then Linux can do it since 2.4 days.
Posted Apr 9, 2015 2:28 UTC (Thu)
by kloczek (guest, #6391)
[Link]
No this is not about this.
If something on VFS layer will be doing two update operations in two different files and those files will be using blocks in separated locations (ie. one fort of block device and second one at the end of the same bdev) COW on block layer will cause that none of these two regions will be used or overwritten during doing random updates inside of these files and new space will be allocated and written using single (bigger) IO.
Again: consequence of using COW is high possibility transforming random write IOs workload to sequential writes characteristics. Less seeks and high possibility of concatenate VFS write IOs on doing IOs on block dev layer.
http://faif.objectis.net/download-copy-on-write-based-fil...
Posted Apr 9, 2015 1:59 UTC (Thu)
by Cyberax (✭ supporter ✭, #52523)
[Link]
> ZFS on lower layers does not care about MTs sources of IOs. From this point of view ZFS is multithread agnostics (from app point of view). In the same time ZFS as in kernel space application internally is multithreaded and able much better balance or spread even single stream of IOs across pooled storage using threads.
Surely, CoW and other tricks in ZFS give ability to easily make snapshots, do checksumming and other tricks.
Except that quite often I don't care about them - right now I'm tuning a Hadoop cluster and ext4 with disabled barriers and journaling totally wins over XFS and btrfs.
Posted Apr 9, 2015 1:44 UTC (Thu)
by kloczek (guest, #6391)
[Link] (2 responses)
Try to estimate how much income is generated by all these HPC systems for all software vendors.
Posted Apr 9, 2015 2:42 UTC (Thu)
by rodgerd (guest, #58896)
[Link] (1 responses)
Posted Apr 9, 2015 4:10 UTC (Thu)
by kloczek (guest, #6391)
[Link]
No I'm not. If I can discuss anything here it can be only financial aspects of supporting Solaris or other OS on HPC platforms from both consumers and hardware/software vendors point of views.
> you'll need to explain why Solaris is so successful that Sun went broke and Oracle refuse to break out their Solaris sales on their earnings reports.
I have no idea about real reasons of above but I know that from Sun time number of developers involved in Solaris development grow few times. I don't think that Oracle hired more developers to work on Solaris only "just for fun". Using only this fact I don't think that you suspicions are correct/relevant.
Posted Apr 10, 2015 0:22 UTC (Fri)
by cesarb (subscriber, #6266)
[Link]
> How it was possible that Ted Tso changed his mind about COW in last few years????
As far as I know, there are still no plans to implement copy-on-write in ext4. So no, he didn't change his mind.
Solaris and Linux
Solaris and Linux
Seems you've lost almost 10 years of new features of Solaris.
Such example is not quite unique.
Solaris and Linux
Yeah, sure. I worked with Slowlaris and ZFS back in 2008-2009 and it definitely was waaaay slower than Linux in many aspects. In particular, we had very concurrent workloads and Linux scheduling was vastly superior.
Solaris and Linux
So maybe you will be able to explain why ext4 has now COW?
Do you really understand impact of using COW semantics?
file level, directory level, or the filesystem level?
***We don't have any plans to implement "copy on write" in ext4***, although
you can create a copy-on-write snapshot at the block device level
using LVM/devicemapper. For many things (database backups, etc.) this
is quite suitable."
Maybe you don't care about using COW but core fs Linux developer does.
Solaris 10 GA it is Jan 2005 .. ~8 years after this email has been send.
Solaris and Linux
Errr... I haven't really parsed the meaning of this.
There's a reason why Solaris disappeared from Top500. Think about it.
Solaris and Linux
However it is not the case in case write IOs. COW can transform random VFS write operation to sequential write IOs or smaller number of such IOs. With clever IO scheduling you are able to reduce number of physical write IOs.
THIS is main advantage as on even SSDs. Doing less bigger write IOs instead of batches small IOs gives you better performance (reducing number interrupts as well).
Solaris and Linux
For example during initial import of mysql database from text dump you can switch volume settings to sync=disabled which can make such import waaaay faster :)
http://milek.blogspot.co.uk/2010/05/zfs-synchronous-vs-as...
Solaris and Linux
WTF is "io concatenation"?
Solaris and Linux
This allows reduce physical layer IO bandwidth.
Solaris and Linux
That's not an advantage if applications themselves can do it. Applications like, you know, Oracle databases.
No, nothing is "multithread agnostic". ZFS has its own complicated metadata that needs additional locking compared to simplistic filesystems like ext4/ext3.
Solaris and Linux
Solaris and Linux
Solaris and Linux
Solaris and Linux