Tridge returns to rsync [LWN.net]

Tridge returns to rsync

Posted Apr 6, 2024 22:06 UTC (Sat) by gerdesj (subscriber, #5446) [Link] (6 responses)

"We're on a mission from God"

_{(Blue's Brothers, 1980)}

Tridge returns to rsync

Posted Apr 7, 2024 0:46 UTC (Sun) by gus3 (guest, #61103) [Link] (5 responses)

They're putting the band back together.

Tridge returns to rsync

Posted Apr 7, 2024 3:43 UTC (Sun) by pepoluan (guest, #144746) [Link] (1 responses)

This is the best comment in the Internet I've ever seen.

FF built-in "font"

Posted Apr 7, 2024 14:01 UTC (Sun) by gus3 (guest, #61103) [Link]

👍

U+1F44D THUMBS UP SIGN

Tridge returns to rsync

Posted Apr 7, 2024 10:37 UTC (Sun) by geuder (guest, #62854) [Link] (2 responses)

Using discord? So I could not join. (Admittedly my contributions to any project are marginal and they are exactly zero in the case of rsync.)

Tridge returns to rsync

Posted Apr 7, 2024 17:05 UTC (Sun) by ttuttle (subscriber, #51118) [Link] (1 responses)

Surely Discord has not arbitrarily chosen to prohibit you from using their service or joining this server?

Tridge returns to rsync

Posted Apr 8, 2024 17:10 UTC (Mon) by jond (subscriber, #37669) [Link]

Perhaps not arbitrarily. Perhaps OP used a third-party client, which is apparently reason enough to ban someone.

Tridge returns to rsync

Posted Apr 6, 2024 22:10 UTC (Sat) by npws (subscriber, #168248) [Link] (30 responses)

I often find myself rsync'ing TBs of data over high latency links (Europe-Asia) and would love to parallelize the transfers. I've tried the commonly suggested methods of using GNU parallel and msrsync, but both didn't work out very well. Does anyone know of a tool to parallelize rsync, ideally while staying compatible with the other rsync options?

Tridge returns to rsync

Posted Apr 6, 2024 22:31 UTC (Sat) by jenders (guest, #98635) [Link] (3 responses)

It doesn’t meet your requirement of rsync option compatibility but I have similar high latency trans oceanic requirements and have had success achieving line rate replacing rsync with rclone.

Having rsync support transfer files over an I/O optimized queue, distributed over a thread pool along with a QUIC/UDP transport would be a dream.

Tridge returns to rsync

Posted Apr 8, 2024 12:43 UTC (Mon) by paulj (subscriber, #341) [Link] (1 responses)

Have you upped your socket buffer sizes? Defaults on Linux will inhibit high-performance on higher latency paths. You need at least 1 BDP worth of socket buffer.

BDP - Bandwidth-delay product

Posted Apr 9, 2024 18:25 UTC (Tue) by Hobart (subscriber, #59974) [Link]

https://en.wikipedia.org/wiki/Bandwidth-delay_product

(for those unfamiliar with the term as I was)

Tridge returns to rsync

Posted Apr 9, 2024 18:31 UTC (Tue) by paulj (subscriber, #341) [Link]

Also, note that QUIC is kind of misnamed... it's not that quick.

Tridge returns to rsync

Posted Apr 7, 2024 17:35 UTC (Sun) by cyperpunks (subscriber, #39406) [Link] (25 responses)

Give axel a try: https://github.com/axel-download-accelerator/axel

Tridge returns to rsync

Posted Apr 8, 2024 7:11 UTC (Mon) by Sesse (subscriber, #53779) [Link] (24 responses)

Just note: There are only two reasons why using multiple TCP connections would be faster:

1. Your TCP congestion avoidance algorithm is not working well (in which case you should probably replace it).
2. Your TCP congestion avoidance algorithm is indeed working well and has detected congestion (in which case all you're doing it pushing other, just as legitimate, traffic out).

The Internet really only works because most traffic is willing to peacefully coexist with others. Having to build in explicit limitations on transit links would probably be very expensive for all parties (and end users would not like the economic implications of the amount of control this gives ISPs).

Tridge returns to rsync

Posted Apr 8, 2024 8:28 UTC (Mon) by atnot (subscriber, #124910) [Link] (1 responses)

There is a third reason, which is some sort of per-connection bottleneck on the webserver. Either deliberate, usually as part of some scheme to make you pay for faster downloads, or because of badly programmed websites that can't keep up with the network.

But both has gotten pretty rare these days in my experience.

Tridge returns to rsync

Posted Apr 8, 2024 8:30 UTC (Mon) by Sesse (subscriber, #53779) [Link]

You could discuss whether it is legitimate to circumvent a speed restriction that was put in place deliberately. :-)

IME as coming from the other side; if the web server speed is the bottleneck (e.g. due to HDD disk speed in particular), it's just another variation on “push others out”, combined with the increased amount of disk seeking.

Tridge returns to rsync

Posted Apr 8, 2024 10:33 UTC (Mon) by PlaguedByPenguins (subscriber, #3577) [Link] (13 responses)

the reason long distance connections are low bandwidth is physics. ballpark 100KB/s is typical for trans-pacific rsync's.

Sydney to LA is a round trip distance of 24000km so at the speed of light it's ballpark 0.08s seconds between when a packet is sent and confirmation comes back. if it's a 1500byte packet then that's a 19KB/s connection at most. larger packets, TCP tuning, and vast buffers of in-flight packets on each end only get you so far.

most people split up large data files into 100+ separate chunks. I've done this many times.
it's pretty much the only way to use rsync over long distances.

Tridge returns to rsync

Posted Apr 8, 2024 10:41 UTC (Mon) by Sesse (subscriber, #53779) [Link] (4 responses)

Any kind of modern TCP certainly does not have a single 1500-byte window! I don't know why you'd think so. There is a limit of 9 kB windows if you have ancient operating systems (e.g. Windows before XP), but this is pretty much irrelevant now.

(FWIW, I've pulled many hundred megabits per second across the Atlantic, on a single TCP connection.)

Tridge returns to rsync

Posted Apr 8, 2024 12:02 UTC (Mon) by Wol (subscriber, #4433) [Link] (1 responses)

> Any kind of modern TCP certainly does not have a single 1500-byte window! I don't know why you'd think so. There is a limit of 9 kB windows if you have ancient operating systems (e.g. Windows before XP), but this is pretty much irrelevant now.

EXCEPT. I am still seeing discussions about how the only *guaranteed* packet size to get through is 1500 bytes. Mostly due to crap hardware. But a combination of (a) routers that drop fragmented packets, and (b) routers that fragment stuff over 1500 bytes, means that there are still plenty of connections that break when you try and increase that window.

Okay, the main backbones almost certainly don't have old kit that does that, but it sounds like there's still a fair bit of kit out there that does do that.

Cheers,
Wol

Tridge returns to rsync

Posted Apr 8, 2024 12:09 UTC (Mon) by Baughn (subscriber, #124425) [Link]

You’re confusing MTU for window size. Those are unrelated concepts.

Tridge returns to rsync

Posted Apr 8, 2024 12:44 UTC (Mon) by farnz (subscriber, #17727) [Link] (1 responses)

There is a limit of 9 kB windows if you have ancient operating systems (e.g. Windows before XP), but this is pretty much irrelevant now.

I thought the ancient limit was 64 KiB (2 ** 16 bytes), and that the issue with ancient OSes is that they don't support RFC 1323 window scaling, which shifts all of the "byte" counters in the TCP packet by a power of 2 set at connection establishment time, to get much larger TCP windows.

Tridge returns to rsync

Posted Apr 8, 2024 13:14 UTC (Mon) by Sesse (subscriber, #53779) [Link]

Sorry, yes, a thinko there. It's obviously 64 kB. (I probably thought of the classic jumbogram size.)

IIRC, there was also some madness in that TCP connections initiated by IE had a lower maximum window scaling option than all other connections…

Tridge returns to rsync

Posted Apr 8, 2024 11:43 UTC (Mon) by farnz (subscriber, #17727) [Link] (7 responses)

Your experience of ballpark 100 KB/s for a trans-pacific rsync does not match mine. Back when I worked somewhere that had high bandwidth trans-atlantic and trans-pacific links, I could merrily rsync at gigabytes per second across either ocean, using a single TCP connection.

The netadmins set us up with a maximum TCP window on the order of 10 GiB - at 200 ms (0.2 seconds) round trip time, that's 50 gigabytes per second, which is more than the network can actually transfer (endpoints didn't have 400G ethernet). TCP congestion control then stops the window growing arbitrarily - it's limited to what's needed to keep the link saturated, but not congested. On short distances (same DC), we'd get actual TCP windows (⇒ buffer space used) down in the kilobytes to megabytes range for saturating a link with RTTs in the 1 ms range, going to gigabytes of TCP window if saturating a long-distance link.

And note that splitting large data files into chunks is "vast buffers of in-flight packets"; it's just that you're doing it at a higher level than TCP.

Tridge returns to rsync

Posted Apr 9, 2024 4:06 UTC (Tue) by PlaguedByPenguins (subscriber, #3577) [Link] (6 responses)

> Your experience of ballpark 100 KB/s for a trans-pacific rsync does not match mine.

you're right. sorry. I should have measured, not tried to remember... today I can rsync at about 5 to 7MB/s Melbourne <-> Toronto or Amsterdam.

FWIW my end is 50Gbit/9000mtu, the other ends are 1Gbit/1500 and ??? (whatever digitalocean has). I have no control over the TCP settings at these other sites.

>And note that splitting large data files into chunks is "vast buffers of in-flight packets"; it's just that you're doing it at a higher level than TCP.

true. but the difference is that if rsync automatically used multiple streams then it would always "just work", regardless of TCP settings at the either end, and the above 5 to 7 MB/s could easily be 10x higher.

Tridge returns to rsync

Posted Apr 9, 2024 8:45 UTC (Tue) by farnz (subscriber, #17727) [Link] (5 responses)

There's a "tragedy of the commons" in here. If everyone creates multiple TCP connections just in case there's a high BDP link in the path, then all that happens is that network settings get changed to reduce the rate at which data can flow down a single connection to protect the servers in the case of low BDP links.

A far better solution is to get default TCP settings changed in places like the kernel; now that 100G links are down below the $1,000 per port mark for server cards, it would be reasonable for default TCP settings to account for BDPs around 50 gigabits (100 Gbit/s * 500 ms). And that improves things not just for rsync users, but for all TCP users, including HTTP, SSH, BitTorrent and others.

Tridge returns to rsync

Posted Apr 10, 2024 9:17 UTC (Wed) by paulj (subscriber, #341) [Link] (4 responses)

Network settings don't need to be changed, the congestion control algorithm in TCP will automatically throttle to the bottleneck, and allocate flow somewhat fairly between all the different connections (if they all get to steady state).

And yeah, the default should allow for much higher peak socket buffer sizes. The kernel does have an auto-sizing ability, but capped to a max setting that is ridiculously low for modern "enterprise" / DC networking. And it's amazing how few operators change it. Even large, very tech-savvy FANGs are running with too-low defaults.

Tech war story: I tried changing base defaults at one of these to allow inter-DC traffic to actually run at something a bit closer to what the network was capable of. The diffs just got lots of FUD-mongering comments from various machine-tier owners - despite it being a series of diffs to /slowly/ and progressively up the default over time. Eventually I just gave up trying to push for this - you just can't reason with "But what if!" FUD. Apparently it's better to let each machine-tier owning, service owner opt in themselves, by creating their own diffs to apply just to their service machines, even though 99.99% of such SWEs will never know to do so (their experience of the world is limited to the IDE on their OS X laptops usually). I bet to this day they've still - by default - got inter-DC traffic artificially throttled by the Linux limits. Sigh.

Tridge returns to rsync

Posted Apr 10, 2024 10:16 UTC (Wed) by farnz (subscriber, #17727) [Link] (3 responses)

Network settings don't need to be changed if you're happy with TCP's congestion control, but where people want to go beyond that for some less visible idea of "fair", they'll do so on the assumption that you're using multiple connections. If you're using fewer than they expect, you lose out, so you'll start using more connections. The result is a loss for everyone, because in this world, the winning move is to have more connections than anyone else so that TCP's fairness gives you an edge (if there are 10 of us accessing a server, and 9 people use 10 connections each, while I use 90 connections, TCP fairness says I should get 50% throughput at the bottleneck).

Or, of course, TCP congestion control changes so that instead of being fair between connections, it becomes fair between IP addresses, or even entire netmasks (it's a lot harder to cheat if there's fairness between /32s, then within each /32, it's fair between /48s, and within each /48, it's fair between /64s, and within each /64, it's fair between /128s, and within each IP, it's fair between connections).

If I had the time and energy to push this, I'd change the default cap on TCP autotuning to be 1 second at the highest speed the fastest running network interface in the machine is capable of, with virtual adapters assumed to be capable of 1 Mbit/s by default. That way, if you have a 100G network card in there, the TCP autotuning algorithm is allowed to raise the window to enough to let one TCP stream saturate the card; I'd then be relying on proper AQM (with associated tiny amount of in-use buffering) in the rest of the network to keep latency and window sizes under control.

Tridge returns to rsync

Posted Apr 10, 2024 13:24 UTC (Wed) by paulj (subscriber, #341) [Link] (2 responses)

Ok, I probably misunderstood your point a little then.

Yes, you could initiate some kind of "more connections" arms race, trying to claim a larger share of some shared bottleneck, in the aggregate of your multiple connections. And someone else could then do the same, etc. In the long run, you and your competitors will end up with greater overheads from connection setup, relative to payload bearing traffic. However, you would indeed crowd out anyone not using this technique.

To some extent, the web went in that direction, with web apps creating ever greater numbers of connections as web pages specified ever greater numbers of resources spread over more and more end-points. Until they started hitting NAT limits, and had to back off a bit. E.g., IVR Google Maps started having issues with parallel downloading of map tiles in certain parts of the world with very dense NATing somewhere in the 00s, and had to re-engineer.

On the other hand, arguably, there are few "shared" bottlenecks anymore. Least, for competing users. DCs may have shared bottlenecks, but these days that's often one operator, who has an interested in ensuring the end-hosts try to remain fair to each other. End-users hosts may wish to engage in competitive, unfair behaviour, however those are typically located on access links where the bottleneck is close to the user and probably actively managed (AQM) by the network operator to enforce some kind of fairness and QoS. One could even argue that perhaps congestion doesn't really matter that much anymore for Internet flow control (an argument made in, e.g., https://dl.acm.org/doi/10.1145/3626111.3628204 ) - that speaks to your point about relying on AQM, more than end-host congestion control algos.

There are also transport protocols out there that get performance by being deliberately unfair. It's very easy for someone new to transport protocols to make network code faster by - unwittingly perhaps - ignoring fairness. I know of media protocols that are - probably unintentionally - engaging in network abuse to try get more bandwidth for themselves. To a certain extent, the "benefits" of BBR were achieved by being "unfair" to CUBIC. Though, on the other hand, is it right to stymie the adoption of better techniques for congestion control, simply because older techniques don't work well in their presence?

And yes, the end-host buffer defaults need revising. But... the bike-shedding potential around any such proposed changes is fast. If you think you could get it changed though, that'd would really help higher-end-network Linux users.

Tridge returns to rsync

Posted Apr 10, 2024 13:38 UTC (Wed) by paulj (subscriber, #341) [Link]

I should clarify "deliberately unfair" versus then saying "unwittingly" or "probably unintentionally". What I mean is someone is working on some new transport protocol and they see opportunities to send more packets, more quickly, so they do that - deliberate engineering in the first order - and those changes result in nice speed-ups immediately in testing. However, in large part because they do not really fully understand the higher-order fairness effects (which may be hard to understand, even somewhat hard to quantify).

So, deliberate in the first-order changes - "wow, networking is so easy, we just unlocked big performance improvements!". Unwitting in the knock-on effects on unfairness in large-scale deployment.

Tridge returns to rsync

Posted Apr 10, 2024 14:01 UTC (Wed) by farnz (subscriber, #17727) [Link]

Unfortunately, I don't have the time and energy to push through the bikeshedding, nor do I currently have a use case (I'm back in the embedded world, in a part where 10M is "incredibly fast", and 100M exists only because 10M Ethernet is basically dead). But I've put my idea out there, and if someone reads this and gets inspired, then I'm happy for my idea to be stolen :-)

Tridge returns to rsync

Posted Apr 8, 2024 14:54 UTC (Mon) by paulj (subscriber, #341) [Link]

Theoretically you are right. In practical terms, the defaults in Linux throttle TCP on even medium-latency high-bw paths, never mind high-latency trans-oceanic.

Joining the streams: https://lwn.net/Articles/968925/

Tridge returns to rsync

Posted Apr 10, 2024 5:10 UTC (Wed) by raven667 (subscriber, #5198) [Link] (5 responses)

It isn't just congestion/contention that limits single-flow throughput, there are many points in a path where a single flow can be effectively limited to say 10Gbit due to the limitations of routing and firewall hardware, the use of link aggregation, which may have a faster interface speed but full utilization is dependent on multiplexing multiple flows. So while you _can_ get a jumbo-flow to 40G or maybe 100G with TCP with enough tuning, it is way simpler and generally more performant with less work to just open 10 connections and make full use of the buffers, input queues, LAGs, fabrics, CPUs, etc. that are horizontally scalable more then they are vertically scalable.

Tridge returns to rsync

Posted Apr 10, 2024 10:10 UTC (Wed) by paulj (subscriber, #341) [Link] (4 responses)

+1 to this comment.

With the caveat that - as per my other comments - outside of local DC traffic, it's unlikely it's network buffer capacity throttling your TCP, but simply your max socket buffer sizes on your Linux server. But the multiple connection technique works around that too.

Tridge returns to rsync

Posted Apr 10, 2024 10:12 UTC (Wed) by paulj (subscriber, #341) [Link] (3 responses)

Finally, to preempt Dave Taht, when it comes to the network you actually want the /fewest/ buffers possible. ;) You need /some/ buffering, but any additional buffering past that (hard to define) point is bad.

Tridge returns to rsync

Posted Apr 10, 2024 10:26 UTC (Wed) by Sesse (subscriber, #53779) [Link] (2 responses)

Traditionally, we expected people to have buffers for about one BDP. However, I've seen plenty of cases where things went through a switch that had something like 64 packets and that this caused loss, severely limiting the TCP speed for high-latency connections. (And these were not rinky-dink Netgear stuff from a store, they were expensive “enterprise” L3 switches with BGP and whatever. But they were not marketed as “routers” and thus did not have super-deep buffers or any reasonable sort of AQM.) And at some point, TCP pacing made a comeback and now you don't really need that much of a buffer, and this kind of random packet loss also doesn't matter as much as anymore (which is good, since Wi-Fi also tends to drop packets for similar not-really-congestion reasons).

…assuming you switch your congestion algorithm to BBR, that is. Which isn't default in any mainstream OS that I know of. :-/

Tridge returns to rsync

Posted Apr 10, 2024 13:30 UTC (Wed) by paulj (subscriber, #341) [Link] (1 responses)

Interesting. Do you remember which vendor or which chip? Was there some asymmetry? (Hot fast input port, sending out to a number of slower ports?).

Tridge returns to rsync

Posted Apr 10, 2024 13:54 UTC (Wed) by Sesse (subscriber, #53779) [Link]

I don't think this is about just one vendor, but I've certainly seen it in Cisco 2960(-S/X), 3560-X, ProCurve 2948, some Cisco 6509 line card (I don't know which, perhaps 6148?), and I have no reason to believe that newer gear in the same grade is different. When I searched around and/or asked networking people, the general gist was just “yeah, this is the cheap stuff, it's only suitable for quiet office LANs, duh”. (On some of them, I've been able to reduce the effect by changing QoS settings to move to a larger queue, e.g. some later 2948 firmware got a simple “single-queue” setting and I've done similar thing on the Ciscos through manual config.) I've seen it happen both on 10gig-to-1gig conversion and plain out 1gig (just happened to collide with other traffic coming out on the same uplink port, I guess—the drop counters on that port certainly went up and SACKs started coming back). At some point, we put a Cisco 4948 in the middle because it happened to have more buffer, and let it do downconversion from 10gig to 4x1gig to get better pacing for some TCP video streams :-)

I generally stopped worrying about this when I adopted packet pacing. First through HTB tricks (put each flow in its own bucket, pace each bucket—of course only works for video streaming or similar), later through fq and BBR. I believe some of the fq work came about partially because I whined enough at the time that people at work started listening, even though all of this was in a couple hobby projects.

Tridge returns to rsync

Posted Apr 10, 2024 13:42 UTC (Wed) by james (subscriber, #1325) [Link]

Just note: There are only two reasons why using multiple TCP connections would be faster...

Also, link aggregation¹ will normally only use one physical link for a single TCP connection. What the switches use for load balancing depends on the switches, but it's sometimes possible to include source and destination TCP/UDP port numbers in the hash (e.g. Cisco documentation, PDF, page 5).

If that's the bottleneck, multiple TCP connections would be more likely to be able to make use of all the available links.

¹ Multiple network links bonded together to act as one, probably using LACP: port channels in Cisco parlance.

Tridge returns to rsync

Posted Apr 6, 2024 22:21 UTC (Sat) by tridge (guest, #26906) [Link] (81 responses)

a huge thank you to Wayne for his great work on rsync over the past 20 years (which is when I handed over maintainership to him). At the time I told Wayne that if he ever needed to step back from maintaining rsync he should reach out to me, which is what has happened.
Note that Wayne is still very much involved at this stage, I'm just lending a hand. It's been a long time since I've worked on rsync so expect it to be a while before I'm fully up to speed.
If you want to join the rsync discord server, follow this link:
https://discord.com/invite/Avfvy9zhdp

Tridge returns to rsync

Posted Apr 7, 2024 12:56 UTC (Sun) by ebiederm (subscriber, #35028) [Link] (38 responses)

May I ask what motivated the choice of Discord?

Especially with Discord being closed source I am surprised by the choice.

Tridge returns to rsync

Posted Apr 7, 2024 14:56 UTC (Sun) by luna (guest, #166424) [Link] (6 responses)

People choose Discord because it's popular. That's really it. It's not got any features that make it uniquely well-suited to software development (and indeed it's actively hostile in a lot of ways). It's really just marketing, and it's disappointing every time a project chooses it (especially now that Discord is going to start showing ads).

Tridge returns to rsync

Posted Apr 7, 2024 15:31 UTC (Sun) by atnot (subscriber, #124910) [Link] (5 responses)

Due to it's long history with large and not always well behaving communities (i.e. gamers), discord has moderation, management a and role-based permission system that far outclasses other options. There are alternatives targeting workplaces that offer somewhat similar functionality (e.g. slack, zulip or mattermost) but they all require you to register a whole new account for the project which discourages drive-by contribution.

Whether rsync is big enough to need this I don't know, but there's very legitimate reasons why many projects chose discord.

Tridge returns to rsync

Posted Apr 7, 2024 16:03 UTC (Sun) by somlo (subscriber, #92421) [Link]

> drive-by contribution

OK, just for grins I tried following the discord link Tridge posted, to see what the fuss is all about.

It asked me for a screen name, and I picked one. It then dumped me into an endless captcha loop, which I just gave up on after 3 iterations. Maybe it didn't like my ad blocker(s). Who knows, who cares (I love rsync, use it religiously, but don't have much to contribute, so I'm not the target audience).

But so much for "drive by contributions"...

Tridge returns to rsync

Posted Apr 9, 2024 19:27 UTC (Tue) by mathstuf (subscriber, #69389) [Link] (3 responses)

> they all require you to register a whole new account for the project which discourages drive-by contribution.

Heh. I keep separate Discord accounts for different groups of related servers. I *prefer* separate accounts for separate uses instead of getting inundated by traffic/notifications from server X, Y, and Z when I'm really only there for purpose P.

Tridge returns to rsync

Posted Apr 11, 2024 11:17 UTC (Thu) by viiru (subscriber, #53129) [Link] (2 responses)

> Heh. I keep separate Discord accounts for different groups of related servers. I *prefer* separate accounts for
> separate uses instead of getting inundated by traffic/notifications from server X, Y, and Z when I'm really only there
> for purpose P.

How do you handle Discord's requirement for a working SMS-accepting phone number per account?

Tridge returns to rsync

Posted Apr 11, 2024 14:10 UTC (Thu) by mathstuf (subscriber, #69389) [Link] (1 responses)

I've never encountered that? AFAIK, Discord has no knowledge of my phone number (I've never given it).

Tridge returns to rsync

Posted Apr 11, 2024 16:12 UTC (Thu) by excors (subscriber, #95769) [Link]

Discord server admins can choose different levels of verification, with the highest requiring a verified phone number (plus a verified email address and being a server member for at least 10 minutes), which is useful for large public servers that have a problem with spam bots, scammers, trolls, etc, as it lets moderators ban people more effectively. (Source: https://support.discord.com/hc/en-us/articles/216679607--...)

Discord might also require verification if it thinks you're behaving in a suspiciously bot-like manner (e.g. rapidly joining many servers and sending many messages, or using "third party clients or modifications to Discord"). (https://support.discord.com/hc/en-us/articles/61817268882...)

If you haven't joined large servers and haven't triggered the suspicion heuristics then you probably won't have encountered the phone verification.

Tridge returns to rsync

Posted Apr 7, 2024 19:52 UTC (Sun) by syrjala (subscriber, #47399) [Link] (29 responses)

I seem to remember a guy that didn't like the closed source nature of BitKeeper and started to reverse engineer it. I wonder if that same guy could do something about Discord...

Tridge returns to rsync

Posted Apr 7, 2024 20:11 UTC (Sun) by atnot (subscriber, #124910) [Link]

There already are such things, the problem is that it's against ToS, discord APIs change a bunch and they are, as I understand it, fairly strict about banning people for using third party clients, ostensibly because of spam.

discord, bitkeeper and choices

Posted Apr 8, 2024 0:29 UTC (Mon) by tridge (guest, #26906) [Link] (27 responses)

I think you may have misunderstood my motivation for working on sourcepuller (the tool I wrote that was compatible with bitkeeper). At the time the bitkeeper license included terms that would have prevented me from ever working on a source code control system in the future if I agreed to the bitkeeper license. I wanted to work on the extfs tools which was only available at the time via bitkeeper. I could not agree to a license which restricted what I worked on in the future, so I wrote sourcepuller to allow me to interact with bitkeeper repos (of which there were many at the time) without restricting my future options on what I worked on.
The fact that this led Linus to write git which is vastly better than sourcepuller is a really nice bonus.
I have often worked on the boundary of free and proprietary software, right from when I was dosemu maintainer for a while, to Samba and many other projects. I make all my projects open, usually GPLv3, but I don't mind using closed tools to help me work more efficiently, even though I am creating free software myself.
I know some people take a stronger stance and refuse to use any closed tools at all. That is a great choice if that is what you want to do. It isn't how I work, and it never has been.

discord, bitkeeper and choices

Posted Apr 8, 2024 18:44 UTC (Mon) by syrjala (subscriber, #47399) [Link]

Thanks for the clarification. I didn't really have any real clue about your motivations, but thought the opportunity for a tongue in cheek comment was too good to pass up ;)

discord, bitkeeper and choices

Posted Apr 9, 2024 0:59 UTC (Tue) by denials (subscriber, #3413) [Link]

tridge, thanks for everything you've done, and continue to do, for the community. So much of what I've accomplished in my career (which isn't much in comparison!) has been enabled by your work.

discord, bitkeeper and choices

Posted Apr 9, 2024 6:19 UTC (Tue) by adamg (guest, #42260) [Link] (8 responses)

That makes me wonder - if there was a clause "if you use bitkeeper, you cannot work on a source control system in the future", and Linus used bitkeeper for a while, how was he allowed to create and work on git? Was it "git is not source control system, it's a version control system" or that such clause in T&C was otherwise hard to enforce (limiting human rights?)?

discord, bitkeeper and choices

Posted Apr 9, 2024 6:50 UTC (Tue) by tridge (guest, #26906) [Link] (7 responses)

I've wondered about that too, although the bitkeeper license was changing rapidly, and it may be that the clause I saw wasn't in the version Linus used.
One version of the license is here:

https://lwn.net/Articles/130758/

(d) No free use for competitors: Notwithstanding any other
terms in this License, this License is not available to You
if You and/or your employer develop, produce, sell, and/or
resell a product which contains substantially similar capa-
bilities of the BitKeeper Software, or, in the reasonable
opinion of BitMover, competes with the BitKeeper Software.

discord, bitkeeper and choices

Posted Apr 9, 2024 8:53 UTC (Tue) by malmedal (subscriber, #56172) [Link]

I believe Linus did not start developing git until the BitKeeper license had been revoked?

discord, bitkeeper and choices

Posted Apr 9, 2024 10:26 UTC (Tue) by chris_se (subscriber, #99706) [Link] (5 responses)

The text itself reads to me (IANAL) more along the following lines: should you or your employer start to work on version control, your BitKeeper license will not be valid anymore. I don't see "you can never again in your life work on VCS" here, and I highly doubt that something that extreme would have been legally enforceable anywhere. (That would go way beyond even employee non-competes, which already have limits.)

Not being allowed to work on other VCS at all in order to be able to contribute to the Linux kernel is bad enough in and of itself though, and I can very much understand why someone wouldn't want to accept those conditions.

discord, bitkeeper and choices

Posted Apr 9, 2024 13:37 UTC (Tue) by kleptog (subscriber, #1183) [Link]

Suppose your employer is a university that employs thousands of people researching all sorts of stuff, how are you supposed to even evaluate if the licence is available to you?

I get what the licence is trying to do, but too many terms are just fuzzy when applied to the real world. ISTM these kinds of clauses work fine in one-on-one contract negotiations, but fail in copyright licences.

discord, bitkeeper and choices

Posted Apr 9, 2024 20:32 UTC (Tue) by tridge (guest, #26906) [Link] (1 responses)

I don't know if that linked license version was the one I saw and refused to accept. Unfortunately I can't find a copy of the license I read at the time. I tend to read licenses for software before I hit accept them, and I remember that the bitkeeper license was one I couldn't accept.
I should note that I could have emailed Ted Ts'o and asked him for a copy of the extfs libs sources, but I realised this was a more general problem and I thought it useful to do something about it. I would also note that I had discussed it with Linus some time before (when he was visiting Canberra after an LCA conference in early 2003 I think?) and Linus was quite positive about creating a free tool that could work with bitkeeper repos.

a bit of email searching later .... I've found the initial email I sent to Linus and Andrew Morton about creating sp in 2005. A long conversation between Linus, Larry and me came after that.

To: torvalds@osdl.org, akpm@osdl.org
Cc: wookie@osdl.org
Date: Thu, 24 Feb 2005 09:24:39 +1100

Linus and Andrew,

As I think Tim has already mentioned to you, I have written a pair
of C libraries that do this:

libsp: talks the bitkeeper remote protocol
libsccs: parses bitkeeper style SCCS files with all details

On top of that I have written a little tool called 'sp' that offers a
command line interface to the libraries. You can do things like "sp
clone URL", "sp co", "sp clean", "sp revlog" etc. I'm working on "sp
pull".

The idea behind doing this is provide a missing feature from
BitKeeper, the ability to export to other formats without losing any
information. I'm not planning on writing a full bk replacement - I
just want to fill in that missing feature, which I think is a
significant stumbling block to anyone who does want to write a good
replacement for bk.

In essence I am doing the opposite of the "cvs-to-bk" tool that Larry
wrote to allow people to migrate away from cvs. I am writing the basic
tool needed to allow people to migrate away from bk if they want to
for whatever reason, without losing any of their meta data.

Note that I am not subject to the bkl as I am not a bitkeeper user,
and I didn't use bitkeeper in writing the above tools (I worked out
what I needed to know by poking at some public servers).

The problem is what Larry's reaction to my releasing this tool might
be. The "anyone in your company" part of the bkl means that he could
claim that you two lose your licenses. Do you think he is likely to do
that? He could also decide to shutdown linux.bkbits.net if he was
upset enough.

So, how do you think he will react?

Cheers, Tridge

discord, bitkeeper and choices

Posted Apr 11, 2024 9:03 UTC (Thu) by adamg (guest, #42260) [Link]

Interesting email, thanks for sharing.

discord, bitkeeper and choices

Posted Apr 11, 2024 9:01 UTC (Thu) by adamg (guest, #42260) [Link] (1 responses)

I thought it was more restrictive, that by using bitkeeper one learned bitmover's IP, thus the restriction was meant to be valid even after license is revoked. But this would probably be illegal construct, so likely it was just restriction for as long as one was using bitkeeper.

discord, bitkeeper and choices

Posted Apr 12, 2024 3:01 UTC (Fri) by tridge (guest, #26906) [Link]

Larry was quite aggressive in enforcing what he thought his license meant, regardless of the legality of it. At one point he threatened to "have me arrested" if I ever visited the USA. He had no basis to do so as I had never agreed to the bkl and what I did was completely legal, but it could still have caused me a lot of inconvenience trying to fight him, so I cancelled one trip to the US. My wife was attending a conference in Hawaii and I had planned to go along to do some tourism. I cancelled my trip as I just didn't want to deal with Larry's craziness. I still haven't seen Hawaii :-)
It seems like Larry had mellowed somewhat since those times.

discord, bitkeeper and choices

Posted Apr 10, 2024 4:13 UTC (Wed) by asheplyakov (guest, #156812) [Link] (15 responses)

Once upon time Microsoft used to claim that 'open source is a cancer'.
Eventually they learned how to cooperate, and there's a Linux kernel (wsl2) in every Windows installation now.

Perhaps it's time for the open source community to do the same.
(For starters stop spreading "proprietary software is immoral" FUD/nonsense).

discord, bitkeeper and choices

Posted Apr 10, 2024 11:45 UTC (Wed) by pizza (subscriber, #46) [Link] (13 responses)

> Eventually they learned how to cooperate, and there's a Linux kernel (wsl2) in every Windows installation now.

They "learned to cooperate" because it makes them a _lot_ of money.

(And then there's the minor detail of effectively making Linux "just another feature" of Windows; the classic EEE play.
I'm willing to bet there are far more _active_ WSL installations than there are native Linux desktops, complete with "Linux" software that can only function under WSL due to needing APIs that are passed through to Windows)

> Perhaps it's time for the open source community to do the same.

Perhaps when the open source community starts getting paid commeasurate with the value they provide?

discord, bitkeeper and choices

Posted Apr 11, 2024 15:55 UTC (Thu) by flussence (guest, #85566) [Link] (1 responses)

WSL2 isn't competing with Linux desktops, it's competing with Macbooks. They're going for the market of yuppies that spend 200 hours decorating their terminal prompt and terminal editor while blogging about the process all on a disposable $2k laptop. Those people are never going to change that workflow to use cmd.exe or powershell, and MS knows it.

discord, bitkeeper and choices

Posted Apr 11, 2024 18:59 UTC (Thu) by pizza (subscriber, #46) [Link]

> WSL2 isn't competing with Linux desktops, it's competing with Macbooks. They're going for the market of yuppies that spend 200 hours decorating their terminal prompt and terminal editor while blogging about the process all on a disposable $2k laptop.

No, WSL2 is going for the corporate market, to ensure IT departments have no reason to buy or deploy anything other than Windows for non-C-suite folks. (And of course that includes Macs)

discord, bitkeeper and choices

Posted Apr 12, 2024 0:49 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link] (10 responses)

> (And then there's the minor detail of effectively making Linux "just another feature" of Windows; the classic EEE play.

The "EEE" has been overhyped for at least 2 decades. It almost never works, unless you have a lot of market force and you keep the changes proprietary.

Basically, you need to convince everyone to use your changed version (which needs to be genuinely better than the status quo), while preventing others from easily duplicating its features. So EEE could have worked with Java, and it almost succeeded with IE6. But it has zero successes afterwards.

How would it look like in the case of WSL? I just don't see it. What can they do that would make users switch to Microsoft Linux?

discord, bitkeeper and choices

Posted Apr 12, 2024 1:25 UTC (Fri) by pizza (subscriber, #46) [Link] (9 responses)

> How would it look like in the case of WSL? I just don't see it. What can they do that would make users switch to Microsoft Linux?

Because (1) it's convenient; (2) it's managed by group policies covering all corporate buzzword salad security/management BS; (3) hardware compatibility no longer matters -- including native DirectX from within Linux[*] (4) nearly entirely painless setup (no dual booting) and data passthrough, (5) You don't have to pay extra for Virtualbox or VMWare or other stuff, etc etc.

It's a breathtakingly brilliant play, especially pushing APIs that only work when running under WSL.

[*] https://devblogs.microsoft.com/directx/directx-heart-linux/

discord, bitkeeper and choices

Posted Apr 12, 2024 1:52 UTC (Fri) by Cyberax (✭ supporter ✭, #52523) [Link]

People are switching to WSL from the regular desktop Linux, that has never been anything but a footnote in the overall market (thanks, GNOME and KDE folks for that!). This market is so secondary right now, that Microsoft is literally switching it to free-to-play model with microtransactions^W ads for monetization.

The main Linux domain are servers and the growing AI/ML market, and Microsoft has nothing to offer there. The WSL itself is designed to stave off Windows' obsolescence in that area, with DirectX passthrough primarily designed for the AI/ML workloads (and not the graphical UI).

discord, bitkeeper and choices

Posted Apr 12, 2024 3:51 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (7 responses)

WSL2 is much less manageable than an actual Linux distribution - it's not possible to enforce a specific distro image (you can ship your own, but you can't prevent users from booting other ones), so you're not guaranteed any management within the Linux image, and there's no way to pass through access to the host certificate store so you can't tie a given WSL instance to its host. To be fair, Crostini in ChromeOS has the same issues (hence the somewhat ironic situation of Google banning the use of Crostini in corp environments).

discord, bitkeeper and choices

Posted Apr 12, 2024 20:08 UTC (Fri) by pizza (subscriber, #46) [Link] (6 responses)

> WSL2 is much less manageable than an actual Linux distribution - it's not possible to enforce a specific distro image (you can ship your own, but you can't prevent users from booting other ones), so you're not guaranteed any management within the Linux image.

It's a _lot_ more manageable than a bare-metal (or dual-boot) Linux installation would be.

All the usual corporate checkboxes can be maintained (Bitlocker/managed FDE, enforced firewall/intrusion detection/data exfiltration suites, bla bla bla). Users get their own wholly-contained sandbox to be root in, but are prevented from mucking around with the super important stuff like the mandated backgrounds on the lock screen.

discord, bitkeeper and choices

Posted Apr 12, 2024 21:37 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (5 responses)

I was involved in helping manage Google's bare-metal corp Linux deployment, which was somewhere on the order of 100,000 systems. We had no difficulty enforcing the same level of management as we did on Windows or Mac. WSL2 is an extremely convenient opaque object that has access to host files and can't meaningfully be monitored by any local agents. I agree that it has all sorts of conveniences, and some thought has been put into how it interacts with complicated corporate environments, but without the ability to enforce specific images the level of control you have is limited to what Windows enforces on it.

discord, bitkeeper and choices

Posted Apr 12, 2024 22:21 UTC (Fri) by pizza (subscriber, #46) [Link] (4 responses)

> [Google] had no difficulty enforcing the same level of management [of bare-metal Linux] as we did on Windows or Mac.

...I don't think Google should be used as an example of the capabilities of a typical medium-to-large corporate IT department.

discord, bitkeeper and choices

Posted Apr 12, 2024 23:03 UTC (Fri) by mjg59 (subscriber, #23239) [Link] (3 responses)

We've got pretty much the same set of capabilities at my current gig, which is at the low end of medium sized.

discord, bitkeeper and choices

Posted Apr 16, 2024 14:24 UTC (Tue) by marcH (subscriber, #57642) [Link] (2 responses)

I think theses "capabilities" are more about knowledge and experience than about company size. Sysadmins across the world and industry have been managing Windows for decades. I wouldn't say they're "good" at it considering how incredibly complex hence unmanageable Windows is but for sure they feel "comfortable" doing it.

discord, bitkeeper and choices

Posted Apr 16, 2024 15:06 UTC (Tue) by pizza (subscriber, #46) [Link] (1 responses)

More than knowledge/experience, it's also a matter of _willingness_ -- $dayjob-1, you had to get director-level signoff (After a complete legal review) to use Linux in _anything_, and the entire (Windows-only) IT department was (doubly) outsourced and run with an attitude that made the "computer says no" trope downright friendly in comparison. Small wonder my research group built our own shadow IT organization, complete with our own internet connection (faster, more reliable, _and_ more secure as it turned out), just so we could do the jobs we were hired for.

At $dayjob-2, despite most of the couple-thousand-strong engineering staff requiring desktop Linux applications to do their jobs, you had to get special Group-manager-level approval to get a native Linux desktop versus remoting into the (massive) compute farm. And there was no officially-sanctioned portable Linux solution (in part because the VPN was Windows-only) Which is a problem when you need direct-attached peripherals and there's a >1s ping at the customer site in Western Elbonia..

discord, bitkeeper and choices

Posted Apr 16, 2024 15:46 UTC (Tue) by farnz (subscriber, #17727) [Link]

And there's a social aspect that gets lost, too. People who prefer using Linux to other OSes correlate well with people who find ways round stupid IT policies that don't work for the business (as you did), and also are more likely to expect a decent explanation when they ask why an IT policy is set the way it is.

That, in turn, puts you in conflict with an IT department that's used to being able to control the business rather than serve it; you're going around their restrictions when they get in the way instead of following whatever arcane and unmanageable process they've built to get the restriction changed (e.g. requiring legal review and sign-off at director level), and you're more likely to ask hard questions about why those restrictions exist when you do go through the arcane process - which, in turn, puts them in the hot seat where they have to actually justify what they do.

In companies where the IT department isn't incompetent (I've worked in some), you don't have these problems, because they consider themselves as supporting the business needs, and you wouldn't be asking for Linux if there wasn't a good business need for it.

discord, bitkeeper and choices

Posted Apr 10, 2024 12:07 UTC (Wed) by Wol (subscriber, #4433) [Link]

> Perhaps it's time for the open source community to do the same.
> (For starters stop spreading "proprietary software is immoral" FUD/nonsense).

In case you hadn't noticed, the Open Source community has NEVER spread that FUD.

The Open Source community is pragmatic, Free and "proprietary" software will always co-exist, because some rather important software wouldn't exist if it wasn't paid for ...

Cheers,
Wol

Tridge returns to rsync

Posted Apr 8, 2024 16:24 UTC (Mon) by wtarreau (subscriber, #51152) [Link]

And not only closed source, but refusing me access without providing an email despite having filled two captchas:

"Verification Required
We've detected something out of the ordinary going on, and we want to keep you and the community safe.
To continue using Discord we will need to verify your account. [Verify by Email]."

(not that I had to re-type this manually since this crap doesn't permit you to select the text).

Discord is one of the worst garbage the internet has ever invented. It confiscates tons of
useful information that could be shared openly, that will never be indexed by search engines
and be lost forever.

It was only meant for teenagers playing games and should never have been adopted by people who
want to share other things than insults.

BTW: thanks to the rsync team for continuing to maintain one of the most useful tool out there that syncs up data everywhere without users even being aware of it!

Tridge returns to rsync

Posted Apr 7, 2024 14:58 UTC (Sun) by luna (guest, #166424) [Link] (40 responses)

I would humbly request that you choose something other than Discord. They've started showing ads among other things and it's a closed platform. Matrix does everything you need from Discord, and the Matrix developers don't need to follow the whims of Chinese investors (i.e. Tencent).

Tridge returns to rsync

Posted Apr 7, 2024 15:15 UTC (Sun) by jzb (editor, #7867) [Link] (39 responses)

We ask that comments be polite and respectful, and nationalism is not something that aligns with those values. It's also worth noting that Discord has something like 34 investors, many of which are U.S.-based.

Tridge returns to rsync

Posted Apr 7, 2024 18:24 UTC (Sun) by epa (subscriber, #39769) [Link] (3 responses)

Really, editor? It’s mere “nationalism”? Chinese developers themselves would be the first to acknowledge that the Party keeps a tight grip on tech companies and attempts to control online discourse. That is, if they were allowed to freely give their views.

Tridge returns to rsync

Posted Apr 7, 2024 22:40 UTC (Sun) by gerdesj (subscriber, #5446) [Link]

That comment would still work fine without a nationality being mentioned.

This is LWN not HN.

Big Jon and co deliver original journalism and are gracious enough to provide a comments section for us to discuss it. For something like 25 years I've been hanging around this parish and barring a very few examples, that discussion has been conducted with good humour and civility.

Mind you I did manage to get Rusty Russel to call me a troll back in the day ...

Tridge returns to rsync

Posted Apr 8, 2024 3:11 UTC (Mon) by PengZheng (subscriber, #108006) [Link] (1 responses)

As a Chinese developer who keeps reading LWN articles for 7 years, I disagree with your comments.

Please do not speak on behalf of people you do not know.

Tridge returns to rsync

Posted Apr 8, 2024 6:52 UTC (Mon) by epa (subscriber, #39769) [Link]

Thanks, noted.

Tridge returns to rsync

Posted Apr 7, 2024 20:39 UTC (Sun) by Wol (subscriber, #4433) [Link] (11 responses)

> We ask that comments be polite and respectful, and nationalism is not something that aligns with those values.

May I ask why? BLIND patriotism (aka "my country right or wrong") is bad, but given the war in Europe would you object to anti-Russian sentiment?

What matters is to separate the government from the people. I'm well known, I suspect, for being somewhat America-hostile but Americans in general are nice people. I just get riled when they assume my values align with theirs.

And this sentiment was merely asking "do we want to get involved with a bunch of people who's values we don't agree with?". Fair question! You may not care, you may care deeply, but you need to discuss it respectfully. Clamping down because you don't like the discussion is not wise. Clamping down because the participants are losing it is fair game.

Take a leaf out of PJ's book - discussion should be respectful, you should have *facts* to back up your opinions, and it should be the sort of discourse that PJ would be happy to host in her own front room. Yeah, I fell foul of those rules occasionally, but you felt free to discuss most things because you knew the moderators would delete anything disrespectful. (And yes, I know LWN doesn't delete stuff ...)

Cheers,
Wol

Tridge returns to rsync

Posted Apr 8, 2024 7:34 UTC (Mon) by LtWorf (subscriber, #124958) [Link] (10 responses)

> would you object to anti-Russian sentiment?

I'd very much object to that yes.

How can we achieve peace with russia while fostering anti-russian sentiment?

Tridge returns to rsync

Posted Apr 8, 2024 8:45 UTC (Mon) by NAR (subscriber, #1313) [Link] (9 responses)

You cannot achieve peace with Russia. Russia needs to loose the war like Germany and Japan did 80 years ago in order to have peace.

Tridge returns to rsync

Posted Apr 8, 2024 9:35 UTC (Mon) by farnz (subscriber, #17727) [Link]

Russia losing the war like Germany and Japan did 80 years ago is achieving peace with Russia - or are you genuinely claiming that Japan and Germany have continued to be at war for the last 80 years?

Achieving peace with Russia may well be hard (I personally believe that it'll need a change in leadership - not just Putin, but all of his allies in government - and I'm not sure how Russia gets that), but it's not impossible. It's just unlikely to be managed by negotiation with the Russian government of today; however, individual Russians are (mostly) not directly involved in that, and saying that you won't work with someone whose parents fled Russia in 2013 because they could see the direction the government was taking and didn't like it is anti-Russian sentiment, and unproductive.

There is a reason that Russia is taking steps to limit communication between Russian ex-pats and emigrants and Russians remaining in the Russian Federation, after all…

Tridge returns to rsync

Posted Apr 8, 2024 12:45 UTC (Mon) by daroc (editor, #160859) [Link]

Joe and I are both new to helping moderate LWN, but I think this is pretty off-topic for an article about rsync. Let's stop discussing international relationships, and avoid making generalizations about people based on nationality. (Which you have not done, but which the general discussion in this thread keeps verging on)

If anyone has questions or concerns about our moderation policy -- or suggestions for how to improve it -- please feel free to write to us privately at lwn@lwn.net.

Anti-russian comments help Russia to win the war

Posted Apr 9, 2024 3:54 UTC (Tue) by asheplyakov (guest, #156812) [Link] (6 responses)

Hello from Donetsk!

FYI: in 2023 about 500.000 volunteers have entered the military service in Russia.
"Russia must loose the war" comment is the best way to encourage even more russians to join the army.
Thus (paradoxically) anti-russian comments in the internet (especially on a technical site like lwn) actually help Russia to win the war.

Anti-russian comments help Russia to win the war

Posted Apr 9, 2024 8:18 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (5 responses)

> FYI: in 2023 about 500.000 volunteers have entered the military service in Russia.

Volunteers or mercenaries?

Anti-russian comments help Russia to win the war

Posted Apr 9, 2024 8:33 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (4 responses)

I think you can't be mercenary if you fight for your own country.

After all, all soldiers get paid…

Anti-russian comments help Russia to win the war

Posted Apr 9, 2024 8:42 UTC (Tue) by Wol (subscriber, #4433) [Link]

So the rumour goes, a lot of soldiers are in fact "criminals" (petty, serious, framed, dunno ...) who were offered a pardon in return for 6 months service.

Cheers,
Wol

Anti-russian comments help Russia to win the war

Posted Apr 9, 2024 15:34 UTC (Tue) by Cyberax (✭ supporter ✭, #52523) [Link] (2 responses)

"Volunteers" are paid about 10x-15x the average salary in Russia.

Anti-russian comments help Russia to win the war

Posted Apr 9, 2024 20:58 UTC (Tue) by LtWorf (subscriber, #124958) [Link] (1 responses)

I don't doubt that. But it's the same in every country I guess. Italian soldiers are paid much more than the average salary when they are on mission. I presume it's the same for the rest of the NATO countries.

You do need some kind of incentive to push people to risk their lives willingly.

Off topic

Posted Apr 9, 2024 21:05 UTC (Tue) by corbet (editor, #1) [Link]

...and this is all far off-topic for LWN. Perhaps we can end this thread here?

Thank you.

Tridge returns to rsync

Posted Apr 7, 2024 20:51 UTC (Sun) by dskoll (subscriber, #1630) [Link] (2 responses)

"Nationalism"? I think it's pretty clear that the Chinese Communist Party is very much opposed to freedom, something that Free Software is supposed to promote. And it's also pretty clear that Chinese companies have no choice but to submit to the will of the CCP.

I think it's enlightened self-interest for Free Software projects to be very careful about relying on infrastructure that has any sort of connection to China, just so long as China is not a free country.

Tridge returns to rsync

Posted Apr 8, 2024 8:06 UTC (Mon) by dullfire (guest, #111432) [Link]

Additionally, that's not even the correct usage of "nationalism". Criticizing a single government[0] without motioning or implying another is not, and can not be, by definition, "nationalism".

[0] There was no criticism in this case, just implication of references to some.

So far it's you who bashes people for their free choice ...

Posted Apr 9, 2024 12:26 UTC (Tue) by asheplyakov (guest, #156812) [Link]

... not the CPC (unless you are the CPC member, that is).

Tridge returns to rsync

Posted Apr 8, 2024 0:31 UTC (Mon) by luna (guest, #166424) [Link] (18 responses)

I was polite and respectful. I didn't make any nationalist comments. I will, however, be canceling my subscription given this comment.

Further, your defense of proprietary software when there are fully open alternatives is worrying.

Escalation to political discussion

Posted Apr 8, 2024 4:56 UTC (Mon) by makendo (guest, #168314) [Link] (17 responses)

The escalation from a discussion on Discord policy to "China" can be irritating for Chinese readers of LWN, many of which holds radical political opinions aligned against the western mainstream. This is not a speech or thought control issue.

Escalation to political discussion

Posted Apr 8, 2024 8:08 UTC (Mon) by epa (subscriber, #39769) [Link]

If the editor had said that, fair enough. But it's not being impolite or disrespectful to raise these concerns, and it's not simply nationalism.

Personally, I am not one of those who'd refuse to use Discord because it might somehow be controlled or monitored by the Communist Party. I think that is excessively tinfoil hat, although it's certainly well documented that Internet services within mainland China (and nowadays the special administrative regions too) are under the Party's thumb. And I can see that diverting a discussion about rsync into one about Internet censorship (or tracking dissidents overseas) is going off-topic. I just don't think the editor is right to dismiss such concerns as nationalistic or to adopt a sanctimonious tone of "polite and respectful" when what's really meant is that we don't discuss those things here.

Escalation to political discussion

Posted Apr 8, 2024 8:25 UTC (Mon) by dullfire (guest, #111432) [Link] (15 responses)

> The escalation from a discussion on Discord policy to "China" can be irritating for Chinese readers of LWN, many of which holds radical political opinions aligned against the western mainstream. This is not a speech or thought control issue.

I think calling their views "radical political opinions" would be just as, or more, offensive as mentioning that using a communications platform that likely is influenced by China may not be in a libre software project's best interest.

So: "Don't talk about things that might irritate Chinese LWN readers", and "This is not a speech [...] control issue".... That does not strike you as a glaring contradiction?

I mean sure: Let us not be assholes. That is fine. I don't think anyone here disagrees with that. This seems to be suggesting that talking about parts of a project governance that the speaker believes is not in the best interest of libre software, just because it also involved concerns about a specific nation, is taboo. If this is the case, just say you don't want people to talk about libre software here, and that you only want "open source software" related conversations.

Escalation to political discussion

Posted Apr 8, 2024 9:07 UTC (Mon) by makendo (guest, #168314) [Link]

Clarification: I am Chinese, and my opinion is same as PengZheng here. "Speech control" refers to allegations on the Chinese government.

Escalation to political discussion

Posted Apr 8, 2024 9:30 UTC (Mon) by farnz (subscriber, #17727) [Link] (13 responses)

I think the biggest issue here is attributing views or outcomes to people (including Discord's owners) simply on the basis of nationality. If Tencent has done something bad, then by all means call out Tencent, just as we call out Meta, Apple and other companies - your actions aren't above criticism just because there might be state influence over you - but don't attribute ill-intent to an entity because it's Chinese (or American, or European, or Indian).

If you can point to somewhere where Discord's done something wrong, that's a good conversation to have, just as a conversation about something Google has done wrong can be worthwhile. But a conversation based purely on stereotypes isn't helpful, because it carries with the implication that being Chinese (or American, or European, or Indian) is the problem, and not the behaviours that you find problematic.

Or, to put it another way, talk about the parts of Discord's governance that are problematic; from that, people can draw their own conclusions about the causes of Discord's bad governance, and infer what they like about a project's use of Discord. Just point to behaviour problems, not to a "problem" nationality.

Escalation to political discussion

Posted Apr 8, 2024 19:57 UTC (Mon) by roc (subscriber, #30627) [Link] (12 responses)

> don't attribute ill-intent to an entity because it's Chinese (or American, or European, or Indian).

Chinese companies are beholden to the Chinese government (the CCP) to an extent that American and European companies (and Indian I think, but I'm less sure about that) are not beholden to their governments. This is true both in principle (the letter of the law) and in practice (see many examples of Chinese CEOs just disappearing when they get offside with the government).

Furthermore, American, European and Indian governments are accountable to their citizens in ways that the CCP is not, again both in principle and in practice.

So, whether or not the individuals involved have good intent, it is logical to be extra wary of Chinese companies. It is certainly reasonable to ask whether, if the CCP demanded Tencent provide access to the private data of Discord users, Tencent would be able to deliver such data.

Nationality as your "mark of evil"

Posted Apr 9, 2024 8:56 UTC (Tue) by farnz (subscriber, #17727) [Link] (11 responses)

Right, but all of what you're saying is theoretical issues with the CCP, until something actually happens; the CIA, NSA, MI5 and MI6 have all gone off-piste in the sorts of ways that you say that the CCP enshrines in government. The US government, for example, under the guise of "national security letters" has demanded all sorts of data from US companies in secret, with no way for those companies to legally tell anyone that they're sharing data with the US government.

Why is it OK for the US government to do this, but not for the Chinese government? If it's not OK for the US government to do this (and yet it does do it), why is partial Chinese ownership of Discord a big problem, but partial US ownership of Discord not a big problem?

These sorts of questions are why I suggest focusing on actual events, not on nationality. Every government does bad things to some extent or other, and it's reasonable to focus on the bad things that are actually happening. Otherwise, this is one short step away from racism - being biased against a product because some of its developers are of the wrong ethnic origin for you.

And note, too, that the big American firms like Google have Chinese nationals employed there; there is nothing that guarantees you safety from Chinese spy agencies, since the fact that Chinese nationals who can get jobs at Google, Facebook, Apple etc exist means that the spy agencies can infiltrate them the way Snowden revealed the NSA had infiltrated them.

Nationality as your "mark of evil"

Posted Apr 9, 2024 16:06 UTC (Tue) by Wol (subscriber, #4433) [Link]

> And note, too, that the big American firms like Google have Chinese nationals employed there; there is nothing that guarantees you safety from Chinese spy agencies

Given the number of aristocratic Brits who ended up spying for the Soviets in the 50s and 60s, who's to guarantee that the CIA et al aren't full of American spies for China ... ?

Cheers,
Wol

Nationality as your "mark of evil"

Posted Apr 10, 2024 6:29 UTC (Wed) by roc (subscriber, #30627) [Link] (9 responses)

> all of what you're saying is theoretical issues with the CCP

Not at all theoretical. See e.g.
https://foreignpolicy.com/2023/02/28/wechat-censorship-ch...

> The US government, for example, under the guise of "national security letters" has demanded all sorts of data from US companies in secret, with no way for those companies to legally tell anyone that they're sharing data with the US government.

I'm no fan of NSLs, but US companies can and sometimes do fight NSLs in court, and they can and sometimes do publish aggregate information about how NSLs are being used. E.g. https://transparencyreport.google.com/user-data/us-nation...
The situation for Chinese companies is much less favourable.

> Why is it OK for the US government to do this, but not for the Chinese government?

I don't think NSLs are "OK". They are, however, more limited in power than what the CCP routinely does.

> And note, too, that the big American firms like Google have Chinese nationals employed there; there is nothing that guarantees you safety from Chinese spy agencies, since the fact that Chinese nationals who can get jobs at Google, Facebook, Apple etc exist means that the spy agencies can infiltrate them the way Snowden revealed the NSA had infiltrated them.

Google's systems are designed to protect user data from malicious Google employees. I work at Google, and if I wanted to hack into user data I don't think I would be able to get away with it. I believe the same is true at Apple, but I'm not sure about Facebook.

Furthermore, there is a big difference between data exfilitrated by a small group of spies --- or provided to the US government via a NSL --- and the company simply providing unfettered access to that data as a matter of policy.

Nationality as your "mark of evil"

Posted Apr 10, 2024 8:55 UTC (Wed) by farnz (subscriber, #17727) [Link] (8 responses)

That's not Discord - it's a different service - and if people were talking about using WeChat, it'd be a whole different matter.

What you're saying, however, is that because TenCent owns a small amount of Discord, it's automatically suspicious because TenCent are Chinese; looking at public share ownership records, however, the same argument applies to Alphabet and all its properties, because some of Alphabet's shares are owned by Chinese entities (including private individuals).

Why is Google safe despite Chinese ownership, but Discord (with a similar degree of Chinese ownership) is not safe?

Nationality as your "mark of evil"

Posted Apr 10, 2024 10:33 UTC (Wed) by roc (subscriber, #30627) [Link] (7 responses)

I see reports that Tencent owns 38% of Discord, e.g. https://eightify.app/media/fact-checking-discord-s-ties-t.... I wouldn't call that a "small amount". 38% in the hands of one entity is enough to have significant influence. I would be surprised if Chinese entities own a comparable amount of Alphabet. (That would be a $740B investment.)

It's possible that those reports are wrong, or that Discord is set up in such a way that Tencent has no influence they could abuse so there's nothing to worry about. Nevertheless it is reasonable to worry about it.

Nationality as your "mark of evil"

Posted Apr 10, 2024 11:36 UTC (Wed) by dullfire (guest, #111432) [Link] (4 responses)

I think this has wander far off-topic. A certain party's screaming of "what about X, or Y, or Z!" is a strawman/whataboutism hybrid. The claims "refute" a point not said by the original poster, and accuses parties nobody talked about of doing things not originally said.

luna originally compared matrix to discord.

Matrix is a federated, open protocol, with multiple server implementations. It also has multiple client implementations. There are multiple opensource &| libre implementations of both client and server.

Discord is a proprietary protocol. It is solely under the control of Discord inc.

Matrix therefore *can not* have any single entity (regardless of who you are concerned about) make unilateral changes that would negatively impact the whole. Meanwhile, as the entity solely in control of discord, Discord Inc COULD unilaterally take any negative action.

luna also pointed out that a company beholden to the Chinese government has a notable about of shares in discord, and thus likely had some influence.

Then we had an LWN editor point out that this was not "polite and respectful", and that it was "nationalism" (At this point I STRONGLY encourage everyone to go lookup that word. It doesn't mean that). There has been no further elaborations on what the editor meant. However at this point I must point out I find the editors' message extremely concerning. Assuming that the editor meant the comment as they said it (if they did not, it would be wonderful if they added clarification), one of the following would have to be taken as not "polite and respectful"

- Requesting someone make a change to the communications platforms a project uses.
- Accusing an communications platform of adding ads.
- Suggesting influence from China(specifically China) is not desirable.
- Suggesting influence from any particular nation is not desirable.
- Suggesting influence from any group is not desirable.
- Suggesting influence from any person(s) is not desirable.

(The last four cover the said part of what luna said, but in progressively more less of a double-standard and more generic/universal)

I must say that ANY of these being true would greatly cripple any free/libre project governance talk on LWN. This is because it effectly says "there is a class of entities commenters are not allowed to cast in anything but the best light". Which means any conversation about, say, who should be debian project leader when multiple people are running... risks not being able to mention why you think person X shouldn't be DPL in a non malicious/hostile way(e.g. if someone wanted to say "X's skill set is best used elsewhere, and a poor fit for DPL", that might contravene the rules).

So I am going to hope/assume that the editor meant something other than what was written.

Finally, as I started with. This thread has veered far off the topic of "what community member would like for their interactions with the rsync project", and should likely be taken elsewhere.

Nationality as your "mark of evil"

Posted Apr 10, 2024 11:53 UTC (Wed) by farnz (subscriber, #17727) [Link]

Specifically what I see as problematic is the assertion that the presence of Chinese investors (regardless of their influence on the company) is bad. Taken as a guiding principle, this asserts that the Linux kernel cannot be trusted, because Huawei has influence over the kernel, and they're a Chinese company, too.

Given that you don't trust the Chinese government (personal decision), it's important to point out why the presence of a Chinese company with some influence is problematic with more than just "I don't like Chinese people having influence" - point to something that Discord (or whatever company you're talking about directly - not its owners, not other Chinese companies, but the company that we're discussing) has done that's problematic and that it's done for the Chinese government.

Otherwise, this all comes down to "my biases say that Chinese people are untrustworthy because they're Chinese, but that similar behaviour by the US and its allies is OK because I'm OK with Americans", and not a genuine cause for concern.

Nationality as your "mark of evil"

Posted Apr 10, 2024 11:54 UTC (Wed) by pizza (subscriber, #46) [Link] (1 responses)

> A certain party's screaming of "what about X, or Y, or Z!" is a strawman/whataboutism hybrid.

It's not "whataboutism" to point out the hystrionics are nearly entirely one-sided, elevating _potential_ problems of one convenient boogeyman with the _demonstrated actions_ of others (notably including their own).

China doesn't have to exercise subtle super secret control over Discord (or Tiktok) to get access to non-Chinese persons' data; they can do what everyone else does and just buy it on the nearly completely unregulated market of data brokers.

> Discord is a proprietary protocol. It is solely under the control of Discord inc.

This, not the ownership structure, is the _real_ problem of Discord.

Nationality as your "mark of evil"

Posted Apr 10, 2024 13:35 UTC (Wed) by farnz (subscriber, #17727) [Link]

As an aside, the substitution rule is a good way to tell if what you're talking about is bias, or actual threat. Take an entity you trust (e.g. the Canadian government), and substitute them for the entity you distrust (e.g. the Chinese government); does your suggestion still sound bad to you?

So, applying that to WeChat censoring some topics outside China at the request of the Chinese government; if WeChat were to censor some topics outside of Canada at the request of the Canadian government, this is still bad. Substituting Canada for China hasn't flipped it from "boogeyman" to "good action", so it's probably not bias.

On the other hand "some proportion of Discord shares are owned by Canadians, and you can't trust the Canadian government to not lean on Canadians to get them to do bad things" doesn't sound sensible if you think the Canadian government is well-run, and makes it clear that the original statement around Chinese instead of Canadians is probably just coming from a place of bias.

Similarly, "Discord ToS has this clause that permits bad things" doesn't have any nationalities in it at all, and is a good reason to distrust Discord. As is "Discord's client is proprietary, and I can't audit it for bad behaviour as a result".

Nationality as your "mark of evil"

Posted Apr 10, 2024 15:27 UTC (Wed) by jzb (editor, #7867) [Link]

First of all, I'd like to apologize for making a mess of things. I've been part of the LWN community for a very long time, and hadn't quite internalized how other folks would respond to a comment from me as an editor vs. all the other times I've joined in discussions. Lesson learned there.

To the point about nationalism, yes - that was a poorly chosen word. What I should've said, aside from "nothing", was that we have a global audience. Calling out one investor's nationality ("Chinese investors") as a negative seemed like something that might alienate some of our readers. I had hoped to stave off the discussion that I instead sparked. My apologies again, I hope we can end this thread / topic of discussion here.

Nationality as your "mark of evil"

Posted Apr 10, 2024 12:56 UTC (Wed) by excors (subscriber, #95769) [Link] (1 responses)

> I see reports that Tencent owns 38% of Discord, e.g. [redacted to avoid helping its SEO]

That looks like a completely unreliable source. Eightify is an "AI YouTube Summary with ChatGPT" tool, and it has published about 5000 articles in /media in the past few weeks, so I bet they're also AI-generated and therefore garbage. The "resources" it cites are largely irrelevant - perhaps it picked them up because they mention e.g. "growing discord between China and the United States" and it's too dumb to realise that's not about Discord.

They probably got the number from https://youtu.be/uvNkdAggUGU?t=555 (linked further down the page in what's presented as a separate article), but that video cites no source, and its fearmongering tone doesn't engender much trust in its accuracy. Maybe the video's author read something like https://seekingalpha.com/article/4371548-tencents-dreams-... which says Tencent owns 38% of DouYu (a Chinese video streaming service) and "~2%" of Discord, and got the numbers mixed up because they both start with D, or something. (That article also cites no source, though at least I think it was written by a human). I can't find any more reliable sources for the 38%.

Since Discord is a private company, it doesn't have to disclose who its investors are, and it has chosen not to, so I wouldn't believe any number you read on the internet. (And especially not any from AI-generated articles.)

Nationality as your "mark of evil"

Posted Apr 10, 2024 16:16 UTC (Wed) by roc (subscriber, #30627) [Link]

Yes, I think you're right. Thanks for pointing that out.

Tridge returns to rsync

Posted Apr 16, 2024 14:47 UTC (Tue) by marcH (subscriber, #57642) [Link]

> We ask that comments be polite and respectful, and nationalism is not something that aligns with those values.

Of course the thread later descended into off-topic politics and proved you right (*sigh*) but luna's initial comment was polite, respectful and not "nationalist". I have no idea whether it was factually correct or not but it was also on-topic considering the news was about rsync development. I use Discord and I like it but whether some Chinese company has some level of control on it is something I find relevant and interesting considering China is currently a strong dictatorship (among many others) exerting a very tight control on speech (among other control). Just as interesting as what the NSA and other western agencies do with tech companies - legally and illegally.

More generally speaking, technology and politics interact a LOT and banning the latter entirely in the comments would make some discussions pretty pointless.

benchmarking as part of tests

Posted Apr 8, 2024 7:58 UTC (Mon) by johnjones (guest, #5462) [Link]

thanks for letting me deal with gnu parallel and TCP cork but really it would be nice if there was a benchmark and compare to other transfer systems thanks - JJ

Tridge returns to rsync

Posted Apr 7, 2024 0:16 UTC (Sun) by dullfire (guest, #111432) [Link] (1 responses)

I've started using rsync, instead of scp, almost exclusively since the "let's make scp secretly sftp, and let us carefully make it totally unable to autofall back if it encounters a host that is incompatible" change.

However a discord server is... kind of off putting.

Tridge returns to rsync

Posted Apr 7, 2024 6:40 UTC (Sun) by WolfWings (subscriber, #56790) [Link]

It's supported the -O (hyphen-capital-o) option to revert to using the RCP (the original non-SFTP SCP protocol) from day one of the change. If a remote server actively reports no support for SFTP (not just denying attempts to access it but still reporting it existing) it falls back just fine?

But some SSH servers even if you disable SFTP (or even if they never support it) don't truly remove it, they just deny the connection the instant its requested on an SSH channel, so it's far less feasible to handle 'transparent fallback' because of how some SSH servers lie like that.

Tridge returns to rsync

Posted Apr 7, 2024 11:42 UTC (Sun) by cthart (guest, #4457) [Link] (4 responses)

Is there a better way to continuously rsync instead of using inotifywait? I'd love if rsync was inotify aware and could be run to continuously monitor some directory and rsync changes as they happen (à la Dropbox).

Tridge returns to rsync

Posted Apr 7, 2024 11:57 UTC (Sun) by zdzichu (subscriber, #17118) [Link] (3 responses)

For that use case I recommend SyncThing (https://lwn.net/Articles/861978/).

Tridge returns to rsync

Posted Apr 7, 2024 19:35 UTC (Sun) by mbunkus (subscriber, #87248) [Link] (2 responses)

SyncThing is an amazing piece of software. Highly recommended for any Dropbox-type thing but where you want storage under your control.

Tridge returns to rsync

Posted Apr 8, 2024 4:19 UTC (Mon) by dilinger (subscriber, #2867) [Link]

Yeah, it's even better than dropbox because I can run it completely peer-to-peer. Sync all pics in the DCIM directory between my phone and laptop (bidirectionally, so if I delete or rotate or edit a pic or video on my laptop, the same happens on my phone). Then sync a separate directory of selected (often hardlinked) pics/vids between my laptop, my wife's laptop, and photoprism server. I love it.

Tridge returns to rsync

Posted Apr 8, 2024 7:22 UTC (Mon) by Sesse (subscriber, #53779) [Link]

Reminded me to check my phone backup, which I had set to run every night through SyncThing and then forgotten under the assumption that it would just work.

This was a good thing, because somehow, it had stopped running in February; the “start automatically on boot” checkbox had just automatically been unchecked! Not very confidence-inspiring. :-)

Tridge returns to rsync

Posted Apr 7, 2024 21:25 UTC (Sun) by birdie (guest, #114905) [Link] (10 responses)

I don't care who returns or not, why is this feature request is still unresolved 20 years after it was filed?

https://bugzilla.samba.org/show_bug.cgi?id=2294

This is literally maddening.

Feature requests

Posted Apr 7, 2024 21:26 UTC (Sun) by corbet (editor, #1) [Link] (1 responses)

So I can only speak for myself, but my sense of the community is that, given the current mood, yelling at maintainers who haven't your pet feature is unlikely to make you a lot of friends — or get your feature merged.

Feature requests

Posted Apr 8, 2024 14:10 UTC (Mon) by birdie (guest, #114905) [Link]

I did not yell at anyone. Literally hundreds of threads on the Internet have been looking for this feature for decades now. What's worse a patch has long been proposed. Maintainers don't even provide solid arguments why they oppose to it.

But yeah, "Let's make it personal and accuse the person, who desperately needs something, at yelling at "poor" developers".

Tridge returns to rsync

Posted Apr 7, 2024 21:43 UTC (Sun) by atnot (subscriber, #124910) [Link]

big Jia Tan energy there

(https://mastodon.gamedev.place/@zarfeblong/11220238151654...)

handling renamed files

Posted Apr 7, 2024 22:48 UTC (Sun) by tridge (guest, #26906) [Link] (2 responses)

As mentioned on that ancient bugzilla report the --fuzzy option is a partial fix for this, but by no means complete. Renaming a whole directory sub-tree isn't handled.
we now use github for managing issues and I've created an issue for this:
https://github.com/RsyncProject/rsync/issues/590
but as I note in the issue this is a surprisingly tricky feature to add.

handling renamed files

Posted Apr 8, 2024 14:10 UTC (Mon) by birdie (guest, #114905) [Link] (1 responses)

> --fuzzy option is a partial fix for this, but by no means complete

Extremely limited in its usefulness. People don't just rename single files in a directory. They move directories around, they do all sorts of stuff.

rsync should have long started working with data blocks independent of directories and files, of course if you use the checksum option.

That means you can detect file renames, you can sync growing files without fully copying them.

handling renamed files

Posted Apr 8, 2024 17:24 UTC (Mon) by Wol (subscriber, #4433) [Link]

> you can sync growing files without fully copying them.

I've not used it, but this has been part of my "optimal backup strategy" for ages. There's an rsync option to "overwrite changed blocks in place", which coupled with some snapshot mechanism like lvm, enables you to store multiple full backups, for the price of one base and multiple incremental backups.

Just do it on a different disk, and preferably different machine, from the master!

Cheers,
Wol

Tridge returns to rsync

Posted Apr 8, 2024 6:56 UTC (Mon) by m0sh1x2 (guest, #142833) [Link] (2 responses)

20 years is nothing. Lets shoot for 100.

Tridge returns to rsync

Posted Apr 8, 2024 14:10 UTC (Mon) by birdie (guest, #114905) [Link] (1 responses)

This feature will be of no use to me in 100 years' time. I will be long dead by then.

Tridge returns to rsync

Posted Apr 8, 2024 16:19 UTC (Mon) by Sesse (subscriber, #53779) [Link]

Perhaps if it is such a matter of life and death for you, you should contribute a patch yourself. Might take less than 20 years :-)

Tridge returns to rsync

Posted Apr 8, 2024 11:08 UTC (Mon) by ras (subscriber, #33059) [Link]

Oh, are we allowed to beg for features here?

Here is mine: a flag that says "copy everything (uids, permissions, acls, xattrs, ...) that is supported by both ends". I have a wrapper script that tries to make that the default now, but it's impossible to know what the other end supports and if you specifically ask for something that isn't supported, rsync barfs (understandably and correctly in my opinion).

As for https://bugzilla.samba.org/show_bug.cgi?id=2294 unison does that. Turns out when I needed it most also happened to be when I used unison to do the copying.

Tridge returns to rsync

Posted Apr 8, 2024 4:07 UTC (Mon) by smfrench (subscriber, #124116) [Link]

This is very exciting. I did some analysis of rsync four or five years ago and noticed problems with its performance e.g. not enough parallelization of writes (and not enough leveraging of offloading copy to the fs - e.g. via copy range and reflink). Also was thinking that it is time for adding support for modern Linux APIs like io_uring in rsync. Will be very interesting to see how rsync continues to improve.