Hello selfhosted! Sometimes I have to transfer big files or a large amounts of small files in my homelab. I used rsync but specifying the IP address and the folders and everything is bit fiddly. I thought about writing a bash script but before I do that I wanted to ask you about your favourite way to achieve this. Maybe I am missing out on an awesome tool I wasn’t even thinking about.
Not gonna lie, I just map a network share and copy and paste through the gui.
Yeah, I mean I do still use rsync for the stuff that would take a long time, but for one-off file movement I just use a mounted network drive in the normal file browser, including on Windows and MacOS machines.
Same lol, somebody please enlighten me on a faster way!
Sounds very straight forward. Do you have a samba docker container running on your server or how do you do that?
I just type
sftp://[ip, domain or SSH alias]into my file manager and browse it as a regular folderYOU CAN DO THAT???
Linux is truly extensible and it is the part I both love and struggle to explain the most.
I can sit at my desktop, developing code that physically resides on my server and interact with it from my laptop. This does not require any strange janky setup, it’s just SSH. It’s extensible.
Dolphin?
Any file manager on Linux supports this
I have two servers, one Mac and one Windows. For the Mac I just map directly to the smb share, for the Windows it’s a standard network share. My desktop runs Linux and connects to both with ease.
Do you really need a container for Samba?
I see the benefits of containers, but a use would be overkill.
Yeah, if OP has command line access through rsync then the server is already configured to allow remote access over NFS or SMB or SSH or FTP or whatever. Setting up a mounted folder through whatever file browser (including the default Windows Explorer in Windows or Finder in MacOS) over the same protocol should be trivial, and not require any additional server side configuration.
What’s wrong with rsync? If you don’t like IP addresses, use a domain name. If you use certificate authentication, you can tab complete the folders. It’s a really nice UX IMO.
If you’ll do this a lot, just mount the target directory with sshfs or NFS. Then use rsync or a GUI file manager.
Just don’t run rsync as a daemon as that’s a security nightmare
Why would you do that? That sounds awful…
The daemon tracks file state, so the transfers start quicker because rsync doesn’t have to scan the filesystem.
I never even set up DNS for things that aren’t public facing. I just keep /etc/hosts updated everywhere and ssh/scp/rsync things around using their non-fqdn hostnames.
You could also use mDNS to the same effect.
sftp
All my machines have my keys, nothing to set up, nothing to tear down.
rclone. I have a few helper functions;
fn mount { rclone mount http: X: --network-mode } fn kdrama {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/KDrama/$x --filter-from ~/.config/filter.txt } fn tv {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/TV/$x --filter-from ~/.config/filter.txt } fn downloads {|x| rclone --multi-thread-streams=8 --checkers=2 --transfers=2 --ignore-existing --progress copy http:$x nas:Media/Downloads/$x --filter-from ~/.config/filter.txt }So I download something to my seedbox, then use
rclone lsd http:to get the exact name of the folder/files, and runtv "filename"and it runs my function. Pulls all the files (based on filter.txt) using multiple threads to the correct folder on my NAS. Works great, and maxes out my connection.Ähm. So your not gonna like this but I just connect with vscode remote-ssh and drag’n drop em from the os file explorer into the vscode one.
So long story short scp I guess.
rsync
- sftp for quick shit like config files off a random server because its easy and is on by default with sshd in most distros
- rsync for big one-time moves
- smb for client-facing network shares
- NFS for SAN usage (mostly storage for virtual machines)
Magic wormhole is pretty dead simple https://magic-wormhole.readthedocs.io/en/latest/welcome.html#installation
I use this a lot at work for moving stuff between different test vms, as you don’t need to check IPs/hostnames
People have already covered most of the tools I typically use, but one I haven’t seen listed yet that is sometimes convenient is
python3 -m http.serverwhich runs a small web server that shares whatever is in the directory you launched it from. I’ve used that to download files onto my phone before when I didn’t have the right USB cables/adapters handy as well as for getting data out of VMs when I didn’t want to bother setting up something more complex.rsync is indeed fiddly. Consider SFTP in your GUI of choice. I mount the folder I need in my file browser and grab the files I need. No terminal needed and I can put the folders as favorites in the side bar.
If you want to use the terminal though, there is
scpwhich is supported on both windows and Linux.Its just
scp [file to copy] [username]@[server IP]:[remote location]That’s essentially the same as rsync
Just slower if you already have some of the files there.
Syncthing and/or ftp.
Syncthing
What do you mean by specifying IP address?
WinSCP for editing server config
Rsync for manual transfers over slow connections
ZFS send/receive for what it was meant for
Samba for everything else that involves mounting on clients or other servers.
As a lazy person, I just prefer
sftpon thunar.
