SS isn't designed for asset management like your computer/filesystem. It is best to keep master copies of all your uploads on your local computer. This is also a good idea if for some reason your site were ever lost.
I wouldn't say you must download them manually but that is the lowest common denominator.
You might be able to do some semi-automatic downloading with a tool like wget. But you have to be familiar with command line tools. I've never used wget for specifically downloading just the assets of an SS site.
