OpenStack Swift client for the fastest synchronisation of huge amount of files? -
i have folder lot of files (~50k, 3gb). need sync folder recursively container in openstak swift-like storage.
i have tried use cli duck (cyberduck), crashes on huge list of files in prepare process. trying use supload utility, slow :(
may recommends me best approach (some cli better) situation?
you should use official python-swiftclient package , :
# load openstack credentials source openrc.sh cd path_to_directory_you_want_to_sync # upload files recursively keeping paths swift upload --changed your_container *
swift not support rsync-like synchronization, use little script delete in container files deleted locally , upload new files without asking swift compare each files :
#!/bin/bash cd $2 diff <(find * -type f -print | sort) <(swift list $1 | sort) | while read x; if [[ $x == \>* ]]; echo "need delete ${x:2}" swift delete $1 "${x:2}" elif [[ $x == \<* ]]; echo "need upload ${x:2}" swift upload $1 "${x:2}" fi done cd -
use :
./swift_sync.sh your_container directory_to_sync
Comments
Post a Comment