AWS CLI: aws sync between 2 different s3 providers
up vote
0
down vote
favorite
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
asked Nov 19 at 19:19
HyperDevil
89742239
add a comment |
up vote
0
down vote
favorite
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
asked Nov 19 at 19:19
HyperDevil
89742239
add a comment |
up vote
0
down vote
favorite
up vote
0
down vote
favorite
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
asked Nov 19 at 19:19
HyperDevil
89742239
Does anybody have a solution to sync a bucket between 2 different s3 providers?
For example one is Amazon S3, second is Wasabi S3?
That involves 2 different endpoints and 2 different sets of credentials.
Preferable without storage data locally first, talking about 1+ million files.
amazon-web-services amazon-s3
amazon-web-services amazon-s3
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
asked Nov 19 at 19:19
HyperDevil
89742239
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
asked Nov 19 at 19:19
HyperDevil
89742239
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
edited Nov 19 at 22:32
John Rotenstein
66.1k773117
66.1k773117
asked Nov 19 at 19:19
HyperDevil
89742239
asked Nov 19 at 19:19
HyperDevil
89742239
asked Nov 19 at 19:19
HyperDevil
89742239
89742239
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
accepted
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
accepted
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
add a comment |
up vote
0
down vote
accepted
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
add a comment |
up vote
0
down vote
accepted
up vote
0
down vote
accepted
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
This would not be possible.
In Amazon S3, it is possible to copy directly between two buckets, even in different regions, because the S3 services in each region communicate with each other.
This wouldn't be possible with a non-Amazon service.
Therefore, your best bet would be to launch an EC2 instance, use aws s3 sync to download the files, then use aws s3 sync (with a different endpoint & credentials) to connect to Wasabi.
It is possible that the S3-compatible providers might offer an easier method, but it appears that Wasabi recommends the 'download and copy' method: How do I copy files from AWS S3 to Wasabi? – Wasabi Knowledge Base
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
answered Nov 19 at 22:35
John Rotenstein
66.1k773117
66.1k773117
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
add a comment |
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Thanks. i was afraid of this, so this keeps me locked in to AWS for now. The sync command after 10 days does not ever finish because of the insane amount of objects.
– HyperDevil
Nov 21 at 8:33
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
Is this the sync to download, or to upload? Another method is to Export Data from Amazon S3 with Snowball - AWS Snowball, but then you'd have to upload from your own computer to Wasabi.
– John Rotenstein
Nov 21 at 9:41
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
This is to sync 2 buckets for backup purposes.
– HyperDevil
Nov 23 at 7:05
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
If you wish to have a second copy of the data (well, Amazon S3 already keeps copies in 3 data centers, but if you want a copy in a different location), then you can use Amazon S3 Cross-Region Replication that will automatically copy the data to a different region (where it would again be kept in 3 different data centers).
– John Rotenstein
Nov 23 at 7:37
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
It has to do with regulatory requirements, one should not only depend on only one technical provider.
– HyperDevil
Nov 26 at 13:45
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53381254%2faws-cli-aws-sync-between-2-different-s3-providers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
