Questions tagged [amazon-s3]
Amazon S3 (Simple Storage Service) is a web service which provides storage and various interfaces for accessing data stored there.
174
questions
0
votes
0
answers
164
views
ffmpeg stream image flow to AWS S3
For a video monitoring solution, I want to push thumbnails of an incoming video (TS/UDP) to a S3 bucket from where those thumbnails are shown on a web page
Currently the following bash loop is working:...
1
vote
1
answer
1k
views
How does FFMpeg read from a remote s3 url
I'm intrigued how does FFMpeg read a video file from a remote url (for example, an aws s3 presigned url).
Does it download the file in chunks and then process it chunk by chunk?
Or does it download ...
- 123
0
votes
0
answers
33
views
Mount or sync an S3 versioned bucket with rewind - back in time
Assuming I have an S3 versioned bucket (backed by Ceph), I would like to restore from it at a specific time. mcli has a --rewind feature that allows some operations to time-travel into the past taking ...
- 151
0
votes
0
answers
56
views
How to route linode object bucket to domain name?
I have a linode object bucket up and running with my files/folders. I have purchased the domain and have set a CNAME to point to [subdomain].[domain].com.website-us-southeast-1.linodeobjects.com. When ...
1
vote
1
answer
148
views
Connect to S3 Web Console with Access and Secret Key
Is it possible to authenticate in the Amazon S3 console (https://s3.console.aws.amazon.com) using an access key/secret key pair? It looks like it only accepts user credentials. If not, is there any ...
- 251
1
vote
0
answers
66
views
AWS Windows EC2 File Transfer - Need to keep session open
I have a rather large file that I need to copy from an S3 bucket to a Windows EC2 and it will not complete before a session ends. I have found lots of suggestions for this for Linux but not Windows.
...
- 11
0
votes
1
answer
394
views
aws sync doesn't download the file and instead create an empty directory
The file I'm trying to download is:
s3://ont-open-data/gm24385_q20_2021.10/analysis/20210805_1713_5C_PAH79257_0e41e938/guppy_5.0.15_sup/align_unfiltered/chr19/calls2ref.bam
This is an open dataset. I ...
- 25
0
votes
1
answer
326
views
How translate s3 url to http in chrome automatically?
Let say I have a link like that:
s3://mybucket/mybucket?prefix=myprefix/file.json
And I open it in chrome browser. I will get an error. Is it possible to automatically translate the s3 url from
...
- 950
2
votes
0
answers
268
views
AWS S3 Bucket CORS Error
I have an amazon s3 bucket called cs-ia.
This is my CORS configuration:
[
{
"AllowedHeaders": [
"*"
],
"AllowedMethods": [
...
- 121
1
vote
1
answer
132
views
Can't delete duplicity s3/glacier archives
For backing up my system to s3, I'm running:
duplicity 0.8.17
Python 3.9.2
boto3 '1.24.36'
My daily backups run fine, as desired they are moved from s3 to glacier after a month, duplicity status ...
- 31
0
votes
2
answers
1k
views
make new objects in s3 bucket public by default
I have an s3 bucket in which the objects within it need to be made public, right now anytime I upload a new file I have to click on it and then click "make public using acl" I wonder if ...
- 113
0
votes
1
answer
139
views
What IAM policy is appropriate for gitlab S3 buckets when using KMS encryption?
I'm setting up GitLab with S3 storage backend and would like to use KMS to encrypt the buckets. I'd like to avoid giving out permissions broadly. I'm using terraform to provision these resources.
How ...
- 1,101
0
votes
1
answer
505
views
How to create AWS S3 key
I've set up an AWS S3 bucket, and it's working for transfers using the web UI, but I want to be able to use the "aws s3 sync" command. The only way I know how to use that command requires ...
- 147
1
vote
1
answer
267
views
Is there a way we can way to minimize or eliminate warning when a recipient receives an email from AmazonSES
When we are sending emails using Amazon SES from the application, recipients of the email are seeing warning messages like below:
I have tried to find possible things that we can do to avoid ...
- 11
2
votes
1
answer
2k
views
Can WinSCP be used to access Google Cloud Storage bucket using the 'Amazon S3' file protocol?
Can WinSCP be used to access Google Cloud Storage bucket using the Amazon S3 file protocol using the HMAC keys?
I tired but got the following error:
Expected request type in credential scope to be &...
- 21
0
votes
1
answer
53
views
Streaming file via HTTP to S3
Currently we have a function on one of our EC2 instances (running C# .NET) that, upon request, opens an HTTP connection to an external site (for the sake of argument, let's say https://www.example.com/...
- 101
0
votes
1
answer
651
views
Whitelist IP addresses for access to AWS S3 static web hosting
I have seen several references that use a Bucket Policy similar in structure to the following
{
"Version": "2012-10-17",
"Id": "Policy1607646663558",
...
- 3,835
0
votes
0
answers
334
views
Why does s3.amazonaws.com respond with 400 Bad Request when I connect over telnet?
This is what my request looks like. I have confirmed that the end of line separator is CRLF:
telnet s3.amazonaws.com 80
GET / HTTP/1.1
Host: s3.amazonaws.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; ...
- 101
0
votes
2
answers
213
views
Windows7 command prompt show number of objects as list from dir command
I'm using Windows7, and I want to know the number of files (or objects) for comparison to objects in Amazon S3 buckets. I'm not interested in size, as the size can differ between file types.
I want ...
- 1
0
votes
0
answers
192
views
GPG piping help for an S3 encrypted postgresql backup into RDS
I was wondering if there was a trick that I am missing.
I am trying to get an encrypted Postgresql backup dump from S3 -> decrypt it in GPG -> pg_restore into RDS.
However, I am not able to get ...
- 101
0
votes
1
answer
569
views
Upload multiple files to S3 from stdin
I found the option to upload to S3 from stdin:
cat test.txt | aws s3 cp - s3://mybucket/test.text
But I have a tgz archive coming in via curl and containing multiple files. I want to write the ...
- 111
5
votes
2
answers
10k
views
How to search for nested files in my S3 bucket?
I'm on the S3 Management Console and I see the instructions to type a prefix to search:
However, when I search, it only finds files in the current folder. Is there a way to search for files that ...
- 157
1
vote
1
answer
350
views
AWS Access Key Issue while accessing s3 bucket on Azure VM
I am trying to access AWS s3 bucket from Azure HDInsight Cluster VM. I generated new keys and added to .aws/credentials. "aws s3 ls" is working fine in Azure VM. If I do hadoop distcp or read a s3 ...
- 11
0
votes
0
answers
178
views
Download all S3 files uploaded within a given time period
I would like to all files uploaded to an S3 button within a given time period.
For all example, everything that was uploaded between 05-11-2017 and 07-11-2018.
Please note that due to the size of ...
- 101
0
votes
1
answer
4k
views
Selectively uploading multiple folders to AWS S3 using CLI
I have around 50 folders, which contain subfolders, I want to upload the folders and subfolders to an S3 bucket, while keeping the subfolders structure.
I read I can use --include command multiple ...
- 179
0
votes
0
answers
14
views
AWS Beanstalk s3 issue
I wanted to update a nodejs app and ruined my s3 server. Instead of updating properly from services -> elastic beanstalk, I deleted the .zip file on s3 and replaced it with a new one. I rebuilt my ...
0
votes
1
answer
51
views
Prepopulated AWS Access Key ID when creating a new profile?
A new counter-party has provided me with an AWS S3 Access Key ID
and a Secret Key so I can send files to them.
I was expecting to run
$ aws s3 configure
and enter those values when prompted.
...
- 1,163
1
vote
1
answer
85
views
Creating folders by file names
I'm trying to find a way to create a folder for every file in a directory. I then want to copy each file into its created folder. These files are currently hosted on AWS (S3).
So for example if I ...
1
vote
1
answer
2k
views
How to configure amazon S3 bucket for read-only access to host a static website?
I have a very small (static) website hosted on Amazon S3. Yesterday I received the following message, that had this subject line:
Notification of Amazon S3 buckets configured for public access
And ...
- 567
1
vote
1
answer
2k
views
Setting up a custom idp for AWS SFTP using Active Directory
I've been digging into setting up custom auth for AWS SFTP and it's one hell of a black hole. There's lot's of references to being able to use a custom idp, but no concrete examples.
We're using ...
- 13