Questions tagged [large-data]
The large-data tag has no usage guidance.
24
questions
1
vote
0
answers
132
views
High RAM usage when providing large file on apache
I'm using Apache 2.4 on AlmaLinux 8. When I publicate a hundreds GB file and if someone starts downloading it, Apache RAM usage (especially VIRT) becomes bigger and bigger, starts swapping and ...
- 23
-1
votes
2
answers
306
views
MySQL replcating huge changing data
Hi I have a master MySQL server with 950 gb of data and slave has stopped for some reason and there is a difference of 80gb of data.
so I started to setup new slave .
Since the data is changing and ...
8
votes
2
answers
3k
views
How to overwrite a very large hard drive (18TB) with random data using shell commands in Linux
I would like to overwrite a very large hard drive (18TB) with random bytes, to then check smart data for reallocated sectors or other errors.
Since badblocks has some limitations on number of blocks ...
- 282
0
votes
1
answer
362
views
GCSFuse v0.33.2 - Large objects (11GB) upload to GCSBucket Fails
Receiving below errors when we try to upload 11 GB file to bucket; also, filesystem is going into "????????" as we can't list the objects available in the bucket.
This requires umount and ...
google-cloud-platform
1
vote
1
answer
2k
views
Varnish crashes on bigger files
Overall, I serve very small files. Think about images and small videos. Caching these with Varnish is like a breeze and doesn't give me any issues.
The problem I am having, is when I am downloading a ...
- 113
1
vote
1
answer
823
views
Multiple disks acting as one filesystem, while still allowing each disk to be accessed alone
I have a fairly large dataset (~160TB) that need to be delivered to a client every so often. This dataset consists of fairly large files, usually between 2Gb and 20Gb each. They exist on a BeeGFS ...
- 535
4
votes
3
answers
608
views
Can I validate a large file download piecemeal over http
I'm downloading a large file over http via wget, 1.2TB. The download takes about a week and has contained corruptions twice now (failed md5 check, which takes days to run by itself).
Is there a good ...
- 1,038
1
vote
2
answers
2k
views
Moving very large (~100 Gb) from one server to the other
We're moving servers, and I need to transfer all the data from Server A to Server B.
I have a tar.gz of about 100Gb that contains all the Server A files.
I'd really like to avoid downloading the ...
- 113
2
votes
1
answer
2k
views
Reading 65k rows hangs PHP/MySQL
I am developing a PHP application that processes IP adresses. I am working with mysql tables containing up to 4 billion rows.
I have a script that currently needs to fetch 65536 adresses from this ...
- 21
1
vote
2
answers
6k
views
Is it better to use rsync over SSHFS or over CIFS as remote repository, having no option for rsyncd?
I have a NAS that is only capable of CIFS, AFS, SSH (no rsyncd capabilities, no NFS).
I have to backup very large files (vm images) and I usually set a rsync server on the backup device then I do a ...
- 63
1
vote
1
answer
405
views
mysqld : multiple tmpdir & balancing
Our 1Tb tempdir can sometimes be completely used by mysqld, resulting in disk-full and query errors. This can be due to lots of mid-size queries, or couple of very big queries.
We have a 5Tb raid ...
- 153
0
votes
0
answers
126
views
Tips on Copying 3 Database files nearing 250 GB?
We are upgrading from SQL Workgroup 2005 to SQL Standard 2012. We have plenty of databases of smaller sizes that copied over with straight windows file copy in a reasonable time frame. We have one ...
- 11
1
vote
1
answer
633
views
Offsite backup solutions for 40TB of data/35 Million Files on single drive [closed]
I've got a server with 40TB of data and 35 million files. While the server itself has raid and all that jazz, I'm concerned about what would happen should something physically destroy the server (fire,...
- 21
4
votes
2
answers
5k
views
4TB HGST SATA drive only shows 1.62 TB in Windows Server 2012
I'm using a Supermicro X9SRE-3F motherboard with the latest BIOS and 2x 4TB drives connected to the on-board SATA controller.
If I set the BIOS to RAID and create a RAID 1 array, the array shows up ...
- 41
4
votes
2
answers
608
views
Tomcat Denial of Service due to large packets
I had asked this question on ITSecurity, but I felt this question is better placed here.
On a recent assesment, I found that sending large (>5 MB) requests to a tomcat server causes 100% CPU usage on ...
- 143