Questions tagged [etl]
Extract, Transform, Load - process in a database
53
questions
3
votes
1
answer
407
views
What are some design ideas for a data mapping and transformation application?
Here is a high level outline of the project:
We frequently need to convert data from a new incoming system to our in house system (sort of a basic ETL process)
We would prefer to do this dynamically, ...
- 141
1
vote
1
answer
124
views
What does 'data coverage' mean when talking about ETL processes?
I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says
ETL tools have been ...
- 19
1
vote
0
answers
836
views
Is MapReduce a correct framework for Extract, Transform, Load of data?
EDIT
I am working on a project to update a legacy ETL infrastructure that supports a number of clients, each with a slightly different setup.
Constraints that cannot be changed:
Source data can ...
- 1,406
2
votes
2
answers
405
views
Automating tests for ETL flow
I created am ETL that parse various files, transform file's data and then push the lines into a DB, until now I did a manual tests and check that all the values parsed correctly and all the lines (...
- 197
1
vote
2
answers
314
views
Data integration from heterogeneous sources
A client has requested that we build a platform for integrating data from partner to their central data store. This will not be "Big Data" scale.
The data from each partner will be accessible through ...
- 205
1
vote
1
answer
1k
views
How to implement ETL with MySQL?
I have a legacy MySQL Database (A), and a new reviewed structure for MySQL data base (B).
Problem number one is that Database has to be alive and keeps receiving data from legacy apps.
What I need is ...
- 513
3
votes
2
answers
253
views
Translate data between inconsistently-matched data structures
How can my program best represent a translation between imperfectly-matched data structures?
I am tasked with a one-way translation of data from one system to another. Both systems are established, I ...
- 191
0
votes
1
answer
670
views
Exporting data to file share vs calling a web service to handle the export
I would like to hear opinions/best practices for the following scenario:
I have an application A (C# app from a vendor that does not have an accessible database, it has a sqllite db but we don't have ...
0
votes
2
answers
53
views
what is a better approach in replicating data from table to another? triggers or a third party ETL tool?
We have multiple tables which we need to retrieve data from and dump to one centralized table. Currently what we are doing is running an ETL job made from Pentaho, retrieve the records from the source ...
- 239
3
votes
1
answer
1k
views
How is one or more aggregate function implemented in most SQL engines?
In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march.
This make sense. What it means is that for calculating the maximun, average ...
- 336
5
votes
1
answer
15k
views
Difference between ESB and ETL
When should a ESB vs ETL tool be used? I have worked on ESB projects using Tibco Businessworks quite a few years ago. The message bus that we built used to consume messages from a source system , ...
- 875
2
votes
2
answers
1k
views
Using ESB for database synchronisation / replication
We're starting to look at implementing an ESB / Microservices architecture. I (think) I know about the concepts, but there's one thing I don't seem to be able to get a good idea about: data ...
- 205
4
votes
2
answers
2k
views
Is ReST useful in Read/Write Operations that involve over 100 Gig
I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a time)...
0
votes
1
answer
359
views
Coordinating a complicated data migration process
A project I'm involved in has suffered a change in scope, and before I set about trying to cook up some homegrown solution, I'm wondering if there is something out there -- some framework, for example ...
- 151
4
votes
1
answer
3k
views
What is the right way to process inconsistent data files?
I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from 50-...
- 61