Questions tagged [etl]
Extract, Transform, Load - process in a database
53
questions
8
votes
3
answers
6k
views
How to do data synchronisation between two applications?
Our client has a CRM application (let's call it A) and a custom built Windows application (VB.NET) (let's call it B) that directly writes to and reads from the CRM database.
Application A is used (...
- 3,120
7
votes
5
answers
376
views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture:
queue -> message processor (horizontal scaling) -> RDBMS
Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
6
votes
2
answers
322
views
Disagreement Concerning Data Integration (I may not understand enterprise ETL tools)
I have been in an ongoing conversation concerning a project we are about to undertake at my place at work. The project concerns data integration. Our customers want to be able to integrate our data ...
- 321
5
votes
2
answers
1k
views
How can large data handling be a special requirement for job?
I chanced to see some job requirements of Oracle SQL programmer or C# developer jobs, who want a person who can handle terabytes of data in ETL or data migration.
I have worked on data migration to ...
quantcoder
5
votes
1
answer
15k
views
Difference between ESB and ETL
When should a ESB vs ETL tool be used? I have worked on ESB projects using Tibco Businessworks quite a few years ago. The message bus that we built used to consume messages from a source system , ...
- 875
4
votes
2
answers
2k
views
Is ReST useful in Read/Write Operations that involve over 100 Gig
I work in Healthcare and we use SAS to Extract and Transform medical and pharmacy claims data for use in downstream reporting applications. For a given Report Request(usually 40 are running at a time)...
4
votes
2
answers
995
views
How should a data warehouse be maintained for a quickly changing schema
I am currently in a process of maintaining a data warehouse for a quickly growing start up company. There is a lot of reporting demands from the clients, and this is usually handled by a data ...
- 151
4
votes
1
answer
3k
views
What is the right way to process inconsistent data files?
I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from 50-...
- 61
4
votes
2
answers
116
views
Designing an ETL with where there are a few points of entry
I'm trying to think of a scalable solution for my current system.
The current system is
3 microscopes
1 processing machine
1. 60-100GB Files come from 2-3 microscopes every 30 minutes
2. That data ...
- 49
4
votes
1
answer
3k
views
Designing a Content-Based ETL Process with .NET and SFDC
As my firm makes the transition to using SFDC as our main operational system, we've spun together a couple of SFDC portals where we can post customer-specific documents to be viewed at will. As such, ...
- 165
3
votes
2
answers
253
views
Translate data between inconsistently-matched data structures
How can my program best represent a translation between imperfectly-matched data structures?
I am tasked with a one-way translation of data from one system to another. Both systems are established, I ...
- 191
3
votes
1
answer
1k
views
How is one or more aggregate function implemented in most SQL engines?
In the book Database Fundamentals, Silberschatz. It is explained that aggregate functions can be calculated on the march.
This make sense. What it means is that for calculating the maximun, average ...
- 336
3
votes
2
answers
1k
views
Data warehouse architecture for mutating schema
I am setting up an ETL process and small data warehouse for querying the data in a few different dimensions. One issue is that the schema for the objects can mutate over time - mainly that some fields ...
- 231
3
votes
1
answer
221
views
Enterprise Wide Keys [closed]
I have for a long time been working on an ODS as well as Data Warehouse. Both are integrating a wide variety of data sources from stove pipe applications. One of the uses of the ODS is to provide ...
- 206
3
votes
1
answer
407
views
What are some design ideas for a data mapping and transformation application?
Here is a high level outline of the project:
We frequently need to convert data from a new incoming system to our in house system (sort of a basic ETL process)
We would prefer to do this dynamically, ...
- 141