All Questions
6
questions
1
vote
1
answer
126
views
Data pipeline design - robust and resilient to future variations
I need to build a data pipeline to populate a database from various files. This is a common scenario. However, I want to have expert opinions for implementing a pipeline that is robust, modular and ...
- 23
1
vote
2
answers
81
views
Should data be pre-processed before being handled by an ETL framework?
So I was discussing coding with an associate of mine at work, and was mentioning how I was working on a project where I'd need to transform the data that was provided into a standardized format before ...
- 871
0
votes
1
answer
354
views
Data Integration Design Using Microsoft SSIS
I am working on a data integration project, where I need to extract data from oracle source and load it to XML file. The requirement is to get the list of customers and foreach customer create an xml ...
- 109
3
votes
1
answer
407
views
What are some design ideas for a data mapping and transformation application?
Here is a high level outline of the project:
We frequently need to convert data from a new incoming system to our in house system (sort of a basic ETL process)
We would prefer to do this dynamically, ...
- 141
4
votes
1
answer
3k
views
What is the right way to process inconsistent data files?
I'm working at a company that uses Excel files to store product data, specifically, test results from products before they are shipped out. There are a few thousand spreadsheets with anywhere from 50-...
- 61
1
vote
1
answer
192
views
Production or Custom Test Data for Unit Testing?
I've recently had a little disagreement with fellow developers. We're transforming various ontologies from the original source format (Pica+, RDF, etc) into our data format and have several converters ...
- 1,460