From a single droplet to a full bottle, our journey to Hadoop at Coca-Cola East Japan

Editor's Notes

1. > Coca-cola East Japan responsible to produce and distribute all the product you love from Coffee to carbonated drinks like Fanta or Sprite  Vending Machines > 5 different bottlers and soon 6 >
2. Data in silos: 3 layers of technologies: mainframe and its eco system / highly customized SAP instances per bottler / single instance of SAP for CCEJ Redundancy of capabilities at each level Duplicated data, no single source of truth Replication going both ways Point to point : 2 endpoints one or several interfaces Multiple ETL tools / ESB tools, servers that stage data No governance: Data structure based on system requirements and vendors convenience: e.g: 2 different vendors working on the same project 2 IDs for the same object Requirements  solution developed by the vendors  Knowledge of the meaning of the data kept on the vendor side Master data managed in each systems  no completeness of the master data / no single source of truth Batch oriented: Data transfer through flat files / fixed width files orchestrated by schedulers (once a day) Little to no event driven flows Multiple intermediate systems that receive the data / load then send to another system
3. Started humbly: Focused on analytics around vending machines & data exploration HDP on Azure with Centos VMs (First in Japan)  HDInsight without Spark difficult to install Data manually uploaded to the cluster through FTP
4. Focused on analytics around vending machines data exploration Teradata : datamart NiFi to integrate between our on prem environment and the cloud environment leveraging the site-to-site connection and using certificates
5. First release and production use of Hadoop Spark program running every night to generate forecast for all vending machines Integration with NiFi on multiple systems to retrive transactional data and master data First attempt to integrate with BW using JDBC (not Vora as BW was not in the same data center and we had no requirements to push back data from BW to Hadoop) NiFi : ExecuteSQL modified to leverage setfetchSize to stream data (10,000 records) Re-encoding to UTF8 in the processor Fixed-width parser Governance around data: Partition data based on delta extraction Naming convention to easily identify data source systems Atlas and AD for access rights NiFi : guidelines around processing groups, e.g: master data / transactional data NiFi: Trigger oriented with webService, error handling: bubbling errors to the top, retry logic to ensure extraction
6. Scheduling & orchestration can be implemented through NiFi Most of our data are linked to a date therefore we partition tables keeping the tabular structure of the tables Hive was the de-facto solution Many data transformation can be implemented directly in HQL Presto to integrate heterogeneous system / good performance on querying data Drill to easily format in JSON and rapidly integrate with Web interfaces NiFi (250 flows) & Boomi (700 flows) integration to have full access to SAP ECC functional modules & IDOCS Program profiling: Fire Defined the use cases: Analytics Data hub and aggregate & processing data Central repository for Master Data 20TB in HDFS Pulling 20GB per day 250 in prod daily extractions in NiFi (ExecuteSQL) other in Dev for CokeOne DM : 0.249885409 HHT : 3.571921535 CMOS : 0.657761362 SC : 4.140049335 DME : 10.447961232 MM : 0.260761793 104 cores 9 datanodes 486GB of RAM HIVE tables: Prod: 402 BI:148 Default: 123 DemandForecast: 42 dm: 7 mdm: 26 sc: 40 vm: 16 Dev: 604 bi: 36 fusion:18 mdm:306 rtr:7 vm:237
7. NiFi in production in October
8. CCEJ Intro Hadoop Not only a datalake, integration platform: business enabler: 1 MDM 2 write off 3 ERP integration Landscape
9. 1: High number of Vending machines Online Vending machines / Offline vending machines 2: Lines up of 50 products in average 25 products in the vending machines Hot and cold product seasonality New product every 3 months 3:External factors Vending inside or outside (in office / stations / sport center) Easily accessible: on top of a montain / on top of a building / in a station underground Wide variety of customers: regulars, events: baseball 4: vending routes Limited number of truck & filler  go replenish the 20 VMs that really needs it what product to put on the truck
10. Processing everyday 14M items (VM x products): 5GB: today’s sales information, stock level, settlement information, … 300GB as we look into the past
11. Producing that report took 6 to 8 hours and had errors
12. Instead of duplicating data (silos) we can reuse the same data to generates those report IDOCS, JDBC, flat files, fixed width files Complex transformation as we are combining legacy data with new number and definition of master data to aggregate data during the rollout period of the new systems
13. Whenever possible we implement extractions triggered by a web service call from the source system We group extractions by flow together under logical “Process Group” We try to have logical subdivisions as well to help readability and maintainability (e.g: extractor for Master data / transactional data or monthly extractors / daily extractors) Site-to-site communication is encrypted using a single root CA certificate shared across all keystore
14. We always implement fail over that retry to perform a processing We always implement error management In each processing group we implement an Output Port called “onError” that are linked with the parent “OnError” Output port this ensure that notification are propagated until reaching the root canvas. The top “OnError” is a Remote Output port available on NiFi Azure side and implement a common handler to send errors Each processor is linked with that “Output port”, at the beginning of the flow branch has their own set of parameters defined: Service Process Priority Those parameters are used by the Error handling process to send notification to Administrators
15. Key Custom Features: Leverage cursors on tables to stream batch of data (10000 rows at a time) Comma separated delimiter CSV file output , by default filename <yyyy-MM-dd_hh:mm:ss.SSS>.csv with LF as line separator. Default UTF-8 file encoding Option to save the extracted file to folder location ( Integrated PutFile Processor functionality ) A Boolean value for user to choose to keep the source file or remove from folder location. UPDATE SQL functionality Boolean option to choose output CSV file as Windows CSV file format with line separator CRLF or Unix CSV file format with line separator as LF. Also a fixed width processor to CSV format
16. Dev environment accessible to anyone (after requesting an account