Toolkit for Apache Spark ML for Feature clean-up, feature Importance calculation suite, Information Gain selection, Distributed SMOTE, Model selection and training, Hyper parameter optimization and selection, Model interprability.
Big Data Modeling, MapReduce, Spark, PySpark @ Santa Clara University
This project was a joint effort by Lucas De Oliveira, Chandrish Ambati, and Anish Mukherjee to create a song and playlist embeddings for recommendations in a distributed fashion using a 1M playlist dataset by Spotify.
Powerful rapid automatic EDA and feature engineering library with a very easy to use API 🌟
Workshop Big Data en Español
Projects and studies regarding Data Engineering Area
End-to-end data engineer project
This project provides an end-to-end data processing and visualization of visa numbers in Japan using PySpark and Plotly. The spark clusters are set up within a Docker container on Azure.
Twitter Spark Streaming using PySpark
Movie Recommendation Engine using PySpark
parallel implementation of hierarchical clustering algorithm based on pyspark
Transformation of Akamai Logs with Spark ETL and discover of Values and similarities in logs used SparkML and H2O ML
This is the final project for the Data Scientist Nanodegree, where our goal is to predict churn for a fictional streaming service called Sparkify.
Samples for Azure Databricks Orientation
[大数据课程作业] Bilibili 助手: 视频推荐 + 热门预测
The goal was to perform predictive maintenance on commercial turbofan engine. The approach used here is a data-driven approach, meaning that data collected from the operational jet engine is used to perform predictive maintenance modeling. To be specific, to build a predictive model to estimate the Remaining Useful Life ( RUL) of a jet engine ba…
Add a description, image, and links to the pyspark topic page so that developers can more easily learn about it.
To associate your repository with the pyspark topic, visit your repo's landing page and select "manage topics."