applied-ml

Curated papers, articles, and blogs on data science & machine learning in production. ⚙️

HitCount

Figuring out how to implement your ML project? Learn how other organizations did it:

• How the problem is framed 🔎(e.g., personalization as recsys vs. search vs. sequences)
• What machine learning techniques worked ✅ (and sometimes, what didn't ❌)
• Why it works, the science behind it with research, literature, and references 📂
• What real-world results were achieved (so you can better assess ROI ⏰💰📈)

P.S., Want a summary of ML advancements? 👉ml-surveys

P.P.S, Looking for guides and interviews on applying ML? 👉applyingML

Table of Contents

1. Data Quality
2. Data Engineering
3. Data Discovery
4. Feature Stores
5. Classification
6. Regression
7. Forecasting
8. Recommendation
9. Search & Ranking
10. Embeddings
11. Natural Language Processing
12. Sequence Modelling
13. Computer Vision
14. Reinforcement Learning
15. Anomaly Detection
16. Graph
17. Optimization
18. Information Extraction
19. Weak Supervision
20. Generation
21. Audio
22. Privacy-Preserving Machine Learning
23. Validation and A/B Testing
24. Model Management
25. Efficiency
26. Ethics
27. Infra
28. MLOps Platforms
29. Practices
30. Team Structure
31. Fails

Data Quality

1. Reliable and Scalable Data Ingestion at Airbnb Airbnb 2016
2. Monitoring Data Quality at Scale with Statistical Modeling Uber 2017
3. Data Management Challenges in Production Machine Learning (Paper) Google 2017
4. Automating Large-Scale Data Quality Verification (Paper)Amazon 2018
5. Meet Hodor — Gojek’s Upstream Data Quality Tool Gojek 2019
6. Data Validation for Machine Learning (Paper) Google 2019
7. An Approach to Data Quality for Netflix Personalization Systems Netflix 2020
8. Improving Accuracy By Certainty Estimation of Human Decisions, Labels, and Raters (Paper) Facebook 2020

Data Engineering

1. Zipline: Airbnb’s Machine Learning Data Management Platform Airbnb 2018
2. Sputnik: Airbnb’s Apache Spark Framework for Data Engineering Airbnb 2020
3. Unbundling Data Science Workflows with Metaflow and AWS Step Functions Netflix 2020
4. How DoorDash is Scaling its Data Platform to Delight Customers and Meet Growing Demand DoorDash 2020
5. Revolutionizing Money Movements at Scale with Strong Data Consistency Uber 2020
6. Zipline - A Declarative Feature Engineering Framework Airbnb 2020
7. Automating Data Protection at Scale, Part 1 (Part 2) Airbnb 2021
8. Real-time Data Infrastructure at Uber Uber 2021
9. Introducing Fabricator: A Declarative Feature Engineering Framework DoorDash 2022
10. Functions & DAGs: introducing Hamilton, a microframework for dataframe generation Stitch Fix 2021
11. Optimizing Pinterest’s Data Ingestion Stack: Findings and Learnings Pinterest 2022
12. Lessons Learned From Running Apache Airflow at Scale Shopify 2022
13. Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training Meta 2022
14. Data Mesh — A Data Movement and Processing Platform @ Netflix Netflix 2022
15. Building Scalable Real Time Event Processing with Kafka and Flink DoorDash 2022

Data Discovery

1. Apache Atlas: Data Goverance and Metadata Framework for Hadoop (Code) Apache
2. Collect, Aggregate, and Visualize a Data Ecosystem's Metadata (Code) WeWork
3. Discovery and Consumption of Analytics Data at Twitter Twitter 2016
4. Democratizing Data at Airbnb Airbnb 2017
5. Databook: Turning Big Data into Knowledge with Metadata at Uber Uber 2018
6. Metacat: Making Big Data Discoverable and Meaningful at Netflix (Code) Netflix 2018
7. Amundsen — Lyft’s Data Discovery & Metadata Engine Lyft 2019
8. Open Sourcing Amundsen: A Data Discovery And Metadata Platform (Code) Lyft 2019
9. DataHub: A Generalized Metadata Search & Discovery Tool (Code) LinkedIn 2019
10. Amundsen: One Year Later Lyft 2020
11. Using Amundsen to Support User Privacy via Metadata Collection at Square Square 2020
12. Turning Metadata Into Insights with Databook Uber 2020
13. DataHub: Popular Metadata Architectures Explained LinkedIn 2020
14. How We Improved Data Discovery for Data Scientists at Spotify Spotify 2020
15. How We’re Solving Data Discovery Challenges at Shopify Shopify 2020
16. Nemo: Data discovery at Facebook Facebook 2020
17. Exploring Data @ Netflix (Code) Netflix 2021

Feature Stores

1. Distributed Time Travel for Feature Generation Netflix 2016
2. Building the Activity Graph, Part 2 (Feature Storage Section) LinkedIn 2017
3. Fact Store at Scale for Netflix Recommendations Netflix 2018
4. Zipline: Airbnb’s Machine Learning Data Management Platform Airbnb 2018
5. Feature Store: The missing data layer for Machine Learning pipelines? Hopsworks 2018
6. Introducing Feast: An Open Source Feature Store for Machine Learning (Code) Gojek 2019
7. Michelangelo Palette: A Feature Engineering Platform at Uber Uber 2019
8. The Architecture That Powers Twitter's Feature Store Twitter 2019
9. Accelerating Machine Learning with the Feature Store Service Condé Nast 2019
10. Feast: Bridging ML Models and Data Gojek 2020
11. Building a Scalable ML Feature Store with Redis, Binary Serialization, and Compression DoorDash 2020
12. Rapid Experimentation Through Standardization: Typed AI features for LinkedIn’s Feed LinkedIn 2020
13. Building a Feature Store Monzo Bank 2020
14. Butterfree: A Spark-based Framework for Feature Store Building (Code) QuintoAndar 2020
15. Building Riviera: A Declarative Real-Time Feature Engineering Framework DoorDash 2021
16. Optimal Feature Discovery: Better, Leaner Machine Learning Models Through Information Theory Uber 2021
17. ML Feature Serving Infrastructure at Lyft Lyft 2021
18. Near real-time features for near real-time personalization LinkedIn 2022
19. Building the Model Behind DoorDash’s Expansive Merchant Selection DoorDash 2022
20. Open sourcing Feathr – LinkedIn’s feature store for productive machine learning LinkedIn 2022
21. Evolution of ML Fact Store Netflix 2022
22. Developing scalable feature engineering DAGs Metaflow + Hamilton via Outerbounds 2022
23. Feature Store Design at Constructor Constructor.io 2023

Classification

1. Prediction of Advertiser Churn for Google AdWords (Paper) Google 2010
2. High-Precision Phrase-Based Document Classification on a Modern Scale (Paper) LinkedIn 2011
3. Chimera: Large-scale Classification using Machine Learning, Rules, and Crowdsourcing (Paper) Walmart 2014
4. Large-scale Item Categorization in e-Commerce Using Multiple Recurrent Neural Networks (Paper) NAVER 2016
5. Learning to Diagnose with LSTM Recurrent Neural Networks (Paper) Google 2017
6. Discovering and Classifying In-app Message Intent at Airbnb Airbnb 2019
7. Teaching Machines to Triage Firefox Bugs Mozilla 2019
8. Categorizing Products at Scale Shopify 2020
9. How We Built the Good First Issues Feature GitHub 2020
10. Testing Firefox More Efficiently with Machine Learning Mozilla 2020
11. Using ML to Subtype Patients Receiving Digital Mental Health Interventions (Paper) Microsoft 2020
12. Scalable Data Classification for Security and Privacy (Paper) Facebook 2020
13. Uncovering Online Delivery Menu Best Practices with Machine Learning DoorDash 2020
14. Using a Human-in-the-Loop to Overcome the Cold Start Problem in Menu Item Tagging DoorDash 2020
15. Deep Learning: Product Categorization and Shelving Walmart 2021
16. Large-scale Item Categorization for e-Commerce (Paper) DianPing, eBay 2012
17. Semantic Label Representation with an Application on Multimodal Product Categorization Walmart 2022
18. Building Airbnb Categories with ML and Human-in-the-Loop Airbnb 2022

Regression

1. Using Machine Learning to Predict Value of Homes On Airbnb Airbnb 2017
2. Using Machine Learning to Predict the Value of Ad Requests Twitter 2020
3. Open-Sourcing Riskquant, a Library for Quantifying Risk (Code) Netflix 2020
4. Solving for Unobserved Data in a Regression Model Using a Simple Data Adjustment DoorDash 2020

Forecasting

1. Engineering Extreme Event Forecasting at Uber with RNN Uber 2017
2. Forecasting at Uber: An Introduction Uber 2018
3. Transforming Financial Forecasting with Data Science and Machine Learning at Uber Uber 2018
4. Under the Hood of Gojek’s Automated Forecasting Tool Gojek 2019
5. BusTr: Predicting Bus Travel Times from Real-Time Traffic (Paper, Video) Google 2020
6. Retraining Machine Learning Models in the Wake of COVID-19 DoorDash 2020
7. Automatic Forecasting using Prophet, Databricks, Delta Lake and MLflow (Paper, Code) Atlassian 2020
8. Introducing Orbit, An Open Source Package for Time Series Inference and Forecasting (Paper, Video, Code) Uber 2021
9. Managing Supply and Demand Balance Through Machine Learning DoorDash 2021
10. Greykite: A flexible, intuitive, and fast forecasting library LinkedIn 2021
11. The history of Amazon’s forecasting algorithm Amazon 2021
12. DeepETA: How Uber Predicts Arrival Times Using Deep Learning Uber 2022
13. Forecasting Grubhub Order Volume At Scale Grubhub 2022
14. Causal Forecasting at Lyft (Part 1) Lyft 2022

Recommendation

1. Amazon.com Recommendations: Item-to-Item Collaborative Filtering (Paper) Amazon 2003
2. Netflix Recommendations: Beyond the 5 stars (Part 1 (Part 2) Netflix 2012
3. How Music Recommendation Works — And Doesn’t Work Spotify 2012
4. Learning to Rank Recommendations with the k -Order Statistic Loss (Paper) Google 2013
5. Recommending Music on Spotify with Deep Learning Spotify 2014
6. Learning a Personalized Homepage Netflix 2015
7. The Netflix Recommender System: Algorithms, Business Value, and Innovation (Paper) Netflix 2015
8. Session-based Recommendations with Recurrent Neural Networks (Paper) Telefonica 2016
9. Deep Neural Networks for YouTube Recommendations YouTube 2016
10. E-commerce in Your Inbox: Product Recommendations at Scale (Paper) Yahoo 2016
11. To Be Continued: Helping you find shows to continue watching on Netflix Netflix 2016
12. Personalized Recommendations in LinkedIn Learning LinkedIn 2016
13. Personalized Channel Recommendations in Slack Slack 2016
14. Recommending Complementary Products in E-Commerce Push Notifications (Paper) Alibaba 2017
15. Artwork Personalization at Netflix Netflix 2017
16. A Meta-Learning Perspective on Cold-Start Recommendations for Items (Paper) Twitter 2017
17. Pixie: A System for Recommending 3+ Billion Items to 200+ Million Users in Real-Time (Paper) Pinterest 2017
18. Powering Search & Recommendations at DoorDash DoorDash 2017
19. How 20th Century Fox uses ML to predict a movie audience (Paper) 20th Century Fox 2018
20. Calibrated Recommendations (Paper) Netflix 2018
21. Food Discovery with Uber Eats: Recommending for the Marketplace Uber 2018
22. Explore, Exploit, and Explain: Personalizing Explainable Recommendations with Bandits (Paper) Spotify 2018
23. Talent Search and Recommendation Systems at LinkedIn: Practical Challenges and Lessons Learned (Paper) LinkedIn 2018
24. Behavior Sequence Transformer for E-commerce Recommendation in Alibaba (Paper) Alibaba 2019
25. SDM: Sequential Deep Matching Model for Online Large-scale Recommender System (Paper) Alibaba 2019
26. Multi-Interest Network with Dynamic Routing for Recommendation at Tmall (Paper) Alibaba 2019
27. Personalized Recommendations for Experiences Using Deep Learning TripAdvisor 2019
28. Powered by AI: Instagram’s Explore recommender system Facebook 2019
29. Marginal Posterior Sampling for Slate Bandits (Paper) Netflix 2019
30. Food Discovery with Uber Eats: Using Graph Learning to Power Recommendations Uber 2019
31. Music recommendation at Spotify Spotify 2019
32. Using Machine Learning to Predict what File you Need Next (Part 1) Dropbox 2019
33. Using Machine Learning to Predict what File you Need Next (Part 2) Dropbox 2019
34. Learning to be Relevant: Evolution of a Course Recommendation System (PAPER NEEDED)LinkedIn 2019
35. Temporal-Contextual Recommendation in Real-Time (Paper) Amazon 2020
36. P-Companion: A Framework for Diversified Complementary Product Recommendation (Paper) Amazon 2020
37. Deep Interest with Hierarchical Attention Network for Click-Through Rate Prediction (Paper) Alibaba 2020
38. TPG-DNN: A Method for User Intent Prediction with Multi-task Learning (Paper) Alibaba 2020
39. PURS: Personalized Unexpected Recommender System for Improving User Satisfaction (Paper) Alibaba 2020
40. Controllable Multi-Interest Framework for Recommendation (Paper) Alibaba 2020
41. MiNet: Mixed Interest Network for Cross-Domain Click-Through Rate Prediction (Paper) Alibaba 2020
42. ATBRG: Adaptive Target-Behavior Relational Graph Network for Effective Recommendation (Paper) Alibaba 2020
43. For Your Ears Only: Personalizing Spotify Home with Machine Learning Spotify 2020
44. Reach for the Top: How Spotify Built Shortcuts in Just Six Months Spotify 2020
45. Contextual and Sequential User Embeddings for Large-Scale Music Recommendation (Paper) Spotify 2020
46. The Evolution of Kit: Automating Marketing Using Machine Learning Shopify 2020
47. A Closer Look at the AI Behind Course Recommendations on LinkedIn Learning (Part 1) LinkedIn 2020
48. A Closer Look at the AI Behind Course Recommendations on LinkedIn Learning (Part 2) LinkedIn 2020
49. Building a Heterogeneous Social Network Recommendation System LinkedIn 2020
50. How TikTok recommends videos #ForYou ByteDance 2020
51. Zero-Shot Heterogeneous Transfer Learning from RecSys to Cold-Start Search Retrieval (Paper) Google 2020
52. Improved Deep & Cross Network for Feature Cross Learning in Web-scale LTR Systems (Paper) Google 2020
53. Mixed Negative Sampling for Learning Two-tower Neural Networks in Recommendations (Paper) Google 2020
54. Future Data Helps Training: Modeling Future Contexts for Session-based Recommendation (Paper) Tencent 2020
55. A Case Study of Session-based Recommendations in the Home-improvement Domain (Paper) Home Depot 2020
56. Balancing Relevance and Discovery to Inspire Customers in the IKEA App (Paper) Ikea 2020
57. How we use AutoML, Multi-task learning and Multi-tower models for Pinterest Ads Pinterest 2020
58. Multi-task Learning for Related Products Recommendations at Pinterest Pinterest 2020
59. Improving the Quality of Recommended Pins with Lightweight Ranking Pinterest 2020
60. Multi-task Learning and Calibration for Utility-based Home Feed Ranking Pinterest 2020
61. Personalized Cuisine Filter Based on Customer Preference and Local Popularity DoorDash 2020
62. How We Built a Matchmaking Algorithm to Cross-Sell Products Gojek 2020
63. Lessons Learned Addressing Dataset Bias in Model-Based Candidate Generation (Paper) Twitter 2021
64. Self-supervised Learning for Large-scale Item Recommendations (Paper) Google 2021
65. Deep Retrieval: End-to-End Learnable Structure Model for Large-Scale Recommendations (Paper) ByteDance 2021
66. Using AI to Help Health Experts Address the COVID-19 Pandemic Facebook 2021
67. Advertiser Recommendation Systems at Pinterest Pinterest 2021
68. On YouTube's Recommendation System YouTube 2021
69. "Are you sure?": Preliminary Insights from Scaling Product Comparisons to Multiple Shops Coveo 2021
70. Mozrt, a Deep Learning Recommendation System Empowering Walmart Store Associates Walmart 2021
71. Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training (Paper) Meta 2021
72. The Amazon Music conversational recommender is hitting the right notes Amazon 2022
73. Personalized complementary product recommendation (Paper) Amazon 2022
74. Building a Deep Learning Based Retrieval System for Personalized Recommendations eBay 2022
75. How We Built: An Early-Stage Machine Learning Model for Recommendations Peloton 2022
76. Lessons Learned from Building out Context-Aware Recommender Systems Peloton 2022
77. Beyond Matrix Factorization: Using hybrid features for user-business recommendations Yelp 2022
78. Improving job matching with machine-learned activity features LinkedIn 2022
79. Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training Meta 2022
80. Blueprints for recommender system architectures: 10th anniversary edition Xavier Amatriain 2022
81. How Pinterest Leverages Realtime User Actions in Recommendation to Boost Homefeed Engagement Volume Pinterest 2022
82. RecSysOps: Best Practices for Operating a Large-Scale Recommender System Netflix 2022
83. Recommend API: Unified end-to-end machine learning infrastructure to generate recommendations Slack 2022
84. Evolving DoorDash’s Substitution Recommendations Algorithm DoorDash 2022
85. Homepage Recommendation with Exploitation and Exploration DoorDash 2022
86. GPU-accelerated ML Inference at Pinterest Pinterest 2022
87. Addressing Confounding Feature Issue for Causal Recommendation (Paper) Tencent 2022

Search & Ranking

1. Amazon Search: The Joy of Ranking Products (Paper, Video, Code) Amazon 2016
2. How Lazada Ranks Products to Improve Customer Experience and Conversion Lazada 2016
3. Ranking Relevance in Yahoo Search (Paper) Yahoo 2016
4. Learning to Rank Personalized Search Results in Professional Networks (Paper) LinkedIn 2016
5. Using Deep Learning at Scale in Twitter’s Timelines Twitter 2017
6. An Ensemble-based Approach to Click-Through Rate Prediction for Promoted Listings at Etsy (Paper) Etsy 2017
7. Powering Search & Recommendations at DoorDash DoorDash 2017
8. Applying Deep Learning To Airbnb Search (Paper) Airbnb 2018
9. In-session Personalization for Talent Search (Paper) LinkedIn 2018
10. Talent Search and Recommendation Systems at LinkedIn (Paper) LinkedIn 2018
11. Food Discovery with Uber Eats: Building a Query Understanding Engine Uber 2018
12. Globally Optimized Mutual Influence Aware Ranking in E-Commerce Search (Paper) Alibaba 2018
13. Reinforcement Learning to Rank in E-Commerce Search Engine (Paper) Alibaba 2018
14. Semantic Product Search (Paper) Amazon 2019
15. Machine Learning-Powered Search Ranking of Airbnb Experiences Airbnb 2019
16. Entity Personalized Talent Search Models with Tree Interaction Features (Paper) LinkedIn 2019
17. The AI Behind LinkedIn Recruiter Search and recommendation systems LinkedIn 2019
18. Learning Hiring Preferences: The AI Behind LinkedIn Jobs LinkedIn 2019
19. The Secret Sauce Behind Search Personalisation Gojek 2019
20. Neural Code Search: ML-based Code Search Using Natural Language Queries Facebook 2019
21. Aggregating Search Results from Heterogeneous Sources via Reinforcement Learning (Paper) Alibaba 2019
22. Cross-domain Attention Network with Wasserstein Regularizers for E-commerce Search Alibaba 2019
23. Understanding Searches Better Than Ever Before (Paper) Google 2019
24. How We Used Semantic Search to Make Our Search 10x Smarter Tokopedia 2019
25. Query2vec: Search query expansion with query embeddings GrubHub 2019
26. MOBIUS: Towards the Next Generation of Query-Ad Matching in Baidu’s Sponsored Search Baidu 2019
27. Why Do People Buy Seemingly Irrelevant Items in Voice Product Search? (Paper) Amazon 2020
28. Managing Diversity in Airbnb Search (Paper) Airbnb 2020
29. Improving Deep Learning for Airbnb Search (Paper) Airbnb 2020
30. Quality Matches Via Personalized AI for Hirer and Seeker Preferences LinkedIn 2020
31. Understanding Dwell Time to Improve LinkedIn Feed Ranking LinkedIn 2020
32. Ads Allocation in Feed via Constrained Optimization (Paper, Video) LinkedIn 2020
33. Understanding Dwell Time to Improve LinkedIn Feed Ranking LinkedIn 2020
34. AI at Scale in Bing Microsoft 2020
35. Query Understanding Engine in Traveloka Universal Search Traveloka 2020
36. Bayesian Product Ranking at Wayfair Wayfair 2020
37. COLD: Towards the Next Generation of Pre-Ranking System (Paper) Alibaba 2020
38. Shop The Look: Building a Large Scale Visual Shopping System at Pinterest (Paper, Video) Pinterest 2020
39. Driving Shopping Upsells from Pinterest Search Pinterest 2020
40. GDMix: A Deep Ranking Personalization Framework (Code) LinkedIn 2020
41. Bringing Personalized Search to Etsy Etsy 2020
42. Building a Better Search Engine for Semantic Scholar Allen Institute for AI 2020
43. Query Understanding for Natural Language Enterprise Search (Paper) Salesforce 2020
44. Things Not Strings: Understanding Search Intent with Better Recall DoorDash 2020
45. Query Understanding for Surfacing Under-served Music Content (Paper) Spotify 2020
46. Embedding-based Retrieval in Facebook Search (Paper) Facebook 2020
47. Towards Personalized and Semantic Retrieval for E-commerce Search via Embedding Learning (Paper) JD 2020
48. QUEEN: Neural query rewriting in e-commerce (Paper) Amazon 2021
49. Using Learning-to-rank to Precisely Locate Where to Deliver Packages (Paper) Amazon 2021
50. Seasonal relevance in e-commerce search (Paper) Amazon 2021
51. Graph Intention Network for Click-through Rate Prediction in Sponsored Search (Paper) Alibaba 2021
52. How We Built A Context-Specific Bidding System for Etsy Ads Etsy 2021
53. Pre-trained Language Model based Ranking in Baidu Search (Paper) Baidu 2021
54. Stitching together spaces for query-based recommendations Stitch Fix 2021
55. Deep Natural Language Processing for LinkedIn Search Systems (Paper) LinkedIn 2021
56. Siamese BERT-based Model for Web Search Relevance Ranking (Paper, Code) Seznam 2021
57. SearchSage: Learning Search Query Representations at Pinterest Pinterest 2021
58. Query2Prod2Vec: Grounded Word Embeddings for eCommerce Coveo 2021
59. 3 Changes to Expand DoorDash’s Product Search Beyond Delivery DoorDash 2022
60. Learning To Rank Diversely Airbnb 2022
61. How to Optimise Rankings with Cascade Bandits Expedia 2022
62. A Guide to Google Search Ranking Systems Google 2022
63. Deep Learning for Search Ranking at Etsy Etsy 2022
64. Search at Calm Calm 2022

README 内容较长，此处已截断，完整内容请查看 GitHub 仓库。