Whirlpool Corporation, a fortune 500 company headquartered in Benton Harbor, Michigan, is the world’s leading home appliance company with over 50 manufacturing and research centers. Whirlpool is in constant pursuit of improving life at home through their reliable appliances.

In this spirit, Whirlpool is working to make cooking more accessible to all through the development of a smart oven. An oven of this nature provides users with insight and instruction to improve the result of the dish.

Our DeepOven system contributes to the larger Whirlpool smart oven goal by estimating volume and quantity of food inside a Whirlpool smart oven. Through a camera inside their oven, users see a livestream of their food cooking and leverage the camera and our tool to improve their cooking experience.

Connecting to the oven brings the user to the livestream view from inside the oven. Using either a frame from the livestream or a pre-captured image, the user initiates our system, which uses advanced machine learning to automatically determine the oven’s rack level, the food quantity, and the total estimated volume of the food in the oven.

Our system produces a 3D reconstruction of the food in the oven, accompanied with the original image and any statistics and data determined by our software. Whirlpool uses this data to improve the performance of their smart ovens, enabling them to better estimate the cooking time required in many scenarios.

The web application is built with React as the front end and utilizes Flask and Python for the back end. Food quantity detection is achieved with a custom trained YOLOv8 instance segmentation model. A convolutional neural network (CNN) model determines the rack level, and a differentiable volumetric rendering model calculates the volume and creates the 3D image of the food.