Building a Mask Generation App with Streamlit and Segment Anything Model (SAM)

## Introduction Image segmentation is one of the most important tasks in modern computer vision. Unlike traditional image classification, which assigns a single label to an entire image, segmentation identifies and separates individual objects or regions at the pixel level. This capability enables machines to understand image content with much greater precision. Recent advancements in AI have led to powerful segmentation models capable of generating object masks automatically. One such breakthrough is Meta AI's Segment Anything Model (SAM), which can identify and segment objects in images without requiring task-specific training. The **Mask Generation App** is a Streamlit-based application that leverages the Segment Anything Model (SAM) to automatically generate segmentation masks from uploaded images. Users can upload an image, generate object masks, and visualize segmented regions directly within a web browser. This project demonstrates how modern foundation models can be integrated into interactive applications to perform advanced image understanding tasks with minimal user effort. --- ## Problem Statement Many computer vision applications require accurate identification and separation of objects from images. Traditional segmentation techniques often require: * Manual annotation * Object-specific training * Complex preprocessing pipelines * Large labeled datasets These requirements make segmentation difficult to implement and scale. The challenge is to create a system that can automatically identify and segment objects from images without requiring custom model training or extensive configuration. The Mask Generation App addresses this challenge by utilizing the Segment Anything Model (SAM), which can generate segmentation masks for various objects and regions automatically. --- ## Features The application provides several powerful features for image segmentation. ### Image Upload Support Users can upload images in popular formats: * PNG * JPG * JPEG ### Automatic Mask Generation The application automatically detects and generates segmentation masks for objects within the image. ### Interactive Streamlit Interface A clean and intuitive user interface allows users to generate masks with a single click. ### Multiple Mask Output The model generates multiple candidate masks for different objects and regions. ### Visual Mask Display Generated masks are displayed directly in the browser for easy inspection. ### Efficient Model Loading The SAM model is loaded once and cached using Streamlit's resource caching mechanism. ### AI-Powered Segmentation Powered by Meta AI's Segment Anything Model (SAM), enabling state-of-the-art segmentation performance. --- ## Technologies Used The project integrates several modern AI and web development technologies. | Technology | Purpose | | ---------------------------- | ----------------------------- | | Python | Core programming language | | Streamlit | Web application framework | | Transformers | Model inference framework | | Segment Anything Model (SAM) | Image segmentation | | Hugging Face | Model hosting and integration | | Pillow (PIL) | Image processing | | NumPy | Numerical array processing | | PyTorch | Deep learning backend | These technologies work together to provide efficient image segmentation capabilities. --- ## How It Works The application uses the following pretrained segmentation model: ```python facebook/sam-vit-base ``` The Segment Anything Model (SAM) is designed to identify and segment objects within images automatically. The process works as follows: 1. The user uploads an image. 2. The image is passed to the SAM model. 3. The model analyzes image features and object boundaries. 4. Multiple segmentation masks are generated. 5. The masks are converted into displayable image formats. 6. The masks are shown within the Streamlit interface. Unlike traditional segmentation pipelines, SAM can generalize across a wide range of objects and scenes without requiring additional training. --- ## Application Workflow ### Step 1: Upload Image The user uploads an image through the Streamlit interface. ### Step 2: Image Preview The uploaded image is displayed for verification. ### Step 3: Generate Masks The user clicks the **Generate Masks** button. ### Step 4: AI Segmentation The SAM model processes the image and detects meaningful object regions. ### Step 5: Mask Extraction The generated masks are extracted from the model output. ### Step 6: Tensor Conversion The masks are converted from PyTorch tensors into NumPy arrays for visualization. ```python mask_np = mask.cpu().numpy() mask_np = (mask_np * 255).astype("uint8") ``` ### Step 7: Display Results The generated masks are displayed as separate grayscale images. --- ## Example Input ### Uploaded Image An image containing: ```text A dog sitting on grass with a tree in the background. ``` The image may contain multiple objects such as: * Dog * Grass * Tree * Sky --- ## Example Output ### Generated Masks #### Mask 1 ```text Object: Dog ``` A segmentation mask highlighting only the dog. #### Mask 2 ```text Region: Grass ``` A segmentation mask isolating the grassy area. #### Mask 3 ```text Object: Tree ``` A segmentation mask highlighting the tree region. The Streamlit interface displays each generated mask individually. Example display: ```text Generated Masks Mask 1 [Dog Segmentation] Mask 2 [Grass Segmentation] Mask 3 [Tree Segmentation] ``` --- ## Use Cases The Mask Generation App has applications across numerous industries and domains. ### Autonomous Vehicles Identify roads, pedestrians, vehicles, and traffic signs. ### Medical Imaging Segment organs, tumors, and anatomical structures. ### Robotics Enable robots to recognize and interact with objects. ### E-Commerce Remove backgrounds and isolate products. ### Augmented Reality Detect objects for real-time visual overlays. ### Content Creation Create object cutouts and image editing effects. ### Satellite Imaging Segment land, water bodies, vegetation, and urban regions. ### Research and Education Demonstrate modern segmentation techniques using foundation models. --- ## Future Improvements Several enhancements could further improve the application. ### Colored Mask Overlays Display masks directly on top of the original image. ### Object Labels Automatically assign names to segmented objects. ### Interactive Selection Allow users to select specific objects for segmentation. ### High-Resolution Processing Support large images while maintaining performance. ### Batch Processing Enable multiple image uploads simultaneously. ### Download Options Allow users to download generated masks. ### Video Segmentation Extend functionality from images to video streams. ### Multi-Class Segmentation Categorize segmented regions into predefined object classes. --- ## Conclusion The Mask Generation App demonstrates how modern foundation models can simplify complex computer vision tasks. By integrating Meta AI's Segment Anything Model (SAM) with Streamlit, the application provides an intuitive platform for generating segmentation masks from uploaded images. The project showcases the power of automatic image segmentation and highlights how advanced AI models can be deployed in user-friendly web applications. Whether used in research, education, robotics, healthcare, or content creation, segmentation technology plays a critical role in enabling machines to understand visual information. As computer vision continues to evolve, applications built on foundation models such as SAM will become increasingly valuable for building scalable, intelligent, and highly adaptable image understanding systems.