Local AI object removal in Rust for PhotoCraft

I wanted to add local AI model support to the Remove and Smart Fill tools in PhotoCraft, an open-source image editor written in Rust. The goal was to remove unwanted content and reconstruct the area from the surrounding image, with inference running on the user's machine.

That work led me to OSOR, or One-Step Object Removal. OSOR-rs is my native Rust inference implementation of its SDXL-Inpainting Phase II model. I built the library so I could integrate that model into PhotoCraft's native Rust application.

Two tools, different rules

With Remove, you paint over unwanted content. Smart Fill starts with a selection and fills that selected area. Both need image completion, but they have different rules for which pixels the result may change.

In the PhotoCraft integration I'm working on, Smart Fill's AI path currently uses LaMa. Remove also has a combined LaMa and OSOR workflow. Its automatic routing uses the geometry of the painted marks: long, narrow footprints go to LaMa, while compact footprints go to OSOR. Detail and object settings can override that choice. This is a geometric rule, not recognition of what the object is.

My original goal covered both tools. OSOR became especially interesting for removal because an object's effects can extend outside the area you painted. The model service and editing workflow need to handle that difference.

The object can leave a shadow

Removing an object doesn't always remove the evidence that it was there. If you paint over a person, their shadow may still be on the ground. A reflection can have the same problem. Filling only the painted pixels leaves those effects behind.

OSOR predicts an edit region as well as the completed image. That region can include effects outside the original mask. This was the part I wanted to explore for PhotoCraft. A removal tool needs to account for the surrounding scene, and the user's brush stroke won't necessarily describe every pixel that should change.

That doesn't mean every OSOR removal should change pixels outside the mask. The integration makes effect removal an explicit previewed operation, while ordinary removal stays within the painted area. The editor still has to respect selections and protected pixels. The useful result is the one you can review and apply deliberately.

Keeping inference in Rust

PhotoCraft already uses Rust for its image engine and native interface. I wanted model inference to fit into that application without starting and managing a separate Python process.

OSOR-rs implements the SDXL-Inpainting Phase II version of OSOR and loads its model bundle from local files. The crate is named osor; the repository remains osor-rs. Candle handles CPU, Metal, and CUDA execution, with an optional Burn backend for WGPU. Once the weights are prepared, inference can run offline.

The library is separate from PhotoCraft. It takes image and mask tensors and returns RGB with the predicted edit alpha. PhotoCraft owns installation, image preparation, and the document edit. Another Rust application can use the same inference code with a different editing workflow.

Making it work like an editing tool

Getting an image out of the model is one part of the work. PhotoCraft also needs to manage memory, run inference away from the UI, and handle cancellation. The integration retains loaded models between jobs when its memory budget permits.

I want to inspect a result before changing the document. If I discard the preview, the source should stay as it was. If I apply it, Undo should restore the previous image. Those paths are part of the integration work, alongside cancellation and model reuse.

This is work in an integration branch, rather than a claim that every released PhotoCraft build includes it. The current OSOR bridge limits model context to a 512-pixel long side and uses a conservative 40 GiB model working-set estimate for memory admission. Local inference still has substantial resource requirements, and image quality depends on the scene and mask.

The OSOR-rs library guide covers the API, tensor inputs, model bundles, and backend setup. The source is on GitHub, and you can follow the application work in the PhotoCraft repository.