From Photo to JSON: How I Built a Receipt Scanner Using Gemini and Swift

The shoebox of faded receipts is a universal symbol of financial disorganization. For freelancers, small business owners, and meticulous budgeters, manually entering expenses is a tedious and error-prone task. Traditional Optical Character Recognition (OCR) tools offered a partial solution, but they often struggled with crumpled, poorly lit, or uniquely formatted receipts. Today, we have better options. This article explores the technology behind Receipt Lens, an iOS app I developed to address this challenge. The app doesn’t just scan receipts—it interprets them. By leveraging Google’s advanced multimodal AI, Gemini, it turns a simple photo into structured, actionable financial data. This is a look at how the process works, from the user’s camera to the AI’s processing and back. Screenshots from the actual iOS app. The Problem with Paper (and Basic OCR) A receipt isn’t just text; it’s a document with inherent structure and context. Traditional OCR scanners might extract the text but often fail to recognize the relationships between items, prices, and dates. They lack the ability to distinguish between different sections of the receipt, such as the total, tax, or individual line items. This limitation makes the data they produce difficult to use for automatic expense tracking or financial reporting. Multimodal AI: Seeing and Understanding Gemini, Google’s powerful AI model, is designed to process multiple types of data, including text and images. This makes it ideal for tasks like receipt scanning, where both visual and textual information are key. The app uses Gemini to analyze the image of a receipt, identify its layout, and extract relevant data points. The process begins when a user takes a photo of a receipt with their iPhone. The image is then sent to Gemini, which processes it using multimodal techniques. This means the AI isn’t just reading text—it’s also understanding the visual structure of the receipt, such as where the total is located, which items are listed, and how the information is organized. Prompt Engineering: Teaching the AI What to Look For To make this work effectively, I spent significant time crafting the prompts that guide Gemini’s analysis. These prompts help the AI understand what to look for in the receipt and how to structure the output. For example, I designed prompts that explicitly ask Gemini to extract the store name, date, items purchased, and total amount, and then format that information into a JSON structure. The JSON output is key because it allows the app to easily integrate with other financial tools and services. It ensures that the data is consistent, organized, and ready for further processing or analysis. Building the App with Swift The app itself was built using Swift, Apple’s programming language for iOS development. It uses the camera API to capture the image, processes it through the AI model, and then displays the structured data to the user. The app also includes features like real-time feedback, allowing users to adjust the image if the AI isn’t capturing the data accurately. The AI’s ability to understand context and layout is what sets Receipt Lens apart from basic OCR tools. It can handle receipts that are not perfectly aligned, have varying fonts, or are partially obscured. This level of adaptability is crucial for making the app useful in real-world scenarios. Why This Matters The rise of generative AI and multimodal models has opened new possibilities for how we interact with digital data. Receipt Lens is a small but meaningful example of how these technologies can simplify everyday tasks and reduce the burden of manual data entry. As AI continues to evolve, we can expect more tools like this that bridge the gap between the physical and digital worlds. By combining the power of AI with the simplicity of a mobile app, Receipt Lens demonstrates the potential of modern technology to make financial management easier, more accurate, and more efficient for users.