Project 6: Build an Image Analysis Application

This project builds an application that classifies and extracts structured information from images, using the vision and multimodal techniques from Unit 18. The scenario is a product-photo intake pipeline for an e-commerce catalog: images arrive from sellers, and the system needs to extract structured attributes, flag quality issues, and detect policy violations before a listing goes live.

Scope and Design Decisions

The application takes an image (a URL or local file) and produces a structured record: detected product category, extracted visible attributes (color, apparent material, condition), a quality flag, and a content-policy flag. It does not attempt image generation or editing — this is a pure analysis pipeline.

Two decisions matter most:

1. Every extraction is a structured output, never freeform description. A catalog pipeline needs machine-usable fields (category, color, condition), not a paragraph a downstream system would need to re-parse — this reuses the structured-output discipline from Unit 6, applied to vision input instead of text.
2. Quality and policy checks are separate calls from attribute extraction. Bundling "what is in this image" with "is this image acceptable" into a single call makes failures ambiguous — a low-quality or policy-violating image should still be intelligible in the extraction step where possible, and each check should be independently retriable and independently loggable.

Structured Attribute Extraction

from openai import OpenAI
from pydantic import BaseModel
from enum import Enum

client = OpenAI()

class Condition(str, Enum):
    new = "new"
    like_new = "like_new"
    used_good = "used_good"
    used_fair = "used_fair"
    damaged = "damaged"

class ProductAttributes(BaseModel):
    category: str
    primary_color: str
    apparent_material: str | None
    condition: Condition
    visible_defects: list[str]

def extract_product_attributes(image_url: str) -> ProductAttributes:
    response = client.responses.parse(
        model="gpt-5.6-terra",
        input=[{
            "role": "user",
            "content": [
                {"type": "input_text", "text": (
                    "Extract catalog attributes from this product photo. If a "
                    "defect is visible (scratches, stains, tears, missing parts), "
                    "list it explicitly in visible_defects."
                )},
                {"type": "input_image", "image_url": image_url},
            ],
        }],
        text_format=ProductAttributes,
    )
    return response.output_parsed

The content list mixes an input_text block with an input_image block in a single user message — this is the standard multimodal message shape from Unit 18, and it matters that the instruction text is included alongside the image in the same message rather than as a separate system prompt turn, since the model needs both pieces of context together to ground its extraction in what it is actually looking at. Condition is an enum rather than a free string, which constrains the model to a fixed, known vocabulary that downstream catalog code can switch on directly without needing to normalize inconsistent phrasing like "gently used" versus "used - good condition."

visible_defects is a list rather than a single optional string because a real product photo can show multiple independent issues (a scratch and a missing button), and collapsing them into one field would force the model to either pick one or awkwardly concatenate — a list is the correct shape for "zero or more of a kind of thing," a modeling choice worth applying generally whenever a field could plausibly have more than one value.

Quality and Policy Checks

class ImageQualityCheck(BaseModel):
    is_acceptable_quality: bool
    quality_issues: list[str]
    is_policy_compliant: bool
    policy_concerns: list[str]

def check_image_quality_and_policy(image_url: str) -> ImageQualityCheck:
    response = client.responses.parse(
        model="gpt-5.6-terra",
        input=[{
            "role": "system",
            "content": (
                "You are a content moderator for an e-commerce image pipeline. "
                "Flag quality issues (blur, poor lighting, watermarks obscuring "
                "the product, wrong aspect ratio) and policy concerns (visible "
                "faces of bystanders, brand logos suggesting counterfeit goods, "
                "prohibited item categories) independently."
            ),
        }, {
            "role": "user",
            "content": [{"type": "input_image", "image_url": image_url}],
        }],
        text_format=ImageQualityCheck,
    )
    return response.output_parsed

Quality and policy are modeled as two independent boolean-plus-reasons pairs in the same schema rather than a single pass/fail flag, because a listing can fail one and pass the other — a perfectly compliant photo can still be too blurry to publish, and a sharp, well-lit photo can still show a counterfeit logo. Keeping them as separate fields, each with its own explanatory list, means the pipeline can route these two failure types to different remediation flows (ask the seller to retake the photo versus escalate to a trust-and-safety review).

Note: What counts as a policy violation is domain- and platform-specific, and the system prompt above is illustrative. A production deployment should encode the platform's actual content policy explicitly, ideally as a shared, versioned document referenced consistently across every moderation call rather than restated ad hoc in each prompt.

The Intake Pipeline

from dataclasses import dataclass

@dataclass
class IntakeResult:
    image_url: str
    attributes: ProductAttributes | None
    quality: ImageQualityCheck
    approved: bool

def process_listing_image(image_url: str) -> IntakeResult:
    quality = check_image_quality_and_policy(image_url)

    if not quality.is_policy_compliant:
        return IntakeResult(image_url=image_url, attributes=None, quality=quality, approved=False)

    attributes = extract_product_attributes(image_url)

    approved = quality.is_acceptable_quality and attributes.condition != Condition.damaged
    return IntakeResult(image_url=image_url, attributes=attributes, quality=quality, approved=approved)

The pipeline short-circuits on a policy violation before ever calling attribute extraction — there is no catalog value in extracting the color and material of an image that will be rejected outright, and skipping that call saves cost and avoids ever storing structured metadata about content that should not be in the system at all. Quality issues, by contrast, do not block extraction: a slightly blurry but otherwise policy-compliant photo can still usefully report its category and color, and the pipeline simply marks it as unapproved for publishing pending a better photo.

Batch Processing Multiple Angles

def process_listing(image_urls: list[str]) -> list[IntakeResult]:
    return [process_listing_image(url) for url in image_urls]

def summarize_listing(results: list[IntakeResult]) -> dict:
    approved_count = sum(1 for r in results if r.approved)
    all_defects = sorted({
        defect
        for r in results
        if r.attributes
        for defect in r.attributes.visible_defects
    })
    return {
        "total_images": len(results),
        "approved_images": approved_count,
        "needs_new_photos": approved_count == 0,
        "aggregated_defects": all_defects,
    }

A real listing typically has several photos, and summarize_listing aggregates across all of them rather than treating each image as an isolated decision — a listing is publishable if at least one photo is acceptable, and defects noticed in any single photo (a scratch visible only from one angle) should surface in the aggregated summary even if other angles look clean. This reflects a broader pattern in multimodal pipelines: individual-item analysis and cross-item aggregation are separate concerns and should be separate functions.

Testing Without Real Images

def test_policy_violation_skips_attribute_extraction(monkeypatch_calls):
    quality_fail = ImageQualityCheck(
        is_acceptable_quality=True,
        quality_issues=[],
        is_policy_compliant=False,
        policy_concerns=["Visible third-party brand logo"],
    )
    monkeypatch_calls(quality_fn=lambda url: quality_fail, attr_fn=None)
    result = process_listing_image("https://example.com/fake.jpg")
    assert result.approved is False
    assert result.attributes is None
    print("PASS: policy-violating images skip attribute extraction entirely")

def test_damaged_condition_is_not_approved(monkeypatch_calls):
    quality_ok = ImageQualityCheck(
        is_acceptable_quality=True, quality_issues=[],
        is_policy_compliant=True, policy_concerns=[],
    )
    attrs = ProductAttributes(
        category="jacket", primary_color="blue", apparent_material="denim",
        condition=Condition.damaged, visible_defects=["large tear on sleeve"],
    )
    monkeypatch_calls(quality_fn=lambda url: quality_ok, attr_fn=lambda url: attrs)
    result = process_listing_image("https://example.com/fake.jpg")
    assert result.approved is False
    print("PASS: damaged condition prevents approval even with acceptable quality")

def _make_monkeypatch():
    import builtins
    module = globals()
    originals = {}
    def apply(quality_fn=None, attr_fn=None):
        if quality_fn:
            originals["quality"] = module["check_image_quality_and_policy"]
            module["check_image_quality_and_policy"] = quality_fn
        if attr_fn:
            originals["attr"] = module["extract_product_attributes"]
            module["extract_product_attributes"] = attr_fn
    return apply

monkeypatch_calls = _make_monkeypatch()
test_policy_violation_skips_attribute_extraction(monkeypatch_calls)
test_damaged_condition_is_not_approved(monkeypatch_calls)

Both tests replace the two model-calling functions with lambdas that return hand-built Pydantic objects, letting process_listing_image's branching logic — the short-circuit on policy failure, the approval rule involving condition — be verified deterministically. This is the core benefit of keeping the model calls as separate, swappable functions: the orchestration logic that decides what to do with the model's output can be tested exhaustively without ever sending an image to the API.

Extending This Project

Add a duplicate-image detector using perceptual hashing to catch sellers reusing stock photos across different listings, and add a size-and-fit extraction pass for apparel categories that cross-references extracted attributes against a category-specific attribute schema.

Common Mistakes

• Combining quality checks and attribute extraction into a single call. This makes it impossible to independently retry or reroute one without the other, and conflates two genuinely different failure categories that need different remediation paths.
• Using free-text fields for attributes that have a fixed, known vocabulary. A condition or category field modeled as a free string produces inconsistent values across images that a downstream filter or search index cannot reliably group.
• Running full attribute extraction on images that fail policy checks. This wastes a model call and, worse, risks persisting structured metadata about content the platform should not be storing at all.

Best Practices

• Model quality and policy as independent flags, each with its own reasons. A photo can fail one without failing the other, and pipelines need to route each failure type differently.
• Short-circuit downstream processing on a policy failure. Don't extract further information from content that will be rejected regardless of what else is found.
• Aggregate across multiple images at the listing level, separate from per-image analysis. Keep the function that judges a single image distinct from the function that judges a listing as a whole.