Intelligent document processing

The flow, step by step

• Upload: the user uploads a file to the REST API.
• Routing: the system sends it to the extractor for its format: python-pptx for PowerPoint, pypdf for PDF, and pandas for Excel.
• Analysis: AI cleans the text and extracts insights in a chain: generic cleaning, Market Analyst Agent, and Brand Analyst Agent, with Groq (gpt-oss-120b).
• Persistence: the result, as structured JSON, is stored in PostgreSQL (Neon) with the JSONB type.
• Response: the API returns the clean JSON.

Design decisions

• Three chained agents: I split the task into a cleaner/extractor (base JSON), a Market Agent (trends and competitors), and a Brand Agent (positioning). Each receives the previous agent’s context. When I tested a single prompt, the model tended to ignore parts of the task; with separate responsibilities, each result is scoped and verifiable.
• Editable prompts: they live in Markdown files (/prompts), separate from the code. A consultant without Python knowledge can adjust the AI instructions without touching the source code.
• JSONB in PostgreSQL: AI may return new properties over time; with JSONB, the schema does not need to be migrated, the complete JSON is preserved for auditing, and it can be queried internally (for example, all analyses for a client).
• Modular extractors and validated API: one extractor per format, and a REST API with FastAPI that validates files and handles HTTP errors.

What gets stored

For each document fragment, the JSON includes topic, client, sentiment, market analysis (competitors, topic, and market size), main insight, and brand analysis (values and perception).

Automation and deployment

• n8n: orchestrates and visualizes. It consumes PostgreSQL data and builds insight dashboards (trends, competitors, market size) for consultants.
• Docker and Azure: the application is containerized and deployed as an Azure App Service demo, with the database on Neon (serverless PostgreSQL).

A real problem: processing large files in the cloud

When I deployed the API to Azure App Service (1 GB RAM plan), long presentations failed with 504 Gateway Timeout and 502 Bad Gateway errors. A 29-slide PPTX makes 3 AI calls per slide, and the process exceeded Azure’s maximum wait time (nearly 4 minutes per request) and the available memory.

What I did

• I adjusted the Gunicorn startup: -w 1 --timeout 1200. A single worker prevents exhausting RAM, and the high timeout prevents Gunicorn from killing the process. The platform request limit still applies.
• I designed the production solution: for large files, the path forward is asynchronous processing (webhooks or queues such as Celery), so the HTTP request is not blocked. This is the next step and has not been implemented yet.
• Today: I process heavy files locally with the Docker container and lightweight files in the cloud.

How I verified it works

• I tested three documents: a real 13-page PDF (a digital transformation white paper), a 29-slide academic presentation with tables and charts, and a test financial Excel file.
• I checked that the model does not invent data: when the document does not mention competitors, it returns Not specified instead of assuming them. I checked this in the final JSON and the database.
• I verified that speaker notes are extracted from presentations and that the n8n workflow, connected to PostgreSQL, displays the table without errors.

Cost

According to the Groq usage dashboard, processing a 13-page PDF with 3 agents (39 calls) cost about 0.02 USD. Therefore, I estimate that a complete document with the 3-agent chain costs less than 0.10 USD.

Use of AI assistants

I used Claude and GitHub Copilot to speed up the base code, debug Docker on Windows (a Python version conflict), and optimize expressions in n8n. I designed and implemented the architecture, pipeline design, prompt strategy, and infrastructure problem-solving myself.