Learn summarized model with
In 2025, there may be maybe no group underneath the solar that isn’t making data-driven selections, or a minimum of claiming to. At Omnisend, information isn’t only a declare — it’s the inspiration upon which the remainder of the home is constructed.
DataOps & Insights is the supply of fact for our selections: operational reporting, predictive analytics, and self-serve information. Our CI/CD pipeline enforces high quality so product groups can transfer quick with out breaking belief.
Nevertheless, after not too long ago becoming a member of the crew as a Software program Engineer, I seen a friction frequent to many information groups: the mechanics of the work have been slowing down the logic. We wanted to optimize not simply our code, however our workflows. After intensive trials, we recognized high-ROI purposes for Giant Language Fashions (LLMs) that compress our time-to-insight from days to minutes.
Right here’s how we did it.
1. The analytics hole: Velocity vs. high quality
The true problem in fashionable information engineering isn’t writing SQL — it’s closing the hole between a pointy enterprise query and a reliable reply earlier than the chance window closes.
At Omnisend, we realized that whereas our logic was sound, the “chores” of knowledge modeling have been making a bottleneck.
The friction: Context switching and boilerplate
Constructing a strong information mannequin requires fixed context switching: leaping between dbt conventions, YAML configurations, testing suites, and documentation requirements. We confronted repetitive scaffolding duties throughout each layer of our transformation pipeline (staging → dims → information → marts).
Each context swap launched an opportunity for error, and sustaining consistency throughout environments grew to become more and more fragile. We wanted a technique to automate the rigorous, repetitive elements of the job so our analysts might deal with structure reasonably than typing.
The answer: Context-aware modeling with Cursor
We turned to Cursor, an AI-powered code editor. Not like customary autocomplete instruments, Cursor indexes our complete repository, permitting it to grasp the precise context of our mission construction, information lineage, and naming conventions.
We arrange the surroundings to help the AI:
- Repo Indexing: Cursor listed our information fashions and documentation, giving it a “map” of our information warehouse
- Guardrails and prompts: We established well-scoped prompts aligned with our SQL model
- Inline critiques: The AI flags anti-patterns — like CTEs that break incremental fashions or fan-out joins — earlier than a Pull Request (PR) is even raised
Implementation: From hours to minutes
With these guardrails in place, the workflow shifted dramatically. When an analyst defines a enterprise requirement, Cursor generates the preliminary information fashions in seconds. It selects applicable supply tables, generates recordsdata in appropriate mission paths (staging/dims/information), and even pulls column descriptions to auto-populate documentation.
What used to take hours of handbook file creation is now accomplished in minutes. The analyst’s position shifts from “author” to “reviewer.”
A be aware on hallucinations: It’s necessary to be life like: the instrument isn’t good. It makes errors when the token chance sequence will get “confused.” Nevertheless, getting 90% of the work accomplished immediately permits us to spend our vitality on the ultimate, crucial 10% of validation.
The affect
- Growth velocity: A 2-5x enhance in mannequin supply velocity by templating YAML and check creation
- Improved consistency: SQL and YAMLs now observe a strict customary, lowering information incidents.
- Higher traversability: The AI enforces a constant hierarchy (staging > dims > information > marts), making the codebase simpler to navigate, perceive, and use when creating a brand new information mannequin
2. Fewer evaluation cycles, fewer incidents
As soon as modeling is completed regionally in Cursor and a PR is raised, the workflow shifts from creation to validation. That is the place we hand the baton to Gemini Code Help.
The problem: The peer evaluation bottleneck
Peer critiques are crucial for high quality, however they’ll turn out to be a bottleneck. A human reviewer — particularly one from a special product group — may miss delicate deviations from our dbt model information or miss out on non-optimal BigQuery features.
We confronted frequent ache factors:
- Context blindness: Struggling to grasp cross-file context in massive diffs
- Fashion drift: Inconsistent formatting making diffs more durable to learn
- Logic gaps: Lacking delicate enterprise logic breaks (e.g., attribution order modifications) that look syntactically appropriate however are functionally mistaken
The answer: Gemini Code Help (with strict tuning)
We deployed Gemini Code Help as our first line of protection. It summarizes diffs by intent, checks in opposition to a repo-specific model information, and proposes concrete fixes.
Nevertheless, out of the field, the AI was noisy. To make it helpful, we needed to arrange the reviewer similar to we arrange the author:
- Noise discount: We tightened the .gemini/config.yaml to prioritize crucial findings over nitpicks
- Context injection: We added a ./gemini/styleguide.md file containing our particular dbt conventions and governance checks
Actual-rorld optimization: The story of three CTEs
The worth of a second AI opinion grew to become clear throughout a latest refactor. We had a mannequin with three duplicated Widespread Desk Expressions (CTEs).
Cursor (the author) flagged them however advised an “if it ain’t damaged, don’t repair it” method, warning that unioning may be slower.
While Gemini (the reviewer) flagged the identical duplication, however really helpful a concrete optimization: consolidating them into one union with a single unnest/be part of.
We examined the Gemini-suggested refactor. The end result was a ~50% discount in runtime. This interaction is crucial: the drafting AI prioritized velocity, whereas the reviewing AI prioritized structure.
The affect
- 30–40% fewer evaluation cycles: Gemini catches syntax and elegance points earlier than a human sees them
- 15–25% discount in logical errors: Fewer post-merge defects tied to inconsistent logic
- Automated governance: The assistant flags PII points and validates source-of-truth tables mechanically
3. Fixing information discovery: The “the place is X?” drawback
As our Superset surroundings scaled to 1000’s of belongings, it grew to become a sufferer of its personal success. A easy query like “The place can I discover our month-to-month recurring income chart for M-segment shoppers?” required deep platform information or a ping to the information crew.
The answer: Embed, index, retrieve
We embedded a Chainlit chatbot immediately into the Superset UI.
- Ingestion: A each day automated pipeline (by way of Dagster) extracts metadata from each dashboard and chart
- Indexing: Metadata is synced to a vector information base on OpenAI
- Retrieval: Chainlit responds by means of the OpenAI Assistant API, returning ranked belongings with direct hyperlinks when out there, or suggesting the place outcomes could also be discovered.
All of it comes all the way down to context
The ability of this method is knowing information relationships. A marketer not too long ago requested: “How lengthy, on common, does it take retailers to activate types from the time of creation?“
No pre-built dashboard answered this. Nevertheless, the Assistant analyzed the intent and accurately recognized the related dataset and columns wanted to calculate the reply. It reworked a “no outcomes” useless finish right into a self-serve win.
The affect
- Silence is golden: A 25–40% drop in “The place is X?” pings to the DataOps crew
- Compelled hygiene: As a result of the bot depends on metadata, “undocumented” grew to become “invisible,” incentivizing the crew to undertake higher documentation requirements
4. Scaling EDA: 76 hours of video in minutes
A few of our most dear information isn’t in a database — it’s in unstructured textual content, resembling buyer conversations. We not too long ago had 76 hours of Quarterly Enterprise Overview (QBR) recordings — a goldmine of shopper suggestions that was virtually unimaginable to investigate manually.
The method: Bypassing the context window
We used Cursor with Claude-4-Sonnet to construct an iterative ETL pipeline for textual content.
- Context definition: We outlined a immediate focusing on particular matters (metrics, benchmarks, suggestions)
- Device technology: Cursor generated a Python script to course of 116 transcript recordsdata
- Iterative extraction: The script iterated by means of recordsdata, extracting related sentences into structured CSVs, which have been then summarized
The affect
This method gave us a blended view of qualitative and quantitative insights: frequency counts of matters alongside exemplar quotes.
Extra importantly, it democratized the workflow. Vytautas Jakštys, our Product Director, a non-technical chief, now makes use of this similar technique. He generates SQL from our dbt docs utilizing Claude, then makes use of Cursor to investigate buyer chats to grasp the “why” behind the numbers.
Last ideas on information as a dialog
We aren’t stapling AI onto our stack for present we’re baking it into how Omnisend asks, solutions, and acts.
The result’s a division that ships fashions sooner, critiques code smarter, and lets everybody discover reliable information with out a guided tour. AI handles the mundane work — constructing new information fashions from enterprise necessities, writing YAML documentation and assessments, checking syntax and proper mannequin use, validating and reviewing, and discovering charts — clearing the runway for us to deal with the actual query:
What’s the following step that strikes us forward?
The subsequent step is to proceed codifying our judgment into the markdown recordsdata: guidelines, tips, types, and extra. It’s an ever-evolving course of. As new LLM fashions emerge, so do new prompting strategies and approaches.
Most significantly, such workflows run totally on well-curated metadata. Your AI is simply pretty much as good as your documentation.
Should you personal a dataset, undertake the model information and certify your belongings. You aren’t simply serving to a human reader at this time, you’re making the assistant smarter for tomorrow.
