Data Engineering

We collect and process scattered data into information that decisions and services can actually use.

Data collection
Data processing
ETL
Data analysis
Large-scale processing
Sensor data
Visualization

The data usually exists already. The problem is that it comes in every shape, the value you need is buried inside a document, and it piles up without anyone looking at it. MEDIANAVI has worked with twenty years of news articles, per-second sensor readings from silos across the country, and the layout structure of textbook PDFs.

What we deliver

Collection pipelines

Bringing data together from the web, APIs, files, sensors and embedded devices. Designed around retries and de-duplication so nothing is lost or double-counted.

Extraction from unstructured documents

Pulling exactly the fields you need out of material with complex layout, such as PDFs. We built and then improved a system that separates units, learning objectives and body text from primary and secondary school textbooks.

Sensor and time-series processing

Designing the path from first-pass processing on the edge device through aggregation and anomaly detection on the server, including the logic that combines readings of different kinds — distance, weight, temperature, humidity — into a single figure.

Dashboards and data export

Screens an operator can use without training — filtering by period and site, alerts on anomalies, Excel export. Built around what people actually do with the data.

How we work

We start from the decision the data has to support, and define what to collect backwards from there.
Validation goes into the pipeline at ingest — what is not checked at collection always becomes a problem later.
Raw data is kept and processed data lives separately, so it can be rebuilt when the rules change.
Results always come out in a form a person can inspect — a dashboard or a spreadsheet.