Data from many systems, checked before any report
We bring data from the ERP, online store, bank statements and spreadsheets into one place, checked at every load, so reports and systems work from the same figures.
Who it's for
For companies where each department gets a different figure from the same report.
What it solves
- Someone spends the morning exporting spreadsheets from several systems to put the report together.
- Finance and sales arrive at the meeting with different figures for the same month.
- A load failed overnight and the dashboard showed stale data without warning anyone.
- Nobody knows where a figure came from or which rule calculated it.
- Customers' personal data travels around in copies and spreadsheets that nobody controls.
What we do
- Source connectors
- ERP, databases, APIs, online store and spreadsheets read automatically, on a schedule.
- Incremental loads
- Only what changed since the last load is brought in, without straining the source systems.
- Calculation rules in code
- Metrics calculated by reviewed, tested code instead of formulas scattered across spreadsheets.
- Quality tests
- Every load checks for duplicates, empty fields and totals; if something does not add up, the data stops there.
- Load monitoring
- An alert when a load fails or runs late, before anyone opens an out-of-date dashboard.
- Personal data kept in check
- With the LGPD in mind, only the fields needed come in, with restricted access and masking.
- Where each figure comes from
- For every metric, the sources and rules that produced it are recorded.
What you get
- Pipelines in production, with scheduled loads and reprocessing
- Report-ready tables, with a data dictionary
- Quality tests running on every load
- Alerts for failed and late loads
- Documentation of sources, rules and who answers for each dataset
How we do it
Inventory
Which sources exist, who uses each figure and where personal data sits.
Definitions
Calculation rules for each metric, agreed with the departments that use them.
First source
Connectors, transformations and tests for one source and one report, end to end.
Expansion
The other sources come in one by one, checked against today's figures.
Operation
Alerts, load monitoring and adjustments when a source system changes.
Technology examples
- Python
- dbt
- Airflow
- Debezium
- Kafka
- PostgreSQL
- BigQuery
In Brazil, we handle
- LGPD
- Brazil's General Personal Data Protection Law (Law 13,709/2018).
Related services
FAQ
Do you build the dashboards and reports?
The focus is the data layer: organised, checked tables ready for the BI tool your company uses. Building the dashboards can be part of the scope; that is agreed during the inventory.
Do we need a data warehouse?
It depends on the volume and how many sources there are. With few sources, a PostgreSQL database separate from the main system may be enough; a cloud data warehouse comes in when volume or queries call for it.
Does personal data need to go into the pipeline?
Only if a report genuinely needs it. We take the LGPD into account: what is not needed stays out, and the rest is masked or access-restricted. The legal basis for each use is for your data protection officer to decide.
Do the loads strain the source systems?
They can, if done badly. That is why we read only what changed, preferably from a replica or the database's change log, at agreed times.
Write to Balkan
Talk to us about your project