You send around last month’s electricity figures. A few days later, someone opens the dashboard and gets a different number. The pipeline is running, the query hasn’t changed, and nobody remembers editing anything.
Now there are two screenshots in the team chat and a very reasonable question: which one should we use?
A dashboard can change because the underlying data has improved. A supplier sends a corrected bill. An estimate gets replaced by a meter reading. Someone finally uploads a missing file. The problem starts when those changes happen quietly and the report gives you no way to explain them.
For a data team, this is a useful place to slow down. Before adding another connector, make it easy to see what arrived, what changed and which version was used in the report.
Start with the bill that arrived late
Imagine a warehouse uses an estimated electricity figure of 12,400 kWh for September. The reporting team closes the month with that estimate. On October 8, the actual bill arrives: 11,900 kWh for the same period.
There are a few easy ways to mishandle this. Append the bill without replacing the estimate and you count the same consumption twice. Overwrite the estimate without keeping its history and September changes with no explanation. Put the bill into October because that’s when it arrived and you move the consumption into the wrong month.
Keep the period covered by the bill separate from the date you received it. A file can arrive today and still belong to last month. That small distinction saves a surprising amount of digging later.
When a bill covers parts of two months, keep its original dates too. If the reporting team allocates consumption between those months, record the agreed method alongside the result. The import job shouldn’t make that decision on its own.
Give each record an identity that survives an upload
A filename is a shaky way to identify a bill. Someone renames the attachment, saves another copy or adds “final” to the end. Your import sees a new file. The business sees the same bill.
Use a stable source identifier where one exists. For invoices, that might mean the supplier account plus the invoice number and line number. For meter readings, it might be the meter identifier and reading timestamp. Check how the source actually works before choosing the key.
Then decide what a repeat means. An identical record arriving twice should leave the accepted total unchanged. A revised record should create a traceable correction. Two legitimate charges on the same date must still be allowed through.
If the source doesn’t provide enough information to tell those cases apart, flag the possible duplicate for review. Deleting everything with a matching amount and date will eventually remove something you needed.
Keep the original, even when it’s messy
It’s tempting to clean a file and keep only the tidy version. Then somebody asks why the original spreadsheet says 11.9 while the database says 11,900.
In this example, the spreadsheet used MWh and the database stores kWh. The conversion is fine. You just need enough context to show it.
Keep the original quantity and unit, the normalized quantity and unit, and the rule used to convert between them. Retain a reference to the source document in the record, with access limited to the people who need it.
For calculated emissions, also keep the factor identifier, version and calculation method used. A result can change because an input changed or because the calculation changed. Those deserve different explanations.
You don’t need to show every field on the dashboard. You do need a way to work back from a number to the records behind it.
Separate the latest view from the report you sent
There’s nothing wrong with a working dashboard updating as better data comes in. That’s useful during the month. The awkward part is using that same moving view as the only record of an approved report.
Keep an identifiable version of what was approved. That could be a versioned reporting table or a saved export with its input versions and calculation settings. A screenshot alone won’t tell you which records were included.
Give the saved version a clear name and date. Label the live view just as clearly. “September, latest available data” and “September, approved October 3” are different things, and a reader should know which one they’ve opened.
If a correction needs to change the approved report, let the reporting owner make that decision. Keep the earlier version and the reason for the revision. Engineers can make revisions easy to follow without deciding the reporting policy.
Make a rerun boring
A useful test is to run the same import twice. If nothing changed at the source, the accepted totals should stay the same.
This is what people mean by making a load idempotent: repeating it doesn’t create an extra business effect. It matters whether you’re loading a monthly CSV or processing changes from a database.
For a table holding current values, update the matching record using a reliable source version. For a history table, keep genuine revisions while filtering out repeated deliveries of the same revision. Those tables serve different purposes.
Be careful with arrival time. An older file can arrive after a newer correction. “Last received wins” can quietly restore an outdated value. If the source has no reliable revision order, send conflicting versions for review rather than guessing.
Check this at the receiving end too. If the destination is a reporting platform such as Dcycle, agree how repeated uploads and corrected records will be handled before automating the handoff. A successful upload message doesn’t tell you whether the final total is right.
Show what’s missing beside the total
A chart can look complete while an entire location is missing. A zero can mean no consumption, no submission or a failed import. Only one of those is a real zero.
Keep those states separate. Alongside a monthly figure, show which locations and periods are covered, where estimates remain, and which records are waiting for a decision.
For example, “nine of ten locations received; one still estimated” tells the reader much more than a green check beside the latest import.
The expected list needs an owner as well. A newly opened site won’t appear as missing if nobody added it to the reporting scope. Review that list with the people collecting the data, especially when locations open, close or change suppliers.
Test the awkward cases before the next close
Pick one reporting period and build a small test set. Include an original bill, an identical copy, a correction, a late record and a record with an unclear unit.
Write down the expected result before running the import. For the warehouse example, replacing the estimate with the actual bill should reduce accepted September electricity by 500 kWh. It shouldn’t add 11,900 kWh to October or leave both September values in the total.
Then check the things people will ask about:
- Can you find the original estimate and the bill that replaced it?
- Does another run leave the result unchanged?
- Does an older file leave the accepted correction intact?
- Is the unclear unit still visible as an unresolved item?
- Can you reproduce the version that was approved before the correction?
Ask someone outside the pipeline team to explain the change using what the system shows. If they need you to open logs and narrate every step, the data may be correct but the handover still needs work.
Leave a short explanation with the change
When a figure moves, write down why in ordinary language. “September electricity decreased by 500 kWh after an estimated warehouse reading was replaced with the supplier bill” is enough to start with.
Add the affected record, the revision date and the person who reviewed it. Keep that explanation close to the report so it survives the next handover.
Late bills and corrections will keep happening. The useful goal is to make the next changed number easy to understand, without another afternoon spent comparing screenshots.