Can we get clarification about differential data sets?

Hugh.Gleaves10 · November 2022

It's not clear to me from reading the documentation if every differential data set contains changes relative to the preceding differential data set or not. Consider: Monday 4am - Full data set is generatedMonday 8am - Diff data set is generatedTuesday 8am - Diff data set is generatedWednesday 8am - Diff data set is generatedThursday 8am - Diff data set is generatedFriday 8am - Diff data set is generatedSaturday 8am - Diff data set is generatedSunday 8am - Diff data set is generatedMonday 4am - Full data set is generatedMonday 8am - Diff data set is generated. What is not clear to is whether data set 10 only contains differences with respect to data set 9 or to the preceding differential data set 8. In other words can we disregard full data sets (once we have the first full data set, i.e. 1) and consider ever differential data set to always contain changes relative to the preceding differential data set? Thanks

Rahul.Desai25 · November 2022

Hi @Hugh Gleaves

Based on https://community.brightspace.com/s/article/Brightspace-Data-Sets-Differential-Data-Sets-Client-Example

"These differential extracts are generated at a set interval, and contains data newly available since the previous differential extracts were generated."

and

https://documentation.brightspace.com/EN/insights/data_hub/admin/bds_intro.htm?tocpath=Administrators%7CBrightspace%20Analytics%7CData%20Hub%20tool%7CBrightspace%20Data%20Sets%7C_____0

"Differential (diff) data sets contain the differences (diffs) of the data that has been updated or inserted in the previous day, and refreshed daily."

it seems every new differential contains differences relative to the previous differential. So differential 10 will have new data relative to differential 9.

Today's full file = Yesterday's full file + Today's differential.

I hope this helps clarify.

Thanks

Rahul.

Kara.Schimmelfing39 · November 2022

Hi @Hugh Gleaves,

Rahul definitely called out some great resources, but I wanted to clarify one point:

Differential 10 will have new data relative to differential 8 (9 in the example is a full data set not a differential).

Once you have consumed a full data set you can proceed using only differentials for the inserts and updates going forward, you would only need to use the fulls again if you miss a file or to capture deleted entities in data sets without a soft deletion field(s). You will not lose any inserts and updates data if you do not consume full data set 9 but consume all the other data sets in the example.

Renee.Judd22 · November 2022

@Rahul Desai , if @Hugh Gleaves does not incorporate Full File 9, will he miss new data that is not delivered in Differential File 8 (because it was generated after Differential 8) OR in Differential File 10 (because it was generated between after 8 and before Full 10 and therefore was inlcuded in 9 andis no longer new)? So to catch what happens in that 4-hr window, Hugh will always need a full dataset?

I ask because we have a similar situation, and have only been collecting differentials after our initial set-up.

Hugh.Gleaves10 · November 2022

Thanks to each of you for taking the time to explain this, very much appreciated.

Renee.Judd22 · November 2022

Ah, thank you, @Kara Schimmelfing! I wrote my question before your answer appeared for me! That is the exact answer I was hoping for, because that is what I promised our data architect. I would have hated to be wrong about something so fundamental!

Rahul.Desai25 · November 2022

Thank you @Kara Schimmelfing for your clarification, it really did help out heaps.

Apologies for any confusion Hugh and Renee.

Thanks,

Rahul.

Can we get clarification about differential data sets?

Answers

Categories