Originally published February 21, 2019
This article introduces using the Data Hub APIs to retrieve a Brightspace Data Set and gives a practical example using the process.
Data Hub houses data exports from Brightspace. Data Hub contains a selection of data set that authorized users can run and export to CSV.
Once Data Hub has been enabled for your Brightspace instance, you can access it in Brightspace or via API.
What information can I currently get from Data Hub?
Currently, Core customers can access more than 120 Brightspace Data Sets. Each data set is available in both a full version and a differential file (updates and changes to the full version) for a specific period of time. The previous 14 differential files are retrievable via the API. Additionally, Advanced Data Sets are also located in the Data Hub but are outside the scope of this article. Details about the contents of Advanced and Brightspace Data Sets can be found in the [BROKEN LINK]Core Analytics Admin Guide.
How do I access Data Hub?
Data Hub can be accessed in Brightspace. For convenience, a Data Hub link can be added to your navbar. Alternatively, you can also navigate to /d2l/datahub/dataexport/list in your browser.
Datahub can also be accessed via API. The following sections will walk you through accessing Data Hub through the API.
What is the expected flow when using Data Hub's API?
Note: authorization is not covered in this section. See the code walkthrough below or the Refresh Token blog post for more information.
The routes below are documented within our API reference documentation.
First, decide which data set you would like to retrieve. There are two different routes to see the available datasets. For a list that is easier to visually interpret you can use /d2l/api/lp/1.15/dataExport/bds/list to see a list of all newest available data sets. This will provide the plugin id needed to retrieve the dataset but doesn't include the clear flag of the data set type (differential/full) outside of the data set name.
Example output:
[
{
PluginId: "df537dc9-8358-4c28-9ab9-ddb8d364a9fc",
Name: "Rubric Object Criteria",
Description: "The rubric object criteria data set returns the basic details for all rubric object criteria.",
CreatedDate: "2019-01-31T07:25:47.430Z",
DownloadLink: "https://myacademy.desire2learn.com/d2l/api/lp/1.15/dataexport/bds/download/df537dc9-8358-4c28-9ab9-ddb8d364a9fc",
DownloadSize: 13075
},
{
PluginId: "26038f5a-539d-4933-86cb-fcb1bbf24cb7",
Name: "Rubric Object Levels Differential",
Description: "The rubric object levels data set differential returns a list of the rubric object levels that have been created for your org units since the last differential was generated.",
CreatedDate: "2019-01-31T05:25:43.159Z",
DownloadLink: "https://myacademy.desire2learn.com/d2l/api/lp/1.15/dataexport/bds/download/26038f5a-539d-4933-86cb-fcb1bbf24cb7",
DownloadSize: 234
},
{
PluginId: "867fb940-2b80-49da-9c8b-277c99686fc3",
Name: "Org Units Differential",
Description: "The org units' data set differential returns details about org units within your organization that have changed since the last differential was executed.",
CreatedDate: "2019-01-31T05:25:43.299Z",
DownloadLink: "https://myacademy.desire2learn.com/d2l/api/lp/1.15/dataexport/bds/download/867fb940-2b80-49da-9c8b-277c99686fc3",
DownloadSize: 241
},
{
PluginId: "428ad0cb-6203-486d-be85-adb01c79578b",
Name: "Content User Progress",
Description: "The content user progress dataset provides details regarding specific user's progress within an org unit.",
CreatedDate: "2019-01-31T07:25:52.038Z",
DownloadLink: "https://myacademy.desire2learn.com/d2l/api/lp/1.15/dataexport/bds/download/428ad0cb-6203-486d-be85-adb01c79578b",
DownloadSize: 4774690
},
… (Note this response is significantly longer but has been truncated for article readability)
]
The data sets that do not have "Differential" in the name are the Full version of the data sets. Brightspace Data Sets, unlike the Advanced Data Sets do not take filter parameters, so you only need the plugin-id to retrieve the most recent version of the data set you want. Additionally, the Brightspace Data Sets have already been generated so you do not need to check their status before retrieving to confirm they are complete.
Another call that will give you more details, including all the older versions of a data set still available for retrieval, is d2l/api/lp/1.18/dataExport/bds.
Example Output (selection of results with the previous data sets collapsed):
{
PreviousDataSets:
[…],
PluginId: "df537dc9-8358-4c28-9ab9-ddb8d364a9fc",
Name: "Rubric Object Criteria",
Description: "The rubric object criteria data set returns the basic details for all rubric object criteria.",
FullDataSet: true,
CreatedDate: "2019-02-11T05:00:46.360Z",
DownloadLink: "https://learninganalytics.brightspace.com/d2l/api/lp/1.18/dataexport/bds/df537dc9-8358-4c28-9ab9-ddb8d364a9fc/1549861245",
DownloadSize: 110696,
Version: "3.2",
QueuedForProcessingDate: "2019-02-11T05:00:45.000Z"
},
{
PreviousDataSets:
[…],
PluginId: "01d3b529-5929-4f54-8348-d9e92fdb726a",
Name: "Competency Activities Differential",
Description: "The competency activities data set differential will return information on activities associated with learning objectives that have been created for your org units since the last differential was generated. This includes activities that have been aligned but may not have been assessed yet.",
FullDataSet: false,
CreatedDate: "2019-02-11T05:00:48.844Z",
DownloadLink: "https://learninganalytics.brightspace.com/d2l/api/lp/1.18/dataexport/bds/01d3b529-5929-4f54-8348-d9e92fdb726a/1549861245",
DownloadSize: 282,
Version: "3.2",
QueuedForProcessingDate: "2019-02-11T05:00:45.000Z"
}
… (Note this response is significantly longer but has been truncated for article readability)
This view gives a flag value for the data set type, the version of the data set, and a download link that directs to that specific dataset (based on the processing timestamp). You will note the above call give the download link for the most recent file for that data set which is slightly different. Based on your processes, one may be more suited for your needs.
Once you have picked a data set there is only one step remaining – retrieving it.
If you wanted to download the Content User Progress data set with a plugin id of: "428ad0cb-6203-486d-be85-adb01c79578b" You would simply use /d2l/api/lp/1.15/dataExport/bds/download/428ad0cb-6203-486d-be85-adb01c79578b to download the zipped Full data set.
For an example of writing a script that uses the API to download Brightspace Data Sets please refer to the following articles:
Need further help working with data? Stuck on any of the above steps? Lacking one of the skill sets needed? D2L provides flexible levels of Data Solutions Consulting services that can assist with any or all steps in the process detailed in this post. If you are interested, please contact your D2L Customer Success representative.