Retrieving Files from SharePoint in Azure Data Factory (ADF)

This document provides a comprehensive guide on creating an Azure Data Factory pipeline named CopySharePointFromFolder. This pipeline facilitates the retrieval of files from a designated folder in SharePoint using a series of well-defined activities.

Pipeline Overview

This section gives an overview of the pipeline activities, detailing each step involved in retrieving files from SharePoint and storing them in Azure Blob Storage.

Overview — Azure Data Factory pipeline

Feel free to customize the pipeline or incorporate additional activities to align with your specific use case and integration requirements.

Pipeline Execution Steps

This section outlines the sequential steps executed during the pipeline’s runtime, explaining each stage’s functionality.

1. Get Token (Web Activity)

This subsection explains the activity responsible for acquiring the OAuth token essential for SharePoint authentication.

- URL: `https://accounts.accesscontrol.windows.net/[Tenant-ID]/tokens/OAuth/2`
- Method: POST
- Headers: Content-Type: `application/x-www-form-urlencoded`
- Body: `grant_type=client_credentials&client_id=[Client-ID]@[Tenant-ID]&client_secret=[Client-Secret]&resource=00000003-0000-0ff1-ce00-000000000000/[Tenant-Name].sharepoint.com@[Tenant-ID]`

2. Get List File (Web Activity)

This subsection describes the activity responsible for retrieving the list of files from the specified SharePoint folder.

- URL: `https://[sharepoint-domain-name].sharepoint.com/sites/[sharepoint-site]/_api/web/GetFolderByServerRelativeUrl('/sites/[sharepoint-site]/[relative-path-to-folder]')/Files`
- Method: GET
- Headers:
- Authorization: `Bearer @{concat('Bearer ', activity('Get Token').output.access_token)}`
- Accept: `application/json`

3. Get ServerRelativeUrl (ForEach Activity)

Within this subsection, the ForEach activity processes the list of files and sets the ‘file’ variable with the file’s ServerRelativeUrl.

- Setting: `@activity('Get List File').output.value`

3.1 Set Variable — file (Set variable Activity)

This sub-subsection details the activity setting the ‘file’ variable.

3.2 Copy SharePoint Data (Copy Activity)

This sub-subsection explains the Copy activity used to transfer files from SharePoint to Azure Blob Storage.