Connecting Microsoft Azure Blob Storage with Lightcast Data Shares

Lightcast’s labor market trends, job posting, workforce profiles, compensation, career pathways, skills projections, demographic insights, and related insights offer the most comprehensive labor market data in the world. Quickly slicing, dicing, and analyzing large datasets is now easier than ever thanks to the launch of Lightcast Data Shares. 

What are Lightcast Data Shares?

Lightcast data is now available through direct data shares on Microsoft Azure Blob Storage and other leading file storage platforms, data warehouses, and marketplaces.  

Data sharing through Lightcast is seamless, flexible and secure. Data is shared directly with existing file storage platforms or data warehouses, without requiring technical lifting from clients’ internal teams.

Delivering data through data sharing is effortless. Customers slice, modify, or prepare data before integrating it into their internal or external solutions. Shared data is always current and secure, because data is shared directly with each customer’s existing cloud storage or warehouse as platform-native sharing, a data feed, or both. 

Accepting data shares from Lightcast requires few technical resources, which leads to decreased costs and faster processes. Data shares make the process of using Lightcast data for ingestion into large language models (LLMs), AI solutions, and predictive analytics models easier  for customers.

Lightcast Data Shares can also be tailored based on each customer’s specific needs and each data share integration can be fully operational in less than two business days.

How Data Shares and Microsoft Azure Blob Storage Work Together

Microsoft Azure Blob Storage, an object storage service offered by Azure, is one of many delivery destinations that Lightcast supports. Lightcast will transfer the data to a Lightcast-managed destination Azure Blob Storage and grant permissions to the Azure application(s) configured in the destination section of a given share.

The permissions granted allow each Azure application to perform read and copy operations on the Bobsled-managed destination container.

Bobsled grants the following permissions to all Azure applications that are granted access to the data in the share:

• Storage Blob Data Reader: Read and list Azure Storage containers and blobs

• Storage Blob Delegator: Get a user delegation key, which can then be used to create a shared access signature for a container or blob that is signed with Azure AD credentials.

What if I Don’t Use Microsoft Azure Blob Storage? 

Lightcast supports several data share destinations beyond Microsoft Azure Blob Storage, including Snowflake, Databricks, Google BigQuery, Google Cloud Storage, Amazon S3, and SFTP.

How to Connect Microsoft Azure Blob Storage with Lightcast Data Shares

To access deliveries made to the Lightcast-managed container, you need to configure Lightcast to grant an Azure application with read access to the Bobsled-managed destination container. There are two types of Azure applications that you may configure to gain access to the container:

1. Lightcast-managed application

2. Consumer managed application

Before consuming a data transfer, a data transfer must be sent to the Azure Blob Storage destination. One of the following access methods must be configured in Lightcast:

• Bobsled Managed Application: a secret must be generated in Lightcast to grant access to the Bobsled Managed Application used to access the data.

• Consumer Managed Application: an Azure Multi-tenant application must be configured access in Lightcast. To consume the data via this method, you must know your application's access credentials.

Consuming a data share 

1. From the Shares list page, click on the share that you would like to access.

2. Once a data transfer in the share has been completed, select the button Access Data.

There are two main ways to consume your transfer via the access data:

• AzCopy

• Azure CLI (Command line)

Option 1: Accessing via AzCopy

Using the Azure Blob Storage command-line tool, you can list, copy, and sync the contents of the data transfer in Azure Blob Storage. To use the following commands, you will need to copy the Storage Container URI located in the access data dialog as pictured above. All of the commands described below are provided within the access data dialog.

Step 1: Login to the CLI

1. Set your client secret

Use the environment variable AZCOPY_SPA_CLIENT_SECRET for the client secret.

• Mac/Linux command: export AZCOPY_SPA_CLIENT_SECRET=<client-secret>

• Windows command prompt: set AZCOPY_SPA_CLIENT_SECRET=<client-secret>

• PowerShell: $Env:AZCOPY_SPA_CLIENT_SECRET=""

2. Run the Login command

The application-id (Bobsled Managed or Consumer Managed) and tenant ID (Bobsled Tenant ID) are provided in the access data dialog.

azcopy login --service-principal --application-id <application-ID> --tenant-id <Bobsled tenant-ID>

Step 2: List the contents

To list the blobs in storage container, you will use the command azcopy list. For more information, visit Azure documentation.

azcopy list <storage-container-URI>

Step 3: Generate SAS Token

1. Login to Azure CLI

The application-id (Bobsled Managed or Consumer Managed) and tenant ID (Bobsled Tenant ID) are provided in the access data dialog.

az login --service-principal -u <application-id> -p <client secret> --tenant <Bobsled tenant-id>

2. Generate SAS Token

You can find the storage account name and container name from the Storage Container URI: https://<storage account name>.blob.core.windows.net/<container name>/<data transfer>

az storage container generate-sas --account-name <storage-account-name> --name <container-name> --permissions lr --expiry <expiry date> --auth-mode login --as-user

Step 4: Switch to your Tenant to Copy and Sync the data

For Bobsled-managed Application:

azcopy login --tenant-id <your-tenant-id>

Consumer managed Application:

azcopy login --service-principal --application-id <application-id> --tenant-id <your-tenant-id>

Step 5: Copy the contents

To copy the blobs in the Bobsled storage container to your target storage container, you will use the command azcopy copy. For more information, visit Azure’s documentation.

azcopy copy "<storage-container-URI?[SAS]>" "<target-container-URI>"

Step 6: Sync the contents

To sync blobs from the Bobsled storage container to your target storage container and only copy modified blobs from the source container, you will use the command azcopy sync.

azcopy sync "<storage-container-URI?[SAS]>" "<target-container-URI>"

Option 2: Accessing Data via Azure CLI

Step 1: Login to the CLI

az login --service-principal -u <application-ID> -p <client-secret> --tenant <Bobsled-tenant-ID>

Step 2: List the Contents of the Storage Container

To list the contents of the storage container, you will use the az storage blob list command.

When using the Azure CLI, you will use three important parameters: --blob-endpoint, --container name, and --delimiter.

The parameters can be derived from the provided Storage Container URI:
https://<storage account name>.blob.core.windows.net/<container name>/<data transfer>

• blob-endpoint = https://<storage account name>.blob.core.windows.net

• Container name = <container name>

• Delimiter = <data transfer>

Discover more about Lightcast Data Shares or contact our team to discuss your specific needs and how to connect Microsoft Azure Blob Storage with Lightcast Data Shares.