Follow the below steps to enable the Automatic Deletion of Blobs from Azure Storage Account.

1. Connect to the Azure storage account on which you need to enable the automatic blob deletion process.

2. Click on Lifecycle management under the Data management section, as shown in the image below.

3. Click on Add a rule button to create a new deletion policy.
4. Provide a name to the rule and select the Rule scope. If you want to apply a filter to limit the rule to specific blobs, you can do it with the ‘Limit blobs with filters’ option.

5. Specify the Blob type and Blob subtypes and click Next.

6. To automatically delete blobs, you can create a rule and click “Add”. For instance, if you wish to remove blobs that are over 14 days old, you can specify a rule as demonstrated in the image below. Additionally, you have the option to transfer old files to cool storage instead of deleting them.

That’s it, we have successfully set a rule to Automatically Delete old files from Azure Storage Account.

Pro tips:
1. Be careful with above settings to delete older files in case of delta lake files.
2. Learn how to mount and unmount a data lake storage in Databricks.

See more

The post Automatically Delete Old Files from Azure Storage Account appeared first on AzureOps.

Pre-requisites:
To mount a location, you would need the following:
1. Databricks service in Azure, GCP, or AWS cloud.
2. A Databricks cluster.
3. A basic understanding of Databricks and how to create notebooks.

What is Mounting in Databricks?

Mounting object storage to DBFS allows easy access to object storage as if they were on the local file system. Once a location e.g., blob storage or Amazon S3 bucket is mounted, we can use the same mount location to access the external drive.

Generally, we use dbutils.fs.mount() command to mount a location in Databricks.

How to mount a data lake in Databricks?

Let us now see how to mount Azure data lake gen2 in Databricks.

First thing first, let’s create blob storage and container. Blob storage should look like in the below image.

New Container should look like in the below image.

To mount an ADLS gen2 we will need the below details to connect to a location.

ContainerName = "yourcontainerName"
azure_blobstorage_name = "blobstoragename"
mountpointname = "/mnt/azureops"
secret_key ="xxxxxxxxxxx"

Once we have this information, we can use below code snippet to connect the data lake with Databricks.

dbutils.fs.mount(source = f"wasbs://{ContainerName}@{azure_blobstorage_name}.blob.core.windows.net",mount_point = Mountpointname ,extra_configs = {"fs.azure.account.key."+azure_blobstorage_name+".blob.core.windows.net":secret_key})

How to check all the mount points in Databricks?

dbutils.fs.mounts()

How to unmount a location?

dbutils.fs.unmount(mount_point)

Let’s use all the above commands in action.

The objective is to add a mount point if it does not exist.

if all(mount.mountPoint != archival_mount_name for mount in dbutils.fs.mounts()):
     dbutils.fs.mount(source = f"wasbs://{ContainerName}@{azure_blobstorage_name}.blob.core.windows.net",mount_point = Mountpointname ,extra_configs = {"fs.azure.account.key."+azure_blobstorage_name+".blob.core.windows.net":archival_secret_key})

Pro tips:
1. Instead of using a storage account key, we can also mount a location using a SAS token URL or service principal
2. Databricks provide a free community version where you can learn and explore Databricks. You can signup here.
3. If you’re aiming to obtain the Databricks certified Data Engineer Associate certification, take a look at these helpful tips.

Notebook Reference

See more

The post Mount and Unmount Data Lake in Databricks appeared first on AzureOps.