Automatically Delete Old Files from Azure Storage with Lifecycle Management

Azure Blob Storage can automatically manage aging data by using lifecycle management policies. Instead of manually deleting old files, you can create rules that delete blobs after a defined period or move them to a cooler access tier. This is useful for logs, temporary files, exports, and other data with a defined retention period.

Quick answer: Open the storage account in the Azure portal, go to Data management > Lifecycle management, create a rule, select the blobs it should apply to, and configure a delete action based on blob age.

How to automatically delete old files from Azure Storage

1. Open Lifecycle management

Open your storage account in the Azure portal. Under Data management, select Lifecycle management.

2. Create a lifecycle management rule

Select Add a rule and give the rule a descriptive name. Choose the rule scope. You can apply the rule broadly or use filters to limit it to specific blobs.

3. Select blob types and filters

Select the blob types to which the rule should apply. You can use filters to target specific containers or blob prefixes. This is especially important when the storage account contains both temporary and long-lived data.

4. Configure the delete condition

Configure the delete action based on the age of the blob. For example, the following policy deletes block blobs that have not been modified for more than 14 days.

Replace 14 with the retention period appropriate for your workload. You can also use filters such as blob index tags when retention needs to depend on application metadata.

Once the rule is saved, Azure Storage evaluates the lifecycle policy periodically and applies the configured action to matching blobs.

If you manage the lifecycle policy through automation or infrastructure-as-code, update the policy JSON in the lifecycle management policy definition you submit to Azure (for example, through the Azure CLI, REST API, or ARM/Bicep workflow); you do not paste this JSON into the normal Azure portal rule form.

{
  "rules": [
    {
      "name": "delete-old-blobs",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"]
        },
        "actions": {
          "baseBlob": {
            "delete": {
              "daysAfterModificationGreaterThan": 14
            }
          }
        }
      }
    }
  ]
}

Related Fabric cleanup: If your files are part of a Microsoft Fabric Lakehouse or Delta workload, see How to Delete Stale Delta Files in Microsoft Fabric Lakehouse before applying a generic storage lifecycle policy.

Delete old files or move them to a cooler tier?

Deletion is not the only lifecycle action. If data is rarely accessed but still needs to be retained, a lifecycle policy can transition blobs to a cooler access tier instead of deleting them. This can help reduce storage costs while keeping the data available.

Important considerations before deleting old files

• Test the rule carefully: Start with a narrow prefix or tag filter rather than applying an aggressive deletion rule to the entire account.
• Check the age condition: Make sure the selected retention period matches the actual business requirement.
• Consider blob versioning: Previous versions can continue to consume storage and may need their own lifecycle cleanup rule.
• Understand soft delete: When blob soft delete is enabled, a deleted blob can remain recoverable for the configured retention period.
• Be careful with data lake files: Do not automatically delete files that may still be required by Delta Lake or another data-processing workload.
• Allow for policy execution time: Lifecycle policies are evaluated periodically, so a new or modified rule does not necessarily act immediately.

For workloads that use blob versioning or soft delete, review those settings alongside the lifecycle policy so that your retention and recovery requirements are consistent.

Pro tips:
1. Be careful with automatic deletion of older files when working with Delta Lake or other data-processing workloads.
2. Learn how to mount and unmount a data lake storage in Databricks.

See more

Visual Studio Marketplace

SSIS Catalog Migration Wizard

Extend Visual Studio with an easy way to migrate SSIS Catalog projects.

Get it from Visual Studio Marketplace

Kunal Rathi

With over 15 years of experience in data engineering and analytics, I've assisted countless clients in gaining valuable insights from their data. As a dedicated supporter of Data, Cloud and DevOps, I'm excited to connect with individuals who share my passion for this field. If my work resonates with you, we can talk and collaborate.