how to read .ss files from Unity Catalog Enabled Azure Databricks Cluster

Question

how to read .ss files from Unity Catalog Enabled Azure Databricks Cluster

Prakash Arumugam (LTIMINDTREE LIMITED) 0 Microsoft External Staff

Hello Team,

We are regular users of the Azure Databricks cluster, and due to security requirements, we are currently transitioning from Service Principal (SPN) authentication to Managed Identity.

Until now, we have been using .ss files (Structured Stream format) through the structuredstreamforspark library, and this has been a critical dependency in many of our workflows.

When we install the same JAR library in a Unity Catalog (UC) enabled cluster using Volumes, the installation completes successfully. However, when we attempt to read the data using:

spark.read.format("sstream")

we encounter the following error:

AnalysisException: [UC_COMMAND_NOT_SUPPORTED.WITHOUT_RECOMMENDATION] The command(s): Data source v2 are not supported in Unity Catalog. RelationV2[...] sstream

Since our deliverables rely heavily on this Structured Stream binary format, this limitation is currently blocking our migration to the UC-enabled cluster with Managed Identity.

Could you please advise if there is a supported workaround or alternative approach to read .ss files in a Unity Catalog environment?

Any guidance would be greatly appreciated, as this format is a critical dependency for our pipelines.

Thank you in advance for your support.

Pilladi Padma Sai Manisha 5,240 Reputation points Microsoft External Staff Moderator

2026-03-06T22:11:23.2433333+00:00

Hi Prakash Arumugam (LTIMINDTREE LIMITED),

This behavior is expected when using Unity Catalog (UC) enabled clusters. Unity Catalog currently does not support certain external Data Source V2 connectors, and the structuredstreamforspark library relies on a custom Data Source V2 implementation (sstream). Because of this, when the cluster is UC-enabled, Spark blocks the command and returns the error UC_COMMAND_NOT_SUPPORTED – Data source v2 are not supported in Unity Catalog.

Even though the JAR installs successfully on the cluster, Unity Catalog enforces restrictions at the execution layer, which prevents unsupported data source connectors from being used.

As a workaround, the workloads that depend on .ss files can be executed on a non-Unity Catalog cluster, where custom Data Source V2 connectors are supported. Another possible approach is to read the .ss files using a non-UC cluster and convert the data to a Unity Catalog supported format such as Delta, Parquet, or CSV, and then store the converted data in UC volumes or tables for downstream consumption.

Many teams migrating to Unity Catalog follow a similar architecture where legacy formats are processed outside UC and the processed results are then published into Unity Catalog tables for further workloads.

At this time, there is no supported method to enable unsupported Data Source V2 connectors directly in Unity Catalog clusters, so processing the .ss files outside UC and storing the results in a supported format is the recommended approach.
Prakash Arumugam (LTIMINDTREE LIMITED) 0 Reputation points Microsoft External Staff

2026-03-06T22:24:05.71+00:00

Hello Pilladi Padma Sai Manisha, Thanks a lot for your time and suggestion and I do agree with your suggestion however the concern is due to security requirements, we are currently transitioning from Service Principal (SPN) authentication to Managed Identity. Even if I have a Cluster which is not enabled with Unity Catalog I won't be able to access the data and the other catch is Most of my data read processing were made through .ss format.
Pilladi Padma Sai Manisha 5,240 Reputation points Microsoft External Staff Moderator

2026-03-09T21:24:52.8733333+00:00

Hi Prakash Arumugam (LTIMINDTREE LIMITED)
Thank you for the additional clarification.

The behavior you are encountering is related to Unity Catalog limitations rather than the authentication method (Service Principal vs Managed Identity). Unity Catalog currently restricts the use of custom Data Source V2 connectors, and the structuredstreamforspark library implements the sstream format using a custom Data Source V2 provider. Because of this restriction, UC-enabled clusters block the command and return the error:

UC_COMMAND_NOT_SUPPORTED – Data source v2 are not supported in Unity Catalog.

Regarding the concern about Managed Identity, it is important to note that using Managed Identity does not require Unity Catalog. A non-UC cluster can still access ADLS or Blob storage using Managed Identity, provided the cluster identity has been granted the appropriate RBAC permissions such as Storage Blob Data Reader or Storage Blob Data Contributor on the storage account.

One possible approach is to use a non-Unity Catalog cluster configured with Managed Identity to read the .ss files using the structuredstreamforspark library and then convert the data to a Unity Catalog supported format such as Delta or Parquet. The converted data can then be written to storage locations governed by Unity Catalog and registered as UC tables for downstream workloads.

This pattern is commonly used when migrating workloads that depend on legacy or custom formats that are not yet supported in Unity Catalog.

At this time, Unity Catalog does not provide a mechanism to enable unsupported custom Data Source V2 connectors, so .ss files cannot be read directly on UC-enabled clusters. Processing the files outside UC and storing the results in a supported format before consuming them in Unity Catalog remains the recommended approach.
Pilladi Padma Sai Manisha 5,240 Reputation points Microsoft External Staff Moderator

2026-03-10T18:32:19.0466667+00:00

Hi Prakash Arumugam (LTIMINDTREE LIMITED),

I hope you had a chance to review the information shared earlier, and I hope this information has been helpful! If you still have questions, please let us know what is needed in the comments so the question can be answered.
Prakash Arumugam (LTIMINDTREE LIMITED) 0 Reputation points Microsoft External Staff

2026-03-11T20:14:58.46+00:00

Hello Pilladi Padma Sai Manisha, Thanks a lot for your suggestion it is really helpful. My understanding is that we will be able to access to the Storage account using Managed Identity only using the Unity Catalog Enabled cluster. If you have any document to bypass this scenario, could you please help us with any supporting document. It would be really helpful.

Thanks a lot once again

Regards,

Prakash
Pilladi Padma Sai Manisha 5,240 Reputation points Microsoft External Staff Moderator

2026-03-11T21:28:55.3+00:00

Hi Prakash Arumugam (LTIMINDTREE LIMITED),
Thank you for your response.

Managed Identity does not require a Unity Catalog–enabled cluster. A non-UC Azure Databricks cluster can also access the storage account using Managed Identity, provided the identity has the required RBAC permissions (for example, Storage Blob Data Reader or Storage Blob Data Contributor) on the storage account.

You can refer to the following documentation for configuration details:
https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/azure-managed-identities#managed-identity

Please note that the limitation related to the sstream Data Source V2 connector applies specifically to Unity Catalog–enabled clusters, which is why .ss files cannot currently be read directly in UC environments.