Package dev.vortex.spark

Class VortexSessionCatalog

java.lang.Object
org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension
dev.vortex.spark.VortexSessionCatalog
All Implemented Interfaces:
org.apache.spark.sql.connector.catalog.CatalogExtension, org.apache.spark.sql.connector.catalog.CatalogPlugin, org.apache.spark.sql.connector.catalog.FunctionCatalog, org.apache.spark.sql.connector.catalog.SupportsNamespaces, org.apache.spark.sql.connector.catalog.TableCatalog
public final class VortexSessionCatalog extends org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension
A session catalog extension that resolves USING vortex tables through the Vortex DataSource V2 connector.

Spark 3.5's built-in session catalog resolves the tables it stores through the V1 DataSource path, which rejects DataSource-V2-only connectors like Vortex, so CREATE TABLE ... USING vortex tables cannot be read back. (Spark 4 resolves them through the V2 provider directly and needs none of this.) Registering this extension as the session catalog fixes that on Spark 3.5: 

spark.sql.catalog.spark_catalog=dev.vortex.spark.VortexSessionCatalog

All operations are delegated to the built-in session catalog — table metadata lives wherever it normally would, including the Hive metastore — but any table whose provider is vortex is loaded as a Vortex DataSource V2 table, backed by the files at the table's location. Tables of every other provider are untouched.

  • Field Summary

    Fields inherited from class org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension

    delegate

    Fields inherited from interface org.apache.spark.sql.connector.catalog.SupportsNamespaces

    PROP_COLLATION, PROP_COMMENT, PROP_LOCATION, PROP_OWNER

    Fields inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

    OPTION_PREFIX, PROP_COLLATION, PROP_COMMENT, PROP_EXTERNAL, PROP_IS_MANAGED_LOCATION, PROP_LOCATION, PROP_OWNER, PROP_PROVIDER, PROP_TABLE_TYPE

  • Constructor Summary

    Constructors
    Constructor
    Description
    VortexSessionCatalog()
    Creates a new session catalog extension.

  • Method Summary

    Modifier and Type
    Method
    Description
    org.apache.spark.sql.connector.catalog.Table
    createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, Map<String,String> properties)
    Creates the table in the delegate session catalog, then returns it resolved through the Vortex connector when its provider is vortex.
    org.apache.spark.sql.connector.catalog.Table
    loadTable(org.apache.spark.sql.connector.catalog.Identifier ident)
     

    Methods inherited from class org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension

    alterNamespace, alterTable, capabilities, createNamespace, createTable, defaultNamespace, dropNamespace, dropTable, functionExists, initialize, invalidateTable, listFunctions, listNamespaces, listNamespaces, listTables, loadFunction, loadNamespaceMetadata, loadTable, loadTable, name, namespaceExists, purgeTable, renameTable, setDelegateCatalog, tableExists

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

    Methods inherited from interface org.apache.spark.sql.connector.catalog.TableCatalog

    createTable, listTableSummaries, loadTable, useNullableQuerySchema

  • Constructor Details

    • VortexSessionCatalog

      public VortexSessionCatalog()
      Creates a new session catalog extension.

      This no-argument constructor is required for Spark to instantiate the catalog through reflection from the spark.sql.catalog.spark_catalog configuration.

  • Method Details

    • loadTable

      public org.apache.spark.sql.connector.catalog.Table loadTable(org.apache.spark.sql.connector.catalog.Identifier ident) throws org.apache.spark.sql.catalyst.analysis.NoSuchTableException
      Specified by:
      loadTable in interface org.apache.spark.sql.connector.catalog.TableCatalog
      Overrides:
      loadTable in class org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension
      Throws:
      org.apache.spark.sql.catalyst.analysis.NoSuchTableException

    • createTable

      public org.apache.spark.sql.connector.catalog.Table createTable(org.apache.spark.sql.connector.catalog.Identifier ident, org.apache.spark.sql.types.StructType schema, org.apache.spark.sql.connector.expressions.Transform[] partitions, Map<String,String> properties) throws org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException, org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException
      Creates the table in the delegate session catalog, then returns it resolved through the Vortex connector when its provider is vortex.

      Spark does not route table creation through this overload — DelegatingCatalogExtension sends the Column[] overload straight to the delegate, and that is the one both supported Spark versions call. It is kept because the delegate may return null on the normal path, which asVortexTableIfVortex(org.apache.spark.sql.connector.catalog.Table) now tolerates; CREATE TABLE ... AS SELECT gets its Vortex table from loadTable(org.apache.spark.sql.connector.catalog.Identifier) instead.

      Specified by:
      createTable in interface org.apache.spark.sql.connector.catalog.TableCatalog
      Overrides:
      createTable in class org.apache.spark.sql.connector.catalog.DelegatingCatalogExtension
      Throws:
      org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException
      org.apache.spark.sql.catalyst.analysis.NoSuchNamespaceException