Package dev.vortex.api

Class DataSource

java.lang.Object
dev.vortex.api.DataSource
public final class DataSource extends Object
A set of Vortex files opened through a Session. Data sources are cheap to open (only the first file is read eagerly, to determine the schema) and can be scanned multiple times.

Native resources are released automatically via VortexCleaner when the data source becomes unreachable.

  • Method Details

    • open

      public static DataSource open(Session session, String uri)
      Open a single URI.

    • open

      public static DataSource open(Session session, String uri, Map<String,String> properties)
      Open one or more URIs or globs. When a glob is used, the first match is opened eagerly; subsequent matches are opened lazily on scan.
      Parameters:
      session - open session
      uri - single URI or glob
      properties - object-store credentials / options

    • open

      public static DataSource open(Session session, List<String> uris, Map<String,String> properties)
      Open one or more URIs or globs. When a glob is used, the first match is opened eagerly; subsequent matches are opened lazily on scan.
      Parameters:
      session - open session
      uris - URIs or globs to scan
      properties - object-store credentials / options

    • open

      public static DataSource open(Session session, NativeReadable readable)
      Open a single file through a caller-provided byte source. See open(Session, List, int).

    • open

      public static DataSource open(Session session, List<NativeReadable> readables)
      Open files through caller-provided byte sources with the default read concurrency.

    • open

      public static DataSource open(Session session, List<NativeReadable> readables, int readConcurrency)
      Open one or more files through caller-provided byte sources instead of native storage clients. Every read the scan performs becomes an upcall into the corresponding NativeReadable, so this is the integration point for external I/O abstractions (for example Iceberg's FileIO). Each file is identified by NativeReadable.name(), which must be unique within readables.

      The readables must remain open until this data source and all scans created from it are closed; native code never closes them.

      Parameters:
      session - open session
      readables - byte sources
      readConcurrency - maximum in-flight readFully calls across all files of this data source; 0 selects the default. Each in-flight read occupies one native thread and typically one stream on its readable.

    • arrowSchema

      public org.apache.arrow.vector.types.pojo.Schema arrowSchema(org.apache.arrow.memory.BufferAllocator allocator)
      Arrow schema of the data source (and of scans produced from it).

    • rowCount

      public DataSource.RowCount rowCount()
      Row count along with the precision of that estimate. Mirrors the Rust Precision<u64> returned by DataSource::row_count: DataSource.RowCount.Unknown when no estimate is available, DataSource.RowCount.Estimate for an inexact hint, DataSource.RowCount.Exact when the count is authoritative.

    • byteSize

      public DataSource.ByteSize byteSize()
      Sum of the on-storage byte sizes of all files included in this data source along with the precision of that estimate. Mirrors the Rust Option<Precision<u64>> returned by DataSource::byte_size: DataSource.ByteSize.Unknown when no estimate is available (for example when the filesystem listing did not return sizes), DataSource.ByteSize.Estimate for an inexact hint (some files contribute extrapolated sizes), and DataSource.ByteSize.Exact when every file has a known size.

    • scan

      public Scan scan(ScanOptions options)
      Submit a scan.