Package dev.vortex.api

Class VortexWriter

java.lang.Object
dev.vortex.api.VortexWriter
All Implemented Interfaces:
AutoCloseable
public final class VortexWriter extends Object implements AutoCloseable
Writer for Vortex files.

Open one with builder(Session, String, Schema, BufferAllocator) to write to a URI, or builder(Session, NativeWritable, Schema, BufferAllocator) to write into a caller-provided byte sink.

Batches are accepted via the Arrow C Data Interface: callers export an Arrow record batch to an ArrowArray / ArrowSchema pair and pass the addresses to writeBatch(long, long). The writer accepts up to four in-flight batches on the session's runtime thread before back-pressuring the caller.

Call close() to flush remaining batches and finalize the file. If the writer becomes unreachable without an explicit close(), VortexCleaner will flush and release native resources as a backstop — but callers should always finalize explicitly so that I/O errors surface through the normal call path.

  • Nested Class Summary

    Nested Classes
    Modifier and Type
    Class
    Description
    static final class 
    VortexWriter.Builder
    Configures and opens a VortexWriter.

  • Method Summary

    Modifier and Type
    Method
    Description
    long
    bufferedBytes()
    Return the number of uncompressed bytes accepted by the writer but not yet written to the sink.
    static VortexWriter.Builder
    builder(Session session, NativeWritable writable, org.apache.arrow.vector.types.pojo.Schema arrowSchema, org.apache.arrow.memory.BufferAllocator allocator)
    Start configuring a writer that streams the file into a caller-provided byte sink instead of a native storage client.
    static VortexWriter.Builder
    builder(Session session, String uri, org.apache.arrow.vector.types.pojo.Schema arrowSchema, org.apache.arrow.memory.BufferAllocator allocator)
    Start configuring a writer that streams records into the file at uri through a native storage client.
    long
    bytesWritten()
    Return the number of bytes successfully written to the underlying sink so far.
    void
    close()
    Flush any pending batches and finalize the file.
    VortexWriteSummary
    finish()
    Flush pending batches, finalize the file, and return its statistics and physical sizes.
    void
    writeBatch(long arrowArrayAddr, long arrowSchemaAddr)
    Write a batch directly from Arrow C Data Interface addresses.

    Methods inherited from class java.lang.Object

    clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

  • Method Details

    • builder

      public static VortexWriter.Builder builder(Session session, String uri, org.apache.arrow.vector.types.pojo.Schema arrowSchema, org.apache.arrow.memory.BufferAllocator allocator)
      Start configuring a writer that streams records into the file at uri through a native storage client. The path may be a full URI or a plain local filesystem path.
      Parameters:
      arrowSchema - describes the exact layout of every batch written

    • builder

      public static VortexWriter.Builder builder(Session session, NativeWritable writable, org.apache.arrow.vector.types.pojo.Schema arrowSchema, org.apache.arrow.memory.BufferAllocator allocator)
      Start configuring a writer that streams the file into a caller-provided byte sink instead of a native storage client. This is the integration point for external I/O abstractions (for example Iceberg's PositionOutputStream).

      The native side writes and flushes the sink but never closes it: after close() returns, all bytes have been written and flushed, and the caller must close the sink to finalize the file.

      Parameters:
      arrowSchema - describes the exact layout of every batch written

    • writeBatch

      public void writeBatch(long arrowArrayAddr, long arrowSchemaAddr) throws IOException
      Write a batch directly from Arrow C Data Interface addresses.
      Throws:
      IOException

    • bytesWritten

      public long bytesWritten()
      Return the number of bytes successfully written to the underlying sink so far.

      This count does not include queued batches or data still buffered by layout strategies, so it may lag the amount of input accepted by writeBatch(long, long). After finish(), it is the exact completed file size and is equal to VortexWriteSummary.fileSize().

    • bufferedBytes

      public long bufferedBytes()
      Return the number of uncompressed bytes accepted by the writer but not yet written to the sink.

      Together with bytesWritten(), this lets callers estimate the in-progress file size: bytes that reached the sink are already compressed, while buffered bytes are still uncompressed and will shrink by roughly the file's observed compression ratio once flushed. After finish(), this is zero.

    • finish

      public VortexWriteSummary finish() throws IOException
      Flush pending batches, finalize the file, and return its statistics and physical sizes.

      This method is idempotent. Later calls return the same immutable summary.

      Throws:
      IOException

    • close

      public void close() throws IOException
      Flush any pending batches and finalize the file. Idempotent.
      Specified by:
      close in interface AutoCloseable
      Throws:
      IOException