[deniskuzZ]

Stacked on #6654 (HIVE-29773). The second commit is this change.

What changes were proposed in this pull request?

VARIANT becomes an ObjectInspector category of its own, next to STRUCT/MAP/LIST/UNION, instead of a struct with a
recognised type name.

• Inspector:
  • VariantObjectInspector is an interface: category VARIANT, type name variant, Variant getVariant(Object).
  • Flavours: standard, constant, struct-encoded (shared by the Hive serdes) and Iceberg.
  • VariantEncoding defines the two-binary layout in one place.
• OI utilities: ObjectInspectorUtils, the converters and SerDeUtils handle VARIANT explicitly. Compare and
  hash keep the byte-wise results of the struct path, and merge joins use HiveVariantComparator.
• Formats:
  • Text, LazyBinary (+ fast, SerDe2), BinarySortable (+ fast), RCFile, ORC, Parquet and JSON write today's bytes.
  • Avro writes Iceberg's variant record: two non-nullable bytes fields tagged "logicalType":"variant", which
    Iceberg's registered logical type accepts.
  • Hive's Parquet reader finds metadata and value by name, never prunes nested columns inside a variant, and
    rejects a shredded variant (a typed_value child) outside Iceberg, which rebuilds shredded values itself.
• Vectorization: a variant column stays StructColumnVector(metadata, value), now wired from the VARIANT
  category. A VARIANT-typed NULL constant vectorizes.
• Planner:
  • Calcite VARIANT in both directions, including VARIANT-typed NULL constants.
  • AST, stats sizing, and SHOW CREATE TABLE / DESCRIBE print variant.
  • Persistent non-Iceberg tables reject VARIANT at any depth; CTAS and materialized views fail at compile time.
    Views and native temporary tables may hold VARIANT.
• Functions:
  • parse_json, CAST(x AS VARIANT), variant_get and to_json work on every inspector flavour.
  • CAST(v AS STRING) and implicit string conversions: NULL for a JSON null, a string's characters, base64 for
    binary, plain text for other scalars (as variant_get(v, '$', 'string')), JSON for objects and arrays.
    to_json(v) and HS2 values stay JSON.
  • collect_set, collect_list, approx_distinct, IN, = and != accept VARIANT.
• Iceberg: IcebergVariantObjectInspector implements the interface, and the writer stores exactly the encoded
  bytes.
• Clients:
  • HS2 reports a VARIANT result column as string with JSON values until HIVE-30166.
  • The LLAP external client gets variant with JSON values.

Why are the changes needed?

At the inspector level VARIANT was a struct<metadata:binary,value:binary>. Plans, result schemas, views, CTAS,
DESCRIBE and HS2 all saw the struct, and features had to recognise a variant by the struct's name or shape.
HIVE-30166 (client capability) and HIVE-30171 (semantic equality) need a variant that is a variant everywhere.

Does this PR introduce any user-facing change?

Yes.

Construct	Before	After
Result, DESCRIBE, SHOW CREATE TABLE, view and CTAS type	struct<metadata:binary,value:binary>	variant
HS2 result column type	struct	string with JSON values
CAST(v AS STRING), implicit string conversions	struct JSON	NULL for a JSON null, bare string, base64 binary, plain scalars, JSON objects and arrays
IN (constants), collect_set(v)	never matched / never deduplicated	byte-wise, like =
v.metadata, v.value	raw bytes	compile error
named_struct('metadata',…,'value',…) into a variant column	accepted	rejected; no SQL route turns a struct into a variant
CTAS / materialized view of a variant into a non-Iceberg table	struct column	rejected at compile time
Non-Iceberg table with a nested variant (array<variant>)	accepted	rejected
CREATE TEMPORARY TABLE t (v VARIANT) with native storage	rejected	allowed (storage-handler temporary tables still reject it)
parse_json('literal')	folded into a struct constant	evaluated per row
ST_GeomFromJson / ST_GeomFromGeoJson with a LIST/MAP/UNION argument	ClassCastException at evaluation	error at initialization

Unchanged:

• on-disk and shuffle bytes, except Avro's variant record (see Formats) and Iceberg's exact value bytes (below);
• the results of =, GROUP BY, DISTINCT, ORDER BY, joins, min/max and hash(v).

Iceberg values written by Hive 4.2.x carry trailing bytes. Hive 4.2.x wrote the whole reused buffer
(BytesWritable.getBytes()), so each stored value carries trailing zero bytes. This change writes exact bytes, and a
Hive rewrite of old rows (INSERT … SELECT, compaction, copy-on-write MERGE/UPDATE) drops the tail. Until then, or
until semantic equality (HIVE-30171), byte-wise =, GROUP BY, DISTINCT, joins and hash(v) treat an old row and a
newly written equal value as different.

Compatibility:

• llap-client jars from 4.2.x or earlier cannot read VARIANT columns (HIVE-30191).
• VariantObjectInspector is an interface, so code compiled against 4.2.x gets IncompatibleClassChangeError.
  VariantObjectInspector.get() and Variant.from(List) are removed.
• ObjectInspectorFactory.getVariantObjectInspector() is deprecated in favour of
  getStandardVariantObjectInspector(). Its objects are now Variant, not List<byte[]>.
• IcebergVariantObjectInspector.get() returns IcebergVariantObjectInspector; Table.validateColumns has a
  fourth parameter.
• struct<metadata:binary,value:binary> columns created by Iceberg CTAS on 4.2.x or non-Iceberg CTAS on master stay
  structs. No SQL route converts them to VARIANT (CAST(x AS VARIANT) rejects structs) until a follow-up adds a
  function from the two binaries to a variant.

How was this patch tested?

• Unit tests:
  • round trips for every serde, and byte identity with the struct encoding for every serde but Avro, including
    nested variants, descending keys, high-bit bytes, buffer reuse, and SQL NULL vs JSON null;
  • OI utilities across inspector flavours, with compare and hash pinned against the struct path's results; UDFs;
    HiveVariantComparator against the BinarySortable key order; JSON escaping;
  • ORC/Parquet file schemas; Parquet children by name, value-first files and shredded groups; vectorized readers
    and rows;
  • Calcite round trips, constant folding, stats sizing and the table rules; Kryo plan serialization;
  • the Avro schema parsed by Iceberg's Avro with its logical types registered;
  • the HS2 schema and the LLAP client.
• q-tests (vectorization on and off):
  • variant_first_class.q: types, casts, the shuffle, hash(v), ordering, a map join, CTE materialization,
    temporary tables and a TEXTFILE directory, over values with bytes >= 0x80;
  • variant_llap_io.q: Iceberg Parquet and Avro and native temporary ORC and TEXTFILE tables on LLAP;
  • variant_parquet_temporary.q: Parquet temporary tables and CTEs with fetch-task conversion;
  • variant_calcite_nulls.q and the existing variant q-tests;
  • negative q-tests for every rejected statement.
• JDBC: TestJdbcDriver2#testVariantType; BaseJdbcWithMiniLlap#testVariantType, whose only subclass is
  @Ignored (HIVE-23549).
• Mutation check: every new VARIANT branch has a test that fails when the branch is reverted.
• Iceberg cannot write a variant to ORC (OrcMetrics), so there is no Iceberg ORC test.

depends on #6890

[sonarqubecloud]

Quality Gate passed

Issues
261 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.8% Duplication on New Code

See analysis details on SonarQube Cloud