Numeric type precision issues in JSON/Debezium output (BINARY_FLOAT, BINARY_DOUBLE, NUMBER)

[rophy]

Version: d167b033 (master, based on upstream v1.9.0)

Description

Three numeric type issues found by comparing Debezium OLR adapter output against Debezium LogMiner adapter output for the same Oracle redo data.

Issue A: BINARY_FLOAT columns output as null

OLR outputs null for every BINARY_FLOAT column value. LogMiner outputs the correct float.

Input	LogMiner	OLR
0	0.0	null
1	1.0	null
256.0	256.0	null

Issue B: BINARY_DOUBLE truncated to 6 significant digits

BuilderJson.cpp:columnDouble() uses std::ostringstream ss; ss << value; without setting precision. The default is 6 significant digits.

Input	LogMiner	OLR
65536.125	65536.125	65536.1
3.14159265358979...	3.141592653589793	3.14159

Same issue in columnFloat().

Issue C: NUMBER with >16 significant digits loses precision

Oracle NUMBER supports up to 38 significant digits, but OLR truncates to ~16, consistent with conversion through a double intermediate.

Column	Input	LogMiner	OLR
NUMBER(20,10)	9999999999.9999999999	9999999999.9999999999	10000000000 (rounded)
NUMBER(20,10)	1234567890.1234567890	1234567890.123456789	1234567890.1234567
NUMBER	3.1415926535... (35 digits)	full 35 digits	16 digits

Values fitting within 16 significant digits are correct.

Steps to reproduce

1. Create table with numeric types:

CREATE TABLE TEST_NUMBERS (
    id          NUMBER PRIMARY KEY,
    col_number  NUMBER,
    col_decimal NUMBER(20,10),
    col_float   BINARY_FLOAT,
    col_double  BINARY_DOUBLE
);
ALTER TABLE TEST_NUMBERS ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS;

INSERT INTO TEST_NUMBERS VALUES (1,
    3.14159265358979323846264338327950288,
    9999999999.9999999999,
    256.0,
    65536.125);
COMMIT;

2. Run OLR with format.type: "debezium" or "json"
3. Compare output against LogMiner for the same redo

Expected result

• BINARY_FLOAT: 256.0
• BINARY_DOUBLE: 65536.125
• NUMBER(20,10): 9999999999.9999999999 (or equivalent BigDecimal encoding)
• NUMBER: full 35 digits of pi

Actual result

• BINARY_FLOAT: null
• BINARY_DOUBLE: 65536.1 (6 significant digits)
• NUMBER(20,10): 10000000000 (rounded to 16 digits)
• NUMBER: 3.141592653589793 (16 digits)

[rophy]

Investigation Notes

Issue B root cause confirmed: std::ostringstream default precision

BuilderJson.cpp lines 30-50:

void BuilderJson::columnFloat(const std::string& columnName, double value) {
    // ...
    std::ostringstream ss;
    ss << value;       // default precision = 6 significant digits
    append(ss.str());
}

void BuilderJson::columnDouble(const std::string& columnName, long double value) {
    // ...
    std::ostringstream ss;
    ss << value;       // default precision = 6 significant digits
    append(ss.str());
}

std::ostringstream::operator<< defaults to 6 significant digits for floating-point types. Verified:

• 65536.125 → 6 sig digits → 65536.1 ✓ matches OLR output
• 3.14159265... → 6 sig digits → 3.14159 ✓ matches OLR output

Fix: ss << std::setprecision(9) << value for float (IEEE 754 single), ss << std::setprecision(17) << value for double.

Same issue exists in BuilderProtobuf.h:270-274 if protobuf format uses similar serialization.

Issue C root cause: Oracle NUMBER parsed through double intermediate

Verified by examining the truncation pattern:

Input	Significant digits preserved	Truncated digits
9999999999.9999999999 (20 digits)	16 → rounds to 10000000000	digits 17-20 lost
1234567890.1234567890 (20 digits)	16 → 1234567890.1234567	digits 17-20 zeroed
3.14159265... (35 digits)	16 → 3.141592653589793	digits 17-35 lost

~16 significant digits = double precision (IEEE 754 binary64 has 52-bit mantissa ≈ 15.9 decimal digits).

The Debezium output encodes NUMBER as Kafka Connect Decimal type (Java BigDecimal → base64-encoded BigInteger + scale). LogMiner preserves all 38 digits of Oracle NUMBER. OLR truncates to ~16, meaning somewhere in OLR's NUMBER→JSON pipeline, the value passes through a C++ double.

Issue A: BINARY_FLOAT → null (needs more investigation)

OLR's code path for BINARY_FLOAT (Builder.cpp:258-262):

case SysCol::COLTYPE::FLOAT:
    if (size == 4)
        columnFloat(column->name, decodeFloat(data));
    else
        columnUnknown(column->name, data, size);
    break;

decodeFloat() at Builder.cpp:617 correctly handles IEEE 754 single-precision (4 bytes). columnFloat() at BuilderJson.cpp:30 outputs the value.

The null output is likely on the Debezium adapter side, not OLR itself. Debezium's OLR adapter maps OLR JSON fields to Kafka Connect schema types. BINARY_FLOAT may not be mapped to the correct schema type, causing Debezium to output null. This needs verification by inspecting OLR's raw network output (before Debezium processing).

[rophy]

Updated Investigation: OLR golden file analysis

Examined OLR's raw JSON output (golden files from number-precision test fixture) to isolate which issues are in OLR vs the Debezium adapter.

Issue A (BINARY_FLOAT → null): NOT an OLR bug

OLR golden files show correct BINARY_FLOAT values:

{"COL_FLOAT":256}
{"COL_FLOAT":3.14159}
{"COL_FLOAT":-256}

OLR outputs the values correctly. The null must come from the Debezium OLR adapter's deserialization/schema mapping.

Issue B (BINARY_DOUBLE 6 sig digits): OLR bug — fix ready

Confirmed in golden files and code. Fix on branch fix/3:

• columnFloat: setprecision(9) for IEEE 754 single-precision
• columnDouble: setprecision(17) for IEEE 754 double-precision

Issue C (NUMBER >16 digits): NOT an OLR bug

OLR's parseNumber() in Builder.h builds decimal strings directly from Oracle's internal NUMBER format — it does not go through double. Golden files confirm full 38-digit precision:

{"COL_NUMBER":99999999999999999999999999999999999999}
{"COL_PRECISE":12345678901234567890123456789012345678}
{"COL_NUMBER":3.14159265358979323846264338327950288}
{"COL_DECIMAL":9999999999.9999999999}

All values match Oracle's full precision. The truncation to ~16 digits happens downstream in the Debezium OLR adapter (likely parsing JSON numbers into Java double instead of BigDecimal).

Summary

Issue	OLR bug?	Action
A: BINARY_FLOAT → null	No	Debezium adapter issue
B: BINARY_DOUBLE precision	Yes	Fix on fix/3 branch
C: NUMBER >16 digits	No	Debezium adapter issue

[rophy]

Issue A Root Cause: Confirmed Debezium OLR adapter bug

Evidence

Debezium's own test suite explicitly skips BINARY_FLOAT and BINARY_DOUBLE for the OLR adapter:

// OraclePrimaryKeyLobReselectIT.java:100-110
@SkipWhenAdapterNameIs(value = AdapterName.OLR, reason = "BINARY_FLOAT not supported")
public void testBinaryFloatColumn() throws Exception { ... }

@SkipWhenAdapterNameIs(value = AdapterName.OLR, reason = "BINARY_DOUBLE not supported")
public void testBinaryDoubleColumn() throws Exception { ... }

Root Cause Analysis

OLR sends correct JSON values (confirmed by golden files):

{"COL_FLOAT": 256}
{"COL_FLOAT": 3.14159}

Jackson deserializes whole numbers like 256 as Java Integer. The conversion chain:

1. OracleValueConverters.convertFloat() checks: Float? NUMBER? BINARY_FLOAT? Double? String? — none match Integer
2. Falls through to JdbcValueConverters.convertFloat() → convertDouble()
3. convertDouble() handles Number (which Integer implements) → returns Double(256.0)
4. But the schema is float32() (expects Java Float), so Kafka Connect's Struct validation rejects the Double
5. Column is optional → handleUnknownData() returns null

The fix needed in Debezium: OracleValueConverters.convertFloat() or OpenLogReplicatorValueConverter should handle Number input types (e.g., ((Number) data).floatValue()).

No existing Debezium issue

Searched both debezium/dbz GitHub issues and Debezium JIRA — no open issue for BINARY_FLOAT/BINARY_DOUBLE with OLR adapter. The limitation is only documented via test skip annotations.

Summary Update

Issue	OLR bug?	Debezium bug?	Status
A: BINARY_FLOAT → null	No	Yes — type mismatch in converter	Will file upstream
B: BINARY_DOUBLE precision	Yes — fixed (rophy/olr#3)	No	Done
C: NUMBER >16 digits	No	Yes — Debezium adapter parses JSON numbers through double	Known limitation

[rophy]

Issue A: Reproduced and filed upstream

Reproduced the BINARY_FLOAT null issue via Debezium twin-test (number-precision scenario). Key findings:

• Only whole-number values are affected (e.g., 0, 1, 256). Fractional values like 3.14 work fine.
• OLR sends correct JSON (e.g., {"COL_FLOAT": 256}). Jackson deserializes 256 as Integer, which eventually becomes Double in the converter chain, but the Kafka Connect schema expects Float — causing a DataException:

DataException: Invalid Java object for schema with type FLOAT32: class java.lang.Double for field: "COL_FLOAT"

• Filed upstream: Oracle OLR adapter: BINARY_FLOAT whole-number values converted to null debezium/dbz#1679