Infor DataLake – JDBC Error 101: the .ionapi file is invalid (it isn’t)

Part 3 of the series: Ingesting Infor LN at Speed

This post stands on its own. If you have arrived here from a search engine with this error in front of you, you do not need the earlier parts of the series skip to section two and take the experiment.

For everyone else, the context in one paragraph: we had just replaced a per-cell JDBC-to-Python bridge with an Apache Arrow transfer path, which removed a GIL-bound bottleneck that had been making our ingestion slower the more parallelism we gave it. With that gone, the reason to keep concurrency artificially low had gone too. So we turned it back up.

And ingestion immediately began failing across the board with:

JDBC Error 101 (.ionapi file is invalid)

The .ionapi file was not invalid. It was the same file that had worked five minutes earlier, byte for byte, and would work again five minutes later. Re-downloading it from ION changed nothing. Re-provisioning the service account changed nothing. Checking the expiry, the scopes, the tenant identifier, the certificate all fine, all correct, all irrelevant.

The error message was describing a state that did not exist.

1. Why This Is Worth Writing Down

Vendor error messages are a form of documentation, and like all documentation they encode what the vendor’s engineer believed the failure mode was at the time they wrote the string. When that belief is incomplete, the message doesn’t become vaguer it becomes confidently wrong, and it points every user who hits it in a direction that has no answer at the end of it.

“The .ionapi file is invalid” tells you to go and look at your credentials. We looked at our credentials for longer than I would like to admit. The credentials were never the problem.

The only way out of a situation like that is to stop reading the message and start designing an experiment.

2. The Experiment

We wanted to isolate one variable at a time, which meant getting away from the batch entirely and constructing conditions by hand against a quiet tenant.

Three conditions, run deliberately:

Condition A — eight handshakes, strictly sequential, quiet tenant.
All eight succeeded. Each OAuth handshake took roughly two seconds. Then and this is the part that reframed everything all eight resulting sessions issued queries in parallel with no trouble whatsoever. Eight concurrent data-plane streams, zero errors.

Condition B — two handshakes, concurrent, quiet tenant.
Persistent failure. Not intermittent, not load-dependent. Two overlapping logins for the same service account, and one of them fails. Two.

Condition C — one handshake attempted while eight extraction streams are running.
Persistent failure. The handshake is alone there is no other handshake anywhere and it still fails, purely because the data plane is busy.

Sit with those three results for a second, because together they say something quite specific and quite unlike what the error message implies:

• Condition A rules out session count. Eight simultaneous sessions are fine.
• Condition A also rules out the credentials, the file, and the service account the same file authenticated eight times in a row without complaint.
• Condition B isolates handshake concurrency as sufficient on its own to cause the failure, at a threshold of two.
• Condition C shows that data-plane traffic counts against the same budget as the token endpoint.

The conclusion:

ION rejects handshakes that overlap either (a) another handshake for the same service account, or (b) active data-plane traffic. The token endpoint and the data plane share one rate budget. Session count is not the limit. Handshake concurrency is.

That distinction is the entire fix. The error told us we had too many connections. The truth was that we had too many simultaneous logins an entirely different problem with an entirely different remedy. If you believe it is connection count, you shrink your pool, lose your parallelism, and still fail intermittently whenever two threads happen to renew at once. If you know it is handshake concurrency, you keep the pool and serialise one narrow operation.

3. The Connection Choreography

Four changes, none large, each addressing a different way handshakes were colliding.

3.1 Serialise Every Handshake

_conn_create_gate = threading.Semaphore(1)

One handshake at a time, process-wide. Everything else stays fully parallel.

The instinctive objection is that this reintroduces a serialisation point into a pipeline we just spent two posts de-serialising. It doesn’t, and the reason is worth being precise about: connections are created rarely and used constantly. A handshake costs about two seconds and then serves an entire batch. With eight threads, the gate costs a handful of seconds once during batch ramp-up and precisely nothing thereafter. Extraction the part that actually takes twenty minutes never touches the semaphore.

Serialising a rare, cheap, expensive-to-collide operation is not the same as serialising your workload. Know which one you are doing.

3.2 Pre-warm Before the Data Plane Gets Busy

Condition C means that even a perfectly serialised handshake will fail if it happens while extraction is streaming. So handshakes have to happen in a quiet window before any extraction starts.

Every executor thread opens its connection up front:

barrier = threading.Barrier(n_threads)
def _warm():
    ok = warm_connection()
    barrier.wait(timeout=600)
    return ok

The barrier is the non-obvious part, and it is the thing that makes this work at all.

Without it, you submit N warm-up tasks to a thread pool and the pool is entirely within its rights to run all of them on one thread, sequentially, as each finishes. You would get N successful handshakes on one connection object and N-1 threads with no connection and they would then create theirs lazily, during extraction, which is exactly the failure you were trying to prevent.

Barrier(n_threads) holds each thread inside the task until every thread has entered one. A thread cannot finish its warm-up task until all the others have started theirs, which forces the pool to occupy every worker simultaneously. That is what guarantees the warm-up lands on distinct threads rather than one thread servicing the whole pool.

This is wired into all three batch entry points ad-hoc runs, DAG-scheduled runs, and plain scheduled runs. Miss one and that path silently reverts to lazy creation under load.

Warm-up failures are deliberately non-fatal. A thread that fails to pre-warm falls back to lazy creation with the retry ladder below. Pre-warming is an optimisation that removes a failure mode; it should not itself become one.

3.3 Make Connections Outlive the Batch

Connection TTL went from 10 minutes to 60 minutes, with ±10 minutes of jitter.

Ten minutes was chosen back when connections were cheap and disposable, and it guaranteed that a twenty-minute batch would renew mid-run creating a handshake in the middle of heavy data-plane traffic, which is Condition C precisely. Sixty minutes means nothing renews during a normal run at all.

The jitter matters as much as the duration. Without it, threads that were pre-warmed together within a few milliseconds will expire together and stampede the token endpoint simultaneously a self-inflicted Condition B, on a timer, guaranteed to fire eventually. Spreading expiry across a twenty-minute window means renewals are naturally serialised by the gate rather than fighting for it.

This is safe because the Compass driver refreshes its own bearer token internally. A long-lived connection does not go stale as its original token ages; the driver handles that beneath the JDBC layer. The TTL exists to bound connection lifetime for hygiene reasons, not because the credential expires.

3.4 Retry with Backoff for the Collisions That Remain

Belt and braces for anything that still overlaps a lazily created connection, a pre-warm failure, an unlucky renewal:

• 8 attempts
• Backoff of min(2**n, 60) seconds, plus jitter
• Re-copy the .ionapi file between attempts, to pick up a renewed token

And the detail that is easy to get wrong and expensive when you do:

The backoff sleeps happen outside the semaphore.

A thread that acquires the handshake gate, fails, and then sleeps for thirty seconds while still holding it has converted a one-at-a-time gate into a one-every-thirty-seconds gate, and blocked every other thread in the pool behind its own failure. Acquire, attempt, release, then sleep, then re-acquire. A waiting thread must never hold the one handshake slot while it waits.

4. Shipping It Without a Regression

All of this the Arrow path and the connection choreography replaced the mechanism at the very bottom of a pipeline feeding 350-plus tables. That is not something you merge and hope about.

The whole thing sits behind a flag, with per-table fallback:

if settings.etl_use_arrow_jdbc:
    try:
        rows_read, rows_written = _stream(extract_table_arrow)
    except Exception as exc:
        logger.warning(
            "arrow-jdbc extraction failed for %s (%s) — falling back to JDBC extraction",
            table, exc,
        )
    else:
        _write_sidecar(rows_written)
        return ...
rows_read, rows_written = _stream(extract_table)

Two properties make this genuinely safe rather than superficially safe.

The fallback granularity is one table, not one batch. A single table hitting an unmapped type or an allocator problem demotes itself to the legacy path and completes. The other 349 stay on the fast path. There is no scenario where one awkward table takes the run down, and no scenario where you have to choose between the whole fast path and none of it.

There is no half-written-file failure mode. This is the property that makes the pattern work, and it is worth checking in your own pipeline before you copy it: the loader’s write_table_stream truncates the run’s Parquet file on open. So if Arrow writes 40,000 rows and then throws, the fallback path opens the same target, truncates it, and writes all the rows from the beginning. The partial write is cleanly overwritten. Without that truncate-on-open property, this fallback would be a data corruption bug wearing a safety-net costume.

Then the part I liked most, which was an accident:

The fallback counter became our health metric.

We had built the fallback for safety. What we got as a side effect was a single number that answers “is the fast path actually carrying the load?” without any instrumentation of the fast path itself. Zero fallbacks means every table went through Arrow. A non-zero count is both an alert and a work item, pre-labelled with exactly which table to go and look at.

There is a small design principle in that. If your safety mechanism increments a counter when it fires, you have a health metric for free and unlike most metrics, it is one you actively want to be boring.

5. Results

Benchmark conditions: full DAG (Bronze → Silver → Gold), 8-way concurrency, clean environment, single run.

Per-table throughput on the larger tables:

The comparisons that matter:

• 50×+ faster than the single-threaded baseline 663,000 against 12,810 rows/min.
• ln_tfgld205, one of the widest tables in the set, went from roughly an hour and a half to about two minutes.
• Tables that could not complete at all inside the 60-minute watchdog now finish inside the log-flush interval.

ln_tfgld205 is the one to look at twice. At 159,000 rows/min it is the slowest table in that list by a factor of four because it is very wide, and width was the original villain of this story. Width still costs something; the Arrow path did not make it free. It made it survivable.

And zeros in three rows of that first table is the result I would actually defend in a review. Nought failures, nought fallbacks, nought auth retries means the fast path carried everything and the connection choreography held, rather than the system limping to a good average while quietly retrying underneath.

But the number that changed how people work is none of the above:

A full-refresh backfill went from “not possible” to something that fits inside an hourly incremental schedule.

Before, a full refresh was a weekend operation that someone had to plan, babysit, and partially abandon. Now the expensive recovery path fits inside the routine one. That changes what you are willing to do schema changes, re-ingests, adding tables because the cost of being wrong stopped being a weekend.

6. Caveats, Stated Plainly

These belong in the post, not in a footnote.

One benchmark run. Single run, 8-vCPU host, against an Infor TRN (test) tenant. Not a multi-run average, no variance figures. Production LN volumes and data shapes will differ, and test tenants are quieter than production ones in ways that matter specifically for the ION rate budget discussed above.

Concurrency above 8 is untested. The authentication ceiling is solved, but we do not know what the next ceiling is. Two candidates: ION’s data-plane rate budget, or the GIL re-emerging somewhere further along most likely in the Parquet encode and ADLS write path, which is still Python and which we have not profiled under higher concurrency. Anyone reproducing this should expect to find a new wall somewhere above eight and should not assume it is in the same place ours will be.

Check your logging before you check your benchmark. During this work, application INFO logs were not reaching docker logs no root handler was configured, so only WARNING and above escaped via lastResort. Which meant the Extraction complete (arrow) line, the one piece of direct evidence that the fast path had run, was invisible for the entire benchmark. We judged Arrow health by the absence of fallback warnings plus raw throughput sound reasoning, but reasoning we should not have needed. If your logs cannot tell you which code path executed, your benchmark is measuring something you are inferring rather than something you observed.

A JVM startup ordering hazard. If the JVM starts before the driver and credential files have settled on disk, the driver caches that bad state and every subsequent connection fails permanently, until the backend restarts. No amount of retrying at the connection layer recovers it, because the corruption is in a JVM-lifetime cache rather than in any individual connection. This one presents as “authentication broke and stayed broken,” which is a very good disguise for the ION problem this whole post is about, and it needs eliminating before you start the experiment in section two.

7. Two Lessons That Transfer

Vendor error messages lie. Not maliciously they encode a hypothesis someone held while writing the string, and hypotheses can be incomplete. “The .ionapi file is invalid” actually meant “two of your logins overlapped,” which shares no vocabulary with the message and no remedy with the investigation it prompts. When a message and the evidence disagree, believe the evidence and design an experiment. Three controlled conditions and about an hour gave us an answer that an unbounded amount of credential-checking never would have.

Rate limits have shapes. It is tempting to model a limit as a single number requests per second, connections allowed and to respond to hitting one by making your number smaller. Ours was not that shape. It limited concurrent handshakes, not sessions, and it counted data traffic against the same budget as authentication. Sessions were unlimited as far as we could tell; two overlapping logins were fatal. Once we understood the shape, the fix was one semaphore and a pre-warm and critically, not a smaller pool, which is what every instinct and every reading of the error message would have told us to do, and which would have cost us the parallelism we had just spent the whole project earning.

This concludes the series. Part 1, The bottleneck was never the network, covers the diagnosis and the parallelism paradox. Part 2, Arrow as the wire format between Java and Python, covers the transport rewrite and its four production traps.