Elixir makes enterprise integration almost comically easy

I like Elixir for business software because the application does not develop a split personality as soon as some work takes longer than an HTTP request.

Plantronic was a Phoenix application with ordinary business concerns: people, teams, calendars, permissions, planning dates, and reports. Then it also had to import exports from SAP. The files arrived out of band, had several shapes, used real-world encodings, and encoded years of organizational rules in details that looked accidental until one of them was wrong.

A common response would be to introduce an import service, a separate queue, and perhaps an integration platform to connect both systems. Plantronic already had Elixir, PostgreSQL, and its domain model. Adding Oban turned that same application into a durable job processor. No new broker or cache was required. There was no second service to deploy and no API between the import code and the planning rules.

That is the part I find almost comical. Enterprise requirements stayed weird, but the operational architecture remained one application and one database.

Background work is part of the application

The SAP import did not run in a controller request or a detached shell script. Oban ran as a child of Plantronic’s OTP application, next to the Ecto repositories, Phoenix PubSub, caches, and the web endpoint. Elixir’s Supervisor gives those processes a common lifecycle and restarts children according to an explicit policy.

Oban stored jobs in the PostgreSQL database Plantronic already needed. A deployment restart could interrupt a process without erasing the job that described its work. Operators could inspect job state in the same system that held the application data. Database backups covered both business records and pending work.

PostgreSQL is still infrastructure. The useful point is that Plantronic did not gain another kind of infrastructure solely because work had to continue in the background. There was no Redis queue, message broker, queue-specific persistence plan, or separate worker release. The same release could serve LiveView screens and process imports.

The amount of glue was small. One worker scanned a configured directory for CSV files. It scheduled itself again, sorted unseen files, and built one import job for each file. An Ecto.Multi inserted the Oban jobs and the matching import-log records in one transaction. Either both became visible or neither did.

In simplified form, the important part looked like this:

Ecto.Multi.new()
|> Oban.insert_all(:jobs, fn changes -> build_jobs(changes) end)
|> Ecto.Multi.insert_all(:logs, ImportEvent, fn changes -> build_logs(changes) end)
|> Repo.transaction()

That transaction removed an awkward failure mode. The scanner could not record a file as seen while losing the job that was supposed to import it. The next scheduled scan provided another chance if creating the batch failed. Oban handled persistence and execution; Ecto kept the enqueue operation consistent with Plantronic’s own records.

Oban also made execution policy ordinary application configuration. Queue concurrency, scheduled execution, uniqueness, cancellation, and retry limits live beside the worker. The first version deliberately allowed only one attempt while import semantics were still being established. Durable jobs do not remove the need to decide whether repeating a partially understood external operation is safe.

The worker is only a durable shell

A job queue does not solve integration by itself. It gives the work a reliable place to run. The valuable part was what happened inside each import job.

The worker detected the file encoding, opened a database transaction, created a temporary table, and streamed the CSV into PostgreSQL with COPY. Every source column entered that temporary table as text. The next statement transformed those strings into Plantronic’s typed records. The temporary table disappeared when the transaction ended.

This approach kept the raw file shape local to one transaction. The rest of Plantronic never needed to know column positions or SAP date conventions. It only saw employees, availability, absences, teams, and organization-unit mappings.

The Ecto transaction also gave each file a clear outcome. If encoding detection, CSV loading, date conversion, or a domain update failed, the transaction rolled back. A half-imported personnel file would be worse than a failed job because planners could mistake incomplete data for current data.

Elixir made it comfortable to combine different tools at the point where each was strongest. File discovery and streaming used the standard library. Oban owned durable execution. Ecto owned database access and transactions. PostgreSQL handled bulk loading, date expansion, joins, and upserts. Phoenix LiveView provided an admin screen for the organization-unit mappings that affected imports.

None of these parts needed a remote call to the others. They shared one supervised application and one domain model.

Weird requirements became domain rules

The source files were not clean tables waiting to be copied. They represented several related feeds with different meanings. Master data described employees and organization units. Calendar data described working hours. Absence ranges had to become one record per affected day. Their filenames carried timestamps, and processing order mattered because a later export could correct an earlier one.

The importer also had to account for less obvious cases:

Employee references appeared both padded and unpadded, so the importer normalized them before matching.
Decimal hours used commas, while Plantronic stored numeric values.
An end date in the year 9999 meant that employment had no known end date.
A later master-data export could omit an email address because of a source problem, so an empty value must not erase a useful existing value.
Organization units could suggest a team, but an existing local team assignment took precedence.
Absence exports described date ranges, while planning needed daily availability.

Those are not transport concerns. Each one changes what a planner sees or what the scheduling model calculates. Putting them in a generic ETL service would not make them less domain-specific. It would only move Plantronic’s business rules into another repository and force two systems to agree on their meaning.

Keeping the import inside Plantronic made the rules testable against the actual schemas. The worker tests created representative files, performed Oban jobs, and asserted on employees, availability, absences, and team assignments. They checked ordering across multiple files, preservation of existing values, normalized identifiers, expanded date ranges, and organization mappings.

This is what I mean by a tight integration. The import was not tightly coupled to controllers or templates. It was close to the domain operations whose correctness it determined.

Elixir removes the pressure to split too early

Long-running work often pushes a web application toward additional services before the domain warrants them. A request process should not spend an hour importing a file, but that does not imply the import needs another language, deployment, datastore, or team boundary.

Elixir already distinguishes processes inside one runtime. A Phoenix request, a LiveView connection, a scheduled scanner, and an Oban worker can have separate lifecycles without becoming separate applications. OTP supervision handles process lifecycle. Oban adds durable job semantics. Ecto and PostgreSQL provide transactions around the state those jobs change.

This arrangement has practical benefits for a small product team. One release contains the web interface and workers. One migration can update the domain schema and the import path together. One test suite exercises both. Logging and telemetry use the application’s existing setup. A developer can follow a value from a source fixture through the worker into the same context functions and tables used by the UI.

A separate service can become useful when ownership, scaling, security, or deployment requirements truly differ. The SAP import had none of those boundaries at first. Splitting it would have created a distributed system without removing any business complexity.

Boundaries still matter

Keeping the import in one application does not mean letting the source format spread everywhere.

Plantronic still needed a clear inbound boundary. The received file had to remain diagnosable. Encoding and column handling belonged near ingestion. A temporary table represented the external shape. Explicit SQL transformed it into local concepts. From that point onward, routing, permissions, and reporting used Plantronic’s model rather than file columns.

The outbound side needed similar care. Public database views and XML endpoints exposed selected planning data to other consumers. Their columns, ordering, filters, and treatment of missing values became contracts. “Public” inside an organization did not mean unrestricted or accidental.

The architecture mattered, but it was not the most interesting result. The interesting result was how little ceremony Elixir required to implement it. A durable background process, an admin interface, transactions, bulk data handling, telemetry, and domain tests all fit into the existing application.

SAP did not become simple. Its exports still carried odd encodings, special dates, changing organization structures, and incomplete values. Elixir let Plantronic spend its complexity budget on those actual rules instead of queue infrastructure and service coordination.

That is why I consider Elixir such a strong platform for business and web applications. It serves the web request, runs the background job, supervises both, and keeps the work close to the domain that gives it meaning.