Skip to main content
Contact

By Sakshi Shah · 20 October 2026 · 16 min read

API & Middleware Development: Connecting Systems That Don't Talk to Each Other

What good API development services deliver: why systems fail to connect, when to use middleware, which integration pattern fits, and how to handle failure safely.

Photo by Kirill Sh on Unsplash

Introduction

A typical growing business runs a dozen or more systems: an ERP or accounting package, a CRM, an e-commerce platform, a warehouse or logistics tool, a helpdesk, payment gateways, marketing tools, and a few in-house applications. Each one works. The problem is between them. Orders are copied from the shop into the ERP. Customer details are updated in the CRM but not in billing. Stock levels are exported to a spreadsheet each morning and uploaded somewhere else. Somebody's job, whether it is written down or not, is to carry data from one system to another.

Connecting those systems properly is the job of APIs and middleware. API development services cover building the interfaces systems use to exchange data, and the middleware layer that sits between them — translating formats, handling authentication, retrying failures and keeping a record of what moved where. Done well, it is invisible: data simply appears where it is needed. Done badly, it becomes a tangle of scripts that fail silently, create duplicates and are understood by nobody.

This piece explains why systems struggle to connect, when a middleware layer is worth building instead of direct connections, the main integration patterns and when each fits, how to design the contract between systems, and — most importantly — how to handle the failures every integration eventually meets.

Why Systems Don't Talk to Each Other

"They both have APIs, so connecting them should be easy" is one of the most expensive assumptions in software. Even when both systems have modern interfaces, several differences have to be resolved:

  • Different data shapes. One system stores a customer's name as one field, another as two. One records prices including tax, another without. Dates, currencies and units all vary.
  • Different identifiers. The same customer is C-10442 in the CRM, ABC Traders Pvt Ltd in accounting and an email address in the shop. Matching them reliably is often the hardest part of the whole job.
  • Different authentication. API keys, OAuth tokens that expire, certificates, IP allow-lists — each system has its own rules.
  • Different timing. One system can push changes the moment they happen; another can only be asked for changes every so often; a third only produces a nightly export file.
  • Limits and quirks. Rate limits, pagination, fields that are optional in the documentation but required in practice, and error messages that do not say what went wrong.
  • No API at all. Older and on-premise systems may offer only a database, a file export or a screen.

Every one of these has to be handled somewhere. The question is whether that handling is scattered across dozens of one-off scripts or gathered in one place that is designed, tested and monitored.

Point-to-Point Connections vs a Middleware Layer

The simplest way to connect two systems is to connect them directly. That is fine for two systems. It becomes a problem as the number grows, because every system needs a separate connection to every other system it exchanges data with.

Point to point: 10 connectionsERPCRMShopStockBillingMiddleware: 5 connectionsMiddlewaremap · auth · retry · logERPCRMShopStockBillinga sixth system adds up to five more point-to-point links — or one more middleware connection
With direct connections, the number of links grows much faster than the number of systems. A middleware layer keeps it to one connection per system.

With five systems all exchanging data, point-to-point integration means up to ten separate connections, each with its own mapping, authentication and error handling. Add a sixth system and it can mean five more. Replace one system — a new CRM, say — and every connection that touched the old one has to be rebuilt.

A middleware layer changes that arithmetic. Each system connects once, to the middleware. The middleware translates each system's data into a shared internal format, handles authentication, retries, logging and monitoring in one place, and passes data on to wherever it needs to go. Replacing the CRM means rebuilding one connection, not five. That independence — each side able to change without forcing changes on the others — is the core benefit described on our API & Middleware Development page.

Middleware is not always necessary. Two systems with a single, simple data flow between them may be well served by a direct integration or an off-the-shelf connector. The case for middleware grows with the number of systems, the complexity of the data, and how much the business depends on it being right.

Choosing the Right Integration Pattern

"Connecting two systems" can mean several quite different things, depending on how fresh the data must be and how much of it moves. Most real integrations use a mix of patterns.

PatternHow it worksGood forWatch out for
Request and response APIOne system asks another for data or to do something, and waits for the answerLive lookups: stock availability, order status, price checksSlow or unavailable systems holding up the caller; rate limits
Webhooks and eventsA system announces that something happened; interested systems reactKeeping systems in step as changes happen: new order, payment receivedMissed or duplicated events; ordering of events
Message queueWork is placed in a queue and processed reliably, in order, at a steady paceHigh volumes, traffic spikes, work that must not be lostExtra infrastructure to run and monitor
Scheduled batch or fileData is exported and imported on a schedule, often as a fileLegacy systems, bulk data, overnight reconciliationData is only as fresh as the last run; partial files
Database replicationA read-only copy of a system's database is kept for other systems to queryReporting and analytics; legacy systems with no APITight coupling to the source system's internal structure

A useful rule is to match the pattern to the business need rather than to what is technically fashionable. If finance only needs yesterday's sales each morning, a well-run nightly batch is simpler and more reliable than a real-time event system. If a customer is waiting on a screen for stock availability, it has to be a live lookup. Older systems that offer nothing but a database or a file export are covered in Legacy Application Modernization: Rewrite, Replace or Gradually Improve? — sometimes a bridge is the first step towards replacing them.

Designing the Contract Between Systems

Every integration rests on a contract: what data crosses the boundary, in what shape, what each side can expect, and what happens when something is wrong. When the contract is implicit — buried in code, assumed rather than written — every change on either side risks breaking the integration without anyone noticing.

A good contract defines the shared format for each kind of record, independent of any one system's quirks. For example, a normalised order event that the middleware produces from whichever shop or channel the order came from:

{
  "event": "order.created",
  "eventId": "evt_7f3a9c21",
  "occurredAt": "2026-10-20T09:14:02+05:30",
  "order": {
    "reference": "WEB-58213",
    "customerId": "CUST-10442",
    "currency": "INR",
    "totalExcludingTax": 42500.00,
    "taxTotal": 7650.00,
    "lines": [
      { "sku": "PMP-220", "quantity": 5, "unitPrice": 8500.00 }
    ]
  }
}

Several details in that example matter more than they appear. The eventId lets every receiving system recognise a duplicate. The timestamp includes its time zone. Money is split into amounts excluding tax and tax, with the currency stated, so no system has to guess. The customer is referred to by a single agreed identifier, which the middleware maps to each system's own. And the event name includes a version-friendly structure, so new event types can be added without breaking existing receivers.

Write the contract down, version it, and treat changes to it as seriously as changes to a public API. A documented interface is also what lets the next integration — or the next developer — start from a specification rather than reverse-engineering the code.

Handling Failure: The Part That Decides Everything

Integrations fail. Networks drop, systems go down for maintenance, tokens expire, a record arrives with a field nobody expected. The difference between a dependable integration and a fragile one is almost entirely in how it handles those moments.

SourceQueueMiddlewarewith idempotency keyTargetLoggeddoneokRetry withgrowing delaysfailedDead-letter queueretries used upAlert → person reviews,fixes and replays
Nothing is silently lost: transient failures are retried automatically, persistent ones are parked and raised with a person, and every message can be replayed without creating a duplicate.

Retries with growing delays. Most failures are temporary. Retrying after a few seconds, then a minute, then several minutes resolves the majority without anyone noticing, and avoids hammering a system that is already struggling.

Idempotency. If a request times out, the sender often cannot tell whether it succeeded. Retrying without protection can create the same order or payment twice. An idempotency key — a unique identifier sent with each request, which the receiver uses to recognise and ignore repeats — makes retries safe.

The duplicate problem: most integration incidents that cost real money are not lost data but duplicated data — the same invoice posted twice, the same payment captured twice. Idempotency keys are the cheapest insurance available.

Dead-letter handling. Some failures will not fix themselves: a record with invalid data, a customer that does not exist in the target system. After a set number of retries, these messages should move to a separate holding area and raise an alert, so a person can fix the cause and replay them. The alternative — dropping them, or retrying forever — is how data quietly goes missing.

Timeouts and circuit breakers. A slow system should not bring others down with it. Timeouts limit how long any call can wait, and a circuit breaker temporarily stops calling a system that is clearly failing, giving it time to recover.

Reconciliation. For anything involving money or stock, a periodic check that both sides agree — order counts, totals, balances — catches the rare problem that slips past everything else.

Security, Monitoring and Ownership

An integration layer that can read and write across your core systems holds a lot of power, and needs the security to match. Store every credential in a secrets manager, never in code. Give each connection the narrowest permissions it needs. Encrypt data in transit. Validate everything arriving from outside, and rate-limit any API you expose. Log who and what accessed which data, so questions can be answered later.

Treat the integration as production infrastructure, not a background script. That means dashboards showing throughput and error rates for each connection, alerts when failures rise or a queue backs up, and a named owner who is responsible for it. Much of the value of our Systems Integration work comes from this last part: an integration that is monitored and owned keeps working as the systems around it change.

1

Map the data flows

Which records move between which systems, how fresh they need to be, and how identifiers match across systems.

2

Define the contract

A written, versioned format for each record and event, including errors and what each side can expect.

3

Build with failure in mind

Retries, idempotency, dead-letter handling, timeouts and reconciliation, tested against real failure scenarios.

4

Monitor, document, own

Dashboards, alerts and a named owner, with the interface documented for whoever builds the next connection.

Integration work is often what makes other projects possible: a customer portal needs live data from core systems, a spreadsheet replaced by an application needs to feed accounting, and AI features need clean access to business data, as described in How to Integrate AI With Your Existing ERP, CRM and Business Systems.

Conclusion

Systems that do not talk to each other cost a business in re-keying, delays, mismatched records and the quiet risk of data that never arrived. API and middleware development fixes that by giving each system one well-defined connection to a shared layer that translates, authenticates, retries and records every exchange.

Choose the integration pattern that matches the business need, write the contract down, and invest most of your care in failure handling — retries, idempotency, dead-letter queues and reconciliation. Then monitor it and give it an owner. That is what turns integration from a collection of fragile scripts into infrastructure the business can rely on.

Have systems that don't talk to each other?

Let's map the data flow and identify the integration layer you actually need. See API & Middleware Development or start the conversation.

Frequently Asked Questions

What is the difference between an API and middleware?

An API is the interface a system offers so other software can exchange data with it. Middleware is a layer that sits between systems, using their APIs (or files, or databases) to translate data, handle authentication and failures, and route information to where it needs to go.

Should we use an off-the-shelf integration platform instead?

For common connections between popular applications, an integration platform can be quick and effective. Custom middleware makes more sense when you have unusual or legacy systems, complex data mapping, high volumes, strict security needs, or when per-connection subscription costs grow large.

Can you connect a system that has no API?

Often, yes — through a database connection, scheduled file exports, or, as a last resort, automating the system's screens. The least invasive option is chosen after testing what the system actually exposes.

How do we avoid duplicate records when integrations retry?

Use idempotency keys: a unique identifier sent with each request, which the receiving system uses to recognise and ignore a repeat. Combined with periodic reconciliation, this prevents most duplicate orders, invoices and payments.

How long does an integration project take?

A single, well-documented connection can take a few weeks including testing. A middleware layer connecting several systems, with identifier matching, failure handling and monitoring, is usually delivered in stages over a few months, with the flow that matters most live first.

Further Reading