> ## Documentation Index
> Fetch the complete documentation index at: https://docs.streemlined.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Core Concepts

> This page covers the key concepts you need to understand when building Streemlined pipelines.

## Pipelines

A **pipeline** is a directed graph of processing nodes connected by links. Each pipeline:

* Consumes data from one or more Kafka topics
* Processes data through transformation nodes
* Outputs data to sinks (databases, Kafka topics, HTTP endpoints)

```mermaid theme={null}
flowchart LR
    A[Kafka Consumer] --> B[Transform]
    B --> C[Branch]
    C -->|High Value| D[Database]
    C -->|Default| E[Kafka Topic]
```

Pipelines are defined visually in the editor and stored as YAML configuration.

## Nodes

A **node** is a single processing unit in your pipeline. Streemlined provides three types of nodes:

<CardGroup cols={3}>
  <Card title="Sources" icon="arrow-right-to-bracket">
    Ingest data into the pipeline (e.g., Kafka Consumer)
  </Card>

  <Card title="Processors" icon="gears">
    Transform, filter, route, or enrich data
  </Card>

  <Card title="Sinks" icon="arrow-right-from-bracket">
    Output data to external systems
  </Card>
</CardGroup>

Each node has:

* **Input ports** — Where data enters the node
* **Output ports** — Where processed data exits
* **Configuration** — Node-specific settings

## Links

**Links** connect nodes together, defining how data flows through your pipeline.

* A link connects an output port of one node to an input port of another
* Data flows along links as individual records
* Links carry schema information for validation

## Schemas

Streemlined tracks **schemas** throughout your pipeline to help you build correct transformations. Schemas flow through your pipeline in a bidirectional system: **downstream** from sources toward sinks, and **upstream** from sinks back toward sources.

### Schema Sources

Schemas can come from:

* **Schema Registry** — Avro schemas fetched automatically
* **JSON inference** — Schemas inferred from sample data
* **Manual definition** — Schemas you define explicitly

For detailed information on how schemas propagate through your pipeline and how to resolve schema conflicts, see [Working with Schemas](/editor#working-with-schemas) in the Visual Editor guide.

## Error Handling

Streemlined provides **dedicated error outputs** for every domain error. When a deserialization fails, a Lookup finds no match, or a Sink cannot write, you decide what happens: route the record to a Dead Letter Queue, send it to a recovery flow — or fail the pipeline entirely if the error is unacceptable.

This approach keeps your main pipeline clean while giving you full visibility and control over failures. For details on configuring error handling in your pipelines, see [Handling Errors](/editor#handling-errors) in the Visual Editor guide.

## Next Steps

<CardGroup cols={3}>
  <Card title="Visual Editor" icon="window" href="/editor">
    Learn to build pipelines in the UI
  </Card>

  <Card title="Transformations" icon="code" href="/transformations">
    Learn JSONata for data transformation
  </Card>

  <Card title="Nodes Reference" icon="diagram-project" href="/nodes">
    Explore all available processing nodes
  </Card>
</CardGroup>