# Pipe and Filter Pattern

## Problem

A complex data transformation must be applied to input, but implementing it as a single monolithic function makes the code hard to understand, test, modify, and reuse. Different use cases need different subsets or orderings of the transformation steps.

## Solution

The **Pipe and Filter** pattern decomposes processing into a series of independent **filters** connected by **pipes**:

- **Filter** — A self-contained processing step that takes input, transforms it, and produces output. Filters know nothing about adjacent filters.
- **Pipe** — The connector that passes output from one filter as input to the next.
- **Pipeline** — An ordered sequence of filters connected by pipes.

Each filter is independently testable and reusable. Filters can be reordered, added, or removed to create different pipelines.

## When to Use

- Data processing pipelines (ETL, log processing, image processing).
- Compiler stages (lexing, parsing, optimization, code generation).
- Unix-style command chaining.
- When processing steps are independent and reusable across different workflows.

## When to Avoid

- When steps have complex interdependencies that require shared state.
- Interactive applications where processing is not sequential.
- When the overhead of data copying between stages is unacceptable.
- Tiny transformations where a pipeline adds unnecessary structure.

## Pseudocode

```
INTERFACE Filter<In, Out>:
    METHOD process(input: In) -> Out

CLASS Pipeline:
    filters = []

    METHOD add_filter(filter):
        APPEND filter TO filters

    METHOD execute(input):
        result = input
        FOR EACH filter IN filters:
            result = filter.process(result)
        RETURN result
```

## Python

```python
from abc import ABC, abstractmethod
from dataclasses import dataclass, field


@dataclass
class LogEntry:
    timestamp: str
    level: str
    message: str
    source: str
    tags: list[str] = field(default_factory=list)


class Filter(ABC):
    @abstractmethod
    def process(self, entries: list[LogEntry]) -> list[LogEntry]: ...

    @property
    def name(self) -> str:
        return self.__class__.__name__


class LevelFilter(Filter):
    """Keep only entries at or above a given severity."""
    LEVELS = {"DEBUG": 0, "INFO": 1, "WARN": 2, "ERROR": 3, "FATAL": 4}

    def __init__(self, min_level: str) -> None:
        self._min = self.LEVELS.get(min_level, 0)

    def process(self, entries: list[LogEntry]) -> list[LogEntry]:
        result = [e for e in entries if self.LEVELS.get(e.level, 0) >= self._min]
        print(f"    [{self.name}] {len(entries)} -> {len(result)} entries (min={list(self.LEVELS.keys())[self._min]})")
        return result


class SourceFilter(Filter):
    """Keep only entries from specific sources."""
    def __init__(self, sources: list[str]) -> None:
        self._sources = set(sources)

    def process(self, entries: list[LogEntry]) -> list[LogEntry]:
        result = [e for e in entries if e.source in self._sources]
        print(f"    [{self.name}] {len(entries)} -> {len(result)} entries (sources={self._sources})")
        return result


class TagEnricher(Filter):
    """Add a tag to all entries."""
    def __init__(self, tag: str) -> None:
        self._tag = tag

    def process(self, entries: list[LogEntry]) -> list[LogEntry]:
        for e in entries:
            if self._tag not in e.tags:
                e.tags.append(self._tag)
        print(f"    [{self.name}] Tagged {len(entries)} entries with '{self._tag}'")
        return entries


class MessageNormalizer(Filter):
    """Lowercase and strip whitespace from messages."""
    def process(self, entries: list[LogEntry]) -> list[LogEntry]:
        for e in entries:
            e.message = e.message.strip().lower()
        print(f"    [{self.name}] Normalized {len(entries)} messages")
        return entries


class DuplicateRemover(Filter):
    """Remove entries with duplicate messages."""
    def process(self, entries: list[LogEntry]) -> list[LogEntry]:
        seen: set[str] = set()
        result = []
        for e in entries:
            if e.message not in seen:
                seen.add(e.message)
                result.append(e)
        print(f"    [{self.name}] {len(entries)} -> {len(result)} entries (removed {len(entries) - len(result)} dupes)")
        return result


class Pipeline:
    def __init__(self) -> None:
        self._filters: list[Filter] = []

    def add_filter(self, f: Filter) -> "Pipeline":
        self._filters.append(f)
        return self

    def execute(self, data: list[LogEntry]) -> list[LogEntry]:
        result = data
        print(f"  [Pipeline] Starting with {len(result)} entries")
        for f in self._filters:
            result = f.process(result)
        print(f"  [Pipeline] Finished with {len(result)} entries")
        return result


def main() -> None:
    logs = [
        LogEntry("10:00:01", "DEBUG", "Starting service", "auth"),
        LogEntry("10:00:02", "INFO", "User logged in", "auth"),
        LogEntry("10:00:03", "WARN", "Slow query detected", "db"),
        LogEntry("10:00:04", "ERROR", "Connection timeout", "api"),
        LogEntry("10:00:05", "INFO", "Cache miss", "cache"),
        LogEntry("10:00:06", "ERROR", "Connection timeout", "api"),  # duplicate
        LogEntry("10:00:07", "FATAL", "Out of memory", "db"),
        LogEntry("10:00:08", "DEBUG", "  Extra whitespace  ", "auth"),
        LogEntry("10:00:09", "INFO", "User logged in", "auth"),  # duplicate
        LogEntry("10:00:10", "WARN", "Disk usage high", "infra"),
    ]

    print("=== Pipeline 1: Error+ logs, deduplicated ===")
    p1 = Pipeline()
    p1.add_filter(LevelFilter("ERROR"))
    p1.add_filter(DuplicateRemover())
    p1.add_filter(TagEnricher("reviewed"))
    result1 = p1.execute(logs)
    print()
    for e in result1:
        print(f"  [{e.level}] {e.source}: {e.message} {e.tags}")

    print("\n=== Pipeline 2: Auth source, normalized, deduplicated ===")
    p2 = Pipeline()
    p2.add_filter(SourceFilter(["auth"]))
    p2.add_filter(MessageNormalizer())
    p2.add_filter(DuplicateRemover())
    result2 = p2.execute(logs)
    print()
    for e in result2:
        print(f"  [{e.level}] {e.message}")


if __name__ == "__main__":
    main()
```

**Output:**
```
=== Pipeline 1: Error+ logs, deduplicated ===
  [Pipeline] Starting with 10 entries
    [LevelFilter] 10 -> 3 entries (min=ERROR)
    [DuplicateRemover] 3 -> 2 entries (removed 1 dupes)
    [TagEnricher] Tagged 2 entries with 'reviewed'
  [Pipeline] Finished with 2 entries

  [ERROR] api: Connection timeout ['reviewed']
  [FATAL] db: Out of memory ['reviewed']

=== Pipeline 2: Auth source, normalized, deduplicated ===
  [Pipeline] Starting with 10 entries
    [SourceFilter] 10 -> 4 entries (sources={'auth'})
    [MessageNormalizer] Normalized 4 messages
    [DuplicateRemover] 4 -> 3 entries (removed 1 dupes)
  [Pipeline] Finished with 3 entries

  [DEBUG] starting service
  [INFO] user logged in
  [DEBUG] extra whitespace
```

## Go

```go
package main

import (
	"fmt"
	"strings"
)

// --- Domain ---

type LogEntry struct {
	Timestamp string
	Level     string
	Message   string
	Source    string
	Tags     []string
}

// --- Filter interface ---

type Filter interface {
	Name() string
	Process(entries []LogEntry) []LogEntry
}

// --- Filters ---

type LevelFilter struct {
	MinLevel int
	Label    string
}

var levelMap = map[string]int{
	"DEBUG": 0, "INFO": 1, "WARN": 2, "ERROR": 3, "FATAL": 4,
}

func NewLevelFilter(minLevel string) *LevelFilter {
	return &LevelFilter{MinLevel: levelMap[minLevel], Label: minLevel}
}

func (f *LevelFilter) Name() string { return "LevelFilter" }
func (f *LevelFilter) Process(entries []LogEntry) []LogEntry {
	var result []LogEntry
	for _, e := range entries {
		if levelMap[e.Level] >= f.MinLevel {
			result = append(result, e)
		}
	}
	fmt.Printf("    [%s] %d -> %d entries (min=%s)\n", f.Name(), len(entries), len(result), f.Label)
	return result
}

type SourceFilter struct {
	Sources map[string]bool
}

func NewSourceFilter(sources []string) *SourceFilter {
	m := make(map[string]bool)
	for _, s := range sources {
		m[s] = true
	}
	return &SourceFilter{Sources: m}
}

func (f *SourceFilter) Name() string { return "SourceFilter" }
func (f *SourceFilter) Process(entries []LogEntry) []LogEntry {
	var result []LogEntry
	for _, e := range entries {
		if f.Sources[e.Source] {
			result = append(result, e)
		}
	}
	fmt.Printf("    [%s] %d -> %d entries\n", f.Name(), len(entries), len(result))
	return result
}

type MessageNormalizer struct{}

func (f *MessageNormalizer) Name() string { return "MessageNormalizer" }
func (f *MessageNormalizer) Process(entries []LogEntry) []LogEntry {
	for i := range entries {
		entries[i].Message = strings.ToLower(strings.TrimSpace(entries[i].Message))
	}
	fmt.Printf("    [%s] Normalized %d messages\n", f.Name(), len(entries))
	return entries
}

type DuplicateRemover struct{}

func (f *DuplicateRemover) Name() string { return "DuplicateRemover" }
func (f *DuplicateRemover) Process(entries []LogEntry) []LogEntry {
	seen := make(map[string]bool)
	var result []LogEntry
	for _, e := range entries {
		if !seen[e.Message] {
			seen[e.Message] = true
			result = append(result, e)
		}
	}
	removed := len(entries) - len(result)
	fmt.Printf("    [%s] %d -> %d entries (removed %d dupes)\n", f.Name(), len(entries), len(result), removed)
	return result
}

type TagEnricher struct {
	Tag string
}

func (f *TagEnricher) Name() string { return "TagEnricher" }
func (f *TagEnricher) Process(entries []LogEntry) []LogEntry {
	for i := range entries {
		entries[i].Tags = append(entries[i].Tags, f.Tag)
	}
	fmt.Printf("    [%s] Tagged %d entries with '%s'\n", f.Name(), len(entries), f.Tag)
	return entries
}

// --- Pipeline ---

type Pipeline struct {
	filters []Filter
}

func (p *Pipeline) AddFilter(f Filter) *Pipeline {
	p.filters = append(p.filters, f)
	return p
}

func (p *Pipeline) Execute(data []LogEntry) []LogEntry {
	result := data
	fmt.Printf("  [Pipeline] Starting with %d entries\n", len(result))
	for _, f := range p.filters {
		result = f.Process(result)
	}
	fmt.Printf("  [Pipeline] Finished with %d entries\n", len(result))
	return result
}

func main() {
	logs := []LogEntry{
		{"10:00:01", "DEBUG", "Starting service", "auth", nil},
		{"10:00:02", "INFO", "User logged in", "auth", nil},
		{"10:00:03", "WARN", "Slow query detected", "db", nil},
		{"10:00:04", "ERROR", "Connection timeout", "api", nil},
		{"10:00:05", "INFO", "Cache miss", "cache", nil},
		{"10:00:06", "ERROR", "Connection timeout", "api", nil},
		{"10:00:07", "FATAL", "Out of memory", "db", nil},
		{"10:00:08", "DEBUG", "  Extra whitespace  ", "auth", nil},
		{"10:00:09", "INFO", "User logged in", "auth", nil},
		{"10:00:10", "WARN", "Disk usage high", "infra", nil},
	}

	fmt.Println("=== Pipeline 1: Error+ logs, deduplicated ===")
	p1 := &Pipeline{}
	p1.AddFilter(NewLevelFilter("ERROR")).
		AddFilter(&DuplicateRemover{}).
		AddFilter(&TagEnricher{Tag: "reviewed"})

	// Copy logs so pipeline 2 starts fresh
	logs1 := make([]LogEntry, len(logs))
	copy(logs1, logs)
	result1 := p1.Execute(logs1)
	fmt.Println()
	for _, e := range result1 {
		fmt.Printf("  [%s] %s: %s %v\n", e.Level, e.Source, e.Message, e.Tags)
	}

	fmt.Println("\n=== Pipeline 2: Auth source, normalized, deduplicated ===")
	p2 := &Pipeline{}
	p2.AddFilter(NewSourceFilter([]string{"auth"})).
		AddFilter(&MessageNormalizer{}).
		AddFilter(&DuplicateRemover{})

	logs2 := make([]LogEntry, len(logs))
	copy(logs2, logs)
	result2 := p2.Execute(logs2)
	fmt.Println()
	for _, e := range result2 {
		fmt.Printf("  [%s] %s\n", e.Level, e.Message)
	}
}
```

**Output:**
```
=== Pipeline 1: Error+ logs, deduplicated ===
  [Pipeline] Starting with 10 entries
    [LevelFilter] 10 -> 3 entries (min=ERROR)
    [DuplicateRemover] 3 -> 2 entries (removed 1 dupes)
    [TagEnricher] Tagged 2 entries with 'reviewed'
  [Pipeline] Finished with 2 entries

  [ERROR] api: Connection timeout [reviewed]
  [FATAL] db: Out of memory [reviewed]

=== Pipeline 2: Auth source, normalized, deduplicated ===
  [Pipeline] Starting with 10 entries
    [SourceFilter] 10 -> 4 entries
    [MessageNormalizer] Normalized 4 messages
    [DuplicateRemover] 4 -> 3 entries (removed 1 dupes)
  [Pipeline] Finished with 3 entries

  [DEBUG] starting service
  [INFO] user logged in
  [DEBUG] extra whitespace
```

## Related Patterns

- **Chain of Responsibility** — both chain and pipe-filter pass requests along a sequence; the key difference is that every filter processes the data, while in a chain, any handler can stop processing.
- **Decorator** — decorator wraps an object to add behavior; a filter wraps a data stream to transform it.
- **Strategy** — each filter is a strategy for one transformation step; the pipeline composes strategies in sequence.
- **Iterator** — a lazy pipeline of generators is an iterator-based implementation of pipe and filter.
