68 lines
3.3 KiB
Markdown
68 lines
3.3 KiB
Markdown
# Redis Pub/Sub Chain Implementation
|
|
|
|
This module implements a chain of services communicating through Redis Pub/Sub channels. Each service in the chain subscribes to the previous service's channel and publishes to its own channel, creating a processing pipeline.
|
|
|
|
## Architecture
|
|
|
|
The implementation follows a simple chain pattern:
|
|
|
|
```
|
|
READER → PROCESSOR → WRITER
|
|
```
|
|
|
|
- **READER**: Generates mock data with a "red" stage and publishes to `chain:reader`
|
|
- **PROCESSOR**: Subscribes to `chain:reader`, processes messages with "red" stage, updates stage to "processed", and publishes to `chain:processor`
|
|
- **WRITER**: Subscribes to `chain:processor`, processes messages with "processed" stage, updates stage to "completed", and publishes to `chain:writer`
|
|
|
|
## Message Flow
|
|
|
|
Each message flows through the chain with a stage attribute that determines how it's processed:
|
|
|
|
1. READER generates a message with `stage="red"`
|
|
2. PROCESSOR receives the message, checks if `stage="red"`, processes it, and sets `stage="processed"`
|
|
3. WRITER receives the message, checks if `stage="processed"`, processes it, and sets `stage="completed"`
|
|
|
|
## Performance
|
|
|
|
The implementation includes timing information to track how long messages take to flow through the entire chain. Sample output:
|
|
|
|
```
|
|
[READER] 1745176466.132082 | Published UUID: 74cf2312-25ec-4da8-bc0a-521b6ccd5206
|
|
[PROCESSOR] 1745176466.132918 | Received UUID: 74cf2312-25ec-4da8-bc0a-521b6ccd5206 | Published UUID: 74cf2312-25ec-4da8-bc0a-521b6ccd5206
|
|
[WRITER] 1745176466.133097 | Received UUID: 74cf2312-25ec-4da8-bc0a-521b6ccd5206 | Published UUID: 74cf2312-25ec-4da8-bc0a-521b6ccd5206 | Elapsed: 1.83ms
|
|
[READER] 1745176468.133018 | Published UUID: 2ffd217f-650f-4e10-bc16-317adcf7a59a
|
|
[PROCESSOR] 1745176468.133792 | Received UUID: 2ffd217f-650f-4e10-bc16-317adcf7a59a | Published UUID: 2ffd217f-650f-4e10-bc16-317adcf7a59a
|
|
[WRITER] 1745176468.134001 | Received UUID: 2ffd217f-650f-4e10-bc16-317adcf7a59a | Published UUID: 2ffd217f-650f-4e10-bc16-317adcf7a59a | Elapsed: 1.76ms
|
|
[READER] 1745176470.133841 | Published UUID: 87e1f3af-c6c2-4fa5-9a65-57e7327d3989
|
|
[PROCESSOR] 1745176470.134623 | Received UUID: 87e1f3af-c6c2-4fa5-9a65-57e7327d3989 | Published UUID: 87e1f3af-c6c2-4fa5-9a65-57e7327d3989
|
|
[WRITER] 1745176470.134861 | Received UUID: 87e1f3af-c6c2-4fa5-9a65-57e7327d3989 | Published UUID: 87e1f3af-c6c2-4fa5-9a65-57e7327d3989 | Elapsed: 1.68ms
|
|
```
|
|
|
|
The elapsed time shows the total time from when the READER publishes a message until the WRITER completes processing it. In the samples above, the end-to-end processing time ranges from 1.68ms to 1.83ms.
|
|
|
|
## Usage
|
|
|
|
To run the demonstration:
|
|
|
|
```bash
|
|
python -m Controllers.Redis.Broadcast.implementations
|
|
```
|
|
|
|
This will start all three services in the chain and begin processing messages. Press Ctrl+C to stop the demonstration.
|
|
|
|
## Implementation Details
|
|
|
|
The implementation uses:
|
|
|
|
1. A singleton Redis Pub/Sub handler with publisher and subscriber capabilities
|
|
2. Thread-based message processing
|
|
3. JSON serialization for message passing
|
|
4. Stage-based message processing to track progress through the chain
|
|
5. Timing information to measure performance
|
|
|
|
Each service in the chain follows these steps:
|
|
1. Subscribe to the appropriate channel
|
|
2. Define a message handler function
|
|
3. Process incoming messages based on their stage
|
|
4. Publish processed messages to the next channel in the chain
|