Kleinanzeigen-Agent - The intelligent assistant for kleinanzeigen.de
An advanced monitoring and filtering platform for online marketplaces with real-time notifications, advanced search functions, and multi-platform support based on a scalable microservices architecture.
Project Overview
Kleinanzeigen-Agent is a specialized platform for optimizing the search and monitoring of listings on online marketplaces. The system allows users to define highly precise search queries and be informed in real-time about new, relevant listings. A key feature is filtering by seller ratings – a feature that is not natively available on most marketplaces.
System Architecture
The platform is based on a modern, scalable microservices architecture:
Frontend
- NextJS-based single-page application with server-side rendering
- Reactive UI components with optimized state management solution
- Progressive Web App (PWA) functionality for mobile use
- WebSocket integration for real-time updates without polling
Backend
- ExpressJS REST API with modular controller structure
- Distributed crawler system with Playwright for browser automation
- Redis-based caching and pub/sub system for real-time communication
- Scalable worker architecture for parallel data processing
Data Processing
- MariaDB for structured relational data
- Typesense for high-performance full-text search with minimal latency
- Data cleaning processes for automatic removal of outdated listings
- ETL pipeline for data extraction, transformation, and loading
Technical Challenges
Several complex technical challenges were solved during development:
Crawler Optimization
The main challenge was developing a robust crawling system that works efficiently without overloading the target platforms. Implemented solutions include:
- Adaptive rate-limiting algorithms to avoid IP bans
- Fingerprint randomization to disguise automated access
- Incremental crawling with intelligent change detection
- Distributed proxy rotation for load balancing and anonymity
Search Performance
For responsive searching in large amounts of data, the following optimizations were implemented:
- Typesense integration with type-safe schemas and ultra-fast search performance
- Multi-level caching system for frequent search queries
- Asynchronous pre-indexing of new data
- Fuzzy matching algorithms for tolerant search with configurable error tolerance
Notification System
The real-time notification system was implemented with the following components:
- Multi-channel delivery via Telegram, Discord, and in-app notifications
- Prioritization algorithm based on user preferences and listing relevance
- Deduplication mechanisms to avoid redundant notifications
- Delayed delivery with automatic bundling during high volume
Technologies Used
- NextJS for the frontend
- ExpressJS for the backend API
- MariaDB for relational data storage
- Typesense for high-performance search with minimal latency
- Redis for caching and pub/sub communication
- Python for specialized crawler components
- Playwright for browser automation and DOM manipulation
- Docker and Kubernetes for containerization and orchestration
- Prometheus and Grafana for monitoring and alerting
Scalability and DevOps
The infrastructure was designed with a focus on scalability and operational stability:
- Horizontal scaling of all components for even load distribution
- Automated CI/CD pipeline for continuous integration and deployment
- Infrastructure-as-Code with Terraform for reproducible environments
- Comprehensive monitoring with detailed metrics and automatic alerts
Future Development
The following extensions are planned for upcoming versions:
- Integration of additional marketplaces like Willhaben and Quoka
- Implementation of an AI-powered price analysis tool
- Development of a mobile app for iOS and Android
- Extension of the API for third-party integrations
Visual Representation
