Staged event-driven architecture as a micro-architecture of distributed and pluginable crawling platform

Abstract

Crawling systems available on the market are usually closed solutions dedicated to performing a particular kind of task. There is a meaningful group of users, however, which require an all–in–one studio, not only for executing and running Internet robots, but also for (graphical) (re)defining and (re)composing crawlers according to dynamically changing requirements and use–cases. The Cassiopeia framework addresses the above idea. The crucial aspect regarding its efficiency and scalability is concurrency model applied. One of the promising models is staged event–driven architecture providing some useful benefits, such as splitting an application into separate stages connected by events’ queues–which is interesting, taking into account Cassiopeia’s assumptions regarding crawler (re)composition. The goal of this paper is to present the idea and PoC implementation of the Cassiopeia framework, with special attention paid to its crucial architectural element, i.e., design, implementation, and application of staged event–driven architecture.