While massive open online courses (MOOCs) can be effective in scaling education, orchestrating collaborative learning activities for large audiences remains a non-trivial task that introduces a series of practical challenges, such as the lack of adequate human support. Even when collaboration takes place, there is uncertainty whether meaningful interactions will occur among learners. This work presents the architecture of a prototype system called PeerTalk. The system was created to enable instructors to easily incorporate real-time collaborative learning activities into their online courses. Furthermore, PeerTalk employs a conversational agent service that aims to scaffold students’ online collaboration and provide valuable guidance, which can be configured by the course instructor. In order to investigate the user-acceptance of the system, two evaluation studies took place. The first one involved a group of experts, i.e., MOOC instructors who are expected to use such a system in their course, whereas the second study featured 44 postgraduate students. The study findings were encouraging in terms of the system efficiency and usability levels, laying the foundation for a conversational agent service, which can effectively scale the support of the teaching staff and be easily integrated in MOOC platforms, creating further opportunities for valuable social interaction among learners.