For some time now, we have been looking for a consistent solution for the mail system used by our teachers and students. We need a secure environment that’s easy to manage. In a combined effort, ICT Medien and Adfinis SyGroup, a Swiss Service Provider, migrated the schools’ existing mail system with around 32,000 accounts. We connected the current identity management (UCS@school) to the e-mail and groupware solution Open-Xchange. In this article we’re going to describe the initial situation, talk about our considerations, the planning phase, and the requirements for the new mail and groupware solution. We’re also going to tell you about our system architecture and the servers involved. During the migration we encountered some problems – you’re going to read about them and about our solutions. Before we start with the more technical details, we’d like to say how happy we are with the new Open Source solution: The new mail system fulfils all requirements in terms of security, high availability, and the current data protection guidelines.
Initial situation
The ICT Media department of the Pedagogical Centre (PZ.BS) is responsible for the administration of the entire IT infrastructure in around 60 elementary schools of Basel-Stadt canton. We also provide educational services for secondary schools, i.e. e-mail, learning platform, or websites. For several years now, we’ve been using UCS@school (Univention Corporate Server for Schools) as central identity and infrastructure management. It’s also connected to Windows Active Directory Service and manages accounts for various services and virtual desktops. A script automatically sychronizes about 32,000 user accounts between the Adfinis servers and the school administration software. The high fluctuation rate is a real challenge here: Every term, we get around 10% new users, and the same amount is no longer required.
All teachers and students from the third grade onwards have personal e-mail accounts, also managed in UCS@school. For every class in every school we automatically generate address books. In 2017, approximately 2.2 million mails were sent, and around 7.4 million mails were received. While the existing mail system was stable enough, it no longer met our requirements.
Requirements: Enhanced Groupware Features and Browser Compatibility
So, we were looking for a modern, up-to-date solution with an interface to UCS that could handle e-mail and address books, and offer groupware features like calendars, a task manager, shared files, etc. ICT Media always tries to use Open Source Software in schools, when possible. So, we had a look at various Open Source solutions as well as the defacto standard Microsoft Exchange. SOGo and Open-Xchange made it to our shortlist, but only Open-Xchange and Microsoft Exchange met our must-have criteria:
- Browser compatibility (Microsoft Edge, Internet Explorer 11, Firefox, and Chrome for Windows)
- LDAP address book (plus display of multiple address books)
- Sending and receiving HTML mails
- Sending plain text mails
- Configuring forwarding to external and internal mail addresses
- Setting up out-of-office messages and filters
- User settings (more than one signature, select from signatures, set name of sender)
- Managing personal calendars
- Organizing meetings with multiple attendees (send out invitations, view acceptances/rejections)
- User-friendly web app for desktop/laptop (1920 x 1280), netbook (1024 x 600), Android smartphone, Android tablet (1024 x 768, portrait mode), iPhone, iPad (1024 x 768, Retina display, portrait mode)
We ran some tests (installation and usability) and asked teachers for feedback. In the end, we decided on Open-Xchange (OX), one argument being the existing UCS interface. Another reason: the possible integration of OX Drive (online storage for photos, files, documents, and videos). During our try-outs, we also made sure that it was possible to migrate the data of the current solution (mailboxes, contacts, address books).
Infrastructure, Clusters, and Load Balancing
The Univention App Center offers easy installation routines for all components of our setup. Per default setting all services (IMAP, MySQL/MariaDB, LDAP, and OX middleware) are running on one single machine which is good enough for smaller environments. In our setup with over 32,000 accounts, we had to find a new solution. Our requirements: high availabilty, clustering, and load balancing/HA proxy. In close corporation with Univention we were able to extend the Open-Xchange standard installation of UCS@school so that it is easily configurable in a multi server infrastructure. The OX App can now be installed on multiple servers (cluster). Our setup consists of the following parts:
- Load Balancing Reverse Proxy for HTTPS and IMAP 4 (external access)
- OX cluster
- MariaDB cluster
- HA proxy cluster
- mutation server
- Dovecot cluster (IMAP mail storage)
High Availability with XenServerCluster and Full-Flash Storage
The entire mail setup is independent from Open-Xchange. The Dovecot cluster is installed at the same time, though, and it’s being integrated into the environment.
Since we were happy with the existing SMTP infrastructure, we didn’t change it. We did set up a new XenServer cluster, though, to ensure a failsafe, high-availability environment. IMAP and the OX data (not the database!) use a shared storage, connected via NFS with the NetApp storage. The remainder of the data, i.e. the database, is stored in directly attached volumes of the virtual servers, located as well on the NetApp. For performance reasons, the entire storage is implemented as full flash. Multiple OX nodes running on UCS@school allow for fast horizontal scaling. If the load increases significantly, additional servers are being provided. The Dovecot directors act as proxy and forward the clients’ IMAP requests to various IMAP servers (also Dovecot) to guarantee a clean session handling. The Dovecot backend servers use the NFS share as central storage.
All MariaDB database servers are Debian GNU/Linux machines. They act as Galera cluster.
Our multi-level network architecture enhances security. All public application servers are separated from the internal servers that store the data. A dedicated UCS server is used for mutations. The event-based UCS listener/notifier mechanism is responsible for exchanging and syncing between several UCS servers.
Automatic Sync between School Software and UCS Accounts
A script exports data from the school administration software ESCADA2 and automatically generates UCS user accounts. A mailbox basically exists as long as a corresponding data record exists in the administration and thus also in the LDAP directory service. If a data record appears in the exported data for the first time, a mailbox is being created for the respective user in the corresponding school context. A context in Open-Xchange is an independent instance that holds users, groups, and resources. It has a unique domain name. If a record stops being exported, the user receives a notification that the mail account will be deactivated after eight weeks.
OX can handle up to 8,000 accounts per context, so it turned out that distributing users to multiple contexts is more efficient. We created separated contexts for teachers, students at primary, and students at secondary level. During every data comparison, there is a check if the account stays in the same context or has to be moved to another one. In education, we see quite a high fluctuation of accounts: Every year, new students attend school, change or leave school – in total, about 30% of the accounts are affected at the end of term. In order to move them to a different OX context, we implemented an automated transfer mechanism. Unfortunately, this makes the whole setup a bit more complicated and increased the administration costs.
Individual Address Books via Script
The address book in Open-Xchange doesn’t quite meet the schools’ requirements. So, we decided to generate our own address books via scripts. In the end, the users need to be able to send mails to all other teachers or the entire class without much effort.
The script creates several address books of a certain type (teachers, classes, etc.). For each school, the content of the address books is being generated with LDAP queries, exported to a CSV file. The script then logs into every OX context, using a dedicated service account. If there are folders missing, i.e. for new classes, the script generates them. It uploads the CSV files to the OX folders and uses a special configuration file to set the permissions for the user accounts.
It turned out that we cannot update every address book for the 32,000 users on a daily basis. Therefore this feature is only available for teachers and other staff members.
Open-Xchange contexts have a so-called global address list (GAL) for all their users. This list is technically necessary for the groupware features like free/busy, shares, etc. Data protection law says that the GAL must not be readable or exportable, so the Adfinis SyGroup developed a highly complex plug-in that disables this list.
Migrating Pilot Schools, Teachers, and Students
The actual migration of the old mail system to Open-Xchange was also carried out by Adfinis SyGroup. It was quite challenging and happened in three stages:
- Migration of pilot schools (one weekend in May, 2017).
- Migrations of teachers and staff members (bank holiday weekend in June, 2017).
- Migrations of all students (summer holidays, 2017).
Converting Data with Cyrus2Dovecot and IMAP
In a first step, we converted the data with Cyrus2Dovecot (while the users were still using the old system); this rsync-based command line tool transforms mails from the Cyrus format into Dovecot maildir folders. Since the tool doesn’t compare the data itself, the migration is quite fast. It’s important to run the command only once, otherwise you’ll have duplicated data. After that, we created OX user accounts in LDAP and integrated them into the OX user management. In a third step, we used imapsync to synchronize the Cyrus and OX mailboxes. The tool uses the IMAP protocol, so it’s not very fast. On the plus side: It only syncs the data that has been changed since step 1 (delta sync), so it didn’t take us too long.
The migration of the pilot schools, the teacher and staff accounts went smoothly. We encountered only minor problems, like a different naming scheme for the mail folders in different mail clients. For example, the web mailer uses the German word for “sent” (“Gesendet”) whereas Thunderbird uses the English term “Sent”. As a result, the sent messages ended up in two different folders, so the support hotline was a bit busier in the days that followed. The migration of the student accounts had to be postponed at relatively short notice, since it turned out that the migration scripts took a bit longer than intended. Especially, the snchronization of the user accounts between LDAP and OX was affected, so we had to optimize the operation.
During the whole time, we made sure to avoid downtimes and other inconveniences. The timing was crucial: Working on the old system had to continue, and we could only test the productive environment during school holidays. We made sure to involve all schools in the organization. All dates were announced well in advance and we responded to change requests as well. Thanks to Adfinis SyGroup things went as planned, and the whole migration process was finished in January, 2018.
Conclusion: More Features, less Administration Efforts
Compared to the old mail system, operating and support costs are more or less the same. At the same time, a simple mail system with address management has grown into a professional groupware solution with e-mail, address book, calendar, task manager, cloud storage, etc. The whole server architecture has been thoroughly tested, with the help of the manufacturers. Our users like the fact that the OX interface looks a bit like that of a well-known web mailer and that OX provides an up-to-date user manual.
Open-Xchange is being developed by a German company – a real advantage in our opinion, since the software follows all EU guidelines. ICT Media now provides a highly availble mail system for 32,000 users with the nice sice-effect that mails from @edubs.ch stay inside that domain, thus, we comply with current data protection guidelines. The Open-Xchange context system and the expected context changes at the end of every term were a bit tricky to handle, but our technicians managed to find a solution. They developed some scripts and a rather complex program for the extended address book. Looking back, the migration to the new system was significantly more work than expected, but it was worth the effort. After a few months it became clear that the new solution is stable and reliable, that our users appreciate it and that it has all features needed in school environment.
