Real-Time Cloud Email Protection For Businesses

Email Protection Company

Project Overview

As a provider of email security services, the client worked with CodeClouds to improve their existing email protection platform by making their onboarding more seamless and providing critical analytics to better safeguard their customers against the most sophisticated modern threats to privacy and data integrity.

Company Type

Email Security

Industry

Services

Business Analysis
Solution Architecture
Software Development
Quality Engineering

Technologies

Python
Laravel
MongoDB
MySQL
Vue.js

Requirements

The client wanted to have a system to seamlessly onboard new clients using an automated process. There were also no analytics for their security service and it was crucial for their business to have these systems in place to attract new customers. Users needed to be able to see metrics and security reports within the dashboard to help them identify potential risks in real-time.

Challenges

Email log files are very large and have to be rotated daily. Additionally these logs are continuously updated as and when emails come and go. So, the parser script has to be performing accurately to not miss any logs and run very fast at the same time.

The main challenge for this project was to read and parse email logs generated by email filtering software. Parsed logs needed to be converted into JSON data so they could then be stored in a NoSQL database. A portal would then ready that JSON data and prepare the analytical reports for the user.

A system was needed to make configuration changes from the platform which will be reflected in the email filtering software as per the client’s need.

Solutions

The backend is built with Laravel and frontend with Vue.js. This way the portal stays decoupled and only has to read the data from the MongoDB database to generate reports. The entire load of reading, parsing, and storing the logs can be handled by the Python scripts so the portal can run smoothly without any performance issues.

Python was chosen as the best language to build the parser script as it’s fast, can handle large amounts of data, and can utilize multi-threading for parallel processing tasks. We also used a hybrid database approach, MongoDB, for storing logs and MySQL for other transactional records of the portal for the client onboarding process.

We decided that the portal will store the configuration changes in MySQL and use a webhook to inform a Python script that there are new configuration changes. Then the Python script will fetch those changes using an API provided by the portal and finally apply them to the respective config files.

The entire system was architected in such a way the portal, log reading, parsing, storing, and applying configuration changes were fully decoupled from one another and do not have runtime dependency.

Results

As a result, we were able to build stable and well performing email protection system with the portal and Python scripts working seamlessly. The client could now manage email filtering servers and customers from the portal, and can assign a server to a client dynamically. Once the server is assigned to the client, it will then inform them on how to make DNS changes to allow email relaying.

Python scripts parse all the logs a server is generating every day, because each server now runs an instance of the scripts. Each script is capable of running multi-threaded processes to complete the reading, parsing, and storing of incremental logs.

The portal could now show all the possible email filtering analytics, such as the number of incoming emails vs the number of outgoing emails vs the number of emails filtered, as well as spam and virus count, quarantine reports, top email receiver and more.