Security is of prime concern for Merce Broadside installations, for various reasons. Corrupted or malware-infected emails can seriously damage the reputation of the transmitting organisation and trigger blocking actions at recipient mail servers. Intruders can attempt to connect to and corrupt the Merce Broadside installation itself.

Hardened platform. Merce Broadside is built on the Linux OS kernel and layered components, all of which are suitably hardened for high security. All configurations of all software components are audited to close known vulnerabilities, and access mechanisms into the underlying server are configured for tight security. This is necessary for any mission-critical system continuously connected to the Internet.

Archived configuration history. All changes to the system configuration, access rights, etc., are archived, and before-images are retained in the database, to allow complete change audit. These records are stored in the database and are updated through the administration console. Whenever any record needs to be updated, a copy of the old record is inserted into an archive table. These archive records are purged periodically by a background process. Thus, a forensic audit will always be able to tell which user changed the configuration of a specific parameter, and when.

Source control for message origination. Messages submitted to Merce Broadside for onward transmission are only accepted if their originating IP address and sender email address match pre-defined ACL entries. These ACL entries can be edited by authorised users. Each bulk mail and mailing list application can have a set of sender addresses and a set of source IP addresses defined against it. Only email messages originating from these sources will be accepted by the dispatcher of Merce Broadside. This prevents an infected PC from sending out malware-loaded messages accidentally through the Merce Broadside system, or any other possible misuse with malicious intent.

Account and password hygiene policies. Merce Broadside supports enforcement of policies for account expiry, temporary accounts and forced password change. This permits the enforcement of some basic levels of account management hygiene. Merce Broadside user management is implemented on the Merce user management foundation, which allows the system to set the number of old passwords which cannot be repeated when a user changes his password. For instance if this is set to 5, then a user cannot re-use one of his last five passwords when changing his password.

SSL v3 Digital Client ID support. Users can be forced to connect to Merce Broadside's administration console only after strict two-way authentication using Digital Client IDs, sharply enhancing the authentication security. This is implemented and supported by the Web server which serves as the front-end for the Broadside administration console. Digital Client IDs require cryptographic keys to be generated and installed on the browsers of all users, and supercedes the simple username-password based authentication which is prone to password sharing, password theft, etc. Any communication using SSL v3 is also encrypted using 128-bit SSL encryption, preventing snooping of text on the wire.

L1 and L2 access privilege separation. In Merce Broadside, L1 users do not automatically and implicitly get access to L2 privileges, thus supporting clear separation of roles. This allows the L1 users to be given system management rights, but they can be denied any visibility into, or edit rights for the application definitions being operated within Merce Broadside. The L1 user can grant himself L2 privileges, but this granting action will then be archived in the historical records for post facto forensic analysis. In addition, not all L1 users can grant themselves any additional rights --- a right-granting privilege is required for this. This allows for high security and clean separation of roles.

Mailing list hold-and-check. Messages meant for broadcast to a mailing list are always put in a message repository for review and manual re-transmission, eliminating the danger of accidental broadcast of incorrect content. This becomes very necessary because of the danger of a miscreant sending out a potentially damaging email message to millions of users through a mailing list, under the full protection and performance assurance of Merce Broadside. To prevent such misuse, all messages submitted for outward transmission are queued and made available for manual verification. A human operator must manually release the message for outward transmission before it can be sent out.

Virus filters for outgoing email. All outgoing emails are scanned for viruses to ensure that no malware-infected email reaches the Internet. This is essential protection for any high-performance mail transmission system like Merce Broadside, because transmission of infected messages can damage the reputation of the Broadside installation's IP addresses and prevent effective legitimate use for a long time thereafter. Recipient servers will black-list the IP addresses and activate blocks for all mails emanating from the Merce Broadside system, if it is used to transmit infected messages even once.

Policy controls on mail content and size. Merce Broadside supports the enforcement of allowing only a permitted list of MIME types for outgoing mail attachments, and imposes size limits. This is a defensive measure to prevent inappropriate messages being sent out accidentally or deliberately through a Merce Broadside system. Most Merce Broadside installations only require to send out a small list of MIME types in their attachments, e.g. PDF files and spreadsheets. Thus, pre-defining this white-list allows the Merce Broadside system to block messages which do not fit the policies. The same benefits accrue from the size limit policy.

Message archiving. All outgoing messages and incoming bounces can optionally be archived for complete post facto audit. This feature is very effective during testing phases and when sending out highly critical emails to small sets of users. The archiving mechanism works for both outgoing messages and bounces which return to the Merce Broadside system. The administrator can choose whether the entire messages will be archived or just the headers, and the retention duration of such archived data.

Message bounce tracking. Each outgoing message from the Merce Broadside system is given a unique address where bounces may return, and the event of a bounce is tracked and logged. This allows Merce Broadside to report exactly which of the millions of outgoing messages have failed to reach their recipients, and in many cases, why. Other incoming messages, e.g. incoming requests into auto-responders, are handled separately from bounces.

Merce Broadside has been designed from the ground up as a key component of enterprise business application infrastructure. Therefore, it is expected that customers will extend Merce Broadside for tighter integration with their back-office systems and core applications. Merce Broadside expects to be integrated into powerful back-office systems of e-governance projects, government ministries, regulatory bodies, academic and professional institutions, and large enterprises of BFSI verticals. Merce Broadside supports mature facilities to make such integration simple.

All data in relational database. All meta-data, configuration information, and logs are stored in a central relational database to allow easy SQL-based access from in-house applications. This allows queries and MIS reports to be generated beyond the customised reports that are provided by the administration console of Merce Broadside. It also allows direct transfer of data from the Merce Broadside tables to internal applications to track successful sending of messages to recipients, performance statistics, and other details.

Customised pre-processors. The pre-processor component of Merce Broadside is expected to be coded as per customised business requirements of the organisation, to allow tight programmatic integration with back-office message generating systems. This allows the development of custom software which can receive input in arbitrary formats, process it in proprietary ways, and then generate messages to be submitted to the dispatcher for outbound transmission. The vision behind this design is that the pre-processor will understand the business domain knowledge of the Merce Broadside customer, while the dispatcher and downstream components will focus on high-performance message transmission.

The dispatcher as function library. The dispatcher of Merce Broadside is available as a function library, to allow custom software to generate messages and transmit them in real time. This function library has been coded to deliver high throughput, and can be used as a gateway between back-end business applications and the Merce Broadside message funnel to the Internet.

The Merce server synchronisation layer. Merce Broadside is built on Merce, whose multi-server synchronisation layer opens up immense possibilities for asynchronous message-based integration with back-office applications. By extending this synchronisation layer, it is possible to make other systems bypass the administration console of Merce Broadside, access the internal tables and configuration settings of Merce Broadside directly, and pull out status information from Merce Broadside asynchronously using secure message exchange mechanisms. It is thus possible to integrate arbitrary business applications with Merce Broadside, trigger actions within the Broadside system, and perform administrative functions entirely controlled by remote servers. It is also possible to extend Merce Broadside to pass messages using this synchronisation layer to other servers on occurrence of certain events, and trigger reactions on those remote servers.

Resilience refers to the Merce Broadside system's ability to continue functioning and delivering results in the face of partial failure of some components or external obstacles with remote mail servers. Availability refers to the measures built into the system to minimise chances of system failure.

RBL monitoring and alerting. Merce Broadside actively monitors a list of Internet-based black-lists and alerts administrators if any of its own IP addresses are detected in any black list. These black lists identify email-sending servers which are either insecure or have been detected to be sending objectionable messages like spam, malware, etc. By monitoring these black lists actively, Merce Broadside ensures that its own capacity to send out messages is never compromised. Human intervention is often required to remove a black-list entry for an IP address, but in the meantime, Merce Broadside disables the offending transmitter and reroutes traffic through other transmitters, thus minimising impact on transmitting effectiveness.

Resilience against remote server malfunction. Merce Broadside guards against slow or misbehaving remote servers by streaming messages out in multiple independent parallel queues. This is often necessary because a single misbehaving remote server can result in thousands of messages piling up in the Merce Broadside queues. By streaming data out in parallel queues, one or two queues slowing down will not affect overall throughput.

Automatic IP address exposure spreading. Merce Broadside automatically distributes its mail transmission across many public IP addresses, thereby mitigating the threat from any one IP address being accidentally black-listed. Each transmitter in a Merce Broadside system has a unique IP address, and the load distribution across transmitters results in the outgoing traffic emanating from a set of IP addresses. This reduces chances of rate throttling at the recipient end, reduces likelihood of black-listing of a specific IP address affecting the overall mail flow, and increases overall reliability. A Merce Broadside installation can operate with more than 1,000 transmitter processes with their respective IP addresses.

Resilience against DoS attacks. Merce Broadside uses a variety of techniques to block DoS attacks from the Internet, including rate throttling, spam filtering, source reputation checks, etc. These DoS attacks become likely because the Merce Broadside system is the designated mail exchange server for the domains from which it sends out email, in order to receive and process bounce messages. The sophisticated spam filters implemented within Merce Broadside borrow technology from our renowned Merce MailGate perimeter message security product, and this technology blocks more than 80% spams without the need for content inspection. This conserves bandwidth in the face of a spam assault and minimises chances of a processing power overload.

Real-time health monitoring. Merce Broadside comes with a built-in instance of Merce Insight which monitors generic as well as Broadside-specific health parameters in real time. Broadside-specific health parameters include the health of the transmitters, the depths of the outbound mail queues, and the throughput of each component of the system. The essence of a high performance system like Merce Broadside is the elimination of bottlenecks and maintenance of a low-latency email flow pipe. Real-time health monitoring ensures this.

Rate throttling for incoming messages. Incoming messages to the Merce Broadside auto-responder applications are rate-throttled to ensure protection against DoS attacks. This is one of the measures used to protect against any rogue counterparty which attempts to overload an auto-responder application by constant transmission of requests to it. The processing of an auto-responder request has overheads which, if incurred in a malicious context, can degrade system performance. The rate throttling layer chokes out the requests before they can add to the Merce Broadside processing load.

DKIM (DomainKey) support. Merce Broadside implements DKIM standards (RFC 4871, 5016) for outgoing messages to enhance credibility and minimise the danger of forged emails. This mechanism signs outgoing messages which leave Merce Broadside, thus helping to assert the authenticity of the contents. Many applications of Merce Broadside involve sending content (bank account statements, etc.) whose authenticity is critical. DKIM helps to add a level of credibility in such situations, and also assists in reducing blockage by spam filters.

High throughput of outgoing emails is perhaps the most important performance challenge of Merce Broadside. Scalability to deliver ever-higher performance when provided with more hardware and network resources is perhaps the second most important challenge. Both these are addressed successfully.

Automatic load distribution. Merce Broadside automatically monitors all transmitter processes and automatically distributes load evenly across all of them, delivering very high scalability. This allows for large installations on an array of servers and Merce Broadside can send out messages in parallel streams.

Multi-process architecture. Merce Broadside is designed as a set of cooperating processes, allowing high fault isolation and delivering high scalability. The pre-processors, dispatchers, and transmitters are all separate processes which handle their specific roles without shared memory or other tight coupling, thus contributing to a highly fault-resilient architecture. Most communication happens over network connections. Health probes use UDP request-responses for a low-latency, loosely coupled integration. This also contributes to very high scalability. More and more hardware can be thrown at a Merce Broadside system to get ever-higher throughput.

Multi-queue message transmission. Merce Broadside transmits messages in multiple parallel queues, making the overall throughput insensitive to occasional blocks or chokes of individual queues. This becomes specially important when remote servers are malfunctioning, making connections hang or time out repeatedly. Such occurrences affect one queue but leave the others unaffected. Blocks, occasional black-listing, or other impediments to mail flow are also shared due to this multi-queue design.

Full-breadth incoming message handling. All Merce Broadside transmitters are configured to handle all categories of incoming messages, thus allowing load distribution, rapid response, and scalability. All incoming messages are handled by transmitter processes in Merce Broadside. This makes the system more capable to handle incoming messages, resist DoS attacks, and deliver faster responses even in the presence of partial unavailability of transmitter processes.

Streaming of log messages. Merce Broadside processes stream log messages in multiple parallel streams, eliminating any bottlenecks at the central database for log recording. Since Merce Broadside has a detailed central log database, all messages from all processes must eventually be logged in the central log database. Many of these processes execute on physically separate servers. To allow high speed logging of messages from all these processes, the logging sub-system first collects messages in separate streams directly to disk using low-latency file-based streams. It then aggregates these log messages into the central database by the use of a separate log aggregator process. Direct updates to a relational database in real time would have yielded poorer performance.

A system like Merce Broadside must deliver ease and security in administration and control features. Some control features enhance security and prevent accidents; others allow more reliable functioning in a multi-team or shared environment.

Delegated rights management. Merce Broadside allows different levels of users to be delegated different degrees of freedom, thus allowing controlled delegation of rights. Each Merce Broadside installation can operate multiple applications in parallel. Applications can be any of three types: bulk mail, mailing list or auto-responder. Each application can have its own set of administrative users, and each user can be assigned fine-grained rights for administrative tasks. This allows effective department-wise delegation of applications and secure sharing of a single Merce Broadside system as an organisational facility.

Two levels of users. There are two levels of users created in any Merce Broadside installation: Level 1 or global users and Level 2 or per-app users. Level 1 (L1) users can create or delete user accounts, monitor overall system health, start and stop services which will affect all applications, etc. Level 2 (L2) users are associated with specific applications, and can perform all administrative tasks associated with their application(s). This allows a single Merce Broadside system to be used as an organisation-wide resource, where L2 users are from the user departments and L1 users manage the infrastructure.

Fine grained authorization privileges. Merce assigns rights to users in fine-grained increments, instead of the simplistic all-or-nothing model of more primitive systems. The tasks allowed to L1 users have been grouped under distinct privileges. Not all L1 users can perform all L1 tasks --- his rights depend on the authorization privileges given to him. Similarly, the tasks allowed to L2 users depend on their fine-grained authorization privileges. For instance, some users may be allowed to modify the parameters of an application definition, but may not have the rights to start and stop outgoing queues --- a potentially disruptive action. Thus, care has been taken to ensure that there is no equivalent danger to the God-like powers of root of Unix or Administrator of MS Windows.

Flow controls. Both incoming mail acceptance and outgoing mail transmission of a Merce Broadside system can be controlled by an authorised user in real time. She can stop incoming message acceptance into the Merce Broadside system from sources which submit messages. Similarly, she can start and stop transmission of outgoing messages which have been accepted by the Merce Broadside system. Outgoing flow controls can be programmed to start and stop automatically as per a 24x7 time-of-day matrix, so that organisations can take advantage of unused bandwidth at night to execute bulk transmissions. This allows the authorised user to program Merce Broadside to "transmit any queued outbound messages of the banking-Singapore application between 2300 hrs and 0600 hrs every night, Monday to Friday, and all 24 hours on weekends."

Single administration console. All applications are controlled from a single administration console. Any number of administrators can login at the same time on this administrative console in parallel using their browsers. Each administrator only gets access to the information that he is permitted to see.

Merce Broadside has powerful control and logging features. All the data captured through those means must be made visible through powerful reports. It is important to provide a set of pre-designed reports and then leave powerful access mechanisms open for extensions and customised report generation. Merce Broadside takes care of both these aspects.

Standard reports. Merce Broadside supports a set of standard reports which are universally applicable to all installations. These are very useful for getting details for one application, one round of mail transmission (also referred to as one campaign), or overall system activity during a time period. The result will be displayed as a table and can be exported into a spreadsheet.

Customised reports. Merce Broadside can generate highly customisable reports about each message which leaves the system. When the pre-processor submits a message to the dispatcher for onward processing, it also submits a customised set of attributes with the message, which are then logged into the Broadside logs together with the core attributes. Later, reports can be generated from the per-message logs by looking up these customised attributes, without any programming being needed. It is possible to generate reports like "Show me all recipients whose emails got stuck in the outgoing queues and never reached them, for the campaign ID 26AS-PYQ1 of the application bulkmail-retailIndia betweeen 30 Apr 2009 and 2 May 2009."

Direct database access. Merce Broadside permits the development of customised reporting software with direct SQL access to its internal database. In keeping with the overall Merce philosophy of encouraging enterprise integration and providing a foundation for customised extensions, Merce Broadside allows reporting engines and customised applications direct access to the raw database tables where it stores its per-message logs. The data can be queried using industry-standard SQL queries and accessed using the ODBC or JDBC protocols. This allows any degree of customised reporting to be added.

Bounce processing. All incoming messages that reach the Merce Broadside system from the Internet, including bounce messages, are processed and logged. These messages are filtered through spam and virus filters first, to resist DoS attacks, and are put through a content-parsing filter to analyse whether they are bounce messages. If yes, they are correlated against the specific outgoing message which resulted in the bounce, and appropriate reasons and error types are extracted from the bounce and logged in the message log. This information is available for reports later.