1. Introduction And Purpose =========================== This document is describing the technical specification of the Data Synchronisation Service (DSS). The purpose of this document is to give the informations which are needed by the developers for to implement and to integrate the service. The DSS was designed by our team in order to make the customer data to be synchronised between the on-premise database and the cloud storage in a reliable way. It is very important that the requirements which are written in this document are respected by all the peoples who are working on the project. 2. System Overview ================== The system is consisting of three main components. Each component is communicating with the others by the usage of the REST API. The “Sync Agent” is installed at the customer side, and it is reading the changes from the local database each 30 seconds. After the changes are detected, they are being packed and sended to the “Sync Gateway”. The gateway is validating the payload, and then it is forwarded to the cloud storage. If the validation is failing, the error is logged and the customer is notified by the system. 2.1 Key Features Of The Service ------------------------------- - Changes are detected automatically by the agent without any manual intervention from the user - All the data is encrypted by using TLS 1.3 during the transmission - Conflicts are resolved by the gateway in accordance to the ‘last write wins’ strategy - The failed transmissions are retried for maximum five times 3. Functional Requirements ========================== 3.1 Change Detection -------------------- The agent must to detect the inserts, updates and deletes in the monitored tables. For this purpose, a trigger-based mechanism is used. The triggers are installed by the agent during the first start, and they must not be modified by the customer, otherwise the synchronisation can be broken. It should be noticed that the detection of the changes is not happening in real time. A small delay of maximum 30 seconds is normal and it is not considered as a defect. 3.2 Data Transmission --------------------- The data is transmitted in batches of 500 records. If a batch is failing, it will be retried by the agent with an exponential backoff. After the fifth failure the batch is marked as “failed” and the administrator is informed per email. The size of the payload shall not exceed 5 MB, because bigger payloads are rejected by the gateway with the HTTP status code 413. 4. Non-Functional Requirements ============================== 4.1 Performance And Scalability ------------------------------- The gateway should be able to handle minimum 1000 requests per second. The latency, which is measured from the moment of the request till the moment of the response, must be lower than 200 ms in 95% of the cases. For the scalability, the gateway is designed in a stateless manner, so that additional instances can be easily added behind the load balancer when the load is growing. 4.2 Security Considerations --------------------------- Every request must be authenticated by using an API key. The keys are rotated each 90 days, and the old keys are being revoked immediately after the rotation. Furthermore, it is strictly forbidden to store the keys in the source code or in the log files. 5. Open Points ============== Some questions are still open and they will be clarified in the next meeting. For example, it is not yet decided if the audit logs will be stored for 6 or for 12 months. The decision will be taken by the product owner till the end of the next sprint.