Where are your servers located?
All our apps are hosted using AWS serverless architecture. We have environments in the United States, Australia and Europe.
How does CrescoData map data?
CrescoData has an internal mapping language (Cresco Mapping Language – CML) which is used to map product and order data into the CrescoData master format. That master format is mapped into each channel’s format which allows activating all supported channels by mapping new integrations into the CrescoData master format.
Depending on your requirements, CrescoData can either map your product and order data to the CrescoData format or you can choose to map data to the CrescoData format yourself. Data can be uploaded into CrescoData’s APIs either by uploading in the raw system format or the CrescoData master format.
Incoming data (data sent to or retrieved by Cresco Data) is mapped via the in-house Cresco-Mapping-Language (CML) which is primarily focused on transforming data.
Mappings for outgoing data (data sent to channels) is usually maintained by Cresco Data in-house to make sure channel integrations are available generically without requiring any account specific changes. Custom mappings are possible however if required and are done using the Mako templating language. This language is used to support a range of data formas.
What data types do you integrate into?
Each Customer of CrescoData is set up in a dedicated sandboxed account to provide absolute data security. Each account is secured with role based permissions.
All data that is passed and stored is encrypted.
Data can be sent to the CrescoData platform in any a range of data formats such as:
- CSV/TSV
- XML
- JSON
This data can be sent via any of the following transfer methods.
- API
- FTP (Cresco-hosted FTP server or external FTP server supported)
- Google Drive
For any transfer method, CrescoData supports:
- regular pulls to check for updates, and
- real-time notifications via
- full API integrations
- webhooks
How do you connect to APIs?
Authentication Process
Authentication to the CrescoData API is done using a client id / secret as well as a username and password. These credentials will be issued by CrescoData and provided during the on-boarding process.
Standard API
The CrescoData API gives access to all of CrescoData’s services such as:
- Account setup and maintenance
- User setup and maintenance
- Status monitoring and alerts
- Reporting data
- Orders
- Products
All communication uses the CrescoData format which is outlined in the CrescoData API Documentation. This means, orders, products and all other data sent to or provided by the system, will be in the CrescoData format unlike the formats used in the Streams and Webhooks APIs.
APIs – Streams API
The Streams API is a set of API endpoints that supports sending data in a pre-defined custom format to the CrescoData system which will automatically map it into the CrescoData format thereby removing one step that would have to be taken by any integration – mapping data to the CrescoData format.
The Streams API uses the same authentication as the standard CrescoData API unlike the Webhooks API which is a simpler signature based authorisation.
The workflow of the Streams and Webhooks APIs are identical. The Streams API is available to offer a single authentication method for integrations that also use the CrescoData API for account setup.
APIs – Webhooks API
The Webhooks API is a set of APIs that allows sending data to CrescoData in a custom predefined format.
The authentication for the Webhooks API depends on the integrations but is usually signature based. The workflow of the Streams and Webhooks APIs are identical. The Streams API is available to offer a single authentication method for integrations that also use the CrescoData API for account setup.
How do you handle scaling?
Objectives
No data loss: Any spike in traffic beyond allocated capacity should, at worst, cause delays but never result in data loss.
Isolation: Traffic spikes should be contained within their respective accounts or connected channels, ensuring that other channels continue operating at normal speeds.
Data flow:
- Prioritize real-time synchronization.
- Support sudden traffic spikes.
- Scale up quickly to prevent delays and scale down gradually for efficiency.
Architecture
CrescoData’s platform is fully serverless, designed from the ground up with this architecture in mind. The core concept is nano-services, meaning:
- Each action runs within its own isolated container, including every API endpoint, allowing for independent scaling based on traffic demands.
- Each datastore is an isolated NoSQL table, ensuring efficient and scalable data management.
Implementation and Key Concepts
Error handling at the architecture level, not in code:
CrescoData’s platform is designed to handle errors, including retries due to traffic spikes, at the architecture level rather than within application code. By leveraging AWS services, this approach:
- Simplifies logic.
- Enhances scalability.
- Prevents data loss.
Following serverless implementation methodology:
The system is designed to operate without traditional servers, avoiding single points of failure.
For efficient data flow, this means:
- Eliminating states during the synchronization process so that each update contains all the required information for reprocessing. This prevents race conditions or errors caused by state inconsistencies.
- Converting bulk updates into record-level updates at the earliest possible stage, reducing memory load and processing time while ensuring a standardized data flow.
Global platform auto-scaling:
The platform automatically scales by monitoring traffic across multiple areas.
- AWS core components allow each service to scale independently, providing baseline auto-scalability.
- However, since individual components only monitor their own traffic, the platform-wide scaling mechanism predicts and prepares for increased demand before traffic reaches critical components.
- This proactive scaling approach prevents bottlenecks and ensures seamless performance even during traffic surges.