Hevo can replicate collections from your Mongo database. We first do an initial data dump and then Change Data Capture (CDC) post that. This ensures that we don't put unnecessary load on your database server while making sure all data is replicated.
1. Create a new Pipeline
Click on PIPELINES option in the left navigation bar and click on Create New Pipeline.
2. Select Source Type
Select MongoDB from the list on Select Source Type Screen.
3. Provide Connection Settings
Provide your Mongo database connection details on MongoDB Connection Settings page. You will have following options in connection details block:
- Source Name - A unique name for this source
- Mongo DB Host - Mongo DB host's IP address or DNS
- Mongo DB Port - The port on which your Mongo server is listening for connections (default is 27017)
- Mongo DB User - The read-only user that can read the tables in your database.
- Mongo DB Password - Password for the read-only user
- Database Name - The database that you wish to replicate
- Auth DB Name - The authentication database if applicable
- If you want to connect to Hevo using an SSH server, check How to Connect through SSH. Else, you will have to whitelist Hevo's IP addresses which will be highlighted on the screen, For eg. in this case you will have to whitelist following IP addresses:
Alternatively, you copy details from an existing source of MongoDB type. Please note that it will create an independent copy of the selected source.
Click TEST CONNECTION to test the credentials and click CONTINUE once test succeeds.
4. Select Ingestion Mode
On this page, you will have 2 options for Ingestion mode. This will define how you want Hevo to read your data from MongoDB source.
- Collections: In this mode, your collections will be polled individually at a fixed frequency. You can use this mode when you are looking to fetch data from multiple collections in your database and you would like to have control over ingestion for every collection individually.
- OpLog: Data will be polled using MongoDB's OpLog in this mode. This mode is useful when you are looking to replicate complete database as it is. This mode is very efficient in replicating but leaves you with less control and manageability over data ingestion.
Select the Ingestion Mode and hit CONTINUE.
4. Provide Job Settings
You will land on this screen if you selected Collections mode on the last screen.
Here, you will be presented with the list of collections in your Mongo database. You can deselect the collection that you don't want to replicate. For every collection, you will have to specify ObjectId field and Date field for incrementing timestamp. Hevo will detect delta changes in your mongo documents using these fields.
After entering the details, click CONTINUE.
5. Select the Destination
Select the Destination where you want to replicate MongoDB Data or Click on NEW DESTINATION to create a new Destination. Check out How to add Destination tutorial for the detailed walkthrough on steps needed for adding new Destination.
6. Pipeline Created
Your Pipeline will be created when you enter this page and you will have an option to see Sample Data and Map Schema.
While Hevo tries to load your schemas, you can select CONTINUE IN BACKGROUNDif it is taking too much time. Click on CREATE SCHEMA MAPPING to map Source and Destination Schemas, check out Introduction to Schema Mapper to learn about Schema Mapper or you can select DO IT LATER to directly head to Pipeline page. You can map schemas later on Schema Mapper page in your pipeline.
Please note that your data will not start replicating in Destination tables until you map source and Destination schemas.