Hevo ingests your data from sources, applies transformations on it and brings it to Schema Mapper. In Schema Mapper you can define how your data will be stored in your destination warehouse.
You can use Hevo's Schema Mapper to do the following:
- Map an incoming event type to a destination table. All the data for that event type will be stored in the mapped destination table.
- Omit unwanted fields from storing in the destination table. This is especially useful to hide sensitive information like your user's personal details and financial information. You can also use it to omit large text fields like descriptions, summaries, and comments.
- Flatten your nested objects into a de-normalized destination table. Mapper automatically suggests a denormalized flat structure for your incoming nested objects.
Apart from giving you the above functionalities, Schema Mapper can also do the following for you:
- Create tables with compatible and optimal data types in your destination warehouse.
- Park all incompatible incoming events to Replay Queue for investigation.
- Notify you of any changes in your incoming schema and on detection of new event types.
How to use Schema Mapper
1. Select a Pipeline
Select a Pipeline from the Pipelines section in the left navigation menu and go to the Schema Mapper subsection on the pipeline.
2. Select event
EVENTS section on the left will show all the available event types from the source of the pipeline.
To the right of the event name, you can see the mapping status of the event type. This status indicates whether the source event type has been mapped to the destination table or not. To know more about the statuses, check Mapping Statuses.
Select the event type you want to map to a destination table.
3. Select the Destination table
To Map the Source table to the Destination, either select a pre-existing table from Destination table drop-down or you can create a new one by clicking on ADD NEW TABLE.
On clicking ADD NEW TABLE a form will appear asking details to create a new table such as Table Name, Columns, Data Type, Primary keys etc. All the fields from the source are auto-detected and a suggestion is given to you.
- If you want to customize the data type of column you have an option to change it from the Data Type drop-down.
- If you want to change the name of the column altogether just click on the Delete icon towards the end of the column and create a new one by entering the name of the new column in Column Name text box and selecting the data type from Data Type drop-down and finally clicking on Add Field.
Finally, click on Create Table to create the table and it will start appearing in Destination Table drop-down. Or if you want to cancel the creation of the new table click on the Cancel button.
4. Map Fields
To Map a Source Field to a Destination Field you can choose a pre-existing column in your destination table or enter a new name. You can skip mapping a field by unchecking the checkbox in front of the field.
Once you are done with mapping the fields click on Apply, any new fields which are not present in the destination table will be created. Your data will start replicating in the destination as per the mappings defined by you.