This site uses different types of cookies, including analytics and functional cookies (its own and from other sites). To change your cookie settings or find out more, click here. If you continue browsing our website, you accept these cookies.
I need some suggestions or guidance regarding loading very large data sets into MySQL Server. I have been using the Alteryx Input Data tool to download all data from MySQL to a temporary file on my local machine (in yxdb format) and then after applying arithmetic operations using formulae node, I am writing them back to MySQL using Alteryx output data node but because my database is very large around 50GB so it's taking way more time to upload (more than 12 hours).
My issue is if there is any faster way to load million of records (around 50GB) into one table in MySQL. Any faster process or approach? Staging? Any help or suggestions.
Below is what i have done so far:
.1)Enabled the performance profiling in the workflow configuration > Runtime tab
.2) Tried to find Bulk Loader driver as data source for MySQL using Manage In-DB connection but I don't see such option. It is only visible when I select SQL server as the data source.
Please let me know if there is any faster way to do this?
Looking at the Alteryx documentation for MySQL at the link below, I don't see the "Bulk write support is available for standard and in-database workflows." message that is present in some other sources (like SQL Server). The support for bulk loading may not be there, but this might be a good thing to run by email@example.com just the be sure.
Unfortunately we do not currently have a MySQL bulk loader, but I wanted to ask what kind of arithmetic you are performing on the data? Reason being, if you can reproduce those operations with the In-DB tools themselves, you can completely avoid pulling the data down from the database. The In-DB tools only pull data from the database on a Stream Out or a Browse tool, otherwise all the In-DB tools do is build up a SQL statement that then sent to the database to do the work. So if the operations/functions you want to perform are supported by your database, you can use the In-DB tools with the standard MySQL connection and never have to pull the data locally.
For example: use a Connect In-DB tool to select your table, a Formula In-DB tool to do your arithmetic, and then a Write Data In-DB tool to create a new table. This workflow will never pull the data locally, but build a query to create that new table from the query.