I have a couple of fields I keep in my SQL tables... (migrateBatch,migrateStart,migrateEnd) ... migrateBatch is a UUID field that's set with the migrateStart, from there a regular process pushes to a queue, where the data is then fetched from the DB and updated in ES/Mongo/whatever... My sproc checks for updated records against the migrate/export date, and if I ever want to re-submit something, I just clear those fields in the table.
It's not the absolute fastest way to go, but it's very solid... I can usually have 500k records migrated in well under half an hour, the actual lookups for denormalized data are a bit heavy.
It's not the absolute fastest way to go, but it's very solid... I can usually have 500k records migrated in well under half an hour, the actual lookups for denormalized data are a bit heavy.