database - INSERT vs UPDATE: MySQL, 7 million rows -

for opting is not programming question, have 1) code, 2) amount of algorithms of comparing data each row in option #1. i not asking how queries or how database should setup.

i have run bit of pickle. have database follows google gtfs specs , writing automatic update program service database.

the database gets overhaul once every 3 months. table least amount of rows consists of between 1-10 rows , largest table contains 7 million rows. rest have between 10 000 , 80 000.

the files program download .txt files translate in table. in other words: stops.txt = database.stops. database innodb type.

i have come 2 solutions.

• 1) every row id in several .txt documents compared in database, if nothing has changed nothing, if has changed, update.

• 2) insert downloaded files own tables (basically mirroring live tables) , switch table names. example: database.stop_new , database.stop switch names.

another twist: overhaul done @ date defined in 1 of .txt files, means done saturday on peak holiday prehaps, meaning users sending queries @ times.

question: approach should go endure nothing breaks , user experience upheld. (i leaning towards option 2...)

according mysql documentation...

mysql uses table-level locking myisam, memory , merge tables, page-level locking bdb tables, , row-level locking innodb tables.

you able row-level lock table, instead of rendering entire contents unusable...

table updates given higher priority table retrievals.

updates take priority on selects, , based on key values, think better option.

> advantages of row-level locking:

fewer lock conflicts when different sessions access different rows

fewer changes rollbacks

possible lock single row long time

> disadvantages of row-level locking:

requires more memory page-level or table-level locks

slower page-level or table-level locks when used on large part of table because must acquire many more locks

slower other locks if group operations on large part of data or if must scan entire table frequently

however, in general table-locks superior row-level locks according mysql documentation...

another option...

instead of using row-level locks, can employ application-level locks, such provided get_lock() , release_lock() in mysql. these advisory locks, work applications cooperate each other. see section 12.14, “miscellaneous functions”.