multithreading - designing a 10 threaded process -


i have 500 odd data-files lying in directory. processing these files using command . problem picks each serially , of course utility moment process spawns 500 processes , staggers them in batches of 10 each interspersed wait . . 1 file == 1 process per design.

command1 <parameters> > log 2&>1& command2 <parameters> > log 2&>1& command3 <parameters> > log 2&>1& command4 <parameters> > log 2&>1&  /# ....till 10 wait  command11 <parameters> > log 2&>1& command1 <parameters> > log 2&>1& command1 <parameters> > log 2&>1& /# ...till 20  wait command1 <parameters> > log 2&>1&  

so dont pound system away @ same time. command series of shells , c code parses data files , checks, involve connecting oracle. i'd re-design every-time command runs don't open new db connection. lets there 100 files . want open 10 db connections instead of 100 separate connections. way want first count how many files have. divide them 10. each division 'batch' of 10 files-so instead of 100 there 150 files, process should automatically create 15 directories . each directories should move in 10 files , can run

 command1 <parameters> directorypath  > log 2&>1&  /# command can pick files in directory or single file /# ct till command 10 ... ....  command10 <parameters> directorypath  > log 2&>1&  

10 files== 1 process design.

any suggestions visa vis commands how can done. ty

with gnu parallel, can run command on files in directory in parallel, 10 @ time:

parallel --gnu -j 10 yourcommand --yourflags {} ::: * 

and run

yourcommand --yourflags file1 yourcommand --yourflags file2 

and one, ten @ time.


Comments

Popular posts from this blog

php - regexp cyrillic filename not matches -

c# - OpenXML hanging while writing elements -

sql - Select Query has unexpected multiple records (MS Access) -