Java multi thread download file and update array

Viewed 8k times. Brian Brian 6, 15 15 gold badges 52 52 silver badges 70 70 bronze badges. Do you have no experience at all with threads in Java? Have you ever used a Runnable or done anything with the Thread class? Curious as to your starting knowledge here. That something interesting method must be pretty interesting. How many iterations of you are taking. I am curious of what veritas asked too.

Are you worried about time spent iterating or time spent doing something interesting? The answer could vary quite a lot depending on the nature of "do something interesting". Can each thread, "do something interesting" for a subset of the array, without worrying about the other threads, or other parts of the array, or about its part of the array changing?

Show 3 more comments. Active Oldest Votes. Community Bot 1 1 1 silver badge. In the second approach the workers don't have access to the original index of each float-- probably necessary information in many applications. You need to call start on a Thread instance to make it actually do something. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password.

Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Who is building clouds for the independent developer? Exploding turkeys and how not to thaw your frozen bird: Top turkey questions Featured on Meta. Now live: A fully responsive profile.

Reducing the weight of our footer. The possibility that execute might run the command on the calling thread depending on the Executor implementation used, and the fact that there obviously is a way that does not exhibit this problem is a rather compelling rationale to me. Show 1 more comment. All the Above mentioned approach creates Threads but the actual Concurreny is not achieved. Sign up or log in Sign up using Google.

Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Who is building clouds for the independent developer? Exploding turkeys and how not to thaw your frozen bird: Top turkey questions Featured on Meta.

Now live: A fully responsive profile. Reducing the weight of our footer. Visit chat. Linked 2. For example a separate thread can wait for processed chunks. If chunks are processed in order 5, 4, 2, 1, 3 this thread can save chunks 1 and 2, when 2 is processed and 3 to 5 when 3 is processed. Separate chunks will be processed asynchronously, but output will be stored in correct order. That may be a difficult algorithm to implement, do you have any idea of an existing API which can ease me with this.

I have encountered a similar situation before and the way I've handled it is this: Read the file in the main thread line by line and submit the processing of the line to an executor. When the next expected line is done processing, insert into the Queue and verify if the next element is inside the binary-search-tree. In the case that the next line is in the tree, remove the node from the tree and append the content of the node to the Queue and repeat the search if the next line is already inside the tree.

Repeat this procedure until all files are done processing, the tree is empty and the Queue is empty. Hope it helps. Minduca Minduca 1, 7 7 silver badges 18 18 bronze badges. Thanks Minduca for a nice explanation, I like the idea, will update you on this soon.

Beware of OOM issues though. Looks promising, let me give it a try, will update you soon on this. To make sure that you dont encounter OutofMemoryError you should initialize Queues with appropriate size You can use a CyclicBarrier to ensure all thread complete their operation Additionally you can set an Action in CyclicBarrier where you can do your post processing tasks. Good Luck, hoping you get the best design. Sachin Thapa Sachin Thapa 3, 4 4 gold badges 21 21 silver badges 41 41 bronze badges.

Thanks Sachin, I also want to store the data in the same order in which I have read, it means the first line of the file should be the first processed record in the output file, how will I be able to do that?

Here is the design which i followed to solve my problem: Use main program as master, read the whole file in one go but dont start processing. Create one data object for each line with its sequence order. Use one priorityblockingqueue say queue in main, add these data objects into it. Share refernce of this queue in constructor of every thread. Create different processing units i. When we add data objects to this queue, we will call notifyall method.

All threads will process individually. After processing, put all results in single map and put results against with key as its sequence number. When queue is empty and all threads are idle, means processing is done. Stop the threads. Iterate over map and write results to a file.

Sandeep Sandeep 1, 10 10 silver badges 16 16 bronze badges. Sign up or log in Sign up using Google. Sign up using Facebook. Sign up using Email and Password. Post as a guest Name. Email Required, but never shown. The Overflow Blog. Podcast Who is building clouds for the independent developer?

Exploding turkeys and how not to thaw your frozen bird: Top turkey questions Featured on Meta. Now live: A fully responsive profile. Reducing the weight of our footer. Linked Related Hot Network Questions. Question feed. Stack Overflow works best with JavaScript enabled.