python - Downloading a Large Number of Files from S3 -

-

What is the fastest way to obtain a large number of files from Python S3 (relatively small 10-50kB) from Python? (In the order of 200,000-million files)

I am currently using beto to generate signed URLs, and I am using PyCURL to get files one by one.

Will some types of concurrency help? PyCurl.CurlMulti Object?

I'm open to all the tips thanks!

In the case of dragon, because it is IO bound, multiple threads will use the CPU, but it probably Will use only one core if you have multiple cores, you may want to consider the new module. However, you want to use multiple threads for each process.

If you use many threads, then this is a good candidate for class.


Comments

Post a Comment

Popular posts from this blog

c++ - Linux and clipboard -

-
After selecting the copies of the buffer in Linux, after the text, we can click and paste the button between the mouse. I think there is a special buffer for this, I want to use it. Programming Language: C ++ Your Library: Qt Thanks. There is just one more correct answer than Paul Dixon, which answers your needs: / P > QClipboard * clipboard = QApplication :: clipboard (); CastString Selected Text = Clipboard-> Text (QClipboard :: Selection);
Read more

Visual Studio 2005: How to speed up builds when a VSMDI is open? -

-
Using Visual Studio 2005, if I open the test in VSMFI and try to run the tests, then sometimes the projects are ready. However, they run very slow. CPU 99% waste and output window VS Shows things associated with compiling it. Is there any way to fix this? It does not make any sense. It is not necessary to upgrade that we are a viable alternative to those people who have to deal with the legacy code. Did you try to upgrade SP1? I do not know of any specific problem in this area, but this will be the first change I tried to do.
Read more

booting ubuntu from usb using virtualbox -

-
I am trying to install Ubuntu on a USB, the problem is that I do not want to repeat the boot once, I want to keep my windows, but I want to boot Ubuntu using the virtualbox and USB to the Ubuntu Want to use my hard drive, but how? The virtualbox will not recognize USB as a hard drive when installing an ISO In virtualbox, you have to create a virtual hard disk. Make a USB drive and tell the virtual box to install a virtual hard drive that you just created. EDIT: To answer some more questions: In the new Virtual Disk dialog, click the button next to the location field, click on the button Virtual Disk on Flash Drive To access, click the arrow next to Save in the Select File in the window next to Save In, and you can access your flash drive from there. I recommend setting the image to be allocated dynamically, because the size of your flash drive is not available disk space. I name the image, then select that image to install Ubuntu in the Verbill box. The virtualbox will never
Read more