java - Best way to grab website content externally -

-

is a search site whose search result is dynamically generated by JavaScript. The user enters a query, and the site displays without refreshing content on the page

I need to catch those search results programmatically (a Java program or a Pearl / Python From script).

So ideally, I can launch my program with 100 queries as user input, and then the program will hit that website with every query and given by the website on my screen Removes all search results. / P>

The obvious problem is that the site is in Javascript instead of ordinary HTML, so the resultant output for sending a URL request and parsing (this page is not going to work as the source code is always going Is a group of different JS files).

Looking at the above conditions, what are my options?

Javascript makes HTTP request almost like a browser, once you find out what you do , You can try to recreate them in perl / python / etc. With Firefox + Firebug you can see the request in the 'net' panel.

The user-agent string, the cookies, the fact is that sometimes the returned data means run / interpreted by Javascript etc. The language of your choice can use a good httpbrowser class ?

Just put a vision, search for IBM, took post data from Firebug, replace new-rows with & amp; 

  [http://bcode.bloomberg.com/sym/dwr/call/plaincall/searchMgr.search.dwr?callCount=1&windowName=&= and put the request URL after ; C0-sCRIPTNAME script = searchMgr & amp; C0-methodName = Search & amp; C0-id = 0 & amp; C0-E1 = string: IBM & amp; C0-E2 = string: & amp; C0-E3 = Number: 100 & amp; C0-E4 = Number: 0 & amp; C0-E5 = Boolean: False & amp; C0-param0 = Object_SearchCriteria: {Search: Reference: C0-E1,% 20filter: Reference: C0-e2,% 20limit: Reference: C0-E3 & amp;,% 20start: Reference: c0- E4,% 20allSources: Reference: C0-E5} & amp; BatchId = 4 & amp; Page =% 2Fsym% 2F & amp; HttpSessionId = & amp; ScriptSessionId = FBC68693A4E1BC08D6E0DDFBDF6D0860]

but it returns

throw 'allowScriptTagRemoting is incorrect.'; // #DWR-answer if (window.dwr) dwr.engine.remote.handleBatchException ({name: 'java.lang.SecurityException', message: 'this is non-permitted'}); Else if (window.parent.dwr) window.parent.dwr.engine.remote.handleBatchException ({name: 'java.lang.SecurityException', Message: 'This is non-permitted'});

No more data. It seems that you have a post request script, considering your restrictions and guidelines, you should just go in contact and ask if there is a public API ?


Comments

Post a Comment

Popular posts from this blog

c++ - Linux and clipboard -

-
After selecting the copies of the buffer in Linux, after the text, we can click and paste the button between the mouse. I think there is a special buffer for this, I want to use it. Programming Language: C ++ Your Library: Qt Thanks. There is just one more correct answer than Paul Dixon, which answers your needs: / P > QClipboard * clipboard = QApplication :: clipboard (); CastString Selected Text = Clipboard-> Text (QClipboard :: Selection);
Read more

What is expire header and how to achive them in ASP.NET and PHP? -

-
I have examined the performance statistics of my website today. I have received a warning (or error may occur) that is given below Add Terminal Headers Here are 15 static components without a 15-futures end date * (no end) Http://www.example.com/video/css/global.css * (no time limit) http://www.example.com/video/js/global.js * (no expiration ) Http: //www.example.com/video/images/main-bg.png what it means and how to achieve it both in PHP and ASP.Net. I am on a shared hosting server, so please tell me some ways to use the code, because I can not make any modifications at the end of the server. If I expire the header, then there is no chance that if I make changes to the CSS, then the user will not be able to remove them immediately because CSS and other files have a fixed time limit (1 month, week ) Are cached for. Is Is there any harm to using PHP? $ time = time () + 3 * 24 * 60 * 60; // 3 day headings ('End:'. GMDAT ('D, D MYH: i: s \ g \ m \ t',
Read more

sql server - How can I determine which of my SQL 2005 statistics are unused? -

-
After a few years, one of my biggest databases has deposited 73 figures out of its largest tables. With Indexes, I can run many types of reports and queries, how often / heavily specific indexes are used. What is an equivalent for data? We are running SQL 2005. To make statistics automatically managed and honest, I would like to "update statistics", "create statistics" and There is no information about any management except "drop statistics" orders. In theory, UPDATE should handle the addition and removal of statistical data. In addition, I've never heard about the amount of storage or storage taking in large amounts, so I'm not sure whether there is any reason for the alarm or not. If you are seeing a potential problem, then I would suggest running an update before, if it does not clean it, it is leaving and then it appears to be safe operation (because it is the only query optimizer Is for). I will not leave "left" in the
Read more