Scraping data dynamically generated by JavaScript

Question

I tried using PhantomJSDriver for scraping this URL: http://www.tsetmc.com/loader.aspx?ParTree=151311&i=67126881188552864

1) First, in this URL we must click on the upper blue link named: سابقه

2) Then I want scrap this XPath: //*[@id="trade"]/div[2]/table/tbody/tr[9]/td[7]

Here is my code but this code can't find required XPath:

var driverService = PhantomJSDriverService.CreateDefaultService();
var driver = new PhantomJSDriver(driverService);
driver.Url = "http://www.tsetmc.com/loader.aspxParTree=151311&i=67126881188552864";
driver.Navigate();
var source = driver.PageSource;
driver.FindElementByXPath("//*[@id=\"tabs\"]/div/ul/li[3]/a").Click();
Thread.Sleep(5000);
try
{
 var pathElement = driver.FindElementByXPath("//[@id=\"trade\"]/div[2]/table/tbody/tr[9]/td[7]");
 MessageBox.Show(pathElement.Text);
}
catch
{
 driver.Dispose();
 this.Close();
}
driver.Dispose();

Answer 1

I would recommend to directly hit this url:

http://members.tsetmc.com/tsev2/data/InstTradeHistory.aspx?i=67126881188552864&Top=999999&A=0

And then parse the values as this contains all the values basically on all the 209 pages so it will save you also the trouble of moving from page 1 -209.

And it looks easy split by ; gives you all the rows and then split by @ gives you the values per column.

• Scorpian Z: but how can you find this URL?
• confusedMind: @ScorpianZ open firefox (open console) or fiddler hit the URL when it runs you will see the above URL getting loaded (ajax call ).