Generate Web traffic with indy TIdHTTP?

This is the forum for miscellaneous technical/programming questions.

Moderator: 2ffat

Generate Web traffic with indy TIdHTTP?

Postby Ahmed Sayed » Tue Jun 18, 2019 9:29 am

Hi i am trying to generate web traffic through using a bot that should use indy's TIdHTTP to simulate this process I do a normal GET request but google analytics does not show anything (I mean no traffic no sessions or current active users).

But when i use TWebBrowser every thing works fine and google analytics counts 1 active user on the website.

The website is created with Wordpress.
Do i have to set something TIdhttp to achieve this or on Wordpress side?

Also, if this is achievable with TIdhttp how can i hold the session for something lie 10 seconds or so then close it as if the user closed the browser?
Ahmed Sayed
Top Poster
Top Poster
 
Posts: 26
Joined: Thu Nov 08, 2018 4:12 pm

Re: Generate Web traffic with indy TIdHTTP?

Postby rlebeau » Tue Jun 18, 2019 1:04 pm

Ahmed Sayed wrote:i am trying to generate web traffic through using a bot that should use indy's TIdHTTP to simulate this process I do a normal GET request but google analytics does not show anything (I mean no traffic no sessions or current active users).


Google Analytics uses a combination of HTTP cookies and client-side Javascript to collect data. The website owner adds script code to its pages that a web browser would then execute, and the script would gather and send data to Google for analysis.

TIdHTTP is strictly an HTTP client, it does not do everything that a full blown web browser would do, like execute client-side scripts. You will have to make your bot simulate what the scripting code does in order to send your own data directly to Google.

See How does Google Analytics work (Beginner's Guide) for more details.

Ahmed Sayed wrote:Also, if this is achievable with TIdhttp how can i hold the session for something lie 10 seconds or so then close it as if the user closed the browser?


Closing a web browser simply deletes any currently active session cookies. You can do the same by destroying the TIdHTTP object, or at least manually deleting the relevant session cookie from the TIdHTTP's TIdCookieManager object. You can't really end a session on Google's end (and a web browser does not try to do so), but Google will timeout a session after about 30 minutes of inactivity.
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1597
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA

Re: Generate Web traffic with indy TIdHTTP?

Postby Ahmed Sayed » Tue Jun 18, 2019 5:52 pm

Thanks Remy That was very informative. I already managed to create sessions with TWebBrowser like in the code below
Code: Select all
unique_ptr<TWebBrowser> WebBrowser1(new TWebBrowser(this));
WebBrowser1->AddressBar = false;
WebBrowser1->MenuBar = false;
WebBrowser1->RegisterAsDropTarget = false;
WebBrowser1->Silent = true;
WebBrowser1->StatusBar = false;
WebBrowser1->Visible = false;
WebBrowser1->HandleNeeded();
ShowWindow( WebBrowser1->Handle , SW_HIDE);

WebBrowser1->Navigate(L"http://galalaplaza.com/");

while (WebBrowser1->ReadyState != Shdocvw::READYSTATE_COMPLETE)
   {
   Application->ProcessMessages();
   }

//Sleep(5000);

WebBrowser1->Document->Release();
SendMessage(WebBrowser1->Handle ,WM_CLOSE,0,0);

But i don't like the idea that TWebBrowser must have a window handle to navigate to the url and fire the completion event. Is there another way for TWebBrowser to do the same without Assigning a HWND. Because i wanna use that in multi-threaded app like many sessions from different IPs connected through proxy list.

Google Analytics uses a combination of HTTP cookies and client-side Javascript to collect data. The website owner adds script code to its pages that a web browser would then execute, and the script would gather and send data to Google for analysis.

TIdHTTP is strictly an HTTP client, it does not do everything that a full blown web browser would do, like execute client-side scripts. You will have to make your bot simulate what the scripting code does in order to send your own data directly to Google.


How can i send the data directly to Google Analytics like you said is there an indy component that will help me execute that java script code?
Ahmed Sayed
Top Poster
Top Poster
 
Posts: 26
Joined: Thu Nov 08, 2018 4:12 pm

Re: Generate Web traffic with indy TIdHTTP?

Postby rlebeau » Wed Jun 19, 2019 11:36 am

Ahmed Sayed wrote:I already managed to create sessions with TWebBrowser like in the code below
...
But i don't like the idea that TWebBrowser must have a window handle to navigate to the url and fire the completion event. Is there another way for TWebBrowser to do the same without Assigning a HWND.


No. TWebBrowser is a visual control, so it needs an HWND to operate. What you are asking for requires what is known as a "headless browser" instead, which is a non-visual browser that has all of the capabilities of a real web browser but without any UI. Delphi/C++Builder does not have any wrappers for headless browsers, you will have to search around for any Microsoft APIs or 3rd party libraries for that.

Ahmed Sayed wrote:How can i send the data directly to Google Analytics like you said


You need to study Google's client-side scripting code (look at the HTML source of any GA-enabled webpage) and then replicate its logic in your own code, duplicating whatever additional web requests the script makes.

Ahmed Sayed wrote:is there an indy component that will help me execute that java script code?


No. Executing web scripts is outside of Indy's scope. And besides, you can't execute Google's script code as-is from outside of a real browser environment (well, not without a LOT of extra work to manually initialize a Javascript engine and replicate everything a browser environment provides to scripts to interact with a browser).
Remy Lebeau (TeamB)
Lebeau Software
User avatar
rlebeau
BCBJ Author
BCBJ Author
 
Posts: 1597
Joined: Wed Jun 01, 2005 3:21 am
Location: California, USA


Return to Technical

Who is online

Users browsing this forum: Bing [Bot] and 18 guests