Robots on the site

Grimeire · Jun 28, 2017

hey,

just looking at the homepage and noticed there is 330 robots visiting the site.

First of all, how is it known they are robots.
secondly, what would robots have to gain being on the site.
thirdly, they outnumber us 20 to 1.

Grimeire · Jun 28, 2017

forgot to add this

Ferret · Jun 28, 2017

In simple terms most are crawlers for search engines google etc.

And online now means over a set time period sat 15 mins

Spectre · Jun 28, 2017

alankellyeire said:
hey,

just looking at the homepage and noticed there is 330 robots visiting the site.

First of all, how is it known they are robots.
secondly, what would robots have to gain being on the site.
thirdly, they outnumber us 20 to 1.

Polite bots identify themselves through their user agent string. This is something passed from the browser of normal users or it can be passed from software in the case of bots and spiders etc.

If you wanted you can change your user agent string using a Firefox addon and see websites as a bot would see them.

List of User-Agents (Spiders, Robots, Browser)

Mick · Jun 28, 2017

I think we will get that for a week or so.

as all the content is being re downloaded, as all urls have changed from the vbulletin URLS.

We have REGEX running on the nginx platform that catches 80% of URLS, the most important ones like threads.

So if you search google for a digitalworldz (vbulletin) page on google you should get forwarded to the new xenforo page of that thread.

There will be around 3 million pages that an be spidered on DW, so I would assume it would take most of the search engines a few days to spider them all. and there are hundreds of engines...

Cheers
Mick

lescantle · Jun 29, 2017

Mick said:
I think we will get that for a week or so.

as all the content is being re downloaded, as all urls have changed from the vbulletin URLS.

We have REGEX running on the nginx platform that catches 80% of URLS, the most important ones like threads.

So if you search google for a digitalworldz (vbulletin) page on google you should get forwarded to the new xenforo page of that thread.

There will be around 3 million pages that an be spidered on DW, so I would assume it would take most of the search engines a few days to spider them all. and there are hundreds of engines...

Cheers
Mick

The robots seem to stay on all the forums mick,Just one think that would be good,Can you add the arrows to take you back to the to of the page without having to scroll up

Mick · Jun 29, 2017

I will look into that @lescantle

But you can press the [home] button on your keyboard for now if you don't want t scroll up.

Mick

lescantle · Jun 29, 2017

I know that mick,Its not a criticism in any way my friend

Robots on the site

Grimeire

VIP Member

Grimeire

VIP Member

Ferret

Administrator

Spectre

Administrator

Mick

Administrator

lescantle

DW Regular +

Mick

Administrator

lescantle

DW Regular +

Similar threads

We value your privacy