#1 Roulette Forum & Message Board | www.RouletteForum.cc

IT & Software => Coding for Roulette => Topic started by: Bayes on Jul 19, 11:06 AM 2010

Title: Screen scraping
Post by: Bayes on Jul 19, 11:06 AM 2010
I was thinking about how you could make use of the spins at Wiesbaden:

link:://:.spielbank-wiesbaden.de/DE/621/Permanenzen2.php (link:://:.spielbank-wiesbaden.de/DE/621/Permanenzen2.php)

An the moment this is just an idea, I don't know how it would be implemented (maybe javascript + CGI). It would basically be an interface to the Wiesbaden spins, so you would place bets etc just like at an OC, but because the spins are available in real-time (and verifiable), it would ensure that anyone using the software could not possibly be cheating. Obviously there would need to be some time function implemented to make sure you didn't place a bet after the result is known. Possible uses: challenges, competitions, and demonstrations by anyone thinking of selling a system.

Anyway, the major hurdle is in capturing the spins. Does anyone know how to do this? there seems to be some kind of blocking software in effect, which means you can't download the gif image of the spins (try right-clicking over the marquees).

Alternatively, and this might be easier, it might be possible to embed that part of the site (the marquees) into a web page. All you would need to do then is add the code and controls for placing bets.
Title: Re: Screen scraping
Post by: albalaha on Aug 09, 02:21 PM 2010
The idea is great I must say but if it can be worked out.
Title: Re: Screen scraping
Post by: Anima-t3d on Sep 07, 06:10 AM 2010
Quote from: Bayes on Jul 19, 11:06 AM 2010
Anyway, the major hurdle is in capturing the spins. Does anyone know how to do this? there seems to be some kind of blocking software in effect, which means you can't download the gif image of the spins (try right-clicking over the marquees).

Alternatively, and this might be easier, it might be possible to embed that part of the site (the marquees) into a web page. All you would need to do then is add the code and controls for placing bets.
Where is there a gif image? The numbers are plain text, animated with javascript. They are retrieved by using AJAX. All keyboard shortcuts and right mouse click are disabled, but you can bypass the code, since it's javascript based. I don't know but perhaps it's even possible to send their server an ajax request from your site, rather than trying to "read" their spins.

But good idea! :thumbsup:
Title: Re: Screen scraping
Post by: Bayes on Sep 07, 07:14 AM 2010
Ok, that's good to hear. Unfortunately my knowledge of AJAX is NIL, but you seem to have a lot of expertise in this area, would this be difficult to implement?

What do you think about the alternative suggestion - embedding the marquees into a page? I think this would be easier but I'm no web programming guru.
Title: Re: Screen scraping
Post by: Illuzionn on Sep 07, 01:33 PM 2010
That's a great idea.   ;)
I'm a professional JEE Web engineer, maybe I can take a look at what's possible to do.
An AJAX web interface should retrieve easily the numbers in real-time, the problem would be to be sure the "spin" will be executed at the exact time the last number shows up in the permanence, if not it will be easy to cheat.
Currently at work I'm using GWT to build AJAX web applications in Java, I can try to make a little demo if you want.  If you're ok, I'll try when I'll have some free time :)
Title: Re: Screen scraping
Post by: Bayes on Sep 07, 01:58 PM 2010
Hi Mickael,

That would be great if you could make a demo, and you're right that the timing is very important. I haven't checked this but I assume each marquee isn't updated at perfectly regular intervals, because the wheels are operated by real dealers (except that there are a couple of auto roulette wheels). This could make it difficult to get the timing right, but as long as we can get a maximum time until 'no more bets' then it should be ok.
Title: Re: Screen scraping
Post by: Illuzionn on Sep 07, 02:05 PM 2010
Ok, I'll take a look at the source code of the webpage, to check the AJAX scripting part.  I hope I'll be able to make something of it.  Unfortunately the web page is translated in german, and I don't know a word of it, so I hope it won't be too difficult to read and understand the data.  :D
Title: Re: Screen scraping
Post by: Illuzionn on Sep 07, 03:24 PM 2010
I just thought about something that will add some kind of difficulties using AJAX. 
Indeed, it's impossible for an AJAX script to call a server different from the one where the script were downloaded.   It's a policy defined inside the web browser and cannot be changed.   Of course this can be solved, by using for example a proxy servlet on the server layer that will redirect your HTTP call to the Wiesbaden spin website. 
This is not very difficult to code, but that implies we'll need a server part along with our HTTP/AJAX client.   I could easily use Google App Engine for this, as I already worked with that  ;)

PS to forum moderators : we cannot edit our posts ? I had to add a new post instead of editing my last one  :o
Title: Re: Screen scraping
Post by: Bayes on Sep 07, 04:24 PM 2010
This is going over my head, I'll leave it in your capable hands.  :)  :thumbsup:

If you need translation from German, I know someone who might help.
Title: Re: Screen scraping
Post by: Twisteruk on Sep 07, 04:35 PM 2010
Quote from: Illuzionn on Sep 07, 03:24 PM 2010
I just thought about something that will add some kind of difficulties using AJAX.  
Indeed, it's impossible for an AJAX script to call a server different from the one where the script were downloaded.   It's a policy defined inside the web browser and cannot be changed.   Of course this can be solved, by using for example a proxy servlet on the server layer that will redirect your HTTP call to the Wiesbaden spin website.  
This is not very difficult to code, but that implies we'll need a server part along with our HTTP/AJAX client.   I could easily use Google App Engine for this, as I already worked with that  ;)

PS to forum moderators : we cannot edit our posts ? I had to add a new post instead of editing my last one  :o


Yes, that's correct. However you do have a 1hr period after you post in which you can modify your post for errors etc

Hope that helps  :thumbsup:
Title: Re: Screen scraping
Post by: Anima-t3d on Sep 07, 05:50 PM 2010
I tried getting the data by using YQL (Yahoo Query Language) and Yahoo Pipes. But it's my first look at it and there are a few issues:
-the given webpage uses javascript to replace the empty cells, but keeps the last added data on the same spot.
-the page given when you click on a table shows all spins but it's hard to get the last added number, since it "adds" rather than replacing.

I had issues with getting my xpath expression to work...

Perhaps someone else can get this to work and share what I was missing? ???
Good luck! :thumbsup:

As attachment there is the modified sourcepage.

Sources:
link:://paul.donnelly.org/2009/07/08/yql-and-jsonp-x/ (link:://paul.donnelly.org/2009/07/08/yql-and-jsonp-x/)
link:://:.daybarr.com/blog/yahoo-pipes-tutorial-an-example-using-the-fetch-page-module-to-make-a-web-scraper (link:://:.daybarr.com/blog/yahoo-pipes-tutorial-an-example-using-the-fetch-page-module-to-make-a-web-scraper)
link:://:.wait-till-i.com/2009/08/25/tutorial-scraping-and-turning-a-web-site-into-a-widget-with-yql/ (link:://:.wait-till-i.com/2009/08/25/tutorial-scraping-and-turning-a-web-site-into-a-widget-with-yql/)
link:://net.tutsplus.com/tutorials/javascript-ajax/quick-tip-cross-domain-ajax-request-with-yql-and-jquery/ (link:://net.tutsplus.com/tutorials/javascript-ajax/quick-tip-cross-domain-ajax-request-with-yql-and-jquery/)
link:://projects.ischool.washington.edu/tabrooks/343INFOAutumn09/JSONP/jsonpJqueryYQL.htm (link:://projects.ischool.washington.edu/tabrooks/343INFOAutumn09/JSONP/jsonpJqueryYQL.htm)
link:://xpath.alephzarro.com/content/cheatsheet.html (link:://xpath.alephzarro.com/content/cheatsheet.html)
link:://:.w3.org/TR/xpath/ (link:://:.w3.org/TR/xpath/)
Title: Re: Screen scraping
Post by: A3on on Sep 08, 09:48 PM 2010
@Anima-t3d
Nice to see a so helpful member, who gives his bets to help  :)
Just wanted to congrat you for that  :thumbsup:
Title: Re: Screen scraping
Post by: Anima-t3d on Sep 09, 11:25 AM 2010
Quote from: A3on on Sep 08, 09:48 PM 2010
@Anima-t3d
Nice to see a so helpful member, who gives his bets to help  :)
Just wanted to congrat you for that  :thumbsup:
Thanks. I just hope together we can pull it off. :thumbsup:
Title: Re: Screen scraping
Post by: superman on Sep 09, 11:53 AM 2010
Ajax will only run in it's own server environment as far as I know, unless you have user privilages I doubt you will have much joy, if they publish the data to the browser why don't you ask them to publish it as xml, with php and others you can easily grab that data.

Another method would be any tool that can handle browser windows, get x,y then read a specific area on the screen OR convert the area into an image then run a pixel checksum against a database of numbers. In the past I have grabbed link: page source code and read certain elements, these days most web developers use css and such to define areas, the area you are after may be easy to spot, just a thought.

Whatever you choose will take some time to develope and in my opinion, random numbers are random numbers, I know they are live results but still wouldn't spend that much time to grab them, just my opinion.
Title: Re: Screen scraping
Post by: Anima-t3d on Sep 09, 02:28 PM 2010
Quote from: superman on Sep 09, 11:53 AM 2010
Ajax will only run in it's own server environment as far as I know, unless you have user privilages I doubt you will have much joy, if they publish the data to the browser why don't you ask them to publish it as xml, with php and others you can easily grab that data.

-There are work arounds for ajax cross domain.
-We could do here with just reading the "output" to the end user... it's not really needed to have the ajax work on a separate page.. it's the data we are after.
-We can't ask for an xml feed, because the data on the page we are after, is protected by javascript to disable view source. So it's unlikely they want to go to the extend of providing a feed...

Thanks for your input though. :thumbsup:
Title: Re: Screen scraping
Post by: Bayes on Sep 10, 02:20 AM 2010
Quote from: superman on Sep 09, 11:53 AM 2010
Whatever you choose will take some time to develope and in my opinion, random numbers are random numbers, I know they are live results but still wouldn't spend that much time to grab them, just my opinion.

Agreed, but this idea is more about keeping things out in the open as much as possible. e.g; if someone wanted to sell a system and was challenged to provide some evidence that the system worked as claimed, where would the spins come from? If they're not RNG then they would have to come either from someone's personal collection, or be in the public domain (in which case there is a possibility that the testee could have them). There would always be some doubt about whether the spins were 'corrupted' in some way, or that some kind of reverse engineering/back-fitting etc was going on,  but this way everything is transparent and the spins can be verified after the event because they're archived. 

It seems as though it's not so easy to grab the spins, but what about the alternative?

link:://:.dyn-web.com/tutorials/iframes/ (link:://:.dyn-web.com/tutorials/iframes/)

link:://:.howcast.com/videos/291721-How-To-Embed-Another-Website-Into-a-Webpage-Using-an-iFrame (link:://:.howcast.com/videos/291721-How-To-Embed-Another-Website-Into-a-Webpage-Using-an-iFrame)

Title: Re: Screen scraping
Post by: Anima-t3d on Sep 10, 05:27 AM 2010
Quote from: Bayes on Sep 10, 02:20 AM 2010
It seems as though it's not so easy to grab the spins, but what about the alternative?

link:://:.dyn-web.com/tutorials/iframes/ (link:://:.dyn-web.com/tutorials/iframes/)

link:://:.howcast.com/videos/291721-How-To-Embed-Another-Website-Into-a-Webpage-Using-an-iFrame (link:://:.howcast.com/videos/291721-How-To-Embed-Another-Website-Into-a-Webpage-Using-an-iFrame)


iframes are just showing the page on your page... which is the same as visiting the page. There is no "grabbing" it kind of shows a window in a window. The only benefit is that you don't need to go to the wiesbaden site.
So IMHO I'd say it's not what you are after. :-\
Title: Re: Screen scraping
Post by: superman on Sep 10, 07:55 AM 2010
Agreed about IFrames, same page, different location, no better or worse.

OK looked at your page in question, right click is disabled but you can view source from your menu bar, it doesn't help though as the numbers don't show up.

So I clicked on the words, TB1 K1 raucher and a new window pops open and viola the actuals are within the source code, heres a snippet of the pages source with 24 and 10

Quote<TD class="y" style="{background-color:black;border-width:0px;border-left:1px;border-style:solid;border-color:black}" id=T8Z1S> 24</TD>
<TD class="g" style="{background-color:black;}"id=T8Z1G> &nbsp;</TD>
<TD class="r" style="{background-color:black;border-width:0px;border-right:1px;border-style:solid;border-color:white}" id=T8Z1R> &nbsp;</TD>
<TD class="y" style="{background-color:black;border-width:0px;border-left:1px;border-style:solid;border-color:black}" id=T8Z26S> 10</TD>


Hope that offers some insight into your task
Title: Re: Screen scraping
Post by: Bayes on Sep 10, 11:05 AM 2010
hmm...  when I click on the same I just get a graphic popping up. What browser are you using?

Title: Re: Screen scraping
Post by: superman on Sep 10, 02:46 PM 2010
Hi bayes, IE8 browser, see atached screenshot, the red ring is where I clicked, clicking on those words above any column/table does the same thing, opens a new sized browser with many tables in it.

[attachimg=#]
Title: Re: Screen scraping
Post by: Anima-t3d on Sep 10, 05:59 PM 2010
Quote from: superman on Sep 10, 02:46 PM 2010
Hi bayes, IE8 browser, see atached screenshot, the red ring is where I clicked, clicking on those words above any column/table does the same thing, opens a new sized browser with many tables in it.

[attachimg=#]

I use firefox and have it too. Perhaps are you clicking on the background image Bayes?
Also I already referred to this window earlier on.. this issue with grabbing the latest number is that it's being added to the list rather than replaced. :-\
Title: Re: Screen scraping
Post by: Bayes on Sep 11, 04:10 AM 2010
Thanks superman, I get it now.  :thumbsup:

I don't think there's much point in doing this if the spins can't be grabbed. You could have the marquee embedded in a page but then you'd have to get the user to enter the spins manually which would then have to be checked against the spins in the archive, which is a bit long-winded.

Ah well, it was just an idea.
Title: Re: Screen scraping
Post by: Anima-t3d on Sep 11, 06:33 AM 2010
Quote from: Bayes on Sep 11, 04:10 AM 2010
Thanks superman, I get it now.  :thumbsup:

I don't think there's much point in doing this if the spins can't be grabbed. You could have the marquee embedded in a page but then you'd have to get the user to enter the spins manually which would then have to be checked against the spins in the archive, which is a bit long-winded.

Ah well, it was just an idea.
It can be grabbed. I just didn't get my xpath expression to work because I'm not really familiar with it (and their site code is a mess  :-\)

But like I mentioned in my post
Quote from: Anima-t3d on Sep 07, 05:50 PM 2010
I tried getting the data by using YQL (Yahoo Query Language) and Yahoo Pipes. But it's my first look at it and there are a few issues:
-the given webpage uses javascript to replace the empty cells, but keeps the last added data on the same spot.
-the page given when you click on a table shows all spins but it's hard to get the last added number, since it "adds" rather than replacing.

I had issues with getting my xpath expression to work...

Perhaps someone else can get this to work and share what I was missing? ???
Good luck! :thumbsup:

As attachment there is the modified sourcepage.

Sources:
link:://paul.donnelly.org/2009/07/08/yql-and-jsonp-x/ (link:://paul.donnelly.org/2009/07/08/yql-and-jsonp-x/)
link:://:.daybarr.com/blog/yahoo-pipes-tutorial-an-example-using-the-fetch-page-module-to-make-a-web-scraper (link:://:.daybarr.com/blog/yahoo-pipes-tutorial-an-example-using-the-fetch-page-module-to-make-a-web-scraper)
link:://:.wait-till-i.com/2009/08/25/tutorial-scraping-and-turning-a-web-site-into-a-widget-with-yql/ (link:://:.wait-till-i.com/2009/08/25/tutorial-scraping-and-turning-a-web-site-into-a-widget-with-yql/)
link:://net.tutsplus.com/tutorials/javascript-ajax/quick-tip-cross-domain-ajax-request-with-yql-and-jquery/ (link:://net.tutsplus.com/tutorials/javascript-ajax/quick-tip-cross-domain-ajax-request-with-yql-and-jquery/)
link:://projects.ischool.washington.edu/tabrooks/343INFOAutumn09/JSONP/jsonpJqueryYQL.htm (link:://projects.ischool.washington.edu/tabrooks/343INFOAutumn09/JSONP/jsonpJqueryYQL.htm)
link:://xpath.alephzarro.com/content/cheatsheet.html (link:://xpath.alephzarro.com/content/cheatsheet.html)
link:://:.w3.org/TR/xpath/ (link:://:.w3.org/TR/xpath/)

So don't give up Bayes, you just need to find the right button to push and you have what you want. :thumbsup: