Post/Author/DateTime	Post
#1 Blakey Jun 14, 2010 7:38:22	Is this still available?   And is there any (correct, up to date) information on how to access it? I'm specifically interested in programatically being able to log into the compendium and then search for and access monster info in my own Java app.   Nothing commercial, all for personal use only. Cheers Blakey
#2 tasiefer Jun 14, 2010 10:35:30	Is this still available?   And is there any (correct, up to date) information on how to access it? I'm specifically interested in programatically being able to log into the compendium and then search for and access monster info in my own Java app.   Nothing commercial, all for personal use only. The only public APIs are the compendium search APIs that you can find via Google.  There's no official API to authenticate or retrieve an entry in easy-to-parse XML.  The usual method to accomplish what you want is to create an in-app browser session, login to the compendium, then utilize those credentials to retreive the HTML page for whatever entry you want (the URL of which is returned via the search API).
#3 Blakey Jun 15, 2010 7:58:55	The only public APIs are the compendium search APIs that you can find via Google.  You might be able to find them but any I have found simply don't work.   Can you point me to one which is working?
#4 tasiefer Jun 15, 2010 10:26:28	The only public APIs are the compendium search APIs that you can find via Google.  You might be able to find them but any I have found simply don't work.   Can you point me to one which is working? www.encounteraday.com/2009/06/17/exclusi... www.enworld.org/forum/software-computers... community.wizards.com/wotc_frazier/blog/
#5 Blakey Jun 16, 2010 4:44:29	Thanks.  After trawling through all three of those threads (all of which I had previusly read) I am finally able to get this working.  Almost.   Next question:  how do I programatically 'login' to the Compendium?   I have figured this much so far: If I want to go to a given power (say Divine Armor, which is power number 959), and also login in the same breath, I use this URL: ...wizards.com/dndinsider/compendium/login.aspx?page=power&id=959
#6 asmodeuslore Jun 16, 2010 15:38:37	I am not sure, but I think that is one of the things the dev team actually doesn't want third-party programmers doing.  They want each user to log in the first time they request any data, and then remain validated for as long as the session remains active. What I suggest you do is set up a log-in page for your app when it first starts up, use that to begin a session, and then use that session while your app does whatever it needs to do.
#7 tasiefer Jun 16, 2010 17:44:38	Next question:  how do I programatically 'login' to the Compendium? As I said in my first response, you don't.  There is no way to do this.  The only method available is to create a browser session within your application, allow the user to login, then you'll be able to go to any entry in the compendium since the authentication credentials will still be in use from the browser.  That's how inCombat does it and that's how Masterplan did it.
#8 rcuhljr Jun 17, 2010 10:51:49	As I said in my first response, you don't.  There is no way to do this.  What? It took like 2 minutes to see the post header on logging in is <code>method=Login&amp;keyPosition=0&amp;userName=REDACTED&amp;userPassword=REDACTED&amp;rememberMe=fal</code><code>se</code><code> and the response is a set cookie containing
#9 tasiefer Jun 17, 2010 12:36:17	You capture that after your programmatic login and use it as your cookie data and you are logged in programatically any time you want to be. Even if that didn't work you could use selenium or other web testing framework to automate the login. Sigh. And depending on the environment and method you're using to retrieve web results then reusing that cookie data may work, but also may not work either consistently or at all.  It all depends on how your framework handles itself.  There's a reason inCombat uses an in-app browser for this and it's not because I didn't try programmatic login.  You also have to consider the fact that people don't like typing their username and password into a strange program.  Yes, I know it was stated that this app was for private use only, but things have a tendancy to not stay that way. WOTC staff produced two API articles for the compendium and talked about the 3rd being for authentication, but then seemingly shut things down.  For whatever reason they don't seem to want people logging in programatically.  That doesn't mean that it's not possible, it just means that it's far easier to login the way that WOTC seems to want people to login than to reverse-engineer the private API and all the quirks that go along with it.  I spent three nights trying to get programatic login working consistently, then was done in an hour when I gave up and went with the browser method.  If your framework works better for you than mine, then more power to you but you shouldn't waste time reinventing the wheel.
#10 rcuhljr Jun 17, 2010 14:12:00	All I'm saying it don't tell people there's no way to do it when it's possible. It's certainly not easy or straightforward in every language but some make it a cinch. I keep my password to DDI stored in firefox because I'm lazy. Because of this it took me a wopping ten minutes from the command line (minus install time) to have Watir up and running, have it log into the site programatticaly and submit some queries to the compendium, and I've never used Watir before. If someone wants an application for personal use that can connect to the compendium automatically and do searchs for them it's really not that bad. tinypic.com/r/11izgvm/6
#11 tasiefer Jun 17, 2010 16:03:35	All I'm saying it don't tell people there's no way to do it when it's possible. It's certainly not easy or straightforward in every language but some make it a cinch. I keep my password to DDI stored in firefox because I'm lazy. Because of this it took me a wopping ten minutes from the command line (minus install time) to have Watir up and running, have it log into the site programatticaly and submit some queries to the compendium, and I've never used Watir before. If someone wants an application for personal use that can connect to the compendium automatically and do searchs for them it's really not that bad. tinypic.com/r/11izgvm/6 Hehe. You do realize that by using Watir you're just doing the browser method I outlined above, right? You're creating a browser instance, logging in via that browser, and then navigating to the final entry. Masterplan and inCombat let the user manually login and your guy there is performing an automated login, but none of the three are programatically logging in.  To do that we'd have to send raw HTTP commands, and I haven't yet seen a program do that.  Doesn't mean one doesn't exist or that it's not possible, as you say, but it's so much easier to just use the browser method.
#12 rcuhljr Jun 17, 2010 16:11:36	All I'm saying it don't tell people there's no way to do it when it's possible. It's certainly not easy or straightforward in every language but some make it a cinch. I keep my password to DDI stored in firefox because I'm lazy. Because of this it took me a wopping ten minutes from the command line (minus install time) to have Watir up and running, have it log into the site programatticaly and submit some queries to the compendium, and I've never used Watir before. If someone wants an application for personal use that can connect to the compendium automatically and do searchs for them it's really not that bad. tinypic.com/r/11izgvm/6 Hehe. You do realize that by using Watir you're just doing the browser method I outlined above, right? You're creating a browser instance, logging in via that browser, and then navigating to the final entry. Masterplan and inCombat let the user manually login and your guy there is performing an automated login, but none of the three are programatically logging in.  To do that we'd have to send raw HTTP commands, and I haven't yet seen a program do that.  Doesn't mean one doesn't exist or that it's not possible, as you say, but it's so much easier to just use the browser method.  I went with the quickest way to automate a login, When I say programattically I mean without user intervention, which is what he wants to do. If you want to redefine it to raw html requests, it's not that hard. Their site doesn't use HTTPS on the login page so they use DES (old school!) implementation in javascript to encrypt the password before sending it to the server, however all the information used by javascript is visible to the code if your running raw http commands, you just need to pull out the vector and the key offset and send them into their javascript des method along with your password, pipe that into the stringToHex() and tada you have your login command. You can do it that way as well if you want to. And all of this assumes your redefinition of programattically to mean something else. But IBM sure seems to think this method counts as programattically. www.ibm.com/developerworks/java/library/...
#13 rcuhljr Jun 17, 2010 16:43:41	Yeah just to double check that I'm not talking out of my ass, here's a few minutes of ruby that visit the dnd tools page and strips out your challenge and vector, giving you everything you need to run var encryptedPassword = stringToHex(des(challenge, password.value, 1, 1, vector, 1)); Which you'll load the JS into Rhino or spidermonkey or whatever your preference is and voila, login credentials. RubyAhoy i45.tinypic.com/2sbtq2a.jpg
#14 tasiefer Jun 17, 2010 21:31:15	Blakey, if you need any further help you can always PM me.  If you're interested in any parsing or pagination code I can provide some stuff there too.
#15 rcuhljr Jun 18, 2010 15:50:11	If anyone needs it I knocked out code to do omg pure http posts (you need jruby and rhino), although I stand by my earlier post that you're better off automating it with watir/mechano/selenium because the dnd site is sitting on IIS 6 and asp 2.0.X which makes it annoying to do posts against since you're butting heads against header formatting issues and other joys. (also it makes for snappy logout, which I didn't implement, I suspect if you snatched the original viewstate you could post it as a logout view state, but that'd be ugly to do) In hindsight I wish I went with curl for the automated scrapping, it really does a better job of faking web interactions and storing and replaying cookies like a real browser. That's be my recommend route in the future if you're dead set against using automation tools, You'd still need rhino/jruby or c/c++ with spidermonkey to run the DES encryption algorithm to get your encrypted password, but meh. A wild automated authentication appears! data = method=Login&amp;keyPosition=0&amp;userName=redacted%40yahoo.com&amp;userPassword=0xf0229158127e13af&amp;rememberMe=false<br />Code = 200<br />Message = OK<br />date = Fri, 18 Jun 2010 18:09:49 GMT<br />server = Microsoft-IIS/6.0<br />x-powered-by = ASP.NET<br />x-aspnet-version = 2.0.50727<br />set-cookie = iPlanetDirectoryPro=3381b4e2-4054-45d9-9a4d-b2e7b02d55b9; domain=.wizards.com; path=/, .ASPXAUTH=687A93BE1AD76D2993CC2677338F85705774B76983F41DBA82<br />D804B6662665356B11F94593502EBB67303ED0F9A97B22EC062EE33C342B9DD239EBEFF4F7F145436<br />9294289DCAAD3E21A6E433D0168192D99CD73C73D52CCEF22525504364922B10D1EF5D97D0D17EF<br />2BCEA9CEBB781C007D7DC329B1CF352A0092ACB4BA952873A77FB36A61E25EC767317750098623D7<br />08746AC9C84EBA27774AE27034767678FCCEEF4D7E9427EB68BCBCA2834BA2A6ED40B4B4BECF28FC<br />11AEA71CF8437E0BBE721AC79DBFD227E6A9DF117ADCBCAC8D24D1443F1BB22BE6D5EE6DEB4FAA91<br />EF41D822A73254716B78E183E2F0F1C568B39741926D1B453BBEA165422C83E4A1A70E676CDA4C99<br />CADF51993FC2482E306F28B71BFC5F3A67DC1D1D9BC6D946383D2866B968D2C993D18327A6420AF7<br />EEF7C1D14661B4A943C567DD5A768B440FC122D7538AA60A3E4DE1FFF2B5DF3498C3DD6DCF297E7F<br />C911848D7450352301EBE3B44473C39174A579AD0B8DDD20232A9DED26B3D4FB58C6044121B67BB<br />35E71B977B70BBF7D8AA9F442B36694; expires=Fri, 18-Jun-2010 18:39:49 GMT; path=/, A<br />uthPlanetPro=; domain=.wizards.com; expires=Tue, 08-Jun-2010 18:09:49 GMT; path=/, BIGipServerWWWDNDPOOL01=890243338.20480.0000; path=/<br />cache-control = private<br />content-type = application/javascript; charset=utf-8<br />content-length = 87<br />{"OperationResult":true,"Message":null,"Data":"0,33887089,rcuhljr,%2fdnd%2fTools.aspx"}
#16 Blakey Jun 20, 2010 9:02:22	Can any of you java experts help me?  I've built a single class/app that shows my problem connecting to the Compendium API.   When I try and connect it always takes me back to the login form without the two sesison cookies I'm expecting to have enabled.
#17 rcuhljr Jun 22, 2010 16:33:49	Your password needs encrypted via DES using a challenge ID and Vector that they store in hidden inputs on the page they send you, I'll dump some example code (It's in Jruby, but it's running a java library Rhino for executing the javascript, so it should be easy to set up in pure java) gist.github.com/449127 Of note are lines 418 and 422, 418 shows were your password needs to get sent into the javascript, and 422 shows what needs to be in the posted data.
#18 dj2 Jun 25, 2010 0:09:48	I was playing with this a while ago in MacRuby. There weren't really any hoops that needed to be jumped through, just POST the right data to the compendium. You can see the code at: http://github.com/dj2/Touchstone/blob/master/App/Models/DDI.rb The interesting thing I noticed was that the ViewState and EventValid fields never seemed to change. It may not be a valid assumption moving forward, but it's worked so far. Looking at the pasted code, the differences I see are I've go Sign+In (url encoded) and you've got Sign In. The other thing I had to do was force a fake page to load after login. I'm also using that fake page load to determine if the login was successful as I can search for the deity name in the response. Note that the reason I'm not storing the cookie afterwards is that Cocoa just deals with that. I don't have to worry about it.
#19 rcuhljr Jun 25, 2010 11:50:40	Well the other difference is yours is actually coherent code as opposed to 30 minutes of 'someone is wrong on the internet!' mode :P I wasn't aware there was a url encoded login option for the compendium, but then I've never really looked at this stuff, I was just brute forcing the login off a random page (tools in this case).  Thanks for the github link, good stuff for anyone who wants to do the impossible.
#20 tasiefer Jun 25, 2010 17:15:54	I was playing with this a while ago in MacRuby. There weren't really any hoops that needed to be jumped through, just POST the right data to the compendium. You can see the code at: http://github.com/dj2/Touchstone/blob/master/App/Models/DDI.rb Now there's some code I can get behind.  Very nice, dj2, good discovery. To document this API a little more clearly for novices:
#21 vagabundo Sep 10, 2010 15:44:01	Anyone know what I'm doing wrong here in python? <br />import urllib2<br />import urllib<br />import httplib2<br /><br />VIEW_STATE = '%2FwEPDwUKLTMxMzExNzE1NGRkc8%2FFVQcDANmIFQs21%2F7Ecpvbzyo%3D'<br />EVENT_VALID = '%2FwEWBAKxxoafCQKyzcaDDQLyveCRDwK4%2BvrXBUnFA%2FTyvF2WhXwyIyva11C3B5zz'<br />insider='InsiderSignin=Sign+In'<br />email='pxxx@xx.xx'<br />password = 'password'<br />URL='http://www.wizards.com/dndinsider/compendium/login.aspx'<br />login=urllib.urlencode({'page':'ritual','id':'2','__VIEWSTATE':VIEW_STATE,'__EVENTVALIDATION':EVENT_VALID,'email':email,'password':password,'InsiderSignin':'Sign In'})<br />headers={'Content-Type':'application/x-www-form-urlencoded','Content-Length':len(login)}<br />req=urllib2.Request(URL,login,headers)<br />response = urllib2.urlopen(req)<br />data = response.read()<br />print data<br /> I keep getting 505 errors
#22 vagabundo Sep 14, 2010 12:09:56	Bump. Anyone help here. I tried running the ruby script but got some errors on it. Not very familiar with Ruby.
#23 ORC_Wyvern Sep 14, 2010 13:21:39	Hi. I had to remove some content from this thread because it violates the Code of Conduct or because it quoted content that did. Please keep your posts polite, respectful, and on-topic, and refrain from making personal attacks.
#24 yargon Oct 20, 2010 21:06:33	Have you guys seen YQL (developer.yahoo.com/yql/console/) , I think it could do wonders if I could get it pasted the login issue. I was thinking of getting a mash up of a list of feats, powers, prestige classes, Epic destinies etc etc.. and geting a RATE4e site running where you could rate each and every item. (both positive and negative) I think this would have a double benefit, players can see what feats are good to take, and if WoTC took notice then it could identify items that no player thinks are good. can anyone help with that? WoTC are you listening?
#25 lokiare Oct 21, 2010 13:56:23	... WoTC are you listening? Oh wow.....lol..........hahahhdhhdhahhahahhahwhwhwhahhshwhwhahhshhhshahahahahhaha! WotC listening... that's a good one....
#26 ryatziv Feb 26, 2011 23:10:45	I was wording on a program to search the Compendium and a few hours in I'm now getting "unknown host" exceptions.  Am I not allowed to do this for some reason?
#27 unniloct Mar 02, 2011 16:03:22	I was wording on a program to search the Compendium and a few hours in I'm now getting "unknown host" exceptions.  Am I not allowed to do this for some reason? Bump! I'm trying to play around with the Compendium and I'm becoming increasing frustrated as nothing from this thread seems to work anymore.
#28 FloLeBlanc Mar 09, 2011 11:26:37	So is there any API yet? I'd like to play around with a combat tracker for a tablet - but didn't find any working API.
#29 ryatziv Apr 19, 2011 23:20:02	What was working before is no longer working.  For example,  wizards.com/dndinsider/compendium/Item.a... used to be a correct link.
#30 pruano Apr 20, 2011 10:36:35	What was working before is no longer working.  For example,  wizards.com/dndinsider/compendium/Item.a... used to be a correct link.  Some still do. What item did that reference to? Some items, namely the ones from LFR, were removed from the Compendium, and some may have gotten a new id.
#31 Blakey Apr 26, 2011 3:03:44	This method does still work.  You can search the compendium for an XML list of monsters, items, powers, etc using the same mechanism as before.   And you can log into the compendium using cookies in order to scrape it.    All do-able programatically. Cheers Blakey
#32 Jharii Apr 26, 2011 5:06:30	It works.  But there is a rather significant difference between an artifact of web design/HTML looting and an actual API.  Also, the licensing agreement for DDI specifically states that you cannot scrape the database.  An API would essentially eliminate the need to do so.
#33 Blakey Apr 26, 2011 6:34:09	It works.  But there is a rather significant difference between an artifact of web design/HTML looting and an actual API.  Also, the licensing agreement for DDI specifically states that you cannot scrape the database.  An API would essentially eliminate the need to do so. What do you mean by "scrape"?   I used the same term but in my case it was very loose - as I'm not sure of the official meaning of the word in this context. My application (which I keep to myself) "scrapes" the compendium to get monster details.  All it actually does is read the HTML for a monster and then I try and do some clever parsing of that HTML which is a nightmare to get right.  Of course, if we could have a proper API that would be 1000 times better but I think we all agree that is never going to happen.
#34 Jharii Apr 26, 2011 8:02:32	Database scraping is typically any automated form of gathering the data from a database to store locally done by a program that simulates human interaction.  It is meant to gather an abundance of data. en.wikipedia.org/wiki/Web_scraping
#35 Elindra Apr 26, 2011 8:50:33	What was working before is no longer working.  For example,  wizards.com/dndinsider/compendium/Item.a... used to be a correct link.  I know little to nothing about this stuff, but if I change your link to use .../item... instead of .../Item... it works fine.
#36 Blakey May 05, 2011 9:55:28	Database scraping is typically any automated form of gathering the data from a database to store locally done by a program that simulates human interaction.  It is meant to gather an abundance of data. en.wikipedia.org/wiki/Web_scraping Okay thanks.  So I don't think my program "scrapes" the Compendium in that case.    I tell it "Build me a Goblin Blackblade" and it goes off and finds that specific monster and builds a token for it.   There is no facility to say "Go and build me a token for every monster in the Compendium" or any multi-token building like that.    So I think I'm safe with respect to the agreement. I wonder if WotC think they would open up piracy or somehow lose money if they gave is a real API for DDi.    How glorious that would be...
#37 chillhelm May 05, 2011 17:33:52	Actually, anyone using some official DDi-Api would need to have access to the login data of an active DDi-Account, so they might even gain a financial benefit from releasing such an api. On the other hand, examples of WotC-Code sure dont look impressive (Monster Slider, OCB)
#38 pruano May 06, 2011 16:38:43	Okay thanks.  So I don't think my program "scrapes" the Compendium in that case.    I tell it "Build me a Goblin Blackblade" and it goes off and finds that specific monster and builds a token for it.   There is no facility to say "Go and build me a token for every monster in the Compendium" or any multi-token building like that.    So I think I'm safe with respect to the agreement. Actually it still is; the amount of scrapping done does not matter, although that is one of its main uses. The fact that the compendium does not give data in structured form (other than html) makes any use of it web scrapping. You may want to read the wikipedia article on it, its quite useful and informative, even going into several techniques used, some of them not well known; a good but somewhat old book on it is "Spidering Hacks" from O`Reilly www.amazon.com/Spidering-Hacks-Kevin-Hem...
#39 Blakey May 09, 2011 9:20:33	The fact that the compendium does not give data in structured form (other than html) makes any use of it web scrapping. Not quite true.   You can get XML from the Compendium if you use their (tiny) API.  And XML is of course very well structured.
#40 pruano May 09, 2011 9:59:24	The fact that the compendium does not give data in structured form (other than html) makes any use of it web scrapping. Not quite true.   You can get XML from the Compendium if you use their (tiny) API.  And XML is of course very well structured. In that case, its not web scrapping indeed. I was under the assumption that the result would be html from your post above.