User talk:Xerxes: Difference between revisions

Jump to navigation Jump to search
(→‎JALT01: Scraping info)
Line 113: Line 113:


This is the command I used. It only saves a TMD file if there actually is one, which is important because otherwise, it would save the 404 error HTML page as every wrong GameID which would make verification a hassle. My scrape should be done in a couple hours so we'll be able to take full advantage of it. If I was to throw away all assumptions as to what GameIDs are allowed I'd end up with a search space 4,294,967,296 items large which might take until 2019 to fully scrape and would be extremely unlikely to actually find anything more. - [[User:PowerKitten|PowerKitten]] ([[User talk:PowerKitten|talk]]) 01:38, 5 October 2017 (CEST)
This is the command I used. It only saves a TMD file if there actually is one, which is important because otherwise, it would save the 404 error HTML page as every wrong GameID which would make verification a hassle. My scrape should be done in a couple hours so we'll be able to take full advantage of it. If I was to throw away all assumptions as to what GameIDs are allowed I'd end up with a search space 4,294,967,296 items large which might take until 2019 to fully scrape and would be extremely unlikely to actually find anything more. - [[User:PowerKitten|PowerKitten]] ([[User talk:PowerKitten|talk]]) 01:38, 5 October 2017 (CEST)
:Those are good assumptions to make, it lines up with what we have documented about ID structure on [[GameIDs]]. You might want to clean that page up when you've done some analysis. (If you're really really curious, only one ID on the wiki uses a non-uppercase letter and it's [[GNHE5d]], and that's GameCube anyways.) I'm guessing even though this is publicly available, it's probably still not ok to transfer the files around through intermediates as a big archive, so I guess I'll have to run that as well to get my own copy to mess with. I'm really not a lawyer though, but I'd like to err on the side of caution. - [[User:Xerxes|Xerxes]] ([[User talk:Xerxes|talk]]) 02:10, 5 October 2017 (CEST)
6,906

edits

Navigation menu