I've noticed that many bot-detection pages are able to be avoided by using this UA. Unless there's a reason to stay on the old old old one, we may as well update it and reduce our WAF hits.
Attempts were met with a Varnish error page presenting 54113 (possibly Fastly related).
Change to User Agent necessary to avoid Varnish/Fastly issues.
Change to Accept necessary to avoid infinite 302 redirection.
Without BOTH of these changes, attempts will fail.
Both changes being made also permit the use of status_code rather than message.
Error codes module expanded to support arrays of error codes rather than only one.
Using this new functionality, Slides was set to error codes 404 (as standard) AND 204 (non standard), to accomodate for that website's odd edge case.
The following targets were fixed:
Archive[.]org
CGTrader
CNET
Contently
IFTTT
Linktree
xHamster
The following targets were removed:
HexRPG (auth wall)
ModelHub (defunct)
Oracle Communities (auth wall)
ModelHub was not added to ./removed_sites.md as the platform itself is shutting down (and will therefore never return to Sherlock). The other removed targets were documented normally.
BitcoinForum is currently down and suspected to be defunct. Since this is uncertain, however, a test condition was added to suppress false positives while allowing for normal operation upon the forum's return.
Paul Pfeister
Matheus Felipe
David
Piotr Pietrusewicz
thirdkeyword
Gowtham K M
Nicholas Murillo
Edwin Kofler
Siddharth Dushantha