this is not a blog

I Reckon This Must be the Place, I Reckon

Musings about our Apache log file (from stupid to bizarre - the logs not the musings)


These texts here are not in any particular order and change all the time.

Looking at the Log Files


In order to effectively manage a web server, it is necessary to get feedback about the activity and performance of the server as well as any problems that may be occurring.
Apache, on Log Files

Looking at this website's log file is boring, interesting and exasperating all at the same time. The first thing we noticed when we first launched was that oh so many browsers try to load favicon.ico. We do not believe in supporting non-standard IE shit like that, so we stubbornly and stupidly just ignored it — stupidly because Apache would issue a 404 for each request which effects bandwidth and is a waste of Apache's time. So we finally smartened up and created one... of zero length.

Then as the time online increased we noticed that we could tell humans from bots in that a human visit loads favicon.ico and CSS and most of the time the images as well; bots do not, getting just the HTML. And that those bots that do not identify themselves are the crooks and the spammers.

In this section we are going to be posting about those interesting and exasperating things we encounter.

By chronicling the odd things found in our logs certain patterns have emerged which can be used to detect and deny exploiters and spammers. (Without a massive external application like fail2ban.)

Notes

1. But since it has been the case for like fifty years now the case can be made that it's a de-facto standard of some sort. sigh
2. We now use a link rel="icon" thingy.
3. They are so obvious, really. Some use a consistent UA, some vary their UA, some use really odd UAs. But they all never load any image or CSS file and always concentrate on a few links at a time.

The Other Log File


If one, like me, though I am obsessive about it, likes (or dislikes) to peruse Server log files, there may be one you are missing: the error log file.

The error log file is not directly available for download, nor (assuming cPanel) your Server's File Manager. But it is through (again, assuming cPanel) the Errors link in the Metrics section of your Server Dashboard.

Click on it, click in the "Latest web server error log messages" box, then "Select All" and "Copy".

Then you can paste it in to your favourate editor to save it for examination. You might find some interesting stuff.

Who the Fuck is androxgh0st?


I think of it as the "Angry Gh0st". And it POSTs to root ("/") all the time.

See more: POST Log, in which I save POST data via a tiny bit 'o PHP code. The POST data in question is like:

    Array
    (
        [0x] => Array
            (
                [0] => androxgh0st
            )
    )

Oh... It was seen in some "web framework" code called Laravel:

    get_source = requests.post(url, data={"0x[]":"androxgh0st"}, ... ).text

Kinda fucked, ain't it?

0xAbyssalDoesntExist is now Abyssal


In my Robots List, where I 411 on all the Bots that hit this Website, I wrote about the Bot with the user-agent of 0xAbyssalDoesntExist.

It only POSTs to "/editBlackAndWhiteList" which is maybe a hardware CVE or something. And that string can be found in many a Website's online Server Logs. (Why do people do that? It servers no purpose. sigh)

greynoise.io lists it several times – but looking at their data results in more confusion.

But now they/he/she/it goes by just Abyssal. Maybe some day some one will expose them. (Not.)

    173.77.246.136 - - [26/Apr/2022:05:47:36] "POST /editBlackAndWhiteList HTTP/1.1" 404 201 "-" "0xAbyssalDoesntExist"
    121.141.149.197 - - [01/May/2022:22:11:18] "POST /editBlackAndWhiteList HTTP/1.1" 404 201 "-" "Abyssal"

Oh, and Bot Mozila/5.0 also POSTs to "/editBlackAndWhiteList":

    118.101.231.80 - - [30/Mar/2022:08:57:33] "POST /editBlackAndWhiteList HTTP/1.1" 404 201 "-" "Mozila/5.0"

P.S. Referenced in some code on Github: TVT-PoC.py. And Github has a user named Abyssal. (The string is also referenced on some RPG Game sites.)

Notes

1. Tee hee.

Who the Fuck is binance.com?

Well, according to them, "the world's largest crypto exchange".

But, who the fuck is doing this:

    31.210.20.88 - - [30/Nov/2021:12:30:09] "GET /wp-admin/css/ HTTP/1.1" 410 4 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    20.115.4.12 - - [30/Nov/2021:13:49:40] "GET /wp-admin/css/ HTTP/1.1" 410 4 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    195.133.18.227 - - [30/Nov/2021:22:59:54] "GET /wp-admin/css/ HTTP/1.1" 410 4 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    45.144.225.104 - - [04/Dec/2021:11:11:57] "GET /wp-admin/css/ HTTP/1.1" 404 3 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    45.144.225.104 - - [04/Dec/2021:11:12:03] "GET /.well-known/ HTTP/1.1" 403 2 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    136.144.41.85 - - [12/Dec/2021:01:30:46] "GET /wp-admin/css/ HTTP/1.1" 404 3 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    136.144.41.85 - - [12/Dec/2021:01:30:57] "GET /.well-known/ HTTP/1.1" 403 2 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    212.227.12.174 - - [13/Dec/2021:23:23:37] "GET /wp-admin/css/ HTTP/1.1" 404 3 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    212.227.12.174 - - [13/Dec/2021:23:23:51] "GET /.well-known/ HTTP/1.1" 403 2 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    212.227.12.174 - - [15/Dec/2021:02:31:27] "GET /wp-admin/css/ HTTP/1.1" 404 3 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    212.227.12.174 - - [15/Dec/2021:02:31:58] "GET /.well-known/ HTTP/1.1" 403 2 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    2.56.59.142 - - [15/Dec/2021:23:46:45] "GET /wp-admin/css/ HTTP/1.1" 404 3 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"
    2.56.59.142 - - [15/Dec/2021:23:46:56] "GET /.well-known/ HTTP/1.1" 403 2 "binance.com" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.85 Safari/537.36"

Netscum.

A Discovery of Brothers?

I may have had contact by "White Hats"! The good (not) people at LeakIX (leakix.net).

First, it solves one nagging notion – well, points into a certain direction. This was about the very odd, invalid-looking requests like:

    "\x16\x03\x01" 404 - "-" "-"
    "\x16\x03\x01\x01\xfc\x01" 404 - "-" "-"

(But that might be an Apache SSL configuration issue.)

But then I got a repeating offender from IP Address 143.198.136.88, and it
dropped more identifying strings:

    143.198.136.88 - [29/Nov/2021:11:39:43] "\x16\x03\x01" 404 - "-" "-"
    143.198.136.88 - [29/Nov/2021:11:39:43] "GET / HTTP/1.1" 404 - "-" "-"
    143.198.136.88 - [29/Nov/2021:11:39:43] "GET / HTTP/1.1" 403 2 "-" "l9tcpid/v1.1.0"
    143.198.136.88 - [29/Nov/2021:11:39:43] "CONNECT leakix.net:443 HTTP/1.1" 403 2 "-" "Go-http-client/1.1"

Followed by this:

    143.198.136.88 - [29/Nov/2021:11:39:43] \
    "GET /cgi-bin/../../../../../../../../../etc/hosts HTTP/1.1" 404 - "-" \
    "Lkx-TraversalHttpPlugin/0.0.1 (+https://leakix.net/, +https://twitter.com/HaboubiAnis)"

(The dots were "disguised", URL-encoded, here's a leakix log excerpt. That is an explicit attempt to get my Server's /etc/hosts file! And they signed their work!)

All their requests are for known exploits. What they say:

"We intent [sic] to provide a preemptive solution by trusting individual researchers and security companies on the most sensible data we index by delivering a clear report on the incidents, we also help to identify what information has/could be affected and how to resolve the issue."

What the fuck does that mean? They add: "There are a few parts involved, but so far the opensource releases are "

l9tcpid "our TCP protocol inspector"
ip4scout "our random ipv4 space scanner"
l9explore "our deep protocol exploration tool"

Thanks for your block strings!

Their hubris borders on the stupidity. They are looking for exploits on my pathetic little website! Which only slows my site down and increases the size of my Server's log files!

First, people of the "White Hat" variety should ask for permission, no? Which is by using the Standard robots.txt. All of these kind of folks say (something like), "To opt out... contact us." Such laziness.

Now, all it seems LeakIX does with the data is... I don't know. Though they say, "This project goes around the Internet and finds services to index them," that ain't true if you view the log.

Their "About" text is too vague, bordering on meaningless. Look, I'm certain they mean well, but, from this Website's POV — they "gather information on ... the most common security misconfiguration[s] currently open." See? Meaningless.

Just making "lists" of exploits — just like "CVE lists" — ain't doing anyone any good... Maybe they should just leave Ports 80/443 alone.

jerks

P.S. Here is our Disclosure Policy.

Zgrab is Grabbing Me!

"Listen, and understand. That Zgrab is out there. It can’t be bargained with. It can't be reasoned with. It doesn't feel pity, or remorse, or fear. And it absolutely will not stop, ever, until your website is dead!"

(Yeah, that's hyperbole; but still, it's kinda funny if not appropriate.)

Zgrab Log

That link is to an unsorted list of Apache log entries with the string "zgrab" to show what that code is being used for on two websites I manage. Note that 654 requests of just "/" have been removed.

So these are 1,402 requests, as of 12/Mar/2022, for resources that are on some "exploit" lists or something; i.e. shit that ain't on my website BUT that if were would possibly be exploitable (or means the existence of explotable code).

On face value – to my view – what Zgrab is doing is just a waste of time and resources, meaning Internet bandwidth and server log files sizes.

(Of course, that's just me; I am certain that many think what is happening here is a "Good Thing™". But that's Bullshit, no matter how you look at it.)

The first 100 or so requests are boring, but others further on are not, as they are explicit attempts to exploit code. I do not see how anyone can condone that, even from the "We are here to help," perspective. If You have an argument on why this request is a "Good Thing™":

    "GET /owa/auth/logon.aspx?url=https%3a%2f%2f1%2fecp%2f HTTP/1.1"

Go to you know what...

The Zgrab code (not necessarily the weirdos at the Zmap project) only GETs a small number of "objects", like:

    /hudson
    /owa/auth
    /portal/redlion
    /actuator/health
    /manager/text/list
    /ecp/Current/exporttool

And a few others. But it seems the targets are dependent on IP address; i.e. one IP range does those, another IP does /.env, another IP does /forum, etc.

(Zmap is also related to Censys, another "We are here to help!" brain fuck that assumes they too have the right to scan your websites for anything they want.)

  1. But these Exploit List Bots are not going to go away, ever. That's the point.
  2. At least I think that that is what is going on here.
  3. "I know! Let's keep login credentials in a file in a known place on every website!"