Demolishing SPH’s claims in Mr Geoffrey’s misleading article: “Attack on Temasek Review – not SPH”
November 6th, 2009 | 
Author: [Read our latest rebuttal to Mr Geoffrey Pereira's second article on 13 November 2009 here]
Last week, we published an article about a SPH IP address caught “grabbing” content from our site. (read article here)
SPH has since published a reply to our article by Geoffrey Pereira on his ST blog here
Read our latest updated article on the Freudian slip made by Mr Geoffrey Pereira here
First of all, before we began, we must thank Mr Geoffrey Pereira for giving us such extensive publicity on the Straits Times.
We are sorry that he was “arrowed” by his superiors to draft a reply to us with the aim of putting us down and demolishing our credibility altogether.
We have nothing personal against Mr Geoffrey Pereira and we understand he is merely an employee of SPH.
It is most disingenuous of Mr Geoffrey to distort the version of events, put words into our mouths and throw a smokescreen to obfuscate the matter and to confuse and mislead readers into thinking that we have hurled a false accusation at them.
Let us correct the three FACTUAL INACCURACIES in Mr Geoffrey’s article:
FACT #1: We did not accuse SPH of launching a DDOS attack against our site.
In the first place, Mr Geoffrey’s choice of the title of his article – “Attack on Temasek Review: not SPH” is completely off the tangent because the article was never about any “attack”.
Nowhere in the article did we ever accuse SPH of launching a DDOS attack to bring down our site. Mr Geoffrey wrote an entire section on IP sproofing which is totally irrelevant because its was not about DDOS at all.
The server log we published showed an IP address belonging to SPH “grabbing” content from our site. It is easily understood by a layman that “content grabbing” is not equivalent to a DDOS attack which is the point we are trying to get across.
In fact, we took pains to explain what a DDOS attack means in the beginning of our article to prevent readers from getting the wrong idea because it occurred only a day after our site was down by a DDOS attack.
Unfortunately, Mr Geoffrey deliberately misquoted us to create the impression that we are accusing SPH of being culprits of his imaginary “DDOS attack”.
We find it amusing that SPH actually asked a staff of its Network Intrusion Protection Services (NIPS) vendor to check 7 days worth of data who found no DOS activity originating from SPH before reaching the conclusion:
“My opinion of the situation is Temasek Review released the article with very little research into what happened on its server.”
It is SPH who did not bother to do any “research” on our original article.
Mr Geoffrey should read through our entire article carefully again and emailed us for clarifications first before publishing his article.
Had SPH bothered to ask us if we had accused them of launching a DDOS against Temasek Review, we would have told them straight in the face:
“NO, we are fully aware of the fact that SPH DID NOT launch any DDOS attack on us at all. What we are keen to know is whether SPH did “grab” our content”.
FACT #2: Timing of the incident occurred between 31st October 2200 hours to 1st November 0100 hours.
As our article had stated clearly, the flurry of network communication requests from the SPH IP took place on 31st October 2009, around 2200 hours to 1st November 0100 hours.
Our correspondent first received the call from the system administrator on 1st November 2009 at around 12.10am.
He typed the article on the spot which explained why the initial date was published as 1 November.
The exact dates were subsequently amended to between 31st October and 1st November after the relevant portion of the server log is printed out by the system administrator and forwarded to us together with cPanel’s Apache access snapshot.
Mr Geoffrey wrote:
“In fact, from midnight on Nov 1 to about 6 am, (covering a period of the alleged attack) no one from SPH accessed the TR site.”
Of course nobody from SPH accessed the TR site during this period of time because our server log did not show otherwise! This period was not even stated in the initial draft of our article, so where did Mr Geoffrey get it from? Another figment of his imagination?
The key question is, did anybody from SPH access the TR site from 31st October 2200 hours to 1st November 0100 hours? This is the critical time period when the “grabbing” was proven to have taken place by our data center and ISP in China.
Mr Geoffrey got the timing of the incident completely wrong and therefore how can he use it as a basis to disprove our claims about the SPH IP address “grabbing” content from our site?
He should get his facts check first before making such an embarrassing mistake which cast doubts on the logic, consistency and accuracy of his article.
We understand that Mr Pereira is probably not acquainted with IT issues like us, but surely there must be an appropriate person from such a big organization like SPH to proof-read his article before it went to publication?
FACT #3 The “grabbing” has the potential to slow and overload our server
Mr Pereira wrote in his article that nobody in SPH try to “grab” content from our site which will load our server:
“Neither did anyone in SPH try to “grab” TR material in a way that would load its server; nor did any SPH staffer launch any attack on the server.”
The fashion by which contents are being accessed is consistent with search robots or a web grabber – ie – a website is archived so that a string search can be made. While this is perfectly legal, some software uses multiple sockets when downloading content, and CAN potentially hog resources from the web server and slow other user’s access.
It WILL HOG the server’s resources but in this incident, it didn’t because the software firewall on the server itself banned the offending IP address minutes into the action after the IP address exceeded 60 connects per minute, the threshold set by the system administrator.
Technically, if the server were to be not protected by firewall and had been configured poorly, a multiple of requests in excess of 60 connects per minute WOULD HAVE brought the server down and that would technically be classified as an attack.
FACT #4: Our log shows SPH’s IP address accessing material from 2008
As we can see from a snapshot of our log, the IP address 203.116.232.234 which was traced back to SPH by our data center and showed it getting our content from as early as 2008.
Mr Geoffrey claimed that SPH logs showed otherwise:
“SPH logs also determined that no one from the company tried to access material from 2008, as claimed by TR.”
There can only be three possibilities:
1. The SPH logs somehow omitted the period between 31st October 2009, 2200 hours and 1st November 2009, 0100 hours.
2. Our system administrator, who is a Chinese national, had either falsified the server log or made a mistake about it but that is quite impossible as we use cPanel on our servers and can see for ourselves under its GUI that the said IP was indeed registered as accessing our site on the date and time in question.
3. The engineers at China Telecoms, the largest ISP in China who owns and runs the Data Centre where our server is located had falsified both the network and firewall logs as well.
Our servers are hosted with RTG (Asia) Network in a China Data Centre. Our system administrator had double checked and verified the logs with their Data Centre before we decided to publish the article.
The said IP address was registered on our server’s Apache log, cPanel’s access logs and even at Data Centre level.
Frankly, we don’t see any reason why someone will go through all the trouble to spoof an IP address to incriminate SPH instead of launching a DDoS attack if that someone has anything against us, as claimed by SPH. Besides, the Data Centre we use has hardware firewall to detect and drop spoofed IPs and Packets.
[Read why IP sproofing is technically IMPOSSIBLE in this case here]
Furthermore, the company’s staff are all Chinese and have NO vested interests in TR or SPH and stand to gain NOTHING by hurling false accusations against SPH.
There is a discrepancy between what SPH said and what was revealed on our log.
None of us are technical people and it will be unfair for us or Mr Geoffrey himself to continue the exchange online.
We propose a simple solution to get to the bottom of the matter to resolve the impasse:
1. Get the system administrator of SPH to contact our hosting company RTG (Asia) Network for our full server log as well as China Telecoms Data Centre for the network and firewall logs. (Obviously we cannot reveal them here for security reasons)
2. Conduct an investigation to find out why our log showed SPH IP addresses “grabbing” our content at the stated time frame on 31 October 2009, 2200 hours to 1 November 2009, 0100 hours.
If the system administrator at RTG somehow made a mistake or gave us wrong information, request RTG to publish it on its site.
We will follow suit with an unreserved apology immediately under our “TOP NEWS” section continuously for 3 days.
However, if it is indeed true that the perpetuator is a SPH staff, we hope SPH can give us an explanation of what really happened.
SPH should realize that they cannot afford to ride roughshod over us like what they did to others before.
Though their sites still dominate Singapore’s blogosphere, we are no pushovers either and we have a sizable readership to reckon with as they must have realized which explained why SPH felt there is a need to reply to our article.
Unfortunately, Mr Geoffrey’s ill-thought article now leaves us with more questions than answers.
We are more than happy to cooperate with SPH to find out the truth as we are really concerned about the content of our site being “grabbed” in such a covert manner.
IT idiots like us have to depend on our system administrators to provide us with the facts relating to server matters. We will greatly appreciate SPH’s assistance in this matter because based on the resources they have at their disposal, they should be able to find out who the real culprit is (assuming if it’s not a SPH staff).
In fact, since SPH has now blown the matter out of proportions by implicating RTG and China Telecoms, they have to pursue it to the end because the reputations of two companies are at stake.
Mr Geoffrey or any SPH staff can contact us at [email protected]. They are advised to consult us first to have their facts checked before publishing another embarrasing article like this to shoot themselves in the foot.
Related articles:
>> A Freudian slip by Mr Geoffrey Pereira?
>> Attack on Temasek Review: not SPH
>> SPH IP caught grabbing “content” from Temasek Review
>> Debunking Mr Geoffrey’s claims on “IP spoofing”
Posted in Editorial
Coz it’s true to character of...
Smoke and mirrors. Typical of ST.
Yes! KO! SPH, Get your facts right first before we talk again ok?
SPH, you can totally bar access to TR from your corporate lan. If no one can access, there won’t be any complaints from TR.
TR, any other sensitive organisations you don’t wish to see coming to your site? How about all the Temasek companies, civil service and stat boards? You can list them out, and request them to ban access to your site. Remember to include starhub, singtel and m1.
Then, no one can grab anything from your site using these corporate lans. As added precaution, you can also ban their access using your firewall too. If you discover anyone from these sensitive organisations still come here, then just publish out their IP as warning.
Then, there’ll be peace for all and everyone will be happy.
TR, I hope ST replies to you.
I’ve often wondered how far a lie can travel. Perhaps, this will be the unofficial point where a lie is clearly exposed, thereby allowing us to look at the sun with different eyes for the first time. I will continue to hope.
SPH LIARS! Stop hiding the truth!SPH you’ve defamed TR, please apologize like gentlemen!
journalism in singapore’s newspapers have gone to the dogs. in fact many newspaper journalists are no worse than filthy lying dogs who named fellow singaporeans “sg-stupid” or condemn the poor and homeless. i wonder if journalism degrees, mass communications diploma etc in singapore are awarded to students for their integrity or the lack of.
capable, honest journalists do not work in singapore’s newspapers. they go to western countries. those that stay behind in singapore’s newspapers have no choice and no real abilities and are all wannabes who smoke the readers.
very good. jsut delete all comments that disagree with your point of you. that’s the best way. If you could, you would delete the sph website also.
It will be a never ending story. I wont be surprise sph will take extreme measure to protect themselves.
They never do give you a straight reply. I am only glad that fewer and fewer Singaporeans are choosing to work there. Most Singaporeans interested in journalism have gone overseas to work in organizations with more freedom. No decent journalism would want to do propaganda articles.
I can only speculate from observation, my I would assume that 70% of the staff at SPH are now foreigners. I also expect the number of foreigners to increase as more Singaporeans become adverse to writing propaganda.
Wah…are they ignoring you, engaging you or rebutting you.
Wait a minute, they might sue you too.
Media Wars, send in the clones (I mean spoofs).
riiiiiiggggght said: very good. jsut delete all comments that disagree with your point of you. that’s the best way. If you could, you would delete the sph website also.
If TR has wanted to hide and wayang, why bother to post SPH’s reply for the whole world to read?
The Brain is use for thinking and the arse is use for shitting, not the other way round.
BOTTOMLINE is, SPH being ‘heavilly backed and connected’ can come up with all sorts of ‘experts’ to testify that their servers were down on that day for maintainance, upgrading or that day server farm all kenna lock down, etc etc etc. So no one was in the office.
Its basically their word against TR’s admin. Even if TR manages to get their DC Chief Engineer to issue a written statement that the IP was indeed reverse traced to SPH servers, SPH will probably find some forensic experts or someone grossly over-qualified to debate that.
This article is for information and clarification only and readers should draw their own conclusion as to which party to believe.
此地无银三百两
hmmm…..
they say, they only never access TR during the time period,
so they actually do access TR outside the time.
is the problem lies with the timing between SG & PRC?
TR also like to act blur. always use “nowhere in our article… bla bla bla..”
Mr Geoffrey,
Why don’t you write it in a column in the ST print or online edition???? Why discredit yourself and your personal blog by posting such non-sense???? Oh well, who reads your ST blog anyway, huh?????
CPT said: It will be a never ending story. I wont be surprise sph will take extreme measure to protect themselves.
Protect themselves against what and who? TR is a small fly compared to SPH but then TR’s rankings is higher.
lego said: is the problem lies with the timing between SG & PRC?
Nop, they are the same.
@radon, NOWHERE means literally nowhere, not act blur. How to have somewhere actually there is NOWHERE?
Now we know the truth. SPH employees has difficulty in reading comprehension.
No wonder their articles are all flawed.
In this case, I actually believed in SPH because the Group really does have serious internal PC usage policy and it’s network is highly and tightly secured. I am also sure that the network or infrastructure security policies, and the software deployed are much more comprehensive that what TR is already using.
With advancing technologies, IP spoofing and other means of obscuring one’s identity, it raises questions if such attacks from just a list of IP addresses are enough.
TR should consult a security expert, instead of inferencing from a list of IP, and quoting a technical officer, which everyone else could have done.
@Lim Jun Jie, your arguement is flawed in a number of area. You claimed: I actually believed in SPH because the Group really does have serious internal PC usage policy and it’s network is highly and tightly secured. I am also sure that the network or infrastructure security policies, and the software deployed are much more comprehensive that what TR is already using.
SPH is a company that doesn’t deal with networking as its core business wheres TR’s servers is hosted in a DATA CENTRE, which deals with the net everyday.
You are basically comparing a LAN to WAN, thats world apart in terms of size, technology, infrastructure, security, etc etc etc. Unless SPH’s network is used for some high level covert undercover phising operations, their infrastructures and softwares compared to a DATA CENTRE is peanuts.
All TR did was to mention that someone or something using an IP address from SPH was ‘grabbing’ (as in reading or downloading) TR’s website. I don’t see the relevance of ’serious internal PC usage policy and it’s network is highly and tightly secured’ in this incident because the act itself would have got through any internal firewall or policies because its the equivalent of a user using the browser to visit TR’s site, but a hundred times faster.
You further claimed: With advancing technologies, IP spoofing and other means of obscuring one’s identity, it raises questions if such attacks from just a list of IP addresses are enough.
There was NEVER any attack nor did TR claimed that it was. Which IDIOT in the world would want to spoof a SINGLE IP to incriminate SPH? Initiating a DDoS is easier than spoofing an IP address.
You advised: TR should consult a security expert, instead of inferencing from a list of IP, and quoting a technical officer, which everyone else could have done.
You must be joking. Consult a Security Expert because someone is ‘reading’ your site?
Yeeloong said: If you start by leading with the definition of DDOS and go on to talk about an SPH IP being logged as accessing your website, please do not try to deny that you did not accuse SPH of launching an attack please. Just as Mark Anthony in no way suggested that Brutus had anything to do with Caeser’s assassination, but repeatedly begged the questio of Brutus’s honour instead… This would be called a leading question in court.
The same can be argued that the writer’s main intention of starting the article with the definition of DDOS is to ensure that the readers will understand what a DDoS is and make a clear distinction between ‘grabbing’ (as in reading) and DDoS, which is a form of attack.
Its pointless to go round the world arguing because there’s always a counter for any arguement unless you are quoting facts carved in rocks.
As it stands now, TR has released a snapshot of their cPanel’s access log and all I hear from SPH are ‘explainations’ and ‘denial’. My money is on TR.
IF and WHEN SPH releases a snapshot of their LAN’s traffic log or their IP usage log for the period in question to counter TR’s allegations, I will revised my opinion on the new evidence (if any), put forth.
Anyway, there’s really no big deal admitting to someone or something from SPH grabbing TR’s site because the articles is in the public domain INTENDED to be read anyway.
TR did not accused SPH of attacking or performing any malicious activity on its server, so why the need to die die deny if some elite from SPH is actually reading TR’s articles 100 times faster than others?
For the benefit of those unfamiliar with cPanel, I have reproduced below a recent snapshot of their server’s Apache access log with kind assistance from a staff of their host.
I have also included the snapshot taken earlier on the incident so that I can better explain and make comparison between the 2 logs.
#1 Snapshot from the incident
Noticed that the SAME IP address was logged to be simultaneously connecting to the server at a VERY SHORT interval, hence the IP was repeatedly logged immediately one after the other. This is the characteristic of a web grabber kind of software (it may also be sort of a SYNC attack but a SYNC attack would be grabbing the same content instead of multiple), certainly not any browser’s characteristic.
#2 Snapshot taken minutes ago
A normal browser reading the site would also show the same IP address as accessing the site but not as repeatative and as close in timing as shown in the snapshot above in #1.
The IP address of the reader will still be logged but the interval between connects would be greater and not one after another. Example here highlighted would be 202.156.13.246 and 202.156.12.228 which are accessing the site ‘normally’.
So whether was it a lone person casually surfing the site or a grabbing software was used, you be the judge.
Just one question, who is the person grapping content at the mentioned time? Can SPH or TR or anyone else able to provide the answer?
yawn, let move on to other issue rather then media war.
网民的眼睛是雪亮的
Mainstreetcitizen asked: Just one question, who is the person grapping content at the mentioned time? Can SPH or TR or anyone else able to provide the answer?
Your question is best directed at SPH who (assuming the IP is not spoofed as claimed) is the most qualified party to answer it. Every System Administrator will definitely setup the LAN or network in such a way that every inflow and outflow (or down and up) is logged.
Getting the culprit is peanut to them, getting SPH to admit and disclosing his or her intentions is harder than striking 4D since SPH is not answerable to anyone.
Thank you Sinkapore; and yes, obviously, my question is to whoever or whatever_ _ _ that has the answer, and surely, it does not take a genious to know whoever or whatever_ _ _ that has the answer.
Really time wasting in trying to find fault in SPH which already proven that they are not honest, why waste time with people who are not honest or whould be honest to come clean on explaining on this matter.
Mr Geoffrey Pereira,
Now lets down to the technicalities involved and examine the evidence before us:
TR Claims that:
1. TR’s server logs indicates that an IP address from SPH was grabbing or reading (NOT attacking) contents from the site on the material time and date stated in the initial article.
2. The ’source IP address’ inside the packets was reverse traced back to one that belongs to SPH and verified with http://whois.domaintools.com/203.116.231.234
SPH’s Defense:
1. IP spoofing is a common tactic used in a DDoS attack and in this case, the IP allegedly to be from SPH is spoofed.
Without considering which idiot in the world would want to spoof ONLY ONE IP address to grab TR’s site and incriminate SPH, lets get down to technicalities.
IP spoofing in this instance is technically OUT OF THE QUESTION and NOT APPLICABLE. Reason being:
a. The history of the HTML GET messages reveal that website contents are being retrieved in sequential and structured fashion consistant with the websites’ site map.
Basically, it means that the source computer knows which information to retrieve next, hence, it is aware of the URL links inside the TR web pages, and therefore receives the information direct from the TR website.
In IP spoofing, the attacker does not receive information from the attacked website because all ’source IP addresses’ have been spoofed, ie: faked.
b. From information gathered from the hosting provider, their hardware firewall uses Reactive Address Blocking (RAB). RAB allows the firewall to track an IP address as it traverses the network and subsequently associate that address across any number of violations.
One of the most useful functions of RAB is that it allows for monitoring of sanity violations, which is when an IP address breaks a strict conformity standard such as trying to spoof an IP address or modify packet flags. This makes sure that all packets coming from or going through the network conform to strict TCP/IP standards.
Additionally, the firewall is enabled with SynCookie support which plainly put, the server will send out a ’syn-cookie’ when the syn backlog for a socket becomes overflowed, as in the case of a sudden surge in traffic and multiple simultaneous connects. The cookie is used to interrupt the flow of sync transmissions with a hashed sequence number that must be correlated with the sending host (source computer). If the sending host does not validate against the hash then the tcp hand-shake is terminated.
In short, any spoofed packets will be dropped and terminated since a spoofed packet would not be able to validate the hash. So the IP addresses alleged to be spoofed would be long terminated BEFORE it even reaches the server, no possibility of it being logged by the server.
2. Neither did anyone in SPH try to “grab” TR material in a way that would load its server; nor did any SPH staffer launch any attack on the server.
Having dispel the suggestion that the IP was spoofed, the fashion by which contents are being accessed is consistent with search robots or a web grabber – ie – a website is archived so that a string search can be made. While this is perfectly legal, some software uses multiple sockets when downloading content, and CAN potentially hog resources from the web server and slow other user’s access.
It WILL HOG the server’s resources but in this incident, it didn’t because the software firewall on the server itself banned the offending IP address minutes into the action after the IP address exceeded 60 connects per minute, the threshold set by the system administrator.
Technically, if the server were to be NOT protected by firewall and had been configured poorly, a multiple of requests in excess of 60 connects per minute WOULD HAVE brought the server down and that would technically be classified as an attack.
Not to worry thou, the action would NOT have fallen under Section 7 of the Computer Misuse Act (CAP 50A) because TR’s articles being in public domain would have given SPH a ‘lawful excuse’ to grab its content.
Bottomline is, SPH will have to come up with a better explaination and denial than pinning the blame on spoofed IP, not all readers to TR is an IT idiot.
Sinkapore, maybe you’re right. Being the oppressive and communist-like face-above-all-else company that TR put it to be, SPH doesn’t want to admit their computer policy is so lax that someone was grabbing from TR using work computer.
Maybe, SPH already identified and punished the staff internally. And that staff is a singaporean and reservist TR fan. He was just syncing your site on his pda for offline reading.
So, TR just gave SPH a lead and a reason (misusing IT resources for non-official business) to get at a TR fan.
Can it be like that?
Oops…………….
@Ah Gong, If SPH had just NOT responded, this thread would have been closed after 14 days and ‘move on’ but since SPH have responded, I agree that TR should respond out of courtesy and dispel any false claims that may be put forth.
@rolleyes, it would be a stretch too far if one were to accept that a TR fan was actually syncing TR’s site with his PDA, dating as far back as 2008?
My point is very simple, no offence have been committed, no foul play suspected nor any allegations have been made at SPH for attempting any malicious activities. TR merely mentioned that someone or something from SPH was grabbing the site, PERIOD.
IF someone or something had indeed been downloading TR’s entire site, whats the big deal? Just silently admit it and keep quiet since legally its legal anyway. Why bother to wayang and deny and then try to brush it aside with lame excuse like spoofed IP?
MAYBE if I HAD NOT posted my findings on RedNano in the other thread, SPH might have an easier way out by citing RedNano was ‘working’ and indexing TR’s site.
My fault, it was my fault and for that, I apologise to SPH.
@sinkapore, maybe that fan is really fanatic? Or maybe he’s an IT nut who hasn’t read “idiots guide to configuring your PDA syncing”?
TR is dead sure SPH is not admitting to the access, which can only mean someone in SPH is grabbing. Can TR confirm their dead-sure grabber “culprit” isn’t a fanatic fan or an IT nut? Who knows? Maybe he’s already punished secretly in a conspiracy to avoid admission.
It’s anyone’s guess.
“Just one question, who is the person grapping content at the mentioned time? Can SPH or TR or anyone else able to provide the answer?”
In fact everyone in Singapore already know who this mysterious person is and he is very famous for taking the rap of whatever Shit PAP has created.
He is none other than NOBODY. NOBODY took the rap for Mas Selamat’s escape. NOBODY responsible for the Minibomb. NOBODY is responsible for TH and GIC’s stratospheric hundred billions investment loss. So it could be the work of NOBODY who responsible for TR’s DDOS. NOBODY is the best buddy of you-know-who and therefore very impossible to catch him.
@rolleyes, TR cannot confirm anything other than the simple fact that someone or something using SPH’s IP was grabbing the site.
Seems to me that TR may have made a big fuss about nothing due to their tech-ignorance (which, to be fair, they do acknowledge).
- “DDoS”? How’s it “distributed” if it’s from a single IP. But then it gets confusing to me… this article says TR never accused SPH from DDoS’ing, but the first article sure sounds like it. TR is at least complaining that a machine giving an SPH IP is requesting too quickly and slowing down your server.
- You had about 100 requests per minute, and you call that a massive DDoS attack? Maybe the IDS has overly sensitive thresholds, if it’s similar to some crappy AV and firewalls, they like to make noise abt every little thing to give the user the feeling of “wow, so many threats being stopped, this security-software I paid for is working well”.
- Was there any user-agent string? What was it?
- The screencaps I see leaves out important info: timestamps, user agent. Why? (Don’t just say “security”.)
- It could be someone using a browser “accelerator”, these things prefetch pages in the background to (try to) speed up browsing experience.
@Sinkapore, so there’s a possibility TR could’ve implicated a reader who was using a grabber out of fanatic energy (or pure ignorance), who’s now under gag order from the ever-oppressive SPH and has been issued a fine and a warning letter that’ll affect his prospects for good. Oops…………
Good work, but I prefer to block the “offensive” grabber quietly, and keep mum about how a simple single-IP grabber can put so much strain on a server to be noticeable like there’s DDOS. Somemore, the grabber is so easy to block. What need for those advanced features of firewall?
By the way, there’s a way to create a map of the site without detection. Just build the map over a few days. To cover all tracks, use many different proxies over the few days. Perl or other scripts can be used to do this automatically. Once the map is built, the person can traverse the site accurately with spoofed IPs. It’s troublesome but if someone is really all out to fool us, then where there’s a will there’s a way.
And how to detect spoof IPs which are valid?
Hi orange,
At least we did provide a snapshot of our server log. Our system administrator has advised against releasing the full information as it is tantamount to announcing to the entire world our server and firewall specifics – an open invitation to another attack.
Mr Geoffrey did not even bother to publish SPH’s and he still claim that the SPH logs did not show anybody or thing accessing our site on 1 November. The time was also wrong!
That’s why we propose SPH deal with our hosting company RTG Asia and its ISP China Telecoms directly.
Mr Geoffrey is right that we know little about server matters. All the information released as based on what our system administrators gave us. We did not add anything on our own.
Please get your facts right:
1. We did not accuse SPH of DDOS. We even DEFINE DDOS for our lay readers to prevent them from getting the wrong idea because the incident occurs one day after the DDOS.
2. It was Mr Geoffrey who claimed that we are accusing his company of launching DDOS against our site.
3. Mr Geoffrey insisted that nobody from his company visited TR during Nov 1am to 6pm, but NOTHING was mentioned about the period on 31st Oct 10pm to 1st Nov 1am.
The onus is for Mr Geoffrey to produce a snapshot of SPH’s server log to substantiate his allegations.
We have merely stated the facts of what happened: somebody or something from an IP address belonging to SPH was caught “grabbing” content from our site which has the potential to slow the server down.
How Mr Geoffrey came to misconstrue it as an accusation of mounting a DDOS attack is anybody’s guess.
The title of the article is self-explanatory. Do you jump to the conclusion based on the URL without even reading the article?
Ah ha!!! it is lky watergate. soon we will impeach the lky regime! down with lky regime.
Doesn’t the very pose that Geoffrey Pereira adopts for the picture in his blog scream: “I AM PAP APPROVED” – “Who the hell are YOU”?
@Orange, you apprently did NOT read the comments of other readers before posting yours.
As for your claim: – It could be someone using a browser “accelerator”, these things prefetch pages in the background to (try to) speed up browsing experience.
I have heard of Browser Accelerator and thats the same as a web grabber, they both download contents at a fast pace.
The issue of WHAT was used is irrelevant, as long as someone or something at SPH was used to download mass contents from SPH, all TR said was this.
SO, NO NEED to find excuses for what was used or how the connection was made because TR NEVER accused SPH of attacking its site, merely that they, SPH was grabbing TR’s content. Besides, SPH have already DENIED that no one and nothing did visit TR’s site which of course we know its not true based on information obtained.
The keyword is grabbing (downloading) and as long as someone or something at SPH did grab, thats enough. You should not be attempting to speculate what was used instead, if protecting SPH was your aim, you should be disputing the log and proving that someone or something from SPH DID NOT visit TR.
You further claimed: – You had about 100 requests per minute, and you call that a massive DDoS attack?
No, I wouldn’t qualify that as a DDoS attack and neither did TR but it will degrade server performance, the extent depending on the server’s config and hardware. Imagine 100 IPs each doing 100 requests a minute?
A single IP going at 100 requests per minute is harmless but coupled with other users who are also accessing the site at the same time, and if the site is a very busy site like TR. It WILL affect server performance.
I could go on and on and on about the Apache Web Server but I figured it best for you to do your own reading.
http://www.apache.org/
You wanted TR to release more information about their server and I quote: – The screencaps I see leaves out important info: timestamps, user agent. Why? (Don’t just say “security”.)
You obviously have no idea how hackers work. Even showing the minute details of how Apache was configured and what modules compiled would have opened the server up right down its core for a seasoned hacker. The snapshot DID show that one IP address belonging to SPH DID access TR’s servers and grabbed content, time and date have been given. All relevant facts to accuse SPH of grabbing is complete, no further details are needed.
What you have been show is only the GUI version of Apache Access Logs, not the actual server access logs (a couple), which would have shown you everything you have asked for, including but not limited to the socket used, process ID, number for requests from that SINGLE IP alone, time and date, the amount of CPU used, which php processes was run, mysql connects, the server load, all errors encountered AND MUCH MUCH MORE.
If you told your friends that you have a very beautiful sister with a superb body, would you have undressed your sister naked just to prove that she has a superb body if requested by your friends or would have told your friends to fiack off?
In my opinion, TR have already released whatever information thats needed to prove a point, the onus now is on SPH to dispute that and if the need arises, TR will consider releasing more, under advicement.
@rolleyes, one cannot deny that there might be a possibility that the person responsible may have been ’shot’ internally and quietly.
The firewall is there for an obvious reason which needs no further explaination and it is because of this firewall that this incident came to light. Its not the gravity for the downloading causing the load thats the point of contention here.
If the incident were to have been originated from some IP address that cannot be correctly pinpoint to an organisation, then the matter would have been left as it is since the firewall has already done its job.
However, the fact that SPH, the NEWS CORPORATE GIANT was actually grabbing TR’s content speaks volume and raises eye brows. It also raises the question of WHY THE NEED, thus we cannot blame TR for the article asking WHY.
You asked: And how to detect spoof IPs which are valid?
A spoofed IP can NEVER be valid because it is a fake IP. However, there are instances of masked IPs being used for other protocols, mostly chat software to register one as being coming from another IP, but it wont work on http protocol.
In a HTTP connection, there is the sending and the receiving part. A request is submitted to the server and then the server will send back an ACK followed by the requested content for the host (your computer) to receive.
Say for the purpose of discussion, you are now sitting on your computer browing TR’s site with the IP 123.123.123.123 and used some software to mask your IP as 231.231.231.231. The TR’s server would register your request as being from 231.231.231.231 and send an ACK to 231.231.231.231 followed by the content. Since you are actually from 123.123.123.123, you wont be able to receive anything.
However, the above scenario will not be possible at all given that the firewall would have dropped the handshake following the lack of the syncookie hash returned and acknowledge by the requesting host, ie: your computer.
Disclaimer: I am surely no expert and stand corrected.
It looks like the whole nation is up against SPH except for a faithful minority. I wonder what will be the outcome if the majority decided to boycott the newspaper for just 1 day. It will surely have a major negative impact as it has won many awards. I finally realised that the news reported are all slanted , biased and edited to suit some interested parties.
@Sinkapore, Dude, you got to really chill man. I was giving my two-cents worth.
I am writing because I know to a certain extend how SPH’s ITD (Information Technology Division) is like. And yes, it is a media company, and not only is it network tightly secured, it’s internal physical compound has unbelievable security protection.
TR is just a private news website, and people are launching DDOS attacks on it. Would you just imagine how many people are trying to get into a leading mass media company’s network?
In fact, the company’s network is so secured that some online websites they are operating are not hosted in-house. They also have advanced data centers to manage. Again, could you image how their data centers have to manage website that caters to millions of local and international customers. And with that kind of attempts to break it?
These are just some point’s that you might not have known.
The boycott will never happen because most people/company already subscript monthly. And beside is there an alternative to MSM now? its not all about politics news ok.
Let’s hope SPH sues TR for defamation so we can find out the truth.
@Lim Jun Jie, I merely remarked that you were NOT reading other readers comments before putting in your 2 cents.
I appreciate your information on SPH and their so-called DC but compared to DC run by China Telecoms, its an elephant and a mice.
It is safe to assume that SPH certainly does not have 1000s of servers running in their basement nor deal with the kind of traffic that a decent DC would have to deal with catering to traffic from around the world. SPh at most, is a Content Provider as opposed to China Telecom, which is an Internet Service Provider. Ths nearest comparison if one has to be made is to compare SPH to MediaCorp, certainly not to SingTel and definitely not to ChinaTel, the largest ISP in Asia (by infrastructure and asset).
Security wise, no comments there in view of my lack of knowledge and understanding on the matter but since you did comment on infrastructure and security for SPH, then you shuld not make comparison on a lone server which TR is on, in all fairness, the comparison should have been made by comparing SPH’s infrastructure against the Data Centre operated by China Telecom, where TR’s server is located.
Oh, the IP address in this incident is not belonging to their media server farm which you highlighted nor their in-house data centre used for the media server farm. Its assigned to their in house LAN and please DON’T ask me how I know, I just know.
@A Tan, if SPH does sue in Sinkapore, the will be no truth for all. Their lawyers will simply walk into the Judge’s chambers and get the Judge to commit prematurely without hearing any evidence. The trial would have been a wayang.
From what I know, TR will be either registered in Hong Kong, Navis or BVI. Lets see how SPH fairs with a lawsuit under these jurisdiction. Correct me if I am wrong but a state linked organisation or high ranking person have NEVER won any lawsuit out of Sinkapore.
Infact SIA lost in Australia?, LKY’s suit against Ex-President got thrown out of Court in Canada, Ex-Transport Minister Yeo DARE NOT sue the writers of Escape from Paradise, same goes for his wife, Helen Yeo, the lists goes on…
To what extent the article “/2009/11/02/sph-and-recent-ddos-attack-on-temasek-review/” *reads like* TR was accusing SPH of maliciously overloading the TR server, I think we can agree to disagree. I did read that article first, and I acknowledge that in subsequent articles and comments, TR and Sinkapore have clarified that TR is only complaining that something with SPH’s IP was requesting pages at abt 100 per minute. How many requests per minute was the (non-SPH) DDoS attacks then?
RE: publishing more of the TR logs,
timestamps and useragents are two columns that we need to see, these two pieces of info are not going help hypothetical-attackers attack you. please lah.
If there is anything else that you think you need to exclude, please go ahead.
Do you have request response times? It might show say 0.1s to respond during normal loads, and then spike to 1s during SPH’s automated requests. How about load averages? Run-queue length?
These serve to substantiate your claim that at 100 req/min, performance is affected… sure, by how much?
Actually, I am surprised that SPH had the logs to be able to say whether or not there were connections to TR.
If it turns out that it was an SPH staffer who was innocently using a browser with a web-accelerator, I don’t know who would lose more face.
Hi orange,
You miss the point altogether.
All our original article did is to present a fact that an IP address from SPH was caught accessing / grabbing content from our site.
We did not accuse SPH of launching a DDOS attack against us. Neither did we claim that our server is slowed as a result.
We just want to know who is the SPH staffer who is “grabbing” our content. That’s all.
Hi A Tan,
SPH has NO CASE to begin with.
They are the ones who are putting words into our mouths.
Since when did we accuse SPH of launching a DDOS attack on our site?
The gist of the article is: an IP address from SPH was caught accessing our site.
Mr Geoffrey DID NOT deny that.
Admin, you can say what you like abt what you intended to say.
The issue is whether a reasonable person reading yr piece comes to the conclusion that SPH was defamed.
If “The gist of the article is: an IP address from SPH was caught accessing our site.”, why so long-winded leh?
@singkapore, one can also not deny that the person silenced could’ve been a simple innocent reader of TR and so TR has unknowingly “murdered” their own reader out of paranoia.
Some IPs are reserved in the IANA standards. Example 127.0.0.1, 192.168.0.1, 10.0.0.0, etc. For internet traffic, these are invalid IPs. There are other unrelated criteria, but if the spoof IP is any of these, RAB will block them. Otherwise, it won’t know.
Yes, blind spoofing is much more difficult than one-way flooding, but still it’s not impossible with some analysis of valid traffic and some luck.
The syn-ack mechanism won’t help with DDOS. DDOS is not concerned with handshakes. It primary target is bandwidth, not number of connections.
Anyway, neither RAB nor syn-ack has anything to do with stopping the alleged grabbing “attack”. It’s simply stopped because it exceeded a threshold.
@admin, so once you know who your reader in SPH is, you’ll hang him to dry. Oops…………
Come on, this is crazy. So, if I work in Temasek company or civil service or statutory board, I’m an evil person that must be kept under your watchful eyes.
Lets have another scenario. If a SPH reporter wants to research a piece about alternative media, you’re going to cry foul because they’re investigating you? I thought you advocate freedom of the press?
This is so crazy coming from a grabbing that’s made out to be DDOS. Rolleyes!
Hi A Tan,
The issue here is whether a reasonable person who took time to read the article carefully will come to the conclusion that SPH did launch a DDOS attack against us when nowhere in the article did it say so.
Had we intended to mislead our readers into thinking that SPH had attacked us, we will not be so stupid to define DDOS in the beginning of the article.
In fact, the bulk of the article is on the SPH IP addresss “grabbing” our content during the stipulated period.
On the other hand, we can also counter-accuse Mr Geoffrey of defaming us by putting words into our mouth and bringing our site into disrepute.
Hi rolleyes,
Please go and read the post by Sinkapore on 6 November 2009 at 9pm.
This is not “normal” browsing. It is “grabbing” the content from our site from as early as 2008. What does the SPH employee/bot want our archived articles? That’s what we are keen to know.
Without phrasing the article a bit more provocatively, we will never be able to elicit a response from SPH.
Now that they have replied in a way, they should just reveal the identities of the employees who are responsible for the “grab”.
Now I wonder if there’s KPI for QC in TR. If the KPI is just Alexa ranking, then TR will slowly become LianHeWanBao online version. Heck care the objectivity, the accuracy. Just go for the BOOMZ Bravo, this grabbing attack episode certainly has the BOOMZ!
Hi rolleyes,
You are right: the most important KPI for us now is Alexa ranking, not that the rest doesn’t matter, but based on our limitations at the moment, they can be KIV for the time-being.
TR is an evolving project. It will gradually adopt a centrist position to reach out to the mainstream audience. It will not remain stagnant like this forever.
In the near future, such articles will never be published in TR again as we want to become a full-fledged online news daily instead of a mere blog.
This article is as objective as it can get. Everything is based on facts. We did not make any wild accusations against SPH in them.
We simply rebuke the points raised by Mr Pereira one by one. You should be questioning the objectivity and accuracy of Mr Pereira’s article and not ours.
@admin, what does he want?
Can it be offline reading? Or it’s simply a fault of his browser’s prefetching? Or he’s researching about alternative media?
Come on, you believe they want to use your articles for publish? We’ll all strike lottery if they can publish any of your articles. There’re so much grousing and grubbing and anti-this anti-that everywhere. Once all of that is removed through a very thorough deep cleansing, what’s there to publish?
Hi rolleyes,
That’s why we published the article to ask SPH what is going on.
This is our site and we were informed by our system administrator that a IP address traced to SPH was caught “grabbing” our entire site.
It is our right to find out what is happening. Neither you or us or anybody else can answer the question.
This is not “normal” browsing as Sinkapore has already explained earlier.
We reproduce what he posted:
“The SAME IP address was logged to be simultaneously connecting to the server at a VERY SHORT interval, hence the IP was repeatedly logged immediately one after the other. This is the characteristic of a web grabber kind of software (it may also be sort of a SYNC attack but a SYNC attack would be grabbing the same content instead of multiple), certainly not any browser’s characteristic”
Again, it goes back to the questions we asked in the initial article: why is there a need to “grab” the content from our site at one go when all of them are already in the public domain?
There’s indeed a whole lot of shit in TR – grubbing and anti-this anti-that everywhere.
This whole incident shows that if you all have been accusing the Govt of monitoring people’s IP addresses for “security reasons”, you should open your eyes and brain BIG BIG and realise that TR and all its anti-govt yakkers are also dictators, and are monitoring YOU, and your IP address too!
Come on guys, if you are truly anti-establishment, then please REALISE that TR has become an ESTABLISHMENT TOO!!! Its time to whack TR!!! Boomz their servers, woohoo!!! Bring ‘em down…
TR admin said:
> “We just want to know who is the SPH staffer who is “grabbing” our content. That’s all.”
OK then.
Just saying that the original article, “/2009/11/02/sph-and-recent-ddos-attack-on-temasek-review/” seems overly-dramatized, if your server wasn’t affected, etc.
If suppose SPH says they’ve concluded that it was due to normal browsing by a staffer whose browser does prefetching, would you be satisfied?
Would you further want to know the staffer’s identity?
Do you think it is within your rights to demand that info?
@rolleyes, I am really so tired from having to repeat al lthe technical stuff over and over again, so I am going to make this short.
RAB basically tracks an IP address through the entire network and monitor its behaviour, ie: everything it does and whether it was conforming to established TCP/IP protocol.
It also blocks spoofed IP and by spoofed IPs, that includes those that are reserved and invalid, examples of which you have already given. RAB recognises a spoffed IPs not by the IP address alone but by the characteristic of the packets and behaviour exhibited, so your claim that it cannot detect a spoofed IP not listed under the reserved IP range is not true.
The syncookie I do agree is ineffective against a DDoS but it does help in detecting spoofed IPs since a return hash has to be sent by the host computer to make the packet a valid one.
You said: Neither RAB nor syn-ack has anything to do with stopping the alleged grabbing “attack”. It’s simply stopped because it exceeded a threshold.
That is also true. The reason for the mention was because it the IP had been spoofed, it WOULD NOT have been allowed through the firewall at all and I wrote that dispelling the claim by the SPH journalist that the IP address was spoofed.
The said incident was stopped by a feature on the software firewall called connection tracking.
Orange said: If suppose SPH says they’ve concluded that it was due to normal browsing by a staffer whose browser does prefetching, would you be satisfied?
SPH have DENIED that any of their staff was involved in this incident. Would you have accepted anything else?
Furthermore, thats certainly NOT normal browsing unless one of their staff is someone in the soap opera, HERO.
Singapore, detection of IP spoof needs countercheck, either directly with the alleged attacker, or with internet devices along the route beyond own network. If (and only if) RAB can collaborate with external devices beyond its network, then by all means, yes. If not, an accomplished hacker can still spoof what we saw.
Now, the grouse is that Mr G is wrong. TR article say TR and SPH are talking different time periods, so they can obviously be both correct. No?
Aiyah, the very basic main point is: so what if someone is grabbing the site??? Rolleyes! How come your host provider so sensitive? But bravo to TR, this is really BOOMZ!
ok ok … Kit kat time. I have an observation:
Recently got this Indian man, was accused of supplying weapons to Tamil Tigers and the article by ST clearly also link him as a founding member of a opposition Party and the party support for a “wanted man”.
http://www.straitstimes.com/Breaking+News/Singapore/Story/STIStory_438662.html
Not long ago, got this Tan Lead Sane who has family problem
and was stabbed by wife of his brother, who happened to be a member of another political party.
http://www.straitstimes.com/Free/Story/STIStory_252816.html
Then of course there was the example of Steve Chia taken pictures which could have happened to any avid photographers. Again, ST did not failed to link him back as a NMP.
What I want to say is that when there are others who tell TR not to sensationalize this issue, must also consider this as the same medicine for ST.
“Grabbing” u mean “snaking” or downloading volumes of webdata using a program or are we frogetting that ANYONE who accesses a website inadvertantly “grabs” data, such as images and text, flash, whatever may be on the page being viewed, that is it is transferred from server to users pc whenever a user views a webpage, without exception.
So what the Feck is the fuss all about? seriously guys i like the TR but if this is where its heading u can add me to the same unsubscribe list as the Straits times.
SNAP OUT OF IT… for readers sake!!!
Should go to court and get things ironed out.