POODLE in the repeat

by with 2 Comments IBM Connections

POODLE in the repeat

Lately we have heard a lot of POODLE I have already blogged about it twice here and here at ibmconnections.com. But there is much more to tell about it so that the reason I am creating this article.

Nowadays when we talk about POODLE we have to specify which one (to make things easier 🙂 ). Nowadays we have POODLE SSLv3 and POODLE TLS. The one we call POODLE SSLv3 bites IBM Connections the most but POODLE TLS is easier to exploit. So for environments which are connected to the internet I advise you to be sure that both variants of POODLE are not exploitable at your implementation.

I will try to describe both variants in detail and how it/they can be solved. Lets first start with POODLE SSL v3

POODLE SSLv3

The news about this first POODLE exploit was brought to the world as a major problem while in fact the problem wasn’t that big if you ask me. The problem itself was there where still site’s out there who did support SSLv3. SSLv3 has shown in the past that it isn’t sufficient anymore as a secure protocol. I will try to describe why I think it wasn’t really such big news as it was brought by all news agencies. Normally when a clients connect to a server the clients tells the server the highest secure protocol it supports (this is what is generally is called  the ClientHello package). Within this ClientHello package the clients also send also all cipher suites (these cipher suites are the real data encryption packages) it supports for that version of SSL/TLS. The server then responds with 3 things ServerHello and Certificate and ServerHelloDone messages. Within the ServerHello message the server will send the protocol it is gonna use. This is decided on the protocol version send in the ClientHello message and the highest version the server supports. So for the real world (which is kind of dependent on browser behavior and version) this means in general that TLS1.2 is used. But with the POODLE SSLv3 bug if both parties supported SSLv3 it was possible by the attacking code to force the connection down from TLS1.2 to SSLv3 (with all the intermediate protocols because you can always fall back only one version at a time). So when this happens a user would still see it has a secure connection but it is over SSLv3 (and not the most sucre possible way between server and client as they bot did support TLSv1.2). The cipher suites used in SSLv3 are all general seen as insecure (only RC4 seems to be the most secure one) and fairly easy to decrypt unintentionally. So as you can see POODLE SSLv3 wasn’t the real problem if you ask me ! but that the server (and or client) still supported SSLv3 was the real problem :-). I added a screenshot to show how that traffic flows between the client and the server so you have and idea what happens on the network to build a SSL/TLS connection.

dFU6e

With IBM Connections we always had to enable the HTTP server with SSLv3 because the code within IBM Connections was still making use of SSLv3 calls to itself. The reason for that is that in the code they made use for example with code like this SSLContext.getInstance(“SSL”). Within the IBM JDK JSSE interface this SSL Keyword translated to make use of SSLv3. This was basically a combination of 2 problems , one that they still used a bit of outdated code within IBM Connections binaries , and also of the JSSE implementation in the IBM JDK that the keyword SSL still translated to make use of SSLv3. As said and stated by me, SSLV3 shouldn’t be used already for many years if you ask me 🙂

Well for IBM Connections we have a solution now (luckily) I will talk about that a bit later in the article.

POODLE TLS

The original attacking method of POODLE was made so that they made use of the padding bytes. Within SSLv3 nobody (wel thats a assumption) did a check on those padding bytes. Within the TLS specification it is specified to check on those padding bytes but most of the implementations out there didn’t do that (which makes the officially incompatible with the TLS’s RFC’s but okay 🙂 ) . So that makes TLS connections which make use of CBC cipher’s also vulnerable when they don’t check those padding bytes (and the funny thing about this is that most of the devs creating these encrypting/decrypting libararies still used the old functions/implementations within their code base which didn’t check those padding bytes). The IBM HTTP Server (IHS) which makes use of gskit for the data encrypting/decrypting part functionality of the SSL/TLS implementation where the rest of the world makes use of OpenSSL. So in the case of the IBM World we need to make sure if the CBC encrypting/decrypting routine checks on those padding bytes. Well I can tell you that older version of those gskit didn’t do that and are vulnerable to those POODLE on TLS attacks also.

But also for that there is a solution where I talk about a bit later in this article.

Solution POODLE SSLv3

Well as explained above there are 2 ways how this could have been solved. IBM did choose to fix it on the IBM JDK side instead of the original problem, the code itself. To solve it on the IBM JDK side we have to install a Interim Fix on top of WebSphere which updates the IBM JDK so that the SSL Keyword doesn’t force a SSLv3 connection anymore but a TLS connection. For the different IBM Connections version I will list the Interim Fixes needed.

For IBM Connections 3.0.1 and 3.0.1.1 and 4 you have to apply iFix PI28934
For IBM Connections 4.5 you have to apply PI29575
For IBM Connections 5.0 you have to apply PI28920 or PI28437

The strange thing is that for IBM Connections 5 (which is running on WebSphere 8.5.5.x) we have 2 iFixes available. Personally I have tried to use PI28920 and couldn’t get that iFix to work but I got some reaction from IBM Connections community (for instance from Klaus Bild) that they where able to use to iFix to get SSLv3 switched off. I personally have used PI28437 several times now with success. PI28437 (SDK6 (J9 2.6) SR8 FP1) also installs a newer version of the JAVA SDK then the PI28920 (SDK6 (J9 2.6) SR7 FP1) so my advice is to make use of the PI28437 iFix instead of the PI28920

After you have installed this iFix you can safely disable SSLv3 on your IHS (IBM HTTP Server). This can be done with the SSLProtocolDisable SSLv3 in your vHost configuration part of the config file(default httpd.conf) for your IHS.

Solution POODLE TLS

As described above also the gskit implementation has some problems regarding the checking of the padding bytes. To tell the gskit implementation that we want a strict checking on those padding bytes we have to add the following setting
SSLAttributeSet 471 1 to the config file (default httpd.conf) to all VHOST sections which have SSLEnable in them. This setting can only be applied to the following IHS versions.

  • 7.0.0.33 or later
  • 8.0.0.9 or later
  • 8.5.5.2 or later

For other release version which can not be upgraded, for whatever reason, to the above specified versions you can install any of the following iFixes as what I understood from this technote to make use of this SSLAttributeSet 471 1 setting.

  • PI17025
  • PI05309
  • PI08502
  • PI09443
  • PI13422
  • PI19700
  • PI26894

So I hope this makes everthing a bit more clear in the POODLE jungle out there 🙂

If you have any addition to this story please let me know so I will update the article.

IBM Connections CSS issue

Avoiding CSS cache issues in IBM Connections 4.5

by with 2 Comments Misc

Many IBM Connections 4.5 admins may have seen this kind of thing in the past:

IBM Connections CSS issue

Users see the page structure in their browser but with formatting badly mangled.

Often clearing cache and refreshing page, or else just waiting a few minutes and trying again, appears to resolve it.  The issue is browser- and platform-independent, and is very intermittent though appears to be seen most commonly first thing in the morning or after server restarts.  The browser and server logs seemed to indicate that a number of files that should be downloaded to the client could not be accessed.

Seen it?  I had in several environments, but after much diagnosis and the absence of anything more concrete, had put it down to server or network loading.

Well it turns out that there is an inherent caching issue in the Connections 4.5 platform that is causing this issue:

Issue Reported 

Major inconsistencies and distortions seen with the UI on Production. Only a restart of the production servers helped resolve this issue only for them to see a reoccurrence every 2-3 hours. The UI gets distorted at this point and all users are affected.

Root Cause for Situation

The Ajax Framework in Connections caches generated Javascript resources in memory for a duration of 4 hours: The Common application also leverages the application server’s dynacache instance, by declaring in its cachespec.xml that responses of the _js and _style servlets to be cached for 30 minutes.

There is therefore a window of 3 hours 30 minutes in which the application server’s dynacache misses the cached resource, because the 30 minutes cache period elapsed, and requests now hit the application server directly. In these circumstances, if the client sends an If-Modified-Since header that matches the last modified time of the generated resource in the “in-memory cache” (which lasts for another 3 hours and 30 minutes), the Common application responds with a 304 Not Modified and an empty response body. This is fine for the client that issues the request, which reuses the response cached by the browser.

The flip side here is that the 304 Not Modified response is cached by the application server in the dynacache for the next 30 minutes. The application server will serve this stale response to every client that requests the same resource with the same values of components declared in the cachespec.xml.

Ideally, 304 responses should never be cached (by dynacache) which is exactly the opposite of what was observed by the customer. At that point, the only way to clear the application server cache was to restart the Common application which is what was being observed had to be done whenever the customer faced this issue.

Specific Action carried out to Fix 

Apply the custom JVM property com.ibm.ws.cache.CacheConfig.filteredStatusCodes, and set it to “304 404 500 502”. This instructs the application server not to cache responses that have one of the response statuses included in the space separated list. Essentially, the 304 responses are not cached. This customer property required an iFix PM54521 on WAS which needed to be back ported to the WAS 7.0.0 FP21 that the customer’s environment had. (SM: this iFix is in WAS 8 and thus is not required in Connections 4.5 environments)

Once this was carried out, this issue has now been fixed and the customer’s production system is stable.

From a more permanent standpoint, below are the approaches considered
1. To have the JVM parameters and the WAS iFix as a part of the IC installer to exclude the 304 response codes
2. Code change to fix failing test of Etag value sent by client as If-None-Match header enclosed in double quotes, compared with Etag not enclosed in double quotes. We suspect that the performance fix (RTC 60458) to enclose Etag headers in double quotes to accommodate Edge server caching introduced was incomplete. When coded, would reduce the possibility to incorrectly generate 304 responses.
3. Understand whether it is safe to not use dynacache to cache responses from Common, and look to handle caching entirely in the application space without relying on dynacache for caching responses.

So just in case that isn’t entirely clear, the fix is as follows:

  • In the ISC, navigate to Application servers > *server* > Process definition > Java Virtual Machine > Custom properties
  • Click ‘New’.  Set Name to be ‘com.ibm.ws.cache.CacheConfig.filteredStatusCodes’ and Value to be ‘304 404 500 502’.  Add a description as appropriate. Click OKNew configuration variable for Connections CSS issue
  • Repeat for all your Connections-related servers
  • Save the configuration
  • Resync your nodes
  • Restart the WAS servers

In my experience this has resolved the issue for all environments that have experienced it (there have been 4 or 5 so far), and thus I am setting it for all new installs. It may well be that IBM will take steps to prevent this using one of the methods discussed in the quote above, but for now this is still the best solution we have.

A huge thank you to the brilliant David McCarthy in IBM Dublin for helping us track down this nasty issue!

A fix for the Chrome header issue

by with No Comment Misc

Stuart blogged back in January about a known issue with downloading files from the files application using Chrome.

Connections Chrome issue

IBM do not officially support Chrome with Connections 3.0.1 – but our good friend & IBM Champion Sjaak Ursinus has devised a work around.

Header edit Content-Disposition ^(.*)creation-date=(.*);smodification-date=(.*);$ “$1creation-date=”$2″; modification-date=”$3″;”

ensure the mod_header module is uncommented httpd.conf

save and close the file.

Restart the HTTP server to pick up the change

Sjaak’s full explanation can be found on the Connections Forum.

[Originally posted on The Connections Blog by Sharon Bellamy where it had 16 comments]

An issue with custom themes and communities

by with No Comment Misc

I have had a PMR open for some time relating to a bit of a known issue with Custom themes and Connections communities.

There was an issue where you would create a custom theme :

commTheme1

In my case I also customised the coloured community themes also:

commTheme2

When clicking on forums, blogs etc the theme was stripped out displaying:

commTheme3

It appears there was a step missing from the wiki!!

http://www-10.lotus.com/ldd/lcwiki.nsf/dx/Defining_a_community_theme_ic301

Step 1.d – d. Remove the file theme.css from the corporateTheme directory

In my case I had based my custom theme on the default theme and had used it as the default Community theme (which has no theme.css), so all communities with the default theme appeared correctly.

BUT, I had also lightly customised the coloured community themes – removing (or renaming) the theme.css from each coloured theme has resolved the issue.

Thank you mr Kieran Reid for investigating and confirming this was the issue.

[Originally posted on The Connections Blog by Sharon Bellamy where it had no comments]

Issue with Connections media widget timing out

by with No Comment Misc

Whilst building a new Connections environment for a customer we noticed a strange issue when uploading large files to the media gallery.

Connections media gallery timeout
Initially I thought it was related to the size of the file, but the same file will upload to the Connections files application without issue. There is very little errors in the SystemOut.log for the Connections server, so I was baffled.

A PMR was opened and the very helpful Mr Dave McCarthy was the PMR owner and we then started on our investigation. During the testing I noticed that the uploads appeared to timeout after 20 mins, exactly 20 mins. After some experimenting on 4 different Connections systems, it was confirmed that it was a timeout, regardless of the file policy or file library size. So not many people are on a internet connection that may take 20 mins to upload a video, but we know it is an issue as the customer I was building the system for confirmed this.

After much digging through existing PMRs Dave was stumped, so the PMR was passed up the chain to the development team. Who confirmed very quickly that their is a setting in the config.js which is buried in the news ear file which has a time out set to 1200 sec (20 minutes)!! Change this setting and as if by magic the timeout issue is resolved.

To change the time out setting do the following :

/installedApps//News.ear/qkr.lw.war/WEB-INF/pages/js/config.js

Find this section, (Line 450), that specifies some timeout values,
including one for upload that is set to 1200 sec (20 minutes):

timeout: {
request: 60,
update: 200,
upload: 1200,
retrieveFiles: 100,
userSearch: 200,
userTypeahead: 10
},

Raise the upload value from 1200 to what is needed to complete the large file upload on your connection speed and save the file. Then restart the News application to make the change effective. This file should be changed on the primary Connections node if you have more than one and sync the changes around the other nodes.