To begin with, the server component is still looking for the wrong properties in registry ("isSSL" and "SSL_Port" rather than the actually existing "isSslEnabled" and "ssl_port"). This means that, whichever option you select in Configuration Utility, the component will default to ssl-disabled.
If we try to trick it, by manually adding the properties it's looking for, we get the following errors and the service shuts down:
[...] [FM] [ERROR]  [oracle.FM.HSX.SERVER.oracle.epm.fm.hsxserver.HsxServer] [...] [SRC_CLASS: oracle.epm.fm.hsxserver.HsxServer] [SRC_METHOD: startThriftServices] An error occurred while starting HSX_SERVER_NAME service.[[
org.apache.thrift.transport.TTransportException: Either one of the KeyStore or TrustStore must be set for SSLTransportParameters
at org.apache.thrift.transport.TSSLTransportFactory.getServerSocket(TSSLTransportFactory.java:99) at oracle.epm.fm.common.BaseSeviceManager.getSSLServerTransport(BaseSeviceManager.java:121) at oracle.epm.fm.common.BaseSeviceManager.startService(BaseSeviceManager.java:68) at oracle.epm.fm.hsxserver.HsxServer.startThriftServices(HsxServer.java:64) at oracle.epm.fm.hsxserver.HsxServer.init(HsxServer.java:43) at oracle.epm.fm.hsxserver.HsxServer.main(HsxServer.java:27) ]]
[...] An error occurred while starting HSX_SERVER_NAME service.[[ oracle.epm.fm.common.exception.HFMException: EPMHFM-66019: An error occurred while starting HSX_SERVER_NAME service. at oracle.epm.fm.hsxserver.HsxServer.startThriftServices(HsxServer.java:70) at oracle.epm.fm.hsxserver.HsxServer.init(HsxServer.java:43) at oracle.epm.fm.hsxserver.HsxServer.main(HsxServer.java:27) ]]
As you can see, it complains about missing keystores. This was always a possibility, since we never got a chance to specify such a thing during configuration; however, most SSL-aware software usually ships with demo keys preconfigured for tests, so it was not unreasonable to expect HFM would do the same. No such luck!
Hopefully both problems will be comprehensively addressed sooner rather than later. As it is, secure 126.96.36.199 implementations still have to rely on both HFM web and app components running on a single box with a tight firewall, which is far from ideal.
Bonus: Peeking Under The Hood
These errors lift the veil on some of the development strategies Oracle employed to remove hard dependencies between HFM and Windows-specific DCOM.
It looks like they are using Apache Thrift, which can be roughly described as a serializer library to auto-generate network APIs for existing programs. They basically dropped DCOM network code and replaced it with a simpler network interface built on (if my speculation from the previous post is correct) some Weblogic subcomponent; this interface uses Thrift-generated serializations to pass data between appserver processes and the ADF-based web application.
In theory, such an approach could make it easier for third-party components to open connections directly to the HFM appserver, bypassing the need for specific client libraries (with their byzantine requirements) and just using Thrift to figure out necessary calls. However, I'm not really an expert in this field and I don't really know what this effort would entail; at this point, your best bet for custom integration is still likely to be the official HFM SDK.