Wednesday, January 18, 2017

DevOoops: In-Memory Databases (Redis) Part 2

Doing part 2 first as the altcoin mining stuff is interesting with the mongoDB/elasticsearch ransomware stuff currently going on.

A redis developer dropped an interesting piece of info here

“However, the ability to control the server configuration using the CONFIG command makes the client able to change the working directory of the program and the name of the dump file. This allows clients to write RDB Redis files at random paths, that is a security issue that may easily lead to the ability to run untrusted code as the same user as Redis is running”

He goes on to show how someone could echo over SSH keys and use the config command to write them to the appropriate place if you have permissions.  He used a key name of "crackit" so I thought I'd see how prevalent it was....I checked a few and saw it a good chunk of them.

go go shodan

I did find something interesting while looking thru some open redis boxes.  I found:

A cron job? running a shell script. Can you do that from Redis???

What's in the shell script?!

alt coin mining! sweeeeeet.

I had no idea what an XMR is but I wanted to see how this person was doing with the money making. Thankfully you can just query the payouts for any XMR address. So I did:

They've made around $20,000 USB in BTC. I guess crime does pay :-)

To satisfy my curiosity started a miner up on a linode and was getting around 60 H/s. This person is cranking out 70 KH/s, so they have a few boxes working for them.

Extending the idea that a good hack yields plenty more I stumbled across this gem. with several different ways to get code exec on redis.

I created some gists from the previous link in case the post disappears.


Monday, January 16, 2017

DevOoops: Client Provisioning (Vagrant)

Notes from the 2015 Devoops Talk

Vagrant used to ship with a default keypair and was difficult to rotate.

**fixed with new versions of Vagrant. Finding hosts using the default key still pretty likely.

Did you change your SSH keys?

Default Credentials

root/vagrant  vagrant/vagrant

No pass to sudo :-)

Scanning for the default key using metasploit (ssh_login_pubkey module)

Identify real from fake by ssh version scan

Log in with private key

Friday, January 13, 2017

DevOoops: Client Provisioning (Kickstart Files)

Notes from the 2015 Devoops talk. Posting it so i can remove it from the slide deck but still refer to it.  Also relevant from a common problems with devops theme.

Kickstart Files

3 ways to set root password

1. Enter during installation

2. Crypted hash in the kickstart file
“rootpw --iscrypted”

3. Clear text in the kickstart file
“rootpw --plaintext”


 Kickstart Files Takeaways

Don't leave these files in open shares

Use the crypted password option for files

Have a process to change the password after initialization

Rotate the initial root password regularly

Thursday, January 12, 2017

DevOoops: Client Provisioning (Chef)

Notes on Chef from the 2015 Devoops Talk. Posting it so i can remove it from the slide deck but still refer to it.  Also relevant from a common problems with devops theme.

Chef allows you to define the state your servers (local or cloud) should be in and enforces it.

Web Interface

Environment Leakage


knife is a Chef command line utility. The credentials are stored in data bags. Credentials can be encrypted.


$ knife data bag list

Chef/knife (encrypted data bag)

Chef/knife with path to secret file

Chef Takeaways

Be aware of what you put into chef recipes

Protect secrets/passwords

Info on securing chef:

Wednesday, January 11, 2017

DevOoops: Elasticsearch

Notes from the Devoops talk on Elastic Search

Elasticsearch Provides a distributed, multitenant-capable full-text search engine with a RESTful web interface and schema-free JSON documents.

*GET request to port 9200 will show version
"version" : {
"number" : "1.2.4"

No Authentication (initially)

Can search stored data via HTTP API

Update data with PUT request

Join an open cluster and receive all data

RCE prior to 1.2.0 (CVE-2014-3120)
RCE prior to 1.5.0* (CVE-2015-1427)



Searching via curl/browser is cumbersome...Kibana FTW

Edit config.js to point to open Elasticsearch

Open index.html in local browser or host on a server

Viewing the content of the document

Import your own data and visualize

Elasticsearch solutions:

Apply authentication if possible

Segment elasticsearch from Corp (and the public in general)

Be aware of the data you put in elasticsearch
-->anyone can search it

Logs Logs Logs


Tuesday, December 20, 2016

Hacking Complex Systems

Back in the day, you could download a piece of software, reverse engineer / fuzz it, find bugs, notify the vendor, post on Full Disclosure, watch a patch come out, and move on to the next bug.

These days systems have become very complex. A system might include:
  • A HID (Touch screen, keyboard, other devices)
  • Data Inputs (USB key, Bluetooth, Wireless, Satellite, Cell)
  • Firmware (BIOS or other embedded aspects)
  • OS
  • Applications (both OEM and 3rd party)
  • Media Servers
  • Other control systems
  • Telematics interfaces

This collection of components may be very expensive, on the order of 250k in some cases, or say 10-20k for a car. These components may be made by multiple different vendors, all with NDA's and MSA's between them.

This whole system is then certified and tested by numerous bodies such as FAA, TSA, NHTSA, NAFTA OEMs, Avionics Manufacturers such as Boeing and Airbus, Airlines, etc. There may be regulations and requirements around patch cycle timing, disclosure, and legal.

How in this context, can these systems be tested for security issues in a reliable and effective manner? Right now there are several ways this testing occurs:

1.) Via Testing Contracts.

The vendor puts out a bid or otherwise engages a 3rd party security company to test the system. NDAs and MSAs are exchanged, access to the system is provided, testing performed, and results delivered. Fixes are developed and pushed out according to the schedule and requirements agreed upon by all the organizations outlined above.


Vendor has a level of protection that their reputation won't be tarnished via media disclosures, their IP stolen, etc. Vendor has some assurance the testers are competent and there is a level of service expected.


This process is not public and people outside this framework have little to no insight into what is going on, how testing is done (or if), who is doing it, what fixes have been put in place. etc. This also limits the number of bright people who can see and test the system, almost ensuring that some bugs will be missed.

2.) Bug Bounties.

Vendors make some aspect of the system available publicly for anyone to test and pays a bounty for valid vulnerabilities discovered. In some special cases the vendor may make an entire system accessible for a limited amount of time. (Time limited to offset the cost of the system)


Process is public and many eyes are on the product. Raises the exposure of the product to new testers and approaches. Builds a level of trust in the vendor and assurance that the vendor "cares about security".


Costs the vendor time and effort and often produces little more than noise, or bugs already known about through internal testing. (I'm basing this on my personal discussions with vendors in the real world). Testing quality is often very low. Often the holistic system cannot be tested in this way, only components.

3.) Rogue Testing.

This is sort of where I came up in the industry initially before moving more into 1.) above. The way this works is that a researcher (or team of researchers) and/or a security company gain access to a system in some way. Examples include buying a piece of the system on eBay or in the case of publicly available systems such as avionics, testing it live. A car could be bought as well. This is sort of a black box approach as access to all the back end systems, telematics, source, .etc. will not be available.


A researcher can sort of do whatever they want without constraints. A security company can leverage this for media attention (marking / sales), and it drums up interest for conference talks. Real bugs are found this way and the vendor is technically notified, either as a heads up by the finder or via the media.


No trust is developed between the vendor and the bug finder. In fact the relationship is almost always adversarial by its nature. The public receives an unclear picture of the true threat. Do they trust the finder who is often over hyping to get attention or do they trust the vendor who has a material interest in under hyping and disproving the bug.

I'm sure I am missing other pros and cons to each of these, so please feel free to send me ideas. I'm also sure there are other approaches to testing which is why I am making this post. Here are some questions to consider:

  • Are complex systems such as avionics and automotive substantially the same from a testing perspective as windows hosts or endpoint software?
  • Is live testing on a passenger vehicle really the right way to do security testing?
  • Should only professional security companies with contracts in hand be allowed to test?
  • Are bug bounties in their current incarnation really effective for these types of systems?
My answer to the above questions is probably no.

I propose that we, the security community, collectively try to come up with a better way or framework for doing this. Any ideals will be appreciated and considered. Are you already doing something in this arena that is better than what I have outlined? Is there something you thought would work but have not gotten traction on it?

I'd love to hear from vendors, sec companies, and researchers alike.

I also propose that unethical behavior in our industry be called out. Every time a company brushes up against extortion, over hypes a bug, or claims credit for non-employee's work, just for short term sales, it damages the credibility of all of us and makes our jobs harder. Lets require the best of ourselves. Security has become huge, and is about to become bigger. Over the last year think how many times hacking has been in main stream media. Now contrast that with 10 years ago. This is an industry that is about to explode. Do we really want to be found wanting when the world finally is ready to take us seriously?

Sunday, November 6, 2016

On Nation States and Sophistication

Thomas Ptacek made an interesting tweet today about Nation States, and if the term has any meaning, which got me thinking. In light of the numerous breaches that have been occurring, affecting both commerce, government, and potentially even elections, I decided to take some time to write down my thoughts on some of the subjects that come up when these events occur.

First lets talk about victim psychology. When a person or an organization is hacked, they go through similar emotions to victims of any crime. There is shame and guilt, anger, a desire to "do something about it" and to make sure "this can never happen again".

There is also a feeling of need to justify why the breach occurred; "How could this have happened?". Also important to take into consideration is the mindset of investigators. They like catching the bad guy, uncovering the mystery, beating the attacker at their own game. However, its not exciting to investigate or report on a dumb or simple attacker, who did nothing exceptional.  Because of this, people are highly incentivized to look for indicators or confirmation that the attacker was some how exceptional. This makes it more ok that they lost and were compromised and it makes investigator's jobs more exciting. (I know, I've been there.)

Lets talk about a word that gets thrown around a lot by media, government, and intrusion investigators: Sophisticated. This term seems to imply a sort of evil genius, someone who did such outlandishly amazing feats of hacking that there is no way your average organization could have stopped or detected them.
    "We got broken into!"
    "How could this have happened? Didn't you do your job? Didn't we spend all that money on defenses?"
    "Well they were VERY sophisticated"
    "Oh well ok then, nothing we could have done"
This is both true and not true. Defenders really have little hope of keeping attackers out (sophisticated or not), even if they do most everything right. Worse, what it takes to do everything "right" is very expensive, the talent to do so is scarce and hard to find, and the technology involved changes rapidly. In actuality, most breaches aren't really that sophisticated, depending on how you define the term.

In the interest of giving you background let me say I've personally investigated a large number of breaches, and my team even more. I've conducted an even larger number of attacks myself for the purposes of security, even some I would label as sophisticated, so I've worked on both sides of the issue. We have seen breaches which have been verified government attacks (verified by direct human means among a number of other things, giving me high confidence, not just by an IP address or a foreign word in code), organized crime, talented blackhats, vandalizing kids, corporate competitors, and malicious insiders. In all of these investigations, very few did anything that I would personally classify as sophisticated.

Its probably time to define what I mean when I say sophisticated. To me an attack requires a number of elements in order to be considered sophisticated:
  • Is targeted rather than opportunistic. This means someone set out with intent to attack the organization rather than stumbling across a random vulnerability they could take advantage of while looking for anything random to break in to.
  • Is planed. This means someone didn't just say "Let me throw a bunch of attacks at this organization I don't like", but rather put together a plan for getting in, staying in, targeting data or capabilities, getting information out, and hiding their identify. There are clues during an investigation that help you see the difference between a planned attack and a haphazard one.
  • Uses unique technology or technology in a unique way. Unless there is an intentional deception going on, sophisticated attacks don't use off the shelf hacker / auditor tools. They typically use high quality (reliable) custom tools, or tools available as a part of operating systems in unusual or unintended ways.
  • Involves malware that obviously took a team to write. There are very talented individuals who can write custom tools, but most often sophisticated tools are written by teams of specialists who break up and take on different features or capabilities of the tool. If you are looking at code, you can often tell this.
  • May involve anti-analysis or anti-investigation techniques, or target investigators directly.
  • Long term persistence. Random hackers usually want to get in and get out. Sophisticated hackers have more confidence in their tools and abilities, have more resources, and tend to stay a while to extract all the value from the compromise they can.
  • Involves data theft beyond purely financial (not just Credit Card numbers) or impact on critical business functionality.
You may not agree with all of my criteria, but hopefully we can agree on the fact that there must be SOME criteria for classifying an attack as sophisticated. I should note that I have seen sophisticated attacks violate any number of the above requirements. Individually none of them certify that an attack is sophisticated, but if taken all together or in majority, they typically do.

Now lets tackle this term "Nation State". As it turns out, this is much trickier than you might suppose. In the context of computer attacks, most people might define this as an attack carried out purposefully by a government against an organization, individual, or other government. People like very clean, clear cut, black and white definitions so that we know who the bad guy is and who the good guy is. Unfortunately the world doesn't work so simply. I would like to propose that a Nation State attack could be one which incorporates any of the following:

  • A highly talented individual hacker, hacking mostly alone. This person may be monitored by a government, either passively or actively, who benefit from their non-directed actions.
  • A private, non-government employed, hacker group, whose activities get co-opted by a government.
  • Defense contractors and other private business who supply tools and talent, knowingly or unknowingly, to a government and it's interests.
  • Military staff whose purpose is typically more one of disruptive capability, but may collaborate with any of these other groups.
  • Civilian government staff, comprised of intelligence professionals and others, who leverage cyber attacks for intelligence purposes.
  • Any of the above who are acting for other purposes, such as personal financial benefit, not under the direction of a government, but perhaps using government tools and resources.

In light of the above, an attack may use known Nation State tools, but could be carried out by someone who either captured or stole these tools, or is using them on the side, without permission, for personal gain. Imagine, for example, a country where you don't have to be a government or military employee to hack for the government. You are given access to the best tools and training, covert networks, and target lists. You see a lot, you know where money and secrets lie. Then government polices change and your services are no longer needed, or are less needed. Maybe you took copies of the tools home. Maybe you still have accounts or access to jump stations and command and control servers. It might be tempting to leverage this to make a little money on the side. Many investigators will see the IPs you are coming from, the tools you are using, your language preferences, and make the Nation State determination, even though this is clearly not the case. I would venture to say that unless you have the following, attribution is shaky at best:

  • Initial entry vector
  • Copies of the tools used and high end reverse engineering capabilities
  • Full packet capture and netflow of the attack
  • Comprehensive logs
  • Forensic images of compromised hosts
  • Threat Intelligence sharing across multiple organizations or even countries
  • Human intelligence (ex. confessions from the attacker, group infiltrators and spies, people assets in law enforcement or other investigatory organizations)
  • Hack back. Access to attacker systems and infrastructure, or even national network infrastructure in order to monitor the actual sources of attacks.

Now for most private companies, the above is fantastically too expensive to maintain, the talent too scarce, and national laws too unfriendly, and from a business standpoint it doesn't make sense to bother. There are of course exceptions, and multiple companies working in an industry and cooperating with government or law enforcement might get close.

It is also important to say that Sophisticated attacks aren't necessarily Nation States, and Nation State attacks aren't necessarily Sophisticated. Let me give some examples.

I know the story of an individual, who when they were around 14 years old, researched and developed a suite of what I could call sophisticated tools, including hardware firmware persistence, air-gap jumping, and ex-filtrated data analytics. This person then extensively planned out an attack against a government in a country other than their own, and conducted it over the course of around a year. They did this primarily for the intellectual pursuit, and to gain access to specific technologies to help them in further attacks down the road. This attack was eventually discovered, and classified as a Sophisticated Nation State attack by the investigators, when in fact it was a talented kid, acting alone.

I have personally investigated attacks verified to be directed, executed, and managed by a foreign government, which used straight up off the shelf and publicly available hacker tools, in very obvious and even clumsy ways. The attack was successful, but was caught and stopped pretty quickly and was only determined to be Nation State because an outside organization had proof obtained by other investigatory means.

I have also seen (and performed) attacks where a couple of US based blackhats will create or purchase a 0day, modify it, build a suite of custom tools developed with foreign language packs, anonymously purchase or compromise hosts in a foreign country, and conduct a campaign against an organization in the US which has all the hallmarks of being a Sophisticated Nation State attack. But it was actually just us performing an attack simulation for a client, or a group of non-government affiliated blackhats using deception to hide who they are.

A sophisticated attack can be an expensive one (although in the case of the 14 year old maybe not so much). High end attack tools, 0day, etc. are very valuable and take time to produce. You don't want to burn these tools for no reason. This means there is incentive to use the least sophisticated and cheapest means to accomplish the following goals:

    - Gain access to a target.
    - Move freely in the target environment.
    - Maintain access as long as desired.
    - Avoid detection.
    - Transfer data at will.
    - Frustrate investigations if detected.

In many cases, the detection aspects in the list above don't matter, even for nation states. Sometimes if you can get in and get what you need with little to no repercussions, you don't care if you are detected a month later.

If you think about it this way, then the ideal situation might be to watch while a non-affiliated 3rd party performs the attack, using their own tools, and you simply reap the access or data rewards without getting your hands dirty.

The goal of this post was to point out that when you hear the terms Nation State or Sophisticated attack thrown around by the media, or companies who sell investigation / threat intelligence services and tools, you might hesitate before taking it at face value. I'm not saying these organizations are being intentionally or maliciously misleading, just that their criteria for making those statements may be too lose and ill defined.

Val Smith