Who let you in here?!

Scott Percival

- wanted to become a proper RE - the kind that could have any problem thrown at them - annoying that the material online was very narrow - so started trying a whole bunch of things, failed a lot - knew my approach was wrong, but didn't know what question to ask - eventually started seeing results - saw the common threads connecting the techniques of the same type - and that for all the most difficult parts of RE, they were progressions of an easier skill, and practicing the easier skill would build your confidence and experience for tackling the harder one - immediate reaction was "how come no-one talks about these hard skills as a continuation of the easy skills" - more high quality research available at your fingertips than ever before, - when it comes to getting started it's still focused on the view from the top of the mountain rather than the path up the side - so today I'm going to talk a bit about the different types of analysis, and rules of thumb, with luck they will give you the foothold you need

Aims of reverse engineering

Gain an understanding of a system's inner workings
Extract information
Create or improve a new design
Make changes or extensions!

- to start with, figure it's good talk etiquette to lay out a definition of what we mean by RE - RE is a discipline focused around gaining an understanding of how a system works. - we'll be talking about software, as it's the best fit for this audience, but it could be anything! - custom microchip, mechanical process, ice cream flavour - and usually we embark upon reverse engineering in order to achieve a certain goal - common one is just to extract something from inside the system, such as an encryption key, or assets, - alternatively we can use what we've learned to guide or improve the design of a new system we're building, especially if we want the new system to work as a drop-in replacement, or for projects like emulators where it's important to model every edge case. - my favourite reason of all, you can extend the system in ways not intended by the original designer - in my view this is where it becomes pure creativity, the system becomes a writing prompt and some constraints, and computers being what they are the limit is your imagination - now I'm not going to fall into the trap of claiming it's easy, because I hate it when talks do that. - reverse engineering is challenging, but it doesn't have to be impossible, or gated off to just a handful of people. - unfortunately there's been this stagnant elitism around reversing for the longest time, let's smash apart some myths

Poisonous myths to be stamped out

Reverse engineering is a competition
In order to start out...

you have to know everything about the target!
you have to know C!
you have to know assembly language!
you have to be a genius!

It only counts as true reverse engineering if...

no-one has ever researched this before
it looks sufficiently painful and complex

Categories of technique

Black box analysis: probe just the inputs and outputs of a system
Dynamic analysis: examine the internals and state of a system while executing
Static analysis: examine and annotate the internals of a system at rest

Have at least two of these approaches handy!

Reading mystery data

Bytes

Hex editor

Shameless plug

https://bitbucket.org/moralrecordings/mrcrowbar

Magic numbers

Lookup tables

- this is a data file I took from an old DOS game - can see there's a pattern going on here, something in groups of four bytes - given they all end in 00s a big list of 4-byte integers. that in itself is noteworthy, but we can also guess what they are for. - other interesting things, if you read in the first one you get the number 353. by a complete coincidence, the rest of the list is 353 entries long! - rest of the numbers are increasing and within the size of the archive, suggesting they are referring to locations - if we visit one, we're at something that looks like a complete file - and finally if you look at the integer stored in the first four bytes here, it's the size before we reach the next entry in the list - LOOKUP TABLE: efficiency is big design factor. CPU does least work in smallest memory. think in terms of laziness; easiest for the programmer to write, lightest in terms of HW - very common as it's easy to write and quite efficient to move around and copy

Raw graphics

Raw audio

Histogram dump

Compression and encryption

Troubleshooting any problem

Debugger

You only need 4 features!

Stepping (step over, step into, continue)
Breakpoints (add and remove)
View state (e.g. variables)
View current stack trace

The investigation formula

Reproduce the behaviour in a test environment
Establish the boundaries of the system
Observe for points of interest
Find an entry point close to a point of interest
Trace through to the exact moment it happens

- to finish this up, let's talk about the steps of how to investigate with the debugger, ideally with the source code. I figured I'd share my routine; this is not prescripive, you don't have to follow this to the letter, but find a way that works for you - reproduce sometimes the hardest part - establish boundaries; fencing off as much unrelated stuff as possible, if your code uses libraries, you should check if they are involved with the behaviour under test - observe; any symptoms that occur around the time of the behaviour. error messages good example. try and find the code modules responsible. what you want to look for is any variables, or state that could be responsible. - make; start the program in the debugger, pick a spot found earlier, add a breakpoint - trace; requires a few goes, but basically step through the code, keeping an eye on any state you found before, and try and find the point where the system flips from being okay to being broken. e.g. if a check fails, work backwards to find out what caused the check to be busted. - what we just described is reverse engineering with source code. by practicing this, you etch patterns and observations into muscle memory, which you will retain when you do it without code. challenge yourself; pick an open bug in your favourite open source project and try to track it down

Understanding any program

Disassembler

- DISASSEMBLER - job is to convert the machine code for the whole program into assembly, so you can go through and document it with your findings. - some disassemblers just give you the code, the fancier ones are interactive, and do things like allow you to rename all the functions and memory chunks - screenshot is of IDA - undisputed queen of disassemblers. IDA supports a massive number of CPU architectures, but some platforms have more targeted disassemblers, so it's worth doing a search for best practice for your platform. e.g. java and .NET both have decompilers - but at the end of the day, it still looks pretty incomprehensible; in fact earlier I clearly said you wouldn't need to know assembly language, and yet here's pages and pages of it! how are we supposed to start?

- examine how a native program works on a modern system. - address space - complete range of addresses CPU can write to, A MAPPING, segments carved out mapping to differnet hardware like RAM or disk. program doesn't have to care - what's in address space? code, static data, general purpose memory, heap memory from the OS, OS kernel, shared libraries - all the gaps aren't filled with anything; accessing them will throw an exception - CPU - almost every CPU works the same way - one or more general purpose registers, which are big enough to fit the largest supported integer type, or a memory address. can store things or do arithmetic. - special register program counter which points to current place - flags register: contains information about the last operation done by the CPU - used for if statements - 4 registers isn't a lot, what happens when we need more than 4 at once, like calling a function with more than 4 params - this is where the stack comes in useful. temporary memory, last in first out. another special pointer which keeps track of the top of the stack, as the CPU has special instructions to push things onto or pop things off the stack.

Conversion

- but I still haven't mentioned the elephant in the room, how are you supposed to read the assembly language? - showing a snippet here, ideally you'd do it for a whole subroutine - CPU vendors really want you to use their chips, so they provide a instruction set reference - open up your instruction set reference, open a new text file, and go through line by line converting the assembly to your favourite brain neutral format. mine is python with gotos. - the first pass will look rough but afterwards you can chip away at it, adding what you've found out about the inputs and outputs, replace registers with names that make sense, until eventually it looks something like this. - sounds backbreaking, but the good news is you only have to do it for code that's interesting! - really all there is to it, you have an entry point from which you can make more discoveries, and cross check with the live code. - like debugging, skill and confidence comes with experience.

Second-last slide

Every reverse engineering skill has a progression
It is normal to start off knowing barely anything
Find an area of research that interests you
Write up your findings!

Questions

https://moral.net.au - @moralrecordings

P.S. GOOD LUCK!

Reverse Engineering is Good and also For Everyone

@moralrecordings