Ishwor Gurung

Why Considerdone Au

2026-09-17T00:00:00+05:45

Why ConsiderDone.au

Most home-service jobs begin with a small but annoying problem: a leaking tap, a broken appliance, a lockout, or a fault that needs an electrician. The work itself may be simple. Getting it done is not.

The usual process asks the homeowner to search, compare reviews, call several businesses, explain the same problem repeatedly, wait for quotes, coordinate a time, and work out how to pay. That is a lot of effort for something people only want to think about once.

I built considerdone.au around a simple question: what if a person could say, “Find someone reliable, book them, and let me approve the price,” then get on with their day?

The aim is not to replace tradespeople. It is to remove the administrative gap between a homeowner who needs help and a local business that can do the work.

The gap in the current marketplace model

Existing marketplaces are useful discovery tools, but they often stop at the hardest part. A homeowner still has to make the calls and decide what happens next. Many platforms also sell the same lead to several providers, which turns a customer’s problem into a race to respond.

That is frustrating for both sides. Homeowners spend time doing coordination work. Providers pay for leads that may be shared with several competitors and may never become a booking.

considerdone.au takes a different path. It starts with one plain-language request. The system searches for relevant local providers, contacts them by phone, asks about availability and price, and brings the result back to the homeowner for approval.

The important distinction is that the product is designed to finish the workflow, not merely generate a lead.

What the service is actively solving

The product has three jobs to do well:

Find a suitable provider for the task and location.
Turn the request into a real booking without making the homeowner chase people.
Handle money in a way that is clear, controlled, and fair to both parties.

The first version uses local-search data to find and rank candidates. It then uses an outbound voice workflow to speak with businesses in the normal channel they already use: the phone. A provider does not need to join a new marketplace or adopt a new point-of-sale system before they can take a job.

The homeowner stays in control. Before the system makes calls, it asks for approval. When a real quote comes back, it asks again before charging anything. Those two pauses are intentional. Delegation should reduce effort, not remove consent.

How a job moves through considerdone.au

The homeowner makes the important choices. considerdone.au does the coordination work in between, and the provider stays in control of the work, availability, and completion.

Engineering for a real-world workflow

The difficult part is not making an AI place a phone call. The difficult part is making the whole workflow dependable when it touches real people, real calendars, and real money.

That has shaped the engineering decisions behind considerdone.au.

The orchestrator owns the journey

A booking moves through many steps: search, calling, quote collection, approval, payment, provider details, job completion, and payout. One orchestration layer coordinates those steps, while each integration stays narrowly focused on its own job.

For example, the calling service does not decide who gets paid, and the payment service does not decide which provider to call next. Keeping those responsibilities separate makes failures easier to understand and changes safer to make.

Actions are designed to be safe to retry

External systems fail. A server can restart halfway through a task. A network request can time out after the other side has already accepted it. Retrying blindly can create duplicate calls or, worse, duplicate charges.

For that reason, side-effecting operations use persistent idempotency keys and database constraints. The system records its progress and can resume in-flight tasks after a restart instead of treating a restart as a failed job. This is less glamorous than the voice interface, but it is essential infrastructure for a service that makes bookings and moves money.

Structured results, not AI interpretation after the fact

Phone conversations are messy. The application does not use raw transcripts to make booking decisions. Instead, the call workflow returns specific structured fields such as outcome, quote, time slot, and whether the provider needs a site visit.

That boundary matters. It gives the workflow defined data to act on, makes it testable, and avoids having hidden interpretation drive a payment or state change.

Payment follows approval, not assumption

For a standard job, the system first places an authorisation hold for the homeowner’s agreed spending cap. It only captures the exact quoted amount after the homeowner approves it. If the quote is declined or no provider works out, the hold is not captured.

After the job is complete, the provider submits completion through a secure, time-limited link. The homeowner can release payment, dispute it, or allow the stated review period to expire. The provider is then paid by bank transfer, less a clearly disclosed booking fee. The consumer pays the quote they approved, with no surprise surcharge added on top.

This two-leg payment model reflects the real timing of a service job: booking and charging happen first; provider payout happens only after completion and review.

Adapting to how trades actually quote

One of the most useful lessons from the build was that not every job can be priced on the phone.

A tap washer or lockout may be straightforward. A hidden leak, electrical fault, or compliance-sensitive job often is not. In many cases, a provider needs to inspect the site before they can responsibly give a price.

Rather than forcing every task into a one-call quote, considerdone.au supports two paths:

For phone-quotable work, the homeowner can approve a quote and book directly.
For diagnosis-first work, the system can book a call-out visit first. The provider then submits an on-site price, and the homeowner gets a separate approval decision before the actual job is charged.

This is a deliberate product choice. A new, unknown on-site price is never automatically approved. The call-out visit is treated as its own booking, so the provider is still paid for their time even if the homeowner declines the later job price.

The fee model follows that reality too. The call-out fee has a small pass-through fee, while the actual job carries the main booking fee. It is a more honest model than pretending a visit is free or expecting providers to absorb the cost of diagnosis.

Trust is a product feature, not a marketing claim

If a system is going to act for someone, it needs strong boundaries.

considerdone.au keeps card numbers out of its servers by using a browser-based tokenized payment component. It keeps an append-only audit trail of state-changing events while storing event payloads as cryptographic hashes rather than raw data. Provider payout links are random, scoped to one booking, time-limited, and single-use.

The product also has safety screening before a provider is contacted. Normal service requests proceed. Requests that suggest an emergency, harm, or criminal intent are stopped and handled with an appropriate response. The system does not automate reports or make high-stakes decisions from a phone transcript.

There are also areas where the work is intentionally not presented as finished. Provider ABN and licence verification need a practical, state-by-state rollout plan. Disputes are handled manually in the first version. Preventing off-platform repeat transactions will require a better mix of customer protection, provider incentives, and repeat-booking value than a simple discount can offer.

Naming those gaps is part of building trust. The point is to solve the problems that can be solved now, measure the uncertain ones, and avoid pretending a complex marketplace is complete before it is.

Building toward a useful local service agent

considerdone.au is being built as a practical assistant for a very ordinary problem: something at home needs fixing, and someone needs to make the booking happen.

The direction is simple. Make the first request easy. Keep the homeowner in control. Respect providers’ existing way of working. Be precise about money. Make the system recover gracefully when real life does not follow the happy path.

The technology matters, but the standard is simpler: when a homeowner asks for help, the experience should leave them able to say, “consider it done.”

The future here is bright. As x402 and other agentic AI payment rails mature, more of the trusted coordination around a job can happen quietly and safely in the background: matching, permissions, booking, settlement, and the record of what was agreed. That should mean fewer hand-offs and less administrative work for homeowners and providers alike. The human part that matters most remains human: the tradesperson’s judgement, skill, and work on site, while the surrounding workflow gets steadily simpler.

Opentelemetry

2023-05-27T00:00:00+05:45

OpenTelemetry Report

I recently went on a couple of months long migration to OpenTelemetry from DataDog, and here are some thoughts specific to OTLP that may be of interest to you.

TL;DR

Emitting all three signals (metrics, traces and logging) in concert and expecting to be able to easily correlate them in the telemetry backend is not straight forward path (also it highly depends on which telemetry backend you choose).

Metrics

For OpenTelemetry’s OTLP metrics, you can be assured that for a long-running service, this is pretty stable. You can choose to ingest with OTLP and fan-out to other vendors like DataDog; there are no surprises involved.

For FaaS workloads (AWS Lambda), it’s a different story. The OpenTelemetry Collector-based, metrics in a FaaS environment is not reliable. I hope it will eventually get there, but exactly when that will happen is unknown. It depends on some key features being implemented in Lambda runtime to mitigate issues involved with the ‘freeze’ and ‘thaw’ of containers.

Traces

OTLP-based traces are pretty stable in the long-running service world. Again, you can choose to ingest with OTLP and fan-out to other vendors like DataDog, and there are no surprises involved.

However, for FaaS workloads such as AWS Lambda, due to the aforementioned freeze & thaw interaction within the extensions’ environment, is yet to materialize fully.

Logs

For OTLP-based logs are a complicated story. It varies on the log sink involved, execution environment, and other constraints. In general, your mileage will vary. OTLP-based log pipelines are not yet widely adopted and are not production ready. So, this is a case of YMMV.

Continuous Profiling

OTLP-based continuous profiling is unheard of; it currently exists in the proposals’ basket.

Observations

OpenTelemetry’s list of dirty laundry is long, and it will take a while to get it widely adopted. But here are a few key ones I have observed:

Contributing vendors have many self-driven interests and agendas that do not necessarily benefit the OpenTelemetry’s open ecosystem.
OpenTelemetry working groups often fail to come to consensus thus impacting deliveries caused by countless number of mundane arguments on the technical merits of a given implementation and design, in addition to lack of leadership (sure, I understand large opensource efforts are not trivial, but it does not take 2+ years to deliver a feature).
OpenTelemetry development is actually siloed; I probably need not say more.

All of this brings me to the following conclusion:

Closed vendors like DataDog can charge $65 million dollars to one customer (just like a raging bull) because their o11y stack is fully baked (all four signals - metrics, traces, logs and profiling) and production ready.

However, all is not lost. OpenTelemetry has a lot of good things going for it, and I am excited to see where it goes. Here are some of the good things I love about OpenTelemetry:

Wide number of telemetry integrations with you name it vendor
An opportunity to shape the Observability landscape for the foreseeable future
Wide variety of language support, telemetry receivers, processors and exporters to choose from
A very active and responsive core team that is willing to help and contribute
Extreme momentum in the community relating to releasing features and bug fixes
Many future enhancements are shaping as we speak, such as using eBPF, client-side metrics aggregations, exemplars, and many Rust-based :) telemetry components are in the works, and many more.

Xdp Ebpf Firewall In Rust

2020-12-26T00:00:00+05:45

XDP/eBPF IP-layer firewall in Rust!

One of the option available today to do sub-millisecond packet filtering at scale is to harness the facilities afforded by XDP, eBPF, Linux kernel and support provided by various NIC manufacturers.

The two (XDP and eBPF) have grown quite rapidly in the 5.x release of the Linux kernel and there is absolutely no reason you would not want to use it for your low-latency, high-performance use case.

In this post, I will be demonstrating a sample XDP code that performs IP-layer firewall function traditionally done by Netfilter/IPtables (thanks for all the fish Rusty Russell!) but ran as a offloaded XDP program by the eBPF VM in the Linux kernel. The walk-through was thoroughly tested on Ubuntu 20.10 with LLVM version 11.0.0 on QEMU/virtio. LLVM is required today because it can produce eBPF targets.

According to XDP developers, on a benchmark they conducted, XDP far out-performed traditional kernel network stack on Linux on all fronts - tx, rx and forward (in the order of 12-25x!) when used with a NIC that that offloads XDP such as the ones from Intel/Mellanox/Netronome (mlx/ixgbe/nfp etc).

This is mind-blowing on 2018 network hardware.

We show that XDP achieves single-core packet processing per- formance as high as 24 million packets per second, and illustrate the flexibility of the programming model through three example use cases: layer-3 routing, inline DDoS protection and layer-4 load balancing

Dependencies Installation and Rust setup

Install dependencies:

$ sudo apt-get install          \
    build-essential             \    
    libelf-dev                  \
    ca-certificates             \
    ca-certificates-java        \
    zlib1g-dev                  \
    llvm-11-dev                 \
    libclang-11-dev             \
    linux-headers-$(uname -r)

Install Rust (single user) using rustup

$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ . ~/.bashrc # or logout then login.
$ rustup install nightly
$ rustup default nightly
$ rustc --version
 

Install Rust (system-wide) using rustup

$ sudo -i
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs |       \
    env RUSTUP_HOME=/opt/rust/rustup CARGO_HOME=/opt/rust/cargo     \
    sh -s -- --default-toolchain stable --profile default --no-modify-path -y
$ tee -a /root/.bashrc <<HD
# setup Rust environment
export RUSTUP_HOME=/opt/rust/rustup
export PATH=${PATH}:/opt/rust/cargo/bin
HD
$ . /root/.bashrc # or logout then login.
$ rustup install nightly
$ rustup default nightly
$ rustc --version

Install carfgo-bpf:

$ cargo install cargo-bpf

Walk-through

Create the eBPF project:

$ cargo bpf new xdp-ebpf-fw

Add a new eBPF program called fw:

$ cd xdp-ebpf-fw
$ cargo bpf add fw

Source for xdp-ebpf-fw/Cargo.toml:

[package]
name = "xdp-ebpf-fw"
version = "0.1.0"
edition = '2018'

[dependencies]
cty = "0.2"
redbpf-macros = "1.3"
redbpf-probes = "1.3"

[build-dependencies]
cargo-bpf = { version = "1.3", default-features = false }

[features]
default = []
probes = []

[lib]
path = "src/lib.rs"

[[bin]]
name = "fw"
path = "src/fw/main.rs"
required-features = ["probes"]

Source for fw/main.rs:

#![no_std]
#![no_main]

use core::fmt::Error;
use cty::*;
use redbpf_probes::xdp::prelude::*;

program!(0xFFFFFFFE, "GPL");

const TCP_XDP_DROP: XdpAction = XdpAction::Drop;
const UDP_XDP_DROP: XdpAction = XdpAction::Drop;
const XDP_PASS: XdpAction = XdpAction::Pass;

// XDP/eBPF based IP-layer firewall to drop all UDP packets.
// And, also drop all TCP packets destined to port 80.
#[xdp]
pub fn xdp_ip_firewall(ctx: XdpContext) -> XdpResult {
    if let Ok(ip_protocol) = get_ip_protocol(&ctx) {
        match ip_protocol as u32 {
            IPPROTO_UDP => return Ok(UDP_XDP_DROP), // drop it on the floor
            IPPROTO_TCP => {
                if let Ok(transport) = ctx.transport() {
                    if transport.dest() == 80 {
                        return Ok(TCP_XDP_DROP);  // drop it on the floor
                    }
                }
            }
            _ => return Ok(XDP_PASS), // pass it up the protocol stack
        }
    }
    return Ok(XDP_PASS); // pass it up the protocol stack
}

fn get_ip_protocol(ctx: &XdpContext) -> Result<u32, Error> {
    if let Ok(ip) = ctx.ip() {
        // We need to make raw pointer into a u32 so `unsafe` is required.
        unsafe {
            return Ok((*ip).protocol as u32);
        }
    }
    // Anything above `255` is reserved.
    return Ok(0x10000);
}

Build eBPF program:

$ cargo bpf build

Network queues on QEMU/virtio for XDP

On QEMU, we need to add 2N queues (where N is the number of vCPUs). The following <driver> section is required inside <interface> (use virsh edit [vmname]) is required (for 4 vCPUs, allocate 8 queues) as of QEMU version 4.2.1.

<interface ...>
[ ... ]
<driver name='vhost' txmode='iothread' ioeventfd='on' event_idx='off' queues='8' rx_queue_size='256' tx_queue_size='256'>
    <host csum='off' gso='off' tso4='off' tso6='off' ecn='off' ufo='off' mrg_rxbuf='off'/>
    <guest csum='off' tso4='off' tso6='off' ecn='off' ufo='off'/>
</driver>
[ ... ]
</interface>

Load the compiled eBPF program!

Load the compiled eBPF program:

$ cd xdp-ebpf-fw
$ cargo bpf load -i eth0 target/bpf/programs/fw/fw.elf

Profit!

:boom:

External resources

Interesting Links

2020-11-07T00:00:00+05:45

Interesting links

LWN’s documentation on io_uring

How io_uring and eBPF Will Revolutionize Programming in Linux

Make sure you have the latest Linux kernel (something like v5.9.0+) before opening the io_uring goodness.

Got 40GigE stream data to do DPI on x86? Do you have performance issue with the pattern matching library currently in use? No problem. Enter Intel Hyperscan.

Both io_uring and hyperscan has multitude of language bindings, so you are not necessarily stuck in C lalaland.

Until next interesting links, adios!

Capture Kernel Crash

2020-06-01T00:00:00+05:45

Capturing Linux kernel panic message on QEMU

In this small post, I will cover the following:

Capture Linux kernel panic message
Using softdog module to set the watchdog to auto-reboot upon Kernel panic.

Capture the panic

Log into your guest and update /etc/default/grub and add console=tty0 console=ttyS0,9600n8 to GRUB_CMDLINE_LINUX; Then run update-grub.

To capture a kernel panic message, run the virtual machine like so:

$ qemu-system-x86_64 -smp 2,sockets=2,cores=1,threads=1 -m 4096 -nographic -serial mon:stdio ubuntu18.04-1.qcow2

You will be greeted with the usual boot messages.

Upon kernel panic, a full trace message will appear on your attached console.

[  247.009745] BUG: stack guard page was hit at 000000006c1ac1e1 (stack is 00000000487682b5..000000001262eb3a)
[  247.012648] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[  247.012648] Modules linked in: helloworld(POE+) binfmt_misc ppdev kvm_amd kvm irqbypass input_leds serio_raw parport_pc parport sch_fq_codel softy
[  247.012648] CPU: 1 PID: 2102 Comm: insmod Tainted: P           OE    4.15.0-64-generic #73-Ubuntu
[  247.012648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[  247.012648] RIP: 0010:_ZN4core3num23_$LT$impl$u20$usize$GT$11checked_mul17h07ae83026124b5eaE+0x10/0x70 [helloworld]
[  247.012648] RSP: 0018:ffffa84b4169ffd0 EFLAGS: 00000282
[  247.012648] RAX: 00000000000000fe RBX: 0000000000000000 RCX: 0000000000000000
[  247.012648] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[  247.012648] RBP: ffffa84b416a0020 R08: ffffffffc0690d60 R09: 0000000000000003
[  247.012648] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffc059edf0
[  247.070679] R13: ffff9c5d78655a00 R14: 0000000000000001 R15: ffff9c5d51aeb240
[  247.070679] FS:  00007f690a2ff540(0000) GS:ffff9c5d7fd00000(0000) knlGS:0000000000000000
[  247.070679] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  247.070679] CR2: ffffa84b4169ffc8 CR3: 0000000137068000 CR4: 00000000000006e0
[  247.070679] Call Trace:
[  247.070679]  ? _ZN4core3num23_$LT$impl$u20$usize$GT$14saturating_mul17hc7be383430829098E+0x15/0x50 [helloworld]
[  247.070679]  ? _ZN4core5slice18from_raw_parts_mut17hc0afd704cde83aefE+0x62/0xd0 [helloworld]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$17get_unchecked_mut17h8+0x]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$9index_mut17h59d846e1fc+0x]
[  247.070679]  ? _ZN4core3cmp6min_by17h6a1a357c03d17732E+0x3d/0x90 [helloworld]
[  247.070679]  ? _ZN4core5slice77_$LT$impl$u20$core..ops..index..IndexMut$LT$I$GT$$u20$for$u20$$u5b$T$u5d$$GT$9index_mut17h6d899aa41763834eE+0x33/0]
[  247.070679]  ? _ZN79_$LT$linux_kernel_module..printk..LogLineWriter$u20$as$u20$core..fmt..Write$GT$9write_str17hb6b9f63c86335f4fE+0x127/0x250 [he]
[  247.070679]  ? _ZN4core3fmt9Formatter12pad_integral17hea95901f585e791cE+0x260/0x8b0 [helloworld]
[  247.070679]  ? _ZN4core3num23_$LT$impl$u20$usize$GT$14saturating_mul17hc7be383430829098E+0x15/0x50 [helloworld]
[  247.070679]  ? _ZN4core5slice14from_raw_parts17hc54a1a170e771755E+0x8f/0xd0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp7fmt_u6417hb10d5615ab39d4d8E+0x5e6/0x720 [helloworld]
[  247.070679]  ? _ZN4core10intrinsics19copy_nonoverlapping17h66731a07c4e66efbE+0xc2/0xd0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp51_$LT$impl$u20$core..fmt..Display$u20$for$u20$u8$GT$3fmt17h8a7427ec99f7ed22E+0x80/0xa0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num49_$LT$impl$u20$core..fmt..Debug$u20$for$u20$u8$GT$3fmt17h9010dd741ac356b3E+0x64/0xa0 [helloworld]
[  247.070679]  ? _ZN42_$LT$$RF$T$u20$as$u20$core..fmt..Debug$GT$3fmt17hce45b733e98582cdE+0x18/0x30 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders10DebugInner9is_pretty17h61db7b5335d258d0E+0x14/0x30 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders10DebugInner5entry28_$u7b$$u7b$closure$u7d$$u7d$17h17d35c1353ea2e80E+0x23c/0x250 [helloworld]
[  247.070679]  ? _ZN79_$LT$linux_kernel_module..printk..LogLineWriter$u20$as$u20$core..fmt..Write$GT$9write_str17hb6b9f63c86335f4fE+0x1be/0x250 [he]
[  247.070679]  ? _ZN4core6result19Result$LT$T$C$E$GT$8and_then17hc710b76977389ea0E+0x4e/0x70 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders10DebugInner5entry17h92ae2868f0df466bE+0x43/0x70 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders9DebugList5entry17he09db0b952ed03ccE+0x23/0x30 [helloworld]
[  247.070679]  ? _ZN4core3fmt8builders9DebugList7entries17h8eb3996229be728fE+0x88/0xa0 [helloworld]
[  247.070679]  ? _ZN48_$LT$$u5b$T$u5d$$u20$as$u20$core..fmt..Debug$GT$3fmt17h9c57c882f7ac9744E+0x51/0x70 [helloworld]
[  247.070679]  ? _ZN50_$LT$$RF$mut$u20$T$u20$as$u20$core..fmt..Debug$GT$3fmt17hb8d4e0e57dc57deaE+0x2a/0x40 [helloworld]
[  247.070679]  ? _ZN4core3fmt5write17hb916ed05c0beea6fE+0x285/0x640 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN11hello_world4foos17he560009d948fdbe9E+0x189/0x1e0 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN4core3fmt10ArgumentV13new17hf3e64bcfe2a81583E+0x50/0x50 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN4core3fmt10ArgumentV13new17hf3e64bcfe2a81583E+0x50/0x50 [helloworld]
[  247.070679]  ? _ZN4core3fmt3num3imp52_$LT$impl$u20$core..fmt..Display$u20$for$u20$u64$GT$3fmt17h67ec8b7f52fe9da3E+0xa0/0xa0 [helloworld]
[  247.070679]  ? _ZN83_$LT$hello_world..HelloWorldModule$u20$as$u20$linux_kernel_module..KernelModule$GT$4init17hcdf4c87881ba8bf7E+0x55/0xb0 [hello]
[  247.092990]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __switch_to_asm+0x41/0x70
[  247.093536]  ? init_module+0x14/0xd0 [helloworld]
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? kfree+0x165/0x180
[  247.093536]  ? do_one_initcall+0x52/0x19f
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? _cond_resched+0x19/0x40
[  247.093536]  ? kmem_cache_alloc_trace+0x14e/0x1b0
[  247.093536]  ? do_init_module+0x27/0x209
[  247.093536]  ? do_init_module+0x5f/0x209
[  247.093536]  ? load_module+0x1939/0x1f30
[  247.093536]  ? ima_post_read_file+0x96/0xa0
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SyS_finit_module+0xe/0x10
[  247.093536]  ? do_syscall_64+0x73/0x130
[  247.093536]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[  247.093536] Code: eb 08 48 c7 45 d0 00 00 00 00 48 8b 45 d0 48 8b 55 d8 48 83 c4 50 5d c3 00 00 00 55 48 89 e5 48 83 ec 50 48 89 7d e0 48 89 75 e 
[  247.097494] RIP: _ZN4core3num23_$LT$impl$u20$usize$GT$11checked_mul17h07ae83026124b5eaE+0x10/0x70 [helloworld] RSP: ffffa84b4169ffd0
[  247.097494] ---[ end trace b512caf3cbcd6f50 ]---
[  247.097494] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[  247.097494] 
[  247.097494] Kernel Offset: 0x1b400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  247.097494] Rebooting in 60 seconds..

Just for your information, you can pipe the kernel panic message to c++filt to get the demangled symbols (the kernel module is written in Rust and it does not yet seem to support [no_mangle]).

$ cat panic | c++filt 
[  247.009745] BUG: stack guard page was hit at 000000006c1ac1e1 (stack is 00000000487682b5..000000001262eb3a)
[  247.012648] kernel stack overflow (double-fault): 0000 [#1] SMP NOPTI
[  247.012648] Modules linked in: helloworld(POE+) binfmt_misc ppdev kvm_amd kvm irqbypass input_leds serio_raw parport_pc parport sch_fq_codel softy
[  247.012648] CPU: 1 PID: 2102 Comm: insmod Tainted: P           OE    4.15.0-64-generic #73-Ubuntu
[  247.012648] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.12.0-1 04/01/2014
[  247.012648] RIP: 0010:core::num::<impl usize>::checked_mul::h07ae83026124b5ea+0x10/0x70 [helloworld]
[  247.012648] RSP: 0018:ffffa84b4169ffd0 EFLAGS: 00000282
[  247.012648] RAX: 00000000000000fe RBX: 0000000000000000 RCX: 0000000000000000
[  247.012648] RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001
[  247.012648] RBP: ffffa84b416a0020 R08: ffffffffc0690d60 R09: 0000000000000003
[  247.012648] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffc059edf0
[  247.070679] R13: ffff9c5d78655a00 R14: 0000000000000001 R15: ffff9c5d51aeb240
[  247.070679] FS:  00007f690a2ff540(0000) GS:ffff9c5d7fd00000(0000) knlGS:0000000000000000
[  247.070679] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  247.070679] CR2: ffffa84b4169ffc8 CR3: 0000000137068000 CR4: 00000000000006e0
[  247.070679] Call Trace:
[  247.070679]  ? core::num::<impl usize>::saturating_mul::hc7be383430829098+0x15/0x50 [helloworld]
[  247.070679]  ? core::slice::from_raw_parts_mut::hc0afd704cde83aef+0x62/0xd0 [helloworld]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$17get_unchecked_mut17h8+0x]
[  247.070679]  ? _ZN99_$LT$core..ops..range..Range$LT$usize$GT$$u20$as$u20$core..slice..SliceIndex$LT$$u5b$T$u5d$$GT$$GT$9index_mut17h59d846e1fc+0x]
[  247.070679]  ? core::cmp::min_by::h6a1a357c03d17732+0x3d/0x90 [helloworld]
[  247.070679]  ? core::slice::<impl core::ops::index::IndexMut<I> for [T]>::index_mut::h6d899aa41763834e+0x33/0]
[  247.070679]  ? <linux_kernel_module::printk::LogLineWriter as core::fmt::Write>::write_str::hb6b9f63c86335f4f+0x127/0x250 [he]
[  247.070679]  ? core::fmt::Formatter::pad_integral::hea95901f585e791c+0x260/0x8b0 [helloworld]
[  247.070679]  ? core::num::<impl usize>::saturating_mul::hc7be383430829098+0x15/0x50 [helloworld]
[  247.070679]  ? core::slice::from_raw_parts::hc54a1a170e771755+0x8f/0xd0 [helloworld]
[  247.070679]  ? core::fmt::num::imp::fmt_u64::hb10d5615ab39d4d8+0x5e6/0x720 [helloworld]
[  247.070679]  ? core::intrinsics::copy_nonoverlapping::h66731a07c4e66efb+0xc2/0xd0 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u8>::fmt::h8a7427ec99f7ed22+0x80/0xa0 [helloworld]
[  247.070679]  ? core::fmt::num::<impl core::fmt::Debug for u8>::fmt::h9010dd741ac356b3+0x64/0xa0 [helloworld]
[  247.070679]  ? <&T as core::fmt::Debug>::fmt::hce45b733e98582cd+0x18/0x30 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugInner::is_pretty::h61db7b5335d258d0+0x14/0x30 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugInner::entry::::h17d35c1353ea2e80+0x23c/0x250 [helloworld]
[  247.070679]  ? <linux_kernel_module::printk::LogLineWriter as core::fmt::Write>::write_str::hb6b9f63c86335f4f+0x1be/0x250 [he]
[  247.070679]  ? core::result::Result<T,E>::and_then::hc710b76977389ea0+0x4e/0x70 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugInner::entry::h92ae2868f0df466b+0x43/0x70 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugList::entry::he09db0b952ed03cc+0x23/0x30 [helloworld]
[  247.070679]  ? core::fmt::builders::DebugList::entries::h8eb3996229be728f+0x88/0xa0 [helloworld]
[  247.070679]  ? <[T] as core::fmt::Debug>::fmt::h9c57c882f7ac9744+0x51/0x70 [helloworld]
[  247.070679]  ? <&mut T as core::fmt::Debug>::fmt::hb8d4e0e57dc57dea+0x2a/0x40 [helloworld]
[  247.070679]  ? core::fmt::write::hb916ed05c0beea6f+0x285/0x640 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? hello_world::foos::he560009d948fdbe9+0x189/0x1e0 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? core::fmt::ArgumentV1::new::hf3e64bcfe2a81583+0x50/0x50 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? core::fmt::ArgumentV1::new::hf3e64bcfe2a81583+0x50/0x50 [helloworld]
[  247.070679]  ? core::fmt::num::imp::<impl core::fmt::Display for u64>::fmt::h67ec8b7f52fe9da3+0xa0/0xa0 [helloworld]
[  247.070679]  ? <hello_world::HelloWorldModule as linux_kernel_module::KernelModule>::init::hcdf4c87881ba8bf7+0x55/0xb0 [hello]
[  247.092990]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __switch_to_asm+0x35/0x70
[  247.093536]  ? __switch_to_asm+0x41/0x70
[  247.093536]  ? init_module+0x14/0xd0 [helloworld]
[  247.093536]  ? __slab_free+0x14d/0x2c0
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? kfree+0x165/0x180
[  247.093536]  ? do_one_initcall+0x52/0x19f
[  247.093536]  ? __vunmap+0x8e/0xc0
[  247.093536]  ? _cond_resched+0x19/0x40
[  247.093536]  ? kmem_cache_alloc_trace+0x14e/0x1b0
[  247.093536]  ? do_init_module+0x27/0x209
[  247.093536]  ? do_init_module+0x5f/0x209
[  247.093536]  ? load_module+0x1939/0x1f30
[  247.093536]  ? ima_post_read_file+0x96/0xa0
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SYSC_finit_module+0xfc/0x120
[  247.093536]  ? SyS_finit_module+0xe/0x10
[  247.093536]  ? do_syscall_64+0x73/0x130
[  247.093536]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[  247.093536] Code: eb 08 48 c7 45 d0 00 00 00 00 48 8b 45 d0 48 8b 55 d8 48 83 c4 50 5d c3 00 00 00 55 48 89 e5 48 83 ec 50 48 89 7d e0 48 89 75 e 
[  247.097494] RIP: core::num::<impl usize>::checked_mul::h07ae83026124b5ea+0x10/0x70 [helloworld] RSP: ffffa84b4169ffd0
[  247.097494] ---[ end trace b512caf3cbcd6f50 ]---
[  247.097494] Kernel panic - not syncing: corrupted stack end detected inside scheduler
[  247.097494] 
[  247.097494] Kernel Offset: 0x1b400000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[  247.097494] Rebooting in 60 seconds..

I triggered the above Linux kernel panic while I was trying to understand how the Rust LKM worked; In this case, I allocated large kernel stack which was caught by the stack guard which caused an immediate Kernel panic. If you are into Rust and Linux kernel, I definitely recommend checking out linux-kernel-module-rust.

Auto-reboot upon Linux kernel panic

If you often get kernel panic (while developing/debugging kernel module or what not) or want to ensure system uptime in the face of Kernel instabilities, you could consider enabling softdog in-tree linux kernel module. This LKM runs a watchdog timer with a periodic heartbeat. Upon heartbeat failure, the timer associated with the heartbeat expires due to a kernel panic upon which, the system is rebooted. For this to work, a kernel parameter needs to be added to GRUB - GRUB_CMDLINE_LINUX="panic=60" in /etc/default/grub and then update-grub (for Ubuntu-like distro) needs to be run; Then the system needs to be rebooted.

Ubuntu has commented out the automatic loading of softdog kernel module, so it needs to be added back in. Remove softdog module from files that reside in /lib/modprobe.d/*.conf and then add it to auto-load in /etc/modules-load.d. Then run the following:

$ sudo update-initramfs -u
$ sudo depmod -a
$ sudo systemctl reboot

Feel free to drop a comment if anything. As usual, until next time - Adios!

Underrated Socat

2020-05-01T00:00:00+05:45

Socat

Socat is one of the most underrated software in Linux. It is a tool to help in debugging and applying network interops in ways the original author of a software probably had not anticipated. The author of socat describes it as multipurpose relay software (socket cat). People lovingly synonymize it as like cat but for sockets. I first came across socat when I was working as a pentester back in the day (I have worn many hats in the past) whilst trying to work my way through protocol debugging.

TL;DR? Refer to the commands instead.

The socat story

So, why would you need socat? Follow through on the fictionalised story below ¹.

Let’s pretend that we want to have a TCP client sitting locally to be able to connect to a TCP server sitting elsewhere on port 3450; i.e., a TCP network clients connect to a TCP socket on the server at port 3450. This is the trivial part and software libraries have evolved to abstract almost 99% of this game so engineers can have a good sleep.

Now, here’s the catch - say that Bob the sysadmin appeared on the deck of this ship and said that clients can only connect to TCP sockets on the server on port 80 and 443 ²

You can use many tools in Linux and *BSDs (and that’s the beauty of it - each to their own) to achieve the precise outcome (HAProxy, Nginx, iptables DNATs, SSH port forwards, IPsec tunnels, sshuttle, Wireguard etc..) but in this instance I would like to focus only on socat.

So, how could socat be of help here for a quick win? The TCP client will now connect to the server on port 80. That is, the TCP endpoint to the client will look like server.ip.address:80.

On the server, Bob, the sysadmin will spin up socat and run a TCP relay from the listening TCP port 80 to the local loopback address’ TCP port 3450. Thus, Bob, the sysadmin will run this on the server:

$ socat -v tcp-listen:80,reuseaddr,fork tcp-connect:127.0.0.1:3450

This will relay all TCP packets that connect to the server on port 80 to be sent to 127.0.0.1 on port 3450. As socat accepts a TCP client connection on port 80, it will fork a new child process, set up a bi-directional socket pairs on the child process and forward all incoming and outgoing TCP packets via these descriptors. The parent process will continue to service new TCP connections.

Until a permanent solution is in place², Bob can now proceed to block non-standard TCP port to ingress the server.

socat the swiss-army knife

An exhaustive list of socat capabilities is documented in the manual page for socat(1) and it is still the authoritative source. Below are some common ones one might need.

Relay TCP to TCP

$ socat -v tcp-listen:80,reuseaddr,fork tcp-connect:172.16.2.10:8080

Relay TCP to UDP

$ socat -v tcp-listen:80,reuseaddr,fork udp-connect:172.16.2.10:8080

Relay UDP to UDP

$ socat -v udp-listen:80,reuseaddr,fork udp-connect:172.16.2.10:8080

Relay UDP to TCP

$ socat -v udp-listen:80,reuseaddr,fork tcp-connect:172.16.2.10:8080

Relay TCP to Unix Domain Sockets

$ socat -v tcp-listen:80,reuseaddr,fork unix-client:/var/run/uds.sock

Relay Unix Domain Sockets to TCP

$ socat -v -v unix-listen:/var/run/uds.sock,reuseaddr,fork tcp-connect:172.16.2.10:80

Relay UDP to Unix Domain Sockets

$ socat -v udp-listen:80,reuseaddr,fork unix-client:/var/run/uds.sock

Relay Unix Domain Sockets to UDP

$ socat -v -v unix-listen:/var/run/uds.sock,reuseaddr,fork udp-connect:172.16.2.10:80

Relay SCTP to Unix Domain Sockets

$ socat -v sctp-listen:80,reuseaddr,fork unix-client:/var/run/uds.sock

Relay Unix Domain Sockets to SCTP

$ socat -v -v unix-listen:/var/run/uds.sock,reuseaddr,fork sctp-connect:172.16.2.10:80

Tunnel Unix Domain Socket packets inside TLS relay

Please avoid setting verify=0 on production if you can.

$ socat -v                                          \
    unix-listen:/var/run/uds.sock,reuseaddr,fork    \
    openssl:server.domain.name:443,certificate=cert.pem,cafile=cert.pem,verify=1,key=key.pem,commonname=server.domain.name

Relay TLS packets to Unix Domain Socket (aka virtual patching)

$ socat -v                                                                                             \
    openssl-listen:443,reuseaddr,pf=ip4,fork,key=key.pem,cafile=cert.pem,cert=cert.pem,method=TLS1.2   \
    unix-client:/var/run/uds.sock

Capture TLS packets to /dev/stdout for quick debugging

Please avoid setting verify=0 on production if you can.

$ socat -v                                                                                       				\
    openssl-listen:443,reuseaddr,pf=ip4,fork,key=key.pem,cafile=cert.pem,cert=cert.pem,verify=0,commonname=server.domain.name	\
    file:/dev/stdout

Feel free to drop me a comment if anything. Until next time adios!

¹ Everything in this story is fictional and has been trivialised for the purpose of simplicity. I am a big fan of KISS philosophy although I haven’t had the time to listen to the band KISS at all :smile:.

² Let’s pretend that there is a forcing function here.

Dnssec For The Win

2020-04-15T00:00:00+05:45

Fast Intro

DNSSEC uses digital signature to verify the authenticity and integrity of DNS records. In essence, it sets up a non-spoofable chain of trust right from the root zone down to the authoritative nameserver and further to modern caching resolvers (e.g., BIND, Unbound etc.). Of course, there’s more, on it and some more.

Why DNSSEC

There are many reasons to turn on DNSSEC, some of which some of them are outright detrimental to businesses (loss of revenue for instance).

It may surprise you to find a number of household domains do not use DNSSEC.

In general these are some reasons to use DNSSEC:

DNS Protocol Attacks
BGP Hijacking Attack¹
DNS Hijacking (Credential Theft)
Domain Theft
Cache Poisoning

It goes without saying that DNSSEC is not a panacea :pill:

Turn the secure bits on

To have DNSSEC turned on for a domain, you need three parties working in tandem with each other :revolving_hearts:

The Domain Name Registrar (e.g., Namecheap, Godaddy, tucows etc.) of the domain
The Authoritative Name Server (e.g., BYO BIND/NSD/PDNS, Route53, Vultr, NS1 etc.) of the domain
The “abstracted” Root Name Servers (I say “abstracted” as it will appear almost invisible to us :boom:)

For illustrative purpose, I will use my domain ishworgurung.com.

My personal domain ishworgurung.com is registered using Namecheap and the authoritative name server is at Vultr.

So, to setup DNSSEC for my domain, I need to generate a fresh copy of DS Records (Delegation Signer) from Vultr. Of particular use are:

Key Tag / Key Type (A unique key per DS record used for lookups) see here and here
Algorithm (crypto algorithm)
Digest Type (hashing algorithm)
Digest (hexadecimal representation of the digest)

Vultr by default generates three DS records, copy them to Namecheap’s DNSSEC console.

Handy tools

Some handy tools the internets has to offer :100:

https://dnsviz.net
https://dnssec-debugger.verisignlabs.com
Dig from Bind
- dig +short ds ishworgurung.com
- dig +short dnskey ishworgurung.com
- dig +short nsec ishworgurung.com

Conclusion

Someone on the internet said:

DNSSEC is a tool, not a religion. Please try to understand how the tool works before criticizing it.

And I agree. DNSSEC is a good thing - let’s do more of it; not less.

I leave you with Dr. Casey Deccio’s Hello Summer Break :sound:. Dr. Casey is the original author of DNSViz :beers:

¹Is BGP Safe Yet.

Vlan On Openbsd

2019-12-07T00:00:00+05:45

Three minute guide to set up VLAN on OpenBSD

If you live under the rock, go check out OpenBSD. It’s pretty nice. I have been running OpenBSD on my servers for about five years and I absolutely love it. Simplicity, robust, secure and minimal - you name it; It’s all there. If you got an old intel 386 or an RaspberryPI, it will be sufficient to run the humble OpenBSD.

Now onto how I VLAN my OpenBSD guest.

The vio0 NIC is based on a QEMU’ed guest on the usual VIO(4) driver.

Get these files up:

$> cat /etc/hostname.vio0 
up
$> cat /etc/hostname.vlan8
inet 10.2.1.61 255.255.255.0 NONE vlan 8 vlandev vio0
up

Then restart the network (or simply reboot)

$> sh /etc/netstart

Then after the ifconfig should look like this:

$> ifconfig # removed listing of other interfaces for brevity
vio0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	lladdr ab:cd:ef:ab:cd:ef
	index 1 priority 0 llprio 3
	media: Ethernet autoselect
	status: active
vlan8: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	lladdr ab:cd:ef:ab:cd:ef
	index 4 priority 0 llprio 3
	encap: vnetid 8 parent vio0 txprio packet rxprio outer
	groups: vlan egress
	media: Ethernet autoselect
	status: active
	inet 10.2.1.61 netmask 0xffffff00 broadcast 10.2.1.255

Watch out for inet, parent, vnetid and broadcast in the above listing.

NB: Please note that the interfaces that sit between the packet path should all have equal MTUs (e.g. in the above listing, the MTU is 1500). Any switches that sit in between should also support 802.1q protocol so the VLAN’ed packets can be tagged accordingly. Thanks to martian67 for this.

PSA (Public Service Announcement)

If you run a network with IoT (Internet Of shit Things), consider putting them in a filtered VLAN that can only communicate within the LAN and not to the internet. This is because a lot of these IoT (Internet Of shit Things) call home and could potentially ex-filtrate private informations :boom:

Interesting Links 8

2019-12-06T00:00:00+05:45

Interesting links #8

Ftrace

BPF Performance Tools: Linux System and Application Observability

Interesting Links 7

2019-09-24T00:00:00+05:45

Interesting links #7

TL;DR

Watch Can strace make you fail? strace syscall fault injection by Dmitry Levin who is its current maintainer.

Non TL;DR

I find myself needing to use strace(1) to debug stuffs a lot but recently, I found myself re-visiting it. Enter syscall tampering / fault injection - Youtube talk by Dmitry Levin Fault Injection in strace.

The reason fault injection becomes a useful tool to have in the arsenal is because you may need to dynamically inject error path into codes developed using Go / Rust / C (amongst the modern language ecosystem). With strace(1), from the application’s point of view, it is all as if Linux kernel handled the system call as per your application’s need; where in-fact strace(1) dynamically instrumented the target syscall and injected the fault for your application.

The alternative to strace(1) is a little costly - bcc/ebpf that need to be instrumented at compile-time (but bcc/ebpf is extremely powerful!) or perf (extremely heavyweight).

Until next Interesting links, keep testing injecting those faults in your code! :-)