piratepartyca: Two more arrested for spoofing MusicBox data: prtpty.ca/66b

jjdavis: RT @doctorow How long will the Canadian gov’t hold people without trial?

m0rty: This is my jam: t.co/462ab

acklas: @doctorow at least a year, remember? http://freebyron.org

wilkiz: @lucy2 They may charge them with computer crime, but that can’t hold

acklas: @wilkiz @lucy2 Crazier prosecutions have done. enwp.org/Aaron_Swartz

Our protagonist is at his company office. So are officers of the RCMP. They’re interested in his work on the so-called “MusicBox antivirus”. Nicnus is not affiliated with that project in any way, but you may recall that he wrote the code which originally made the blocker possible.

They’d rather he come with them. Does he have to come? Is he being charged with something? No. He’d rather stay here. Is that a problem? They confer. It doesn’t have to be. Good.

They want to know if he’s aware of the recent rash of activity. What activity? People spoofing MusicBox data, targeting the RCMP data collection. Oh, that. Of course he knows about it. Everyone does.

Is he also aware that his software may be used to perform this exploit? What exploit? The one where people spoof data! That’s not an exploit. Is sending a letter with a false return address an “exploit”? Yes, he is aware that his software makes it possible to create such a letter and send it to the RCMP.

Why would he create such software? Well, he wasn’t the first to do so, if they only care about the spoofing. They are not aware of the prior art. He can show them the code and the posts. It is not a secret. They would like that.

16:00 <nicnus> cops are here

Does he know why the earlier code was written? To demonstrate the unreliability of the data. Unreliability? Surely they understand that they cannot trust anything they get from a data stream that can be spoofed.

They take anonymous tips by phone. This is at least as good as that.

16:02 <acklas> Woah! Are you ok?

16:05 <nicnus> calm down. If I’m helpful, they’re happy 😉

They’ve never needed to track down pranksters leaving fake anonymous tips, because it’s hard to do en-masse. It can be done en-masse with VoIP? Well, no one has at the least. What would they do if that changed? They probably can’t apply computer crime laws to telephone calls. Probably. It’s still impeding the work of the police. Might even be fraud. There are laws in place.

16:20 <nicnus> They’re gone.

* Your connection has been secured

16:21 <acklas> Oops. Should have turned that on sooner.

16:21 <nicnus> meh. Those anonymous kiddies following your lead are in trouble, though

16:22 <acklas> Oh?

16:22 <nicnus> Yeah. At the very least, it’s some form of fraud

* acklas shrugs

16:23 <acklas> Their fault for not being more careful

Nicnus is waiting for a bus. He only went in to the office to meet with the officers. They came to his workplace and expected him to be there. So old-school.

There’s a chilly breeze blowing. Nicnus slips his hands up inside the sleeves of his hoodie. When his phone vibrates, he waits a few minutes before checking.

m0rty: Cops were just here. Something about my computer sending them messages?

Oh, man.

@m0rty your computer is probably infected with a botnet virus.

The bus is here. He gets on, but there are no seats available. There is barely a place for him to stand. And some people think no one takes the bus around here! He keeps one hand on a bar for stability, the other on his phone.

m0rty: @nicnus A virus? I have Norton and AVG both.

Nicnus stares at the screen for a moment. The bus lurches, throwing him against some blonde in her twenties. He plants his feet and switches to holding one of the provided handgrips on the bus.

@m0rty having two probably not helping. You can get it anyway. Then your system does what it is told.

People are moving. He slides his phone closed and looks around. Someone is trying to get off the bus. They should have made their way to the door long before now. The door is sitting open, and they are pressing through the crowd. There, they’ve made it. People flow towards the back as someone boards. Now he is wedged against some hefty guy who has a seat. Good thing he didn’t bring his backpack today. He manages to slide open his phone again.

jjdavis: @m0rty @nicnus is right. Running two antivirus not a good idea.

acklas: @m0rty You should probably just always assume your Microsoft systems are infected.

m0rty: @nicnus What should I do to get rid of it?

Nicnus cannot write a reply. No way to manipulate the phone properly just now.

jjdavis: @m0rty Only sure way is to wipe and restore from backup

acklas @m0rty might not be worth it. You’ll just get infected again

jjdavis: @acklas that’s really optimistic of you

The bus lurches again. He is face-to-face with the same blonde. Suddenly aware of his surroundings instead of his phone, our protagonist realises that this is probably awkward. He turns his head. No longer face-to-face, but now he’s staring at her shoulder. That’s probably not better. He looks over her shoulder and out the window. It’ll have to do.

m0rty: @jjdavis @nicnus I don’t really keep backups. Don’t want to lose my family videos! :S

16:31 <acklas> Who is this m0rty guy anyway? Where is he?

16:31 <jjdavis> Some guy nicnus knows. I don’t this he’s near any of us, though.

Now everyone is exiting the bus. Everyone? Oh, no, just most everyone. It’s a major stop, and this is where most of the people were going. Nicnus has two more stops to go, so he finally gets a seat.

16:33 <nicnus> friend from highschool. He lives out west.

@m0rty I wouldn’t worry to much about it. Let me know if you have more trouble

His stop is next. He pulls the signal cord and makes his way to the door. Phone goes into his pocket. Bus is stopped. Door is not opening. Oh, red light. Driver is waiting to pull ahead. Whatever. Open now. Getting off.

Our protagonist trudges towards his home. His mind is on the motion of his feet. Should he take longer strides? Maybe he should try to avoid the cracks in the sidewalk. Careful, there’s something on the ground. Probably don’t want to know what it is. Now he notices the people walking ahead of him. Two abreast and blocking the sidewalk completely. Not walking very quickly. He plods behind them for a few seconds before pulling over onto the grass and walking past.

Home. His phone is still in his pocket, but he doesn’t really need it. His desktop is already on, since he hadn’t expected to be gone long.

* Your connection has been secured

16:45 <nicnus> acklas: have the cops been to see that lady on your street again?

16:45 <acklas> nicnus: yeah. different ones. They were nicer this time. Figured out pretty quick she was not the actual source of the traffic.

16:46 <jjdavis> So someone is being more careful. It’s not all kiddies

16:46 <nicnus> It’s just kiddies getting caught

16:50 <acklas> the kiddies always get caught

16:51 <nicnus> the cops I talked to likened the MusicBox data to anonymous tips. So they don’t expect it to be admissible in court, I guess

16:53 <jjdavis> I guess that answers your brother’s concerns about evidence fabrication

16:55 <nicnus> yeah, maybe

Hungry. Food. First, music. What music? Something energetic. He opens an encrypted tunnel to a computer in the United States so that he can access a music service that only works for “people in the US”. He asks it to play energetic music, based on his past preferences, and it queues up what looks like a stream of evenly-mixed Dubstep and Power Metal.

Food.

His fridge is empty. Typical. Bread. He has bread. Good. Toast. What to put on the toast? Does he need to put anything on the toast? How hungry is he? Toast first, then re-evaluate.

What is he working on today? Oh, right, the discovery process for the encryption box. Some customers have reported that sometimes the boxes cannot find each other, even if they are plugged directly in to the same hardware. That doesn’t make much sense, but he is going to try to reproduce it.

One hour passes. Then two. Nicnus remembers he put toast in the toaster. It’s cold now. Whatever, eat it anyway. He isn’t really closer to finding the problem. Well, he’s a little closer: he’s pretty sure there *is* a problem. He just is not just quite what it is, or how to make the problem happen. It happened once, but then worked when he tried again.

00:10 <jjdavis> Taking the g/f out. Will not be very responsive for a bit

* jjdavis is away: out with the g/f

Nicnus is pulled from what he is working on. 00:10 UTC? What is that in local time? He looks out his window. Early evening, looks like. Where did his day go? He looks at the code that he has been working on. It might be the cause of the problem, but it is hard for him to check since he can’t surface the problem reliably. How will he know if changing this has fixed it?

03:52 <jjdavis> That went pretty well. Maybe I’m getting the hang of this

03:55 <acklas> or maybe she’s just being nice 😉

What time is it? Probably should think about dinner. What has he eaten today? Orange juice for breakfast, and toast for a late lunch. Now that he’s thinking about it, he is very hungry. Why wasn’t he hungry a minute ago? What was he working on? He can reproduce the problem now, but it is not what he thought it was. He has no idea how to fix it, but at least he can test it. Food. He should eat food. Does he have any food? He has some frozen yogourt in his freezer, and some peanuts. Dinner!

Someone sends him a link to a funny video. He watches it twice while he eats. It’s pretty good. The end of the video contains an obscure reference to some product from Suzuki. Now he’s reading Wikipedia articles about the Dutch East India Company.

06:01 <acklas> bed time

* acklas is offline

The issue with the encryption box might be in the hardware, and not in the software at all. How would he test that? Oh! They have an all-software emulator for it, which they built so they could test the software back before the hardware existed. It’s a bit out of date, since they don’t use it anymore. After some testing, he finds that it has the same problem. The problem is definitely in the software.

His music stopped at some point. When did it stop? Seems that his encrypted tunnel to the US has stopped working. He starts up another one.

09:00 <nicnus> Woo! I think I’ve got this bug nailed. Time for some sleep

Let’s say you have some code and you want it to be able to iteract in a very similar way with either a network connection or a named pipe. You might structure this using a sum type (also called a tagged union):

data Stream = Network NHandle | Pipe PHandle

def interaction(h)
	input = case h
		when (Network handle)
			readNetwork(handle)
		when (Pipe handle)
			readPipe(handle)
	end

	output = transform_input(input)

	case h
		when (Network handle)
			writeNetwork(handle, output)
		when (Pipe handle)
			writePipe(handle, output)
	end
end

Since “dynamic typing” is just the practise of giving every value the exact same type, which is a giant sum type, a dynamically typed construction would look like this:

def interactionDynamic(h)
	input = case h
		when is_a?Network
			readNetwork(h)
		when is_a?Pipe
			readPipe(h)
	end

	output = transform_input(input)

	case h
		when is_a?Network
			writeNetwork(handle, output)
		when is_a?Pipe
			writePipe(handle, output)
	end
end

This is basically the same, but it introduces a tradeoff. In the original, if we wanted to add another supported stream, we needed to add a case to the sum type *and* cases to the function, and in return we were guarenteed that we would only ever be passed cases we could handle. With the dynamically typed version, we can just add cases to the function, but any value can be passed to us and if we can’t handle it the code will just crash.

Neither of these solutions are very nice in this case. We really want abstracted operations:

def read(h)
	case h
		when is_a?Network
			readNetwork(h)
		when is_a?Pipe
			readPipe(h)
	end
end

def write(h,output)
	case h
		when is_a?Network
			writeNetwork(h, output)
		when is_a?Pipe
			writePipe(h, output)
	end
end

def interactionDynamicAbstract(h)
	write(h, transform_input(read(h)))
end

Much nicer! One issue here, though, is that these operations really are fundamental to Networks and Pipes, and we have defined them as part of our code. It is likely that other people using Networks and Pipes will end up writing similar code. What if we bundled up all the operations you could do with a Pipe right alongside the data?

def interactionDuckTyped(h)
	h.write(transform_input(h.read()))
end

Now the operations live inside the datatype. This has the added advantage that we don’t even need to write any code to support a new type of stream, as long as the stream comes packaged with a read and a write operation.

There are two issues here, though. One is very similar to the tradeoff we made for dynamic typing, which is that the compiler cannot infer from what we’ve written which operations are necessary and so cannot check that values being passed to our procedure actually implement these operations until runtime. The other problem is that the only thing we know about the operations for sure is their names. We cannot easily document that these operations should have certain properties, because the users of our function cannot easily what operations we are using.

Can we fix these issues while keeping the extensability and power? We could try bundling the operations up seperately:

def interactionRecord(h, ops)
	ops[:write](h, transform_input(ops[:read](h)))
end

Now we can know that these operations will be present, and we can document that the operations the user passes in must satisfy certain properties. This seems like a pretty good solution. We have, however, leaked the management of which operations to use back to our user. Can we keep this safety while regaining the conveniance of having the stream type author specify the operations instead of our user?

This is exactly what typeclasses do. Consider:

typeclass Stream a where {
	void write(handle, bytestring);
	bytestring read(handle);
}

hastypeclass Stream Network where {
	write = ...;
	read = ...;
}

def interactionTypeclass(h)
	Stream::write(h, transform_input(Stream::read(h)))
end

Now we have a natural place to document the properties operations must have (the typeclass definiton), and their relationships with each other. The compiler can look up the correct hastypeclass to figure out what code to run for each type, and while it does so it can check that the operations we need (the ones that are part of Stream) are in fact defined for the input type.

Safety, extensability, and ease of use, all in one solution.

Last time we got IPC working, and used it to build a simple path-resolution module. This time we’re going to write a fully-functional serial port driver.

Generic Interrupts

When we got interrupts working, we sort of hacked the kernel part in to support the timer, which is all we were interested in at the time. Let’s make that a bit more generic. We’re going to need one more syscall, since interrupts have to be handled by the kernel. The simplest one we can implement just allows a process to wait for an interrupt to fire on a particular line:

void interrupt_wait(int intr);

.global interrupt_wait
interrupt_wait:
	push {r7}
	mov r7, #0x5
	svc 0
	bx lr

#define TASK_WAIT_INTR 3

case 0x5: /* interrupt_wait */
	/* Enable interrupt */
	*(PIC + VIC_INTENABLE) = tasks[current_task][2+0];
	/* Block task waiting for interrupt to happen */
	tasks[current_task][-1] = TASK_WAIT_INTR;
	break;

We are already enabling the timer interrupt when the kernel starts. Here, we enable the interrupt that the process is asking to wait on, since if it is not enabled the task will block forever, so this seems like a useful place to do it.

We also want to unblock tasks when the interrupt fires. Get rid of the -4 case that we have hardcoded for the timer and put this in:

default: /* Catch all interrupts */
	if((int)tasks[current_task][2+7] < 0) {
		unsigned int intr = (1 < < -tasks[current_task][2+7]);

		if(intr == PIC_TIMER01) {
			/* Never disable timer. We need it for pre-emption */
			if(*(TIMER0 + TIMER_MIS)) { /* Timer0 went off */
				*(TIMER0 + TIMER_INTCLR) = 1; /* Clear interrupt */
			}
		} else {
			/* Disable interrupt, interrupt_wait re-enables */
			*(PIC + VIC_INTENCLEAR) = intr;
		}
		/* Unblock any waiting tasks
			XXX: nondeterministic unblock order
		*/
		for(i = 0; i < task_count; i++) {
			if(tasks[i][-1] == TASK_WAIT_INTR && tasks[i][2+0] == intr) {
				tasks[i][-1] = TASK_READY;
			}
		}
	}

You’ll notice we also need a new magic number for versatilepb.h:

#define VIC_INTENCLEAR 0x5 /* 0x14 bytes */

Code for this section on Github.

Output Driver

Let’s use this new ability to implement half of our driver: output. This will allow us to write to a specific fifo for output to the serial port instead of using bwputs.

A refresher of serial port related magic numbers:

/* http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dui0224i/Bbabegge.html */
#define UART0 ((volatile unsigned int*)0x101f1000)
/* http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0183g/I18381.html */
#define UARTFR 0x06 /* 0x18 bytes */
#define UARTIMSC 0x0E /* 0x38 bytes */
#define UARTFR_TXFF 0x20
#define UARTIMSC_TXIM 0x20

#define PIC_UART0 0x1000


Let’s make a generic driver that accepts parameters telling it what serial port to drive:

void serialout(volatile unsigned int* uart, unsigned int intr) {
	int fd;
	char c;

We need to create the fifo that processes send bytes on:

	mkfifo("/dev/tty0/out", 0);
	fd = open("/dev/tty0/out", 0);

We also need to tell the UART to trigger an interrupt when it becomes ready to send a byte:

	*(uart + UARTIMSC) |= UARTIMSC_TXIM;

And finally a loop to read bytes and send them to the UART:

	while(1) {
		read(fd, &c, 1);
		interrupt_wait(intr);
		*uart = c;
	}

This seems pretty reasonable, but there is a problem. The UART triggers an interrupt when it becomes ready to send a byte. If it is already ready (such as at boot time), the interrupt will not trigger. So we try this:

	while(1) {
		read(fd, &c, 1);
		*uart = c;
		interrupt_wait(intr);
	}

This is closer, but in both of these versions there is a race condition where the UART might not actually be ready anymore by the time we process the interrupt (depending on what other tasks are doing), so let’s add a check for that:

	while(1) {
		read(fd, &c, 1);
		if(!(*(uart + UARTFR) & UARTFR_TXFF)) {
			*uart = c;
		}
		interrupt_wait(intr);
	}

Now, however, if the UART is not ready, we will skip bytes. So we only want to read a new byte if we actually wrote the current one:

	int doread = 1;
	while(1) {
		if(doread) read(fd, &c, 1);
		doread = 0;
		if(!(*(uart + UARTFR) & UARTFR_TXFF)) {
			*uart = c;
			doread = 1;
		}
		interrupt_wait(intr);
	}

To test this out, let’s set up a new first:

void first(void) {
	int fd;

	if(!fork()) pathserver();
	if(!fork()) serialout(UART0, PIC_UART0);

	fd = open("/dev/tty0/out", 0);
	write(fd, "woo\n", sizeof("woo\n"));
	write(fd, "thar\n", sizeof("thar\n"));
	while(1);
}

Our current pre-emption speed will make this a bit of a pain to run, so you can set the clock value in main to something lower.

We can now print to the serial port by writing to a fifo!

Code for this section on Github.

Input Driver

Some new magic numbers:

#define UARTICR 0x11 /* 0x44 bytes */
#define UARTFR_RXFE 0x10
#define UARTIMSC_RXIM 0x10
#define UARTICR_RXIC 0x10
#define UARTICR_TXIC 0x20


The input driver will be another process that waits on the same interrupt. Unfortunately, because this one interrupt actually can signal one of several things, if two of them are true, and different processes work on processing them, the interrupt will fire constantly and prevent any work from getting done. We solve this by clearing the more specific interrupt in the drivers where we handle them. For example, in serialout:

*(uart + UARTICR) = UARTICR_TXIC;

Next, start with some setup:

void serialin(volatile unsigned int* uart, unsigned int intr) {
	int fd;
	char c;
	mkfifo("/dev/tty0/in", 0);
	fd = open("/dev/tty0/in", 0);

	/* enable RX interrupt on UART */
	*(uart + UARTIMSC) |= UARTIMSC_RXIM;

	while(1) {

We want to wait on the interrupt, and then immidiately clear the specific interrupt we handle, just like in the output driver:

	interrupt_wait(intr);
	*(uart + UARTICR) = UARTICR_RXIC;

And we want to make sure, not only that no race condition has changed the UART status, but that the interrupt was meant for us at all:

	if(!(*(uart + UARTFR) & UARTFR_RXFE)) {

And then just read the byte and put it in our fifo:

	c = *uart;
	write(fd, &c, 1);

This input server has a bit of a bug. If the fifo is ever full, it will block, and maybe miss input bytes from the serial port. Our processes are likely to be way faster than any serial port, but still, in a real kernel you would want to account for this by splitting the driver into two processes: one that waits on the interrupt, and one that buffers the bytes internally.

Code for this section on Github.

Echo Application

To tie all this up, we write a simple application that will echo back every character you type:

void echo(void) {
	int fdout, fdin;
	char c;
	fdout = open("/dev/tty0/out", 0);
	fdin = open("/dev/tty0/in", 0);

	while(1) {
		read(fdin, &c, 1);
		write(fdout, &c, 1);
	}
}

Code for this section on Github.

We’re done!

We now have a working serial port driver that respects the hardware by using interrupts instead of burning CPU time looping on flag checks. Remember, any interactios with the UART will affect these drivers, so using bwputs may have strange side effects now.

To be honest, this is about as far as I expected to get when I originally started this series. I could write a clock driver, to allow for sleep calls and similar, but you should be able to see how that might work. One big topic I have avoided thus far is enabling the MMU and doing virtual memory / memory protection. I may try to figure out how that works on this harware, but it will require quite a bit of doing. Suggestions for others things you would like to see in this series go in the comments!

Code so far on Github.