Alien Coder

GSoC 2011 Project: FreeBSD Path-based file system MAC policy

2011-05-02T20:47:00.000-07:00

I've been accepted into Google Summer of Code (2011) once more. If you don't know and are curious, I was accepted into Google Summer of Code 2008. This summary email to the OpenChange mailing list better describes what I accomplished that summer of 2008 with the OpenChange and KDE projects and my mentor Brad Hards.

This year I applied to the FreeBSD project and you can find a description of my proposal here.

I've already started talking to my mentor (Pawel Jakub Dawidek) and we're in the brainstorming phase.

There are some interesting obstacles that I will be facing this summer that I am looking forward to overcome. To begin with, except for starting to write a small hobby OS for educational purposes a long time ago, I've never really done any kernel-level development in the past. I have read much about OS design and implementation principles and have read through kernel sources. I'm also currently reading The Design and Implementation of the FreeBSD Operating System.

Engineering-wise the first and foremost challenging obstacle that I am currently facing is the fact that the MAC framework doesn't really forward any path metadata to its modules. So, I'm trying to find a way to do this in a clean and modular way. Once I do, I can then use this data to match it with the vnodes of files that access needs to be restricted on.

I will try to keep posting as much as I can as the project develops.

As a bonus, here's a tool I've been using for browsing the FreeBSD sources. It's an LXR-based FreeBSD and Linux Kernel Cross-reference site. It also has the sources for other operating systems such as NetBSD, Darwin, Plan 9 and MINIX.

Preliminary screenshot

2009-12-28T20:07:00.000-08:00

I've been working hard on getting the Files utility to work by working on the Utility Layer.

My goal is to get the (0.1) Files utility finished by the first week of January (and it seems like I should be able to make it).

I just wanted to quickly post a screenshot of where I am so far.
There's a lot to explain as to what's going on behind the scenes because I've added a lot of stuff, so I'll save that for later.

Here's the screenshot:

The only thing that is hard-coded here are the "Static" folder and file shown. Not for long however, as the client and server can already synchronize the files/folders that are in the server The client just doesn't parse and display that information yet.

Also, I would like to push out our 0.1 pre-alpha release out by February 1st. If it all goes, I think we should be able to do it.

Update:
So we've pushed out 1.0, but I forgot to show an update of what the files utility is looking like now.
This is on Ubuntu, instead of Kubuntu:

Utilities++

2009-11-28T21:25:00.000-08:00

I've gotten a lot done this weekend on getting workplace and utility data synchronized between the client and server on login.

The task was easier than I expected, but still a lot had to be done.

It won't seem like much in this screenshot, but believe me, it was quite a bit of work:

It might actually seem like less is going on now than in our previous screenshots.

What happened before was that all those Utilities and Workplaces that were loaded on the GUI, were manually loaded.

What I've done this weekend is that I've added a few messages to our client<->server protocol to synchronize workplaces and utilities over the network when a client logs in. Also, they are actually utility plugins, and not just made up utilities.

Another thing here is that the Utility icon is actually part of the utility plugin binary, and Mira core gets it from the plugin at run-time.

Also, this weekend I finally got a new laptop and I have a dual-boot setup with Windows 7 on it.

So, my next task is to set up a dev environment around MS Visual C++ express 2008 and get Mira to compile on it. I tried once a long time ago, but gave up on it because a lot of code had to be re-written.

However, this code has already been removed, so I don't expect this to take too long this time around.

Also, all the work I've done on the plugin has been tested on linux, but not on windows, which prevents me from pushing these changes to trunk according to the rules we laid out a while ago in a meeting.

After I'm done getting Mira to work with Visual C++, I'll then work on a simple Files utility and get ready for our 0.1 release :).

I expect us to be able to release 0.1 before mid-January if we keep up this pace.

Mira Client Utilities

2009-11-25T18:18:00.000-08:00

I just finished laying the framework for Utilities (Plugins) on the Client.

The design is more Object Oriented and cleaner than it is for the server thanks to QPluginLoader.

This is an example of what a plugin looks like:


#include "UtilityInterface.h"
#include "GuiInterface.h"
#include "NetworkInterface.h"

class FilesUtility : public QObject, public UtilityInterface, public GuiInterface, public NetworkInterface
{
    Q_OBJECT
    Q_INTERFACES(UtilityInterface GuiInterface NetworkInterface)

    public:
        FilesUtility();
        ~FilesUtility();

        bool initialize_gui(QLayout* qt_layout);
        bool receive_message(const std::string& workplace_name, const std::string& message);
        bool load_workspace(const std::string& workspace_name);
};

Basically you have to implement interfaces in order for Mira core to be able to interact with the Plugin.

There are some interfaces that are going to be required, and others are going to be optional.

With this design it is also really easy for utilities to communicate with each other. All that is necessary is for utilities to share interfaces, request the other utility's object, and just use the interface. This is not implemented yet, but it's fairly easy to do.

All that's needed now is a good library for plugins to use to interact with Mira core. This will probably be an ever-evolving library, at least until we get to release our first version of Mira.

After I get the library started, I will propose a merge with trunk. But for now you can follow this code in my Launchpad branch.

Mira Utilities via Plugins

2009-11-08T18:46:00.000-08:00

After reading through a lot of material on the web on dynamic libraries (aka plugins) using C++, I've come to a basic design for plugins in the server.

It consists of using the functions provided by the OS (dlopen/dlsym for Unix-based systems including Mac OS X and LoadLibrary/GetProcAddress for MS Windows).

Each plugin will have to provide two functions with C linkage:
- extern "C" void create_utility(WorkPlace* workplace)
-

extern "C" void receive_message(const std::string& workplace_name, TcpConnection* tcp_connection, const std::string& message)

The first function (create_utility) is a factory function that takes care of creating new instances of the utility.

The way it should work is that a new instance of the utility class is created for each workplace. This Utility class wouldn't have to derive from any other base class as is normally done with C++ plugins.

The second function (receive_message) is used to receive network messages coming from clients, mostly from its utility counterpart on the client-side.

Keep in mind that this implementation will only be for the server. The Qt QPluginLoader framework will probably have to be used on the client. I didn't want to use this on the server to not introduce Qt as a dependency for the server.

Also, the plugin system on the server doesn't require much flexibility. The current design is rather simple and seems to work quite well so far. The UtilityManager class is less than 50 (real) lines of code.

One thing I haven't introduced yet is a way for utilities to communicate with each other internally (within the server). A few ideas that I can think off the top of my head to implement this, might to require the addition of another function with external C linkage to pass internal messages. Not sure how flexible this would be do.

Another option would be to use Boost.Signal2.

With this, and some more work on the Mira framework, I should be able to produce some simple utilities rather easily. Still some design issues that I'm gathering need to be worked out.

Mira client connects to Server

2009-11-07T13:29:00.000-08:00

There isn't much to say about this, so I'll just show some screenshots.

These two screenshots show some error handling I've added to the network code. Apart from this and a handler for error messages coming from the server, I haven't really added anything else to the network code.

The client code has been able to connect to the server, but nothing had been done to be able to do this with the GUI. This is just to show the little progress I've made this weekend.

The code is currently in a separate branch from trunk which Shilpa created for handling network messages on the server. I'm waiting on her to finish up some work on this and then we're going to merge these changes with trunk.

Security in the Directory Layer

2009-10-31T17:17:00.000-07:00

After brainstorming for a while I came up with an idea for the security stuff in the directory layer.

In the existing code it's referred to as an access list.

Basically different resources are registered with the Directory layer. Each resource has an access list which is basically an user id associated with a permission.

Right now the access is an int8_t with the following constants defined for identifying what each value represents:

 static const uint8_t  READ_ACCESS        = 0x01;
 static const uint8_t  WRITE_ACCESS       = 0x10;
 static const uint8_t  READ_WRITE_ACCESS  = 0x11;
 static const uint8_t  NO_ACCESS          = 0x00;

What identifies the resource is a simple std::string object. This makes it flexible enough so that we can provide this functionality to utility developers and new resources can be created at run-time without creating conflicts.

However, because of this we will need to enforce a naming convention for resource names. This will need to be enforced by only allowing utility developers to create resource access lists through our API.

The naming convention should probably be something like this:

- For server resources: "server_resource"

- For workspace resources: "workspace_workspacename_resource"

- For workspace resources created by utilities: "utility_workspace_utilityname_workspacename_resource"

ie: In the Directory database in trunk there is an existing resource called "server_add_new_user". This is a resource that is used to determine who can add new users to the server.

In this case, a read access would only allow to see who has access to the resource, and a write access would allow to actually use the resource (Add new users to the server).

There is one more thing that we've talked about which is Server and WorkPlace roles. I haven't implemented this yet, but I have a good idea of how to do so (probably sometime next week).

I think the Role code should be more like a script (not literally of course). All it will do is know how to give the person the role is given to, the right access, to the right resources. So if a role of Server Administrator is applied to a user, then the Role code will give the user read/write access to most server functionalities, such as stopping the server (server_stop), restarting the server (server_restart), adding new users (server_add_new_user), etc... This should be easy to implement. We just need to find out what resources to give access to for each Role.

Mira connects to peers on LAN

2009-10-12T18:53:00.000-07:00

Ok... so it wasn't as easy as I thought to get the network code going for p2p. I think the discovery part was actually easier.

Either way I do have something to show:

What you see is my regular laptop (clsk@aliensoft (192.168.0.3)) + a virtual machine (aalvarez@alientest (192.168.0.4)) both running a Mira client. You can see that they discover each other and connect to each other as well.

Right now, they create two connections when they discover each other, but I've already got a system down so that they chose which one will connect just haven't implemented it yet. Also, I haven't integrated it to the GUI just yet, but that will be the easy part.

Either way, that's the final product of long hours of scratching my head, trying to figure out how Avahi really worked and re-designing the network code. There's a lot Mira will be able to do from here. Getting Mira to do what Giver does is just a matter of implementing drag&drop and sending the files through the connection already established.

Also, the network code is multi-threaded. Right now the whole application runs 5 concurrent threads which is more than enough. So for those that have multi-core processors, Mira will take advantage of them.

Update: Here's a screenshot with the ContactDockWidget userlist being updated when a connection with a Neighbor (P2P) is established.

Also, I've created a new branch in launchpad for this code called p2p. This will live there until I can port the code to Windows and OS X

Service discovery on the LAN on Mira

2009-10-11T14:50:00.000-07:00

So I don't have much to show, but here is something...

This means the Mira client application can register its own peer-to-peer service and discover other clients on the network.

There's some more to do before I can actually show some useful screenshots. Mainly re-doing the client networking code. Shouldn't be much work since I'm just going to use the Server's network code as a template. Like I did before, but this time taking into account the fact that the client is somewhat of a server as well.

By the way, this only works on linux so far. I plan to do a version using Bonjour so it'll work on windows and Mac OS X. So for now I'll be posting to a separate branch.

Restarting Mira development

2009-10-10T21:12:00.000-07:00

Another project I used to work on, and plan to start working on again is Mira Groupware.

Its development has been inactive for some time, but that won't be for long.

After watching a video on Giver (an application developed by a Novell programmer), it motivated me to start working on it again.

Giver is a great application, and I suppose it does what it is supposed to do, but it has so much potential. However, development for it has stopped completely as it was just a weekend (or week-long) project. I could just expand Giver, but I don't think it was designed to do all the things I have in mind. Which Mira Groupware is. The network code for Mira has been ready for a long time. I'm now working on the Zeroconf (service discovery) on linux using Avahi (the same library Giver uses).

Hopefully I'll have some screenshots to show in a few days/weeks.

libmapi++ not header-only for long

2009-10-09T18:02:00.000-07:00

So after trying to clean up the code in the Akonadi Resource it seems like libmapi++ is not going to be a header-only library much longer.

Either way, the main reason for keeping it that way was to make it easier for other people to use. Now that libmapi++ is part of the openchange build-system, and it's distributed as part of the openchange distribution, that is not really needed anymore, because it gets "built" anyway.

However, that's going to be in the near-future. First I want to concentrate in getting the resource stable and working with the current version of akonadi and whatever client is able to access it (which I'm still trying to figure out).

After a quick trial, Mailody seems not to be very up-to-date with Akonadi development. So right now I'm trying to figure out what to use to test the resource.

Brad Hard's OpenChange/Akonadi Talk

2009-10-08T18:23:00.000-07:00

Here's a link to Brad Hard's (My GSoC Mentor) talk on OpenChange and Akonadi:

http://www.kdedevelopers.org/node/3905

So How Was GSoC 2008 after all?

2009-10-08T18:04:00.001-07:00

So I never used this for GSoC. It's actually been a while since I got accepted and finished GSoC.

So I'll tell a brief story of how it went.

After being asked to decide for one of the two projects, I decided to go with the Akonadi OpenChange resource. I would say that the biggest factor was the fact that I already had a relationship with who would be my mentor before I was even accepted. But I was very interested in both projects.

The summer went well, and I accomplished almost all tasks of my project.

You can see most of what I accomplished here.

After the summer ended I redeployed from Iraq back to the U.S.

I've also moved a lot of times since then, so I never really had a chance to completely settle down until now.

So now that I'm somewhat settled down, I've decided to start working on the Akonadi openchange resource again. Most of what is left is sending information to the Exchange server, as I almost finished pulling information from it during GSoC.

I've already fixed an annoying bug that made the resource crash when an openchange profile database wasn't created (which is the case for all new users).

The bug slipped by because I always had a profile database while I was working on the resource before. I guess starting all over again brought something good to the table :).

So, After installing MS Exchange 2008 R2 Preview, I found out that you can't install Exchange server in it, so now I'm going to try and install MS Exchange Server 2008 (not the R2 version) and see if that works.

I hope to start using this again to keep people updated.

Haiku Project Proposal

2008-03-24T03:31:00.000-07:00

Below is part of my project proposal for Haiku. I have removed the first part of the application that has personal information I would rather not post on this blog. Also this post expands on the content within the application with graphics and more detailed explanation.

Project Idea

What I will be trying to achieve this summer is to implement ICMP and an error-handling mechanism for the network stack. I have been reading the network stack code and already have a good understanding of how it works internally.

The main goals for this project are to:

Implement the ICMP module (for IPv4) as described in RFC792: This will require minor modifications to the IPv4, TCP and UDP modules. Right now the IPv4 header is being removed from the net_buffer before it is being passed to the other protocols (TCP, UDP or ICMP). Some ICMP messages require (in some cases) that the original IPv4 header + 64 bytes of the upper layer header (TCP/UDP in this case) be sent back as part of the ICMP header so that the receiving host can identify which packet caused the error.

Create an error-handling mechanism for IPv4, TCP and UDP: I will use the functions already set for the net_modules. Specifically error() and error_reply(). This will be implemented in such a way that the TCP and UDP modules are independent from the ICMP module and all messages are transmitted through the IPv4 module, like so:

The error-reporting mechanism will work as follows:

An ICMP message is received from a remote host and is passed to the ICMP module from the IPv4 module.

If the message is regarding a previously-sent packet by either the IPv4, TCP or UDP layers, the message will be sent to the IPv4 module using the error() function.

If the message is regarding a previously-sent packet by either the TCP or UDP layers, they are sent to these modules using their error() function and the message is handled accordingly. Otherwise they are processed and handled by the IPv4 module.

To reply to a message the TCP and UDP module will use the IPv4's reply_error() function to pass the reply message down to the ICMP module using the send_data() function.

Below is a graphic representation of this process:

Document the work described above: I plan to document all the work that I do. I will use this blog so that people can track my progress and also to show step-by-step how I implemented the work described above. I will also prepare a separate document which will more likely be based on the contents of this blog by the end of the summer. If time permits I would also like to document the whole inner-workings of the network stack based on this document.
I’m not sure who the author is, so I might have to start from scratch to avoid copyright issues.

First Post + GSoC 2008

2008-03-24T00:30:00.000-07:00

I set up a blog once before, but never really posted much to it since I didn't have much stuff to talk about that other people would even care about.

Now things have changed since I'm planning on applying for Google Summer of Code 2008 and (I think) I have a good chance of getting accepted.

I'll be using this blog mainly so that other people can track my progress on whichever project I get accepted.

As of right now I'm planning on applying for two projects. One is to implement a resource (basically a backend plugin) to Akonadi. The resource I would be implementing in the event that I get picked up is an OpenChange resource to allow users to interact with MS Exchange Servers.

I will also be applying to implement ICMP and an error-handling mechanism for the Haiku network stack. This will also include Path MTU discovery.

I already have a pretty good idea of how to do this since I've spent a couple of hours looking at the network stack code and I must say that Haiku has one of the cleanset network stack implementation I've ever seen in my life. It was relatively easy to follow the code. I think this might have to do wit the fact that it's written in C++. Some might argue that having C++ in the kernel is stupid. I think otherwise, and Haiku has definitely proven those people to be wrong.

For anyone interested here is a basic explanation of the internal design of the Haiku Network Stack that I (unfortunetly) found after I already knew the how things worked. It did help clarify one or two things afterwards though.

As you have seen both projects I've chosen to apply for are in C++. C++ is my native (programming) language. It's the first one I learned (except for BASIC in like 4th grade) and it's the one I have the most experience on. It also happens to be the one I like the best. I've started to appreciate python lately however, but that's another story.