\documentclass[a4paper]{article}
\usepackage{hyperref}
\usepackage[pdftex]{graphicx}
\usepackage[numbers]{natbib}
\usepackage{verbatim}
\usepackage{color}
\usepackage{listings}
\usepackage{float}
\bibliographystyle{plainnat}
\floatstyle{boxed}
\newfloat{code-example}{thp}{lop}
\floatname{code-example}{Code Example}

\begin{document}
\lstset{showstringspaces=true,
		language=C++,
		frame=single,
		breaklines,
		float=tbph,
		prebreak={\space\hbox{\ensuremath\hookleftarrow}},
}

\newcommand{\registrykey}[1]{$#1$}

\title{Implementing Distributed COM in Samba}
\author{Jelmer Vernooij {\em jelmer@samba.org} 
	\\ Samba Team Member}

\maketitle

\begin{abstract}
The ``Component Object Model'' (COM),  is one of the key 
features of the Microsoft Windows platform. COM allows 
developers to use and create interfaces that can have several 
implementations and can be called by different programs, written in 
different languages. The involved parties need only agree upon the interface.

COM is most widely known as the core of other technologies such as OLE, 
ActiveX and Automation. Even the more recent .NET platform supports COM.

DCOM is the distributed version of COM. It is implemented on top of 
DCE/RPC and was documented in an Internet RFC. This document describes both
the way that DCOM works and how it will be implemented in Samba-4.

{\large This document is a draft and is by no means finished yet. However, 
suggestions and comments are always welcome.}
\end{abstract}

\tableofcontents

\section{Introduction to 'plain' COM}
The terminology used in the COM world can be quite confusing. Especially 
all the various ID's \footnote{so far I have seen: MID, OXID, OID, IPID, CID, IID, CLSID, LCID and DISPID.} and the different meanings of the word class tend 
to confuse people. A broader introduction to COM is available on the web at \cite{EssentialCOM} and \cite{StepByStepCOM}.

Basically, there are three entities that are important in COM:

\begin{itemize}
\item Interface
\item Implementation
\item Instance
\end{itemize}

COM has a very clear distinction between interface and implementation. Calls 
are always made by interface, while the specific implementation is only 
mentioned when a new object is created.

\begin{figure}
\includegraphics[scale=.40]{interface-implementation}
\caption{An example of COM's distinction between interface and implementation}
\end{figure}

\subsection{Interfaces}
Interfaces are the basic elements in COM. An interface is nothing more 
than a definition of a list of methods that can be called on an 
object. 

Interfaces support inheritance, so if interface $A$ is based upon $B$ it 
contains all of $B$'s methods as well as its own. All interfaces are based 
upon the ``IUnknown'' interface. 

Interfaces are identifier by Interface ID's (IID's). IID's are 
globally unique (they are GUID's).

Windows hosts keep a list of interface in their registry, at 
\registrykey{HKEY\_CLASSES\_ROOT{\backslash}interface}.

Note that \registrykey{HKEY\_CLASSES\_ROOT} is a ``dynamic'' registry 
key created by merging all keys in 
\registrykey{HKEY\_CURRENT\_USER{\backslash}Software{\backslash}Classes} and 
\registrykey{HKEY\_LOCAL\_MACHINE{\backslash}Software{\backslash}Classes}.

\subsection{Implementations (Classes)}
A COM Class (usually abbreviated to ``coclass'') is an implementation 
of one or more interfaces. 

Classes are identified by Class ID's (CLSID's), which are also supposed to be 
globally unique. Some of them also have a ``Prog ID'', which is a human-readable, 
unique string (e.g. ``MyCompany.MyClass''). Those familiar with Java, Python or C\#, 
might recognize the similarity between Prog IDs and packages.

Windows hosts keep a list of classes they know about in the registry at 
\registrykey{HKEY\_CLASSES\_ROOT{\backslash}CLSID}.

\subsection{Instances (Objects)}
Objects are instances of classes and are identified by object ID's (OID's).

Funny enough, classes by themselves are objects as well (sometimes referred 
to as ``class objects''). The methods on a class object are somewhat equivalent
to the {\em static} methods found in languages like C++, Java and C\#.

The most important method on a class object is the $CreateInstance$ method
that returns a new instance of the class. The $CreateInstance$ method is part of 
the $IClassFactory$ interface, which is implemented by pretty much all class objects.

\subsection{Interfaces Pointers}
When making calls to object, the object is never referenced itself --- only the 
specific interface on a certain instance. Such an ``instance of an interface''
is called an {\em Interface Pointer}. For example, to read from a stream, 
one does (in simplified C++):

\begin{lstlisting}[caption=Creating a COM object by CLSID]
char buffer[20];
IUnknown *instance;
com_create_simple_object(Foo_CLSID, &instance);
((IStream *)instance)->Read(20, buffer);
\end{lstlisting}

\subsection{Calling methods}
The methods in a class are merely implementations of methods defined by 
the interfaces supported by that class. You always use the interface when 
calling methods, at least when calls need to be able to work cross-language.

In order to call a method, you first obtain a handle to the relevant 
interface on your object.  You can obtain such a handle by 
calling the $QueryInterface$ method, which is part of the $IUnknown$ 
interface.

\begin{figure}
\includegraphics[scale=.50]{call-local}
\caption{A local COM call}
\end{figure}

\subsection{Creating an object}
In the COM world, there are three ways for obtaining an instance of 
an object:

\begin{itemize}
\item Binding directly to the ``Class Object''. This can be used to do either a ``static'' call or to call $CreateInstance$ (if the class object supports the $IClassFactory$ interface) to return a new class instance
\item Create an instance based on the CLSID. Windows looks up the CLSID in the registry, finds the associated DLL or EXE and loads it. ($CoGetInstanceFromFile() $)
\item From Persistent State Object. This creates an instance from a $IStorage$ pointer
\end{itemize}

Objects returned by other calls all use one of the three ways described above in
one way or another.

ActiveX controls embedded into browsers can also use URL's to 
get access to an instance of an object.

\section{Defining Interfaces}

In the Windows world, COM structures, interfaces and classes are specified in ODL (Object Definition Language). 
ODL is basically DCE/RPC IDL with a few extensions. Originally, these ODL files were compiled by a tool called 
$mktyplib$\cite{MkTypLib} but later support for ODL was merged into the $midl$\cite{MIDL} tool which is 
used on Windows to compile DCE/RPC IDL files. During this process some ODL features were lost or changed, details are available in \cite{MIDLMkTypLibDiff}.

The main extensions ODL makes to IDL are:

\begin{itemize}
\item inheritance for interfaces. Base classes can be specified in a fashion similar to C++ class inheritance.
\item "object" interfaces, i.e. interface that can have instances. Specified by adding $object$ to the properties of the interface.
\item passing pointers to interface instances. Either an explicit interface name can be specified or the $iid\_is()$ property can be used to pass a pointer to an interface (run-time) identified by a custom IID.
\item writing ``type library files'', i.e. binary files that contain a binary version of the interface definition (extension: $.tlb$
\end{itemize}

\section{Distributed COM}

Distributed COM is nothing more then a 'hack' to make standard COM work 
remotely. It was added later as a way to allow the distribution of COM objects 
over multiple computers. It is basically the glue between DCE/RPC \cite{opengroupdce} and COM, as drawn in \ref{relations}

\begin{figure}
\caption{The relations between COM, DCOM and DCE/RPC}
{\center
\includegraphics[scale=.50]{relations}
}
\label{relations}
\end{figure}

DCOM is not as closed as one might think; there are several documents available 
from Microsoft explaining the wire format \cite{UnderstandDCOM,DCOMarch}. There has even 
been an Internet draft on DCOM \cite{DCOMSpec}.

\subsection{Proxies and stubs}
When doing COM remotely, instead of directly calling the object one 
would like to use, the client calls on a 'stub' object that 
takes care of the network call. 

The stub, which is usually autogenerated, then does one or more 
DCE/RPC calls and passes along the data it receives.

On the server side (where the 'real' object remains) the DCE/RPC subsystem 
forwards the RPC call to a proxy object (also autogenerated, usually) that 
then makes a call to the real object.

In the Windows world, MIDL generates these proxies and stubs. In Samba, 
these classes are generated by pidl.

\begin{figure}
\includegraphics[scale=.50]{call-remote}
\caption{A remote COM call using DCOM}
\end{figure}

\subsection{Object Exporters}
Every object lives in a certain context, known to Windows programmers as 
an {\em apartment}. Calls crossing apartment boundaries need marshalling.
Usually the term apartment maps to a thread. Apartments are identified 
remotely by Object Exporter IDs (OXIDs)
unique to the system on Windows.

\begin{figure}
\includegraphics[scale=.50]{object-exporter}
\caption{The object exporter}
\end{figure}

\subsection{Object RPC}
DCOM is implemented on top of DCE/RPC\cite{opengroupdce} and uses the optional 
$object$ field in the DCE/RPC header. DCOM uses this field 
to store an Interface Pointer ID (IPID) which uniquely identifies an interface pointer.  This extension is usually referred to as Object RPC (ORPC).

Other than this minor change, there is no need for modifications to the 
client-side DCE/RPC implementation. 

All DCOM calls have a mandatory extra \verb'[in]' and a mandatory extra \verb'[out]' 
argument that always occur before the other arguments. These arguments (ORPCTHIS and ORPCTHAT) contain COM version and extension information.

\subsection{Marshalling}
When an interface pointer is sent across the wire using NDR, it is 
transformed into a OBJREF struct by the proxy, after which 
it is sent as a normal NDR structure. 

The proxy can marshall interface pointers into the OBJREF struct using three 
possible methods:

\begin{itemize}
\item Standard marshalling
\item Handler marshalling (Requires the interface to implement IStdMarshalInfo)
\item Custom marshalling (Requires the interface to implement IMarshal)
\end{itemize}

\subsection{Activation}

Before an object can be used, it must be activated (created). Older 
DCOM implements use the ``plain'' $IRemoteActivation$ interface for this 
purpose, while newer implementations (2000 and above) use the object 
interface $ISystemActivator$\footnote{I suspect this interface is implemented by the $System.Activator$ class in .NET}.

This interface can be called with a CLSID and a list of interfaces the 
client would like to use on the new object. The server will then (locally)
do a number of calls:

\begin{enumerate}
\item GetClassObject(CLSID)
\item IClassFactory::CreateInstance()
\item IUnknown::QueryInterface() (once for each interface in the list)
\item IUnknown::Release()
\end{enumerate}

Along with a list of interface pointers, the server returns an 
OXID and a list of SECURITYBINDINGS and STRINGBINDINGS belonging to this OXID.

\subsection{OXIDs}
Objects don't have to live at the same RPC endpoint or even the 
same host they were activated at. For this purpose, every interface 
pointer that is sent over the wire contains an OXID and a resolver address. 

If the client wants to do a call on an interface pointer, it asks the remote 
server listening on the resolver address where the object with the given 
OXID can be reached. The server will respond with a list of SECURITYBINDINGS
and STRINGBINDINGS.

The STRINGBINDING structs, as returned by 
$IRemoteActivation$ and $IOXIDResolver$, can easily be converted to 
DCE/RPC binding strings while the SECURITYBINDING structs contain possible 
account names that can be used to authenticate.

\subsection{Garbage Collection}

In order to make sure objects aren't kept running long after the client that 
created them crashed horribly, clients have to regularly send pings to the 
host they are living on. Pinging is done based on OID (object ID's) rather then 
IPID's. Objects can be grouped together in $PingSets$, which makes it 
possible to ping thousands of objects with a just a very small packet.

Objects are generally presumed dead if none of their clients have pinged 
them for 120 seconds \cite[Section 5.2.3]{DCOMSpec}.

\subsection{Running Object Table}

Every machine keeps a list of the objects that it exports. 
The running object table can be accessed using the IRunningObjectTable\cite{IRunningObjectTable} interface, which 
is available at the same endpoint as the endpoint mapper (which corresponds
to the rpcss.exe process on Windows).

\section{Integration in Samba 4}

The first attempt to implement DCOM in Samba 4 failed because it was aimed at implementing both 
DCOM and COM together, ignoring the concept of proxy classes. This attempt failed because it 
made matters very complicated. 

\subsection{First try}

The second attempt seperated the COM and DCOM implementations (somewhat similar to Windows), making the code simpler.

\begin{itemize}
\item $lib/com$ contains a very simple (but sufficient) COM implementation. 
\item $lib/com/interfaces$ contains pidl generated structure files for the COM interfaces
\item $lib/dcom$ contains the glue between DCE/RPC (NDR) and COM.
\item $lib/dcom/proxies$ contains the pidl-generated proxy files
\end{itemize}

\subsubsection{COM implementation}

Since Samba is written in C, which is not object-oriented, COM calls in Samba 
are made using function pointers in structs. This works 
in pretty much in the same fashion as the GTK+\footnote{http://www.gtk.org/} project's GObject model.

For example, the following Windows code:
\begin{lstlisting}[caption=Simple COM example in Windows]
IClassFactory *pcf = 0;
CoGetClassObject(CLSID_Penguin, CLSCTX_ALL,0, IID_IClassFactory, (void**)&pcf);
IBird *pBird = 0;
pcf->CreateInstance(0, IID_IBird, (void**)&pBird);
pcf->Release();
pBird->EatFish();
pBird->Release();
\end{lstlisting}

would translate to :

\begin{lstlisting}[caption=Simple COM example in Samba]
struct IClassFactory *pcf = NULL;
struct IBird *pBird = NULL;
com_get_class_object(CLSID_Penguin, IID_IClassFactory, &pcf);
IClassFactory_CreateInstance(pcf, mem_ctx, 0, IID_IBird, &pBird);
IUnknown_Release(pcf, mem_ctx);
IBird_EatFish(pBird, mem_ctx);
IUnknown_Release(pBird, mem_ctx);
\end{lstlisting}

in Samba.

\subsubsection{Loading class objects}

As described earlier, objects are usually created by obtaining 
a handle to their class object and then calling $CreateInstance$ on that. 

In the Windows world, class objects are obtained by $CoGetClassObject()$ 
which internally loads them from a .DLL, a .EXE or the current process.

Samba will allow for two methods to provide a class object. First, 
class objects can be registered by internal parts of Samba, using 
the $register\_running\_com\_class$ function. 

It is also possible use custom classes by putting a library file
that provides a $get\_class\_object$ function in Samba's library directory. 
Samba will use the CLSID as the filename to load the appropriate class.

\subsubsection{DCOM implementation}

The $dcom\_IUnknown\_from\_OBJREF$ and $dcom\_OBJREF\_from\_IUnknown$ functions take care of 
unmarshalling and marshalling interface pointers.

\subsubsection{Pidl extensions}

Samba 4 supports ODL files in it's regular IDL compiler rather then creating a seperate 
compiler for ODL files. This is possible because ODL files are not significantly different 
from IDL files and the extensions necessary to support ODL fairly simple. These
extension do not require large amounts of changes in 
the pidl code nor do they increase the complexity of the pidl code.

\newpage
\oddsidemargin=0em
\begin{figure}
\caption{The data flow inside pidl}
\includegraphics[scale=.40]{pidl_structure}
\end{figure}
\newpage

The additional pidl module, $com_header.pm$, generates C header files that contain 
structs with function pointers for each object interface. They also generate wrapper macros for
easier use of the interfaces. 

For example, a header file generated by $com_header.pm$ for an ODL file containing the IUnknown and IStream object interfaces would look like this:

\begin{lstlisting}[caption=COM header example]
/* Interface */

#define IUNKNOWN_METHODS  \
	int (*Release) (struct IUnknown *, TALLOC_CTX *mem_ctx); \
	int (*AddRef) (struct IUnknown *, TALLOC_CTX *mem_ctx); \
	void (*QueryInterface) (struct IUnknown *, TALLOC_CTX *mem_ctx, struct GUID *iid, void **result); 

  
struct IUnknown_vtable {
	IUNKNOWN_METHODS
};

#define ISTREAM_METHODS \
	IUNKNOWN_METHODS \
	void (*Read) (struct IStream *, TALLOC_CTX *mem_ctx);

struct IStream_vtable {
	ISTREAM_METHODS
};

/* Instance */
struct IUnknown {
	struct com_context *ctx;
	struct IUnknown_vtable *vtable;
	void *object_data;x1x
};

#define IUnknown_QueryInterface(i,m,p1,p2) (i->vtable->QueryInterface(i->object_data, m, p1, p2))
		...
\end{lstlisting}

\subsection{DCOM implementation}

\subsubsection{Pidl extensions}

A new Pidl module, $proxy.pm$, generates one proxy class (implementation) per interface. The functions 
in this class each simply call their DCE/RPC equivalents (similar to Windows). 

Another new Pidl module, $stub.pm$, generates one stub with a custom dispatch function 
for each interface. This dispatch function looks up the interface pointer in the running 
object table and then calls the appropriate function on the interface pointer.

\begin{verbatim}
 - Problem: 
 	Need MInterfacePointer in NDR-related stuff, but
	need struct IUnknown / void in com_header, etc. (with IID)

	- Always set iid_is() for Interface names occurring in IDL
	- Need different treatment in com_header.pm and proxy.pm
	- unknown interfaces are a different story, but always require a iid_is(), so always encode a MInterfacePointer when an iid_is() is set.
	- in the proxy, do a conversion from "real interface pointer" <-> MInterfacePointer
	 - when iid_is() was set
	 - what when iid_is() not set?
	   - See if \$NAME is registered as an interface or as a struct. If it's
	   not registered at all, use something like this:

#ifdef COM_IUNKNOWN_UUID
	/* Conversion from MInterfacePointer */
#else 
	/* Plain assignment
#endif

TODO: 
	Split out NDR-related and other stuff in ndr\_*.h
\end{verbatim}

\subsection{Integration in Mono}

COM contains a lot of classes, making it very time-intensive (if not 
impossible) to implement. Since it is object-oriented by nature, it is 
also much easier to implement in an object oriented language, rather than 
C. 

DCOM is currently mostly a legacy technology, with Microsoft 
focussing on the .NET framework. The .NET framework in 
the Windows world has compatibility for COM built in. Rather than 
a hard-coded standard for remoting, the .NET framework supports 
various backends, DCOM being one of them.\cite{DotNETRemotingIntro}

Mono\footnote{http://www.mono-project.com/} is an open source project 
that aims to implement Microsoft's .NET framework. Work is currently in 
progress to implement COM support for Mono.

It should be relatively easy to add support for DCOM 
once the COM support is done, perhaps somewhat 
similar to Remoting.Corba\footnote{http://remoting-corba.sourceforge.net/}, 
based on Samba's DCE/RPC code.

This makes the scope of the problem a lot smaller --- it means no longer 
being required to reimplement COM in Samba. The only thing necessary for 
this, is the DCE/RPC implementation and ORPC extensions from Samba and 
some way to generate Proxy/Stub objects in Mono (see \ref{relations}).
	
\bibliography{dcom}

\end{document}
