WS Data Synch

Topics: Mobile Client Software Factory
Apr 13, 2006 at 3:28 AM
originally posted by: m_a_madero

Is there any guidance on how to do a Data Synchronization using Web Services instead of SQL Server Replication?
Apr 19, 2006 at 3:03 PM
originally posted by: fkc888

There is no such guidance in the current scope of this project.

How important is this to you? Can you describe your business case for this?
Apr 20, 2006 at 3:44 AM
originally posted by: m_a_madero

Really often we need to do Data Synch, but our client doesn't have SQL Server to do a subscription, so we have to manually copy the most recent data to the PDAs DB using WS.
For the most of the use cases this is readonly data from the PDAs user point of view and this made things a lot easier. I feel there are lots of things that can be automated using GAT, and I guess this could be a really common problem.

- In case we need to modify data on the PDA, we should have a row versioning control, similar to what SQL Server does. This could be automated easily, I actually have seen some code generators for doing this.
A nice option would be to choose if you want to put the versioning control columns in the same table or on a different one in case we are not allowed to modify some legacy system's table.
- Create the needed schema for the tables we need in the PDAs DB.
- Create triggers to track changes on the Server's DB.
- Generate code to read from the tables probably using a discriminator (in case we dont want all the data) so that PDA clients can pull the data they want.


Well this is really just and idea...
Apr 22, 2006 at 4:52 PM
originally posted by: johnkattenhorn

Hi,

You've read my mind, i love what has been started here but i echo your requirements, in particular in our situation :

1) We have a back end office system hosted in Oracle.

2) We are not allowed to modify it's schema.

We have built a fairly advanced prototype of a PDA solution connected to this system using Web Services. As you have indicated most of the infromation flows from the back end system in our situation which is read-only.

For the prototype we have taken a view that we will pass 'ownership' of the data using business processes. It's a cop out i know technically but basically the business know that if a particular record is assigned to a PDA then they should'nt update it. It's going to be updated at some point by a PDA user.

Any chance of being able to add non SQL back end into the scope of the project at some time ?

Thanks

John
Apr 25, 2006 at 1:37 AM
originally posted by: EdJez

I just replied on another thread but I'll expand here:

Hi - Absolutely.
The SF recommends an architecture for the app, and in that architecture it's up to you to get data over to the device using either replication or calling web services that return reference data and then saving that in a local cache yourself. You don't need to use the replication block if you don't need it, as a matter of fact, that's why we left it out the blocks as small interindepenmdent units. Just like in desktop clients, there is a decison to take about being database-centric or SOA-ish on your exchange.

To get reference data via web services you would do something like this, if I understand your target design:
1) You have to have the application deal with a local cache of data in a read-only fashion. Both the cache and/or the app can issue a request to refresh data, when the probability of it being stale by next connection becomes too high.
2) Create the web services that return reference data. You might specify 'filters' for the data either as a parameter of the web service or as an optional SOAP header. You populate this header when the web service request is actually dispatched, as well as any final tweaking before you make the actual WS call by implementing your own ProxyFactory. (This is also where we will be populating a SOAP header with a request message-id guid to help the server out with idempotency). The web service could use SQL, Oracle, or whatever it chooses.
3) When the callback for return happens, you clear the cache of the apprppriate elements and re-populate it.

There are some patterns about state management in message-oriented apps that you might want to consider, e.g. versioning reference data; how does a command specify the version of reference data is using (e.g. does an 'Order' call reference by ID, by date, or actually embed 'Catalog' info such as price). I personally recommend Pat's classic article for a drilldown of these considerations: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnbda/html/dataoutsideinside.asp

And to reinforce, if you go web services style in your app, whatever database is being queried for data is up to the implementation of the web service that you build. The client just calls it :)

Check out the latest drop - how DisconnectedServiceAgents work and get used from the app. There is still a little bit of refactoring that we need to do, but the goal is that we would have a VS.NET wizard (implemented using GAT) that would ask you to point to an existing web service proxy or a WSL and generate the proxy code needed for offline (serializing, enqueuing, etc) automatically. You can think of it as a 'Add Offline Web Reference' wizard.
May 1, 2006 at 5:39 AM
originally posted by: johnkattenhorn

Hi,

Thanks for the reply, i agree with all that you have said here and i've started to look through the disconnected service agents. Can i ask - What was the thinking behide building in intelligence into the WebService request as opposed to using the connection manager and background threads to control synchronisation with the WebServices depending on network state.

I love the idea - just wanted to know the thinking behind it.

Thanks

John
May 20, 2006 at 6:33 AM
originally posted by: JohnSocha

I wanted to add more about the synchronization. The problem you're describing of having a non-SQL Server back end is very common. It is also common that the data you need to send to the Pocket PC should be different than what you have on the server. For example, you may want to denormalize the data, remove columns, etc. so you don't send as much information to the Pocket PC.

In these cases you can use something like ETL to pull data from one or more locations that may be Oracle, SAP, etc. into tables that are in a "mobile" data repository. In these cases you could use SQL Server for both the repository and the server that will allow merge replication with your devices.

This approach, of course, might be overkill if you don't have to send a lot of data to the clients. When your data sets are small, you can request the entire dataset each time you "synchronize," and just replace the reference data on the device with the new data set.

It really boils down to whether you need real synchronization or can just overwrite your data as to wether you'll want to look into something like merge replication (which requires SQL server), or web services calls (which are back-end agnostic).

Hope that helps,
-- John Socha-Leialoha
Aug 4, 2006 at 3:29 AM
originally posted by: mrbelk

If we use the DSA and the Agent/AgentCallback infrastructure to manage these WS synchronization attempts, is there any way to associate a Database service that might be created in the root work item with the agentcallback?

Presumably, the agentcallback is where all the magic will have to happen w.r.t. getting the reference data (in the form of a dataset containing multiple tables) from the result of the webservice call.

Would it be advisable to just create a new database instance in the agentcallback to flush and repopulate the reference data?

Thanks,
Matthew
Aug 4, 2006 at 6:28 AM
originally posted by: josegallardo

The current implementation of DSA lives out of CAB and it doesn't use Object Builder for callbacks creation.
Callback methods are called in the WebServiceRequestDispatcher class. You can take a look at the InvokeReturnCommand and InvokeExceptionCommand methods.

The following class is a custom request dispatcher implementation using CAB and OB for the callbacks.

------------------------------------------------

using System;
using System.Collections.Generic;
using System.Text;
using Microsoft.Practices.Mobile.DisconnectedAgent;
using Microsoft.Practices.Mobile.EndpointCatalog;
using Microsoft.Practices.Mobile.CompositeUI;
using System.Net;
using System.Reflection;

namespace AdventureWorks
{
public class CabRequestDispatcher : WebServiceRequestDispatcher
{
WorkItem workItem;

public CabRequestDispatcher(IEndpointCatalog catalog, ServiceDependency WorkItem workItem)
: base(catalog)
{
this.workItem = workItem;
}


protected override OnExceptionAction InvokeExceptionCommand(Request request, Exception realException)
{
if (request.Behavior.ExceptionCallback != null)
{
return (OnExceptionAction)this.CabBuildCallbackInvoke(request.Behavior.ExceptionCallback, request, realException);
}
return OnExceptionAction.Dismiss;
}

protected override void InvokeReturnCommand(object onlineProxy, Request request, object result)
{
if (request.Behavior.ReturnCallback != null)
{
MethodInfo method = onlineProxy.GetType().GetMethod(request.MethodName);
if (method.ReturnType != typeof(void))
this.CabBuildCallbackInvoke(request.Behavior.ReturnCallback, request, request.CallParameters, result);
else
this.CabBuildCallbackInvoke(request.Behavior.ReturnCallback, request, request.CallParameters);
}
}

private Object CabBuildCallbackInvoke(CommandCallback callback, params object[] args)
{
object targetInstance = workItem.Items.AddNew(callback.TargetType);

try
{
MethodInfo method = callback.TargetType.GetMethod(callback.TargetMethodName);
return method.Invoke(targetInstance, args);
}
finally
{
workItem.Items.Remove(targetInstance);
}
}
}
}
Aug 4, 2006 at 6:28 AM
originally posted by: josegallardo

Additionally, in your workitem (the root workitem hopefully), you should add the RequestManager (DSA) service using the full Initialize method including all the parameters, doing something like:

private void AddRequestManagerService()
{
Database database = this.WorkItem.Services.Get<Database>();
IEndpointCatalog endpointCatalog = this.WorkItem.Services.Get<IEndpointCatalog>();
ConnectionMonitor connMonitor = this.WorkItem.Services.Get<ConnectionMonitor>();
DatabaseRequestQueue requestQueue = new DatabaseRequestQueue(database, "Requests");
DatabaseRequestQueue deadLetterQueue = new DatabaseRequestQueue(database, "Dlq");
CabRequestDispatcher dispatcher = this.WorkItem.Services.AddNew<CabRequestDispatcher, IRequestDispatcher>();

IConnectionMonitor connections = new ConnectionMonitorAdapter(connMonitor);
RequestManager requestManager = RequestManager.Instance;
requestManager.Initialize(requestQueue, deadLetterQueue, connections, dispatcher, endpointCatalog);

this.WorkItem.Services.Add<RequestManager>(requestManager);
this.WorkItem.Services.Add<IRequestQueue>(requestManager.RequestQueue);
requestManager.StartAutomaticDispatch();
}
Aug 4, 2006 at 8:14 AM
originally posted by: mrbelk

I had already done all of the setup with the request queue stuff in the rootWorkItem.OnRunStarted event.

This seems like a good start, but I can't make the connection between the CabRequestDispatcher and the DSA Agent callback methods.

Does this give me something in the callback methods where I can have access to the workitem and it's "Services" and "Items" collections?

I apologize for my ignorance, but the CAB + dependency injection techniques represent a fairly substantial paradigm shift for me.

Thanks,
Matthew
Aug 4, 2006 at 8:35 AM
originally posted by: josegallardo

You can use dependecy injection in your callback class.

The CabRequestDispatcher creates a callback object using the object builder. If you need to use any service from your workitem (or even the workitem itself) in the callback methods, just use dependecy attributes in the constructor of the callback class or properties to get those services/items from the current context workitem.

To get access to the current workitem, you can use service dependency in the constructor of your callback class:

public ServiceDisconnectedAgentCallbackBase(ServiceDependency WorkItem workItem)
: base()
{
this.workItem = workItem;
}

Now you can use the workItem private member from the callbacks.
Aug 4, 2006 at 8:46 AM
originally posted by: mrbelk

Duh.

Thanks,
Matthew