Sunday, May 31, 2020

SSL And Certificates For Many-to-Many Servers And Clients

I had a lot of trouble when I was trying to figure out how to configure my Windows Service applications to use https and Certificates. Part of the problem I had was the stuff I was finding when I Googled was often too simplistic.  My Windows Service applications (we call them Gateways) are both server and client, which is where my difficulty in configuration comes from. Each Gateway can send messages to other Gateways. It can also receive messages from other Gateways. We can have a network of many Gateways, all communicating with each other.

I eventually got it all sorted out and documented it so that it was easy to configure the next Gateway that got deployed. Here are the details:

We're using WCF (Windows Communication Foundation), and to illustrate what I'm writing about, here is part of the ServiceModel section of a sample config. First, just the the <services> and <client> :

<services>
  <service name="My.Application.Service" behaviorConfiguration="ServiceBehavior">
    <endpoint address="https://MyComputer/My.MessageService"
          binding="wsHttpBinding" bindingConfiguration="wsHttpServiceConfig"
          contract="My.IMessageService"
          name="ServiceEndpoint" />
  </service>
</services>
<client>
  <endpoint address="https://AddressDoesNotMatter/WeSubstituteWithAddressFromDatabase"
            binding="wsHttpBinding" bindingConfiguration="wsHttpClientConfig"
            contract="My.IMessageService"
            behaviorConfiguration="ClientBehavior"
            name="ClientEndpoint" />
</client>

In case there are different requirements for the Service and the Client, you should use two different BindingConfigurations (wsHttpServiceConfig and wsHttpClientConfig in this example). In our Service applications, there is usually no reason to have different configurations, but by specifying two of them, we have an easy way to change them.

Also note that there are two different behaviorConfigurations, ServiceBehavior and ClientBehavior. More on that later.

Without any kind of security at all, the two binding configurations look like this:

<bindings>
  <!-- NO security, NOT https -->
  <wsHttpBinding>
    <binding name="wsHttpServiceConfig" openTimeout="00:00:59" sendTimeout="00:00:59" 
             maxBufferPoolSize="4000000" maxReceivedMessageSize="4000000">
      <readerQuotas maxDepth="4000000" maxStringContentLength="4000000" maxArrayLength="4000000" 
                    maxBytesPerRead="4000000" maxNameTableCharCount="4000000" />
      <security mode="None" />
    </binding>
    <binding name="wsHttpClientConfig" openTimeout="00:00:59" sendTimeout="00:00:59" 
             maxBufferPoolSize="4000000" maxReceivedMessageSize="4000000">
      <readerQuotas maxDepth="4000000" maxStringContentLength="4000000" maxArrayLength="4000000" 
                    maxBytesPerRead="4000000" maxNameTableCharCount="4000000" />
      <security mode="None" />
    </binding>
  </wsHttpBinding>
</bindings>

Note the <security> tag specifies a mode of "None" (you won't be using https with this set to "None").

Types of Security

There are two types of security when using https://.
  • There is SSL only: the message is encrypted and the client must trust the server’s Certificate (it must be created from a trusted Certificate Authority common to both client and server).
  • There is mutual authentication: in which both parties, client and server, trust each other’s Certificate (and there is additional encryption) This is what I needed to use for my Gateways.

For Both Types

For both types of security (your URLs will obviously be https:// rather than http://), the following must be done for each application:
The Certificate must be bound to a Port. Use netsh in a Command window to do that:

netsh http add sslcert ipport=0.0.0.0:ServicePort certhash=CertThumbprint appid={48d00867-cd90-45a2-b398-0def54c7e671}
  • Where ServicePort is the port associated with the URL endpoint address in the config.
    • SSL uses port 443 by default, in which case, you don't need to specify 443 in the config's endpoint address.
    • But you can use other ports if you need to. Then you would need to specify a port in the config's endpoint address and in the above netsh command.
  • CertThumbprint is the Thumbprint of the Certificate that you're binding to this port.
  • appid can be any valid GUID and the above setting is is just an example. In a .NET application, you could grab a GUID from the project's AssemblyInfo.cs (under the project's Properties in Solution Explorer).
Next question, how do we implement security in the config? It depends on whether you're using SSL only or SSL with authentication, as follows:

SSL Only

Change the <security mode="None" /> tags, in both binding configurations shown above, to the following:

<security mode="Transport">
  <transport clientCredentialType="None" />
</security>

SSL With Authentication

For Authentication, we’ll need to make changes to the same two binding configs as shown above for “SSL Only”, but the security mode will be different, and we will be specifying that we’re using Certificates, as follows:

<security mode="TransportWithMessageCredential">
    <transport clientCredentialType="Certificate" />
    <message clientCredentialType="Certificate" />
</security>

If you remember, I promised more about the behaviorConfigurations. Here's where they come into play. First, here is an example of what the <behaviors> section might look like if you're *not* using https and Certificates:

<behaviors>
  <!-- NO security, NOT https -->
  <serviceBehaviors>
    <behavior name="ServiceBehavior">
      <serviceMetadata httpGetEnabled="true" httpGetUrl="http://MyComputer/My.MessageService" />
      <serviceDebug includeExceptionDetailInFaults="true"/>
    </behavior>
  </serviceBehaviors>
  <endpointBehaviors>
    <behavior name="ClientBehavior">
      <!-- I have nothing, you may have something here already already -->
    </behavior>
  </endpointBehaviors>
</behaviors>

To implement SSL with authentication, we need to add the to the above <behaviors> section to incorporate the actual Certificate credentials. We'll be making changes under both the <serviceBehaviors> section (adding a <serviceCredentials> section) and the <endpointBehaviors> section (adding a <clientCredentials> section).

The ServiceBehavior:

<serviceBehaviors>
  <behavior name="ServiceBehavior">
    <!-- Changed to use https -->
    <serviceMetadata httpsGetEnabled="true" httpsGetUrl="https://MyComputer/My.MessageService" />
    <serviceDebug includeExceptionDetailInFaults="true"/>
    <!-- Added <serviceCredentials> -->
    <serviceCredentials>
      <serviceCertificate findValue="ThumbprintForThisServiceCertificate"
toreLocation="LocalMachine"
                          x509FindType="FindByThumbprint"/>
      <clientCertificate>
          <authentication certificateValidationMode="ChainTrust"
                          revocationMode="NoCheck"
                          trustedStoreLocation="LocalMachine" />
      </clientCertificate>
    </serviceCredentials>
  </behavior>
</serviceBehaviors>

The ClientBehavior (which is "opposite" of the settings in the ServiceBehavior):

<endpointBehaviors>
  <behavior name="ClientBehavior">
    <!-- I have nothing, you may have something here already already -->
    <!-- Added the <clientCredentials> -->
    <clientCredentials>
      <clientCertificate findValue="ThumbprintForThisServiceCertificate"
toreLocation="LocalMachine"
                          x509FindType="FindByThumbprint"/>
      <serviceCertificate>
          <authentication certificateValidationMode="ChainTrust"
                          revocationMode="NoCheck"
                          trustedStoreLocation="LocalMachine" />
      </serviceCertificate>
    </clientCredentials>
  </behavior>
</endpointBehaviors>

That's pretty much it! Not so bad when it's all laid out for you, I suppose.

Now, you may still have some questions, such as how to find the Certificate's thumbprint or what kind of Certificate to use. You can Google for that, if the following stuff I'll show below isn't sufficient for you (although, it should be).

First, a few links for reference. You'll be using the MMC (Microsoft Management Console) snap-in tool for viewing your Certificates:


Next, a shortcut for bringing up the MMC snap-in tool. Run from the Windows Command window:

certlm.msc (for Local Computer Certificates --- you must be an Administrator on the Local Computer)
certmgr.msc (for Current User Certificates --- no Admin rights required)

Here is a screenshot with my test Certificate ... simply double-click on the Certificate you need to look at:


For the purposes in the scenario I've outlined above, we need both Client Authentication and Server Authentication. You can see that under the "Intended Purposes" column. The description in the dialog window further describes what that intended purpose means.

Now click on the Details tab:


You can see the Client and Server Authentication when the "Enhanced Key Usage" property is selected.

The  "Key Usage" property is not selected in this screenshot, but you can see enough of it to see that it says Digital Signature and Key Encipherment.

Notice the Thumbprint just below that (I have redacted most of its value). Click on it and copy it from the textbox to use in your config:


I have seen this thumbprint displayed two different ways ... one, like the screenshot shows, with spaces between each character. I have also seen it with no spaces (on Server OS's I think). If your version has the spaces you need to delete them one by one. And here's one more thing to be aware of: Not only are there spaces between each character, but there is an unprintable character at the beginning of the string also!!!   Be sure to delete that too!

Saturday, April 25, 2020

Saving Your WinForm Data

I wrote a blog post 5 months ago about Printing Your WinForm: https://geek-goddess-bonnie.blogspot.com/2019/11/printing-your-winform.html

Today's post is doing something "almost" similar for saving your WinForm data. I was helping a person on the MSDN forums with his particular need to save data entered by a user on a complex Form to one file, such that the Form could be re-populated by reading the file and processing the info. The Form contained multiple tabs with a picturebox and and a textbox on each tab, along with a button so the user could upload an image for the picturebox.

I suggested the use of DataSets, because the DataSet .ReadXml() and .WriteXml() are so easy to use. This particular person was not a developer, so I wanted to keep it easy. A more experienced developer can take what I lay out here in my blog and do more with it ... such as save the info contained in the DataTable to a database.

I would also recommend the use of Typed DataSets over plain old DataSets. I didn't get into that in my reply to the forum questioner, except to mention that Typed DataSets are better most of the time, IMHO. More on that later after this first set of code. In either case, the DataType for the PictureBoxImage column should be System.Byte[].

We'll need four additional methods on the Form to handle this functionality (with appropriate Save and Repopulate buttons to call the first two methods):

  1. SaveTabPageInfoToFile() to Save all your TabPage information to a DataSet XML file.
  2. FillTabPagesFromFile() to repopulate your TabPage information from that saved XML file.
  3. ImageToByteArray() to take an Image from a PictureBox and create a Byte[] array.
  4. ByteArrayToImage() to take a Byte[] array and create an Image from it.

Code Using Typed DataSets:

Number 1:

public void SaveTabPageInfoToFile()
{
    FormInfoDataSet dsFormInfo = new FormInfoDataSet();

    // Fill the DataTable with info from each TabPage
    FormInfoDataSet.TabPageRow rowTabPage;
    foreach (TabPage page in this.tabControl1.TabPages)
    {
        rowTabPage = dsFormInfo.TabPage.NewTabPageRow();
        dsFormInfo.TabPage.AddTabPageRow(rowTabPage);

        rowTabPage.TabPageName = page.Name;
        foreach (var ctrl in page.Controls)
        {
            if (ctrl is TextBox)
            {
                rowTabPage.TextBoxName = ((TextBox)ctrl).Name;
                rowTabPage.TextBoxText = ((TextBox)ctrl).Text;
            }
            else if (ctrl is PictureBox)
            {
                rowTabPage.PictureBoxName = ((PictureBox)ctrl).Name;
                rowTabPage.PictureBoxImage = ImageToByteArray(((PictureBox)ctrl).Image); // This is Number 3
            }
        }
    }

    // Now write the DataSet to an XML file
    dsFormInfo.WriteXml("FormInfo.xml");
}

Number 2:

public void FillTabPagesFromFile()
{
    FormInfoDataSet dsFormInfo = new FormInfoDataSet();

    // Fill the DataSet from the saved file
    dsFormInfo.ReadXml("FormInfo.xml");

    // And now, repopulate all your TabPages
    // Unfortunately, in using a DataTable.Select(), the resulting array of DataRow[] cannot be cast to
    // a TabPageRow[] array and we have to use the a plain old DataRow[], along with the non-typed syntax.
    DataRow[] rows;
    foreach (TabPage page in this.tabControl1.TabPages)
    {
        foreach (var ctrl in page.Controls)
        {
            if (ctrl is TextBox)
            {
                rows = dsFormInfo.TabPage.Select($"TextBoxName = '{((TextBox)ctrl).Name}'");
                if (rows.Length > 0)
                    ((TextBox)ctrl).Text = rows[0]["TextBoxText"].ToString();
            }
            else if (ctrl is PictureBox)
            {
                rows = dsFormInfo.TabPage.Select($"PictureBoxName = '{((PictureBox)ctrl).Name}'");
                if (rows.Length > 0)
                    ((PictureBox)ctrl).Image = ByteArrayToImage(rows[0]["PictureBoxImage"] as Byte[]); // This is Number 4
            }
        }
    }
}

Number 3:

private Byte[] ImageToByteArray(Image image)
{
    var ms = new System.IO.MemoryStream();
    image.Save(ms, System.Drawing.Imaging.ImageFormat.Png);
    return ms.ToArray();
}

Number 4:

private Image ByteArrayToImage(Byte[] bytes)
{
    var ms = new System.IO.MemoryStream(bytes);
    return Image.FromStream(ms);
}

Another few words about Typed DataSets

In this particular example, a Typed DataSet was useful in Number 1, but not so much in Number 2. The moral of this story is that, in cases like this, unless you have an existing Typed DataSet already defined in your application, possibly for other uses, then you may not want to go through the steps to create one for something as simple as this. In which case, I suggest using a global DataSet variable and defining the DataSet in your Form's constructor to be utilized in the two methods, like this:

Code that does not use a Typed DataSet:


// define two global variables:
private DataSet dsFormInfo;
private DataTable dtTabPage;

// In the Form's constructor:
public Form1()
{
    InitializeComponent();

    // add the code to define your DataSet/DataTable
    this.dsFormInfo = new DataSet();
    this.dtTabPage = new DataTable();
    this.dtTabPage.Columns.Add("TabPageName", typeof(string));
    this.dtTabPage.Columns.Add("TextBoxName", typeof(string));
    this.dtTabPage.Columns.Add("TextBoxText", typeof(string));
    this.dtTabPage.Columns.Add("PictureBoxName", typeof(string));
    this.dtTabPage.Columns.Add("PictureBoxImage", typeof(System.Byte[]));
    this.dsFormInfo.Tables.Add(this.dtTabPage);
}

Now change the bolded code below in your two methods (Number 1 and 2) accordingly:

public void SaveTabPageInfoToFile()
{
    // Start with an empty DataSet/DataTable
    this.dsFormInfo.Clear();

    // Fill the DataTable with info from each TabPage
    // using the non-Typed syntax
    DataRow rowTabPage
    foreach (TabPage page in this.tabControl1.TabPages)
    {
        rowTabPage = this.dtTabPage.NewRow();
        this.dtTabPage.Rows.Add(rowTabPage);

        rowTabPage["TabPageName"] = page.Name;
        foreach (var ctrl in page.Controls)
        {
            if (ctrl is TextBox)
            {
                rowTabPage["TextBoxName"] = ((TextBox)ctrl).Name;
                rowTabPage["TextBoxText"] = ((TextBox)ctrl).Text;
            }
            else if (ctrl is PictureBox)
            {
                rowTabPage["PictureBoxName"] = ((PictureBox)ctrl).Name;
                rowTabPage["PictureBoxImage"] = ImageToByteArray(((PictureBox)ctrl).Image); // This is Number 3
            }
        }
    }

    // Now write the DataSet to an XML file
    this.dsFormInfo.WriteXml("FormInfo.xml");
}


public void FillTabPagesFromFile()
{
    // Start with an empty DataSet/DataTable
    this.dsFormInfo.Clear();

    // Fill the DataSet from the saved file
    this.dsFormInfo.ReadXml("FormInfo.xml");

    // And now, repopulate all your TabPages
    DataRow[] rows;
    foreach (TabPage page in this.tabControl1.TabPages)
    {
        foreach (var ctrl in page.Controls)
        {
            if (ctrl is TextBox)
            {
                rows = this.dtTabPage.Select($"TextBoxName = '{((TextBox)ctrl).Name}'");
                if (rows.Length > 0)
                    ((TextBox)ctrl).Text = rows[0]["TextBoxText"].ToString();
            }
            else if (ctrl is PictureBox)
            {
                rows = this.dtTabPage.Select($"PictureBoxName = '{((PictureBox)ctrl).Name}'");
                if (rows.Length > 0)
((PictureBox)ctrl).Image = ByteArrayToImage(rows[0]["PictureBoxImage"] as Byte[]);
            }
        }
    }
}

Happy Coding!  =0)

Sunday, March 22, 2020

So You Have To Work From Home --- Now What?


The work from home (WFH) experience is going to be new to a whole lot of people, but not for me. I've been working from home (off and on) for more than 30 years after having a few kids in the early 80's. In the late 80's, I was asked if I could write some software (to interface with cash registers and produce back-end reports) and I said sure (with no experience at all with cash registers)! When I was done writing that successful product, I started my own software business from home.

Everything was quite different in those days. I connected to "chat rooms" for technical information using a modem. Those chat rooms eventually became "forums" (where I made many friends). The Internet was there (although it was typically referred to as the World Wide Web), but it was nothing like it is today!

My business morphed into software contracting and finally (after being tired of being a "starving programmer"), I worked as an employee at a number of companies over the next 20 years ... some companies insisted that everyone work on the premises, others (the enlightened companies) allowed work from home. All in all, I've worked from home much more often than not.

In the last 30+ years, I've learned a lot about being productive in a home environment. These things apply to everyone who has to work from home, whether you're an office worker, in management or in the tech industry.

Get A Room!

First, you've got to have a space dedicated to work. A separate room is preferable, but not always possible. I've been lucky to have a separate room that I could dedicate as my "office space" in every house (or apartment) that I've lived in. Sometimes, you just *have to* be able to shut the door!!  ;0) 

At the very least, you should have a space where you can spread out all your work stuff and keep it there. Setting up your work stuff on the dining room table, that needs to be used for eating dinner every evening, is not an ideal set up. But, if it's all you've got, you've got to at least be organized. Maybe keep your stuff in a large Rubbermaid container, or something similar, that can easily be moved.

Get Connected

Every job these days entails having a computer, if only for email if nothing else. You'll most likely need a way to connect to the company network from home. The best way is through a VPN (Virtual Private Network, which is highly recommended for security purposes). There are other methodologies for connecting, such as TeamViewer and LogMeIn. Your company IT people should be able to help you get going with the setup for remote connections.

Endless Meetings? Nah!

You'll also need to be able to conduct and/or attend virtual meetings. It can be done solely with phone conference calls, but frequently a meeting needs to have the ability to share screens or have a camera to show a whiteboard or other things that aren't on a computer (and also, to see your co-workers "face-to-face"). There are plenty of virtual meeting software options, such as Skype, GoToMeeting, Microsoft Teams and more.  

The up-side to this is that you and your co-workers can probably do away with a lot of excess meetings, since a lot of the communication can be easily accomplished with email. If that's not working for a particular problem, for whatever reason, then don't hesitate to have a phone call or use the meeting tools.

There are a lot of distractions in an office environment (people popping their head in your office ... hey, ya got a minute for a quick question?) These kinds of distractions interrupt your work, but they're so common in an office environment that people don't even think about them as disruptive to your concentration.

The company that I currently work for (we all work remotely) has instituted  "Office Hours" from 9:00-10:00 AM and from 2:00-3:00 PM ... during each of those one hour time slots, everyone is fair game for interruptions with work-related questions. And when you're really busy, you can ignore email outside of those two time slots and address them only during the appropriate time slot (unless, of course, it's an urgent matter that needs addressing immediately).

Balance Working Hours With Private Hours

Figure out what will make you feel like you're "at work". Some people like to get dressed in work clothes every morning (well, maybe "Casual Friday" kind of work clothes). For some, it's that first cup of coffee, or sorting through your email or reading a work-related website. For me, it's the coffee ... I usually work in my bathrobe for a couple of hours first.  (I know, TMI ... ha ha)

You should figure out what your work hours will be and then stick to it! If you don't, it can impact you in two different ways:

One way is distraction ... oh, I just need to do this one thing first before I start work (which may balloon to a few more things), etc.  Well, think about it ... when you actually had to drive to the office every day, that "one thing" didn't get done before you went to work, did it? So, don't make it any different now that you're home all day. (Although, granted, you might have to be more flexible if you have children who are home from school).

The flip side of distraction is working too much. It's an easy trap to fall into ... your work is right there at home and it's easy to be tempted to keep working longer than you would at the office. Don't do it ... at the end of your self-appointed office hours, that's it. Don't even think about work. Now is your personal/family time and work will wait until tomorrow morning. 

Bottom line: it's all about good work habits. In other words, discipline!

Make Time For Yourself

Take a lunch break. A real lunch break ... don't make a sandwich and take it back to eat at your desk. Eat lunch in the kitchen, or on your back deck/balcony/whatever if it's a nice day.

Exercise. You'll be a lot more sedentary working from home (without those trips to the water cooler to gossip with your co-workers). Take a quick 10 or 15 minute walk, run or bike ride at lunchtime or mid-afternoon. If the weather's not cooperating, just walking around your house will work too (especially if you have stairs you can go up and down).

And Finally ...

This emergency situation won't last forever (hopefully). You may find that working from home suits you just fine! You may find that you're way more productive at home than you could ever be at the office (without sacrificing your "me time"). And your employer may discover that many of their employees *can* work productively from home and start allowing it on a regular basis.

Stay Safe!  =0)









Saturday, February 29, 2020

Organize Your Projects With Partial Classes

I was recently asked if I had any type of procedure or process that I follow for creating Classes. I responded, "No, I don’t have anything concrete for doing that. Sometimes whatever you’re trying to accomplish can dictate how you will handle creating classes".

But here's one tip that I find can often be useful and that is, as the title of this post indicates, using partial classes for project organization. It can also be useful when you have a few different team members that need to work on different parts of the class. Let me explain ...

Let's say you have an application that processes various types of messages that it receives. This is what MyMessage class might look like:

public class MyMessage
{
public string MessageType { get; set; }
public string MessageData { get; set; }
public DateTime CreatedDatetime { get; set; }
public string ReceivedFrom { get; set; }
public string SendTo { get; set; }
}

Now, let's say that you have three categories of messages: Incidents, Requests, Resources. You might organize this by having four partial class files. These four files could be named MessageProcessor.cs, MessageIncidentProcessor.cs, MessageResourceProcessor.cs and MessageRequestProcessor.cs. Note that all these files start with the word "Message" ... this effectively groups them all together in your project making it obvious that they belong together and making them easy to find in your project.

One could also use project folders for organizing similar functionality, but I often find that it's easier to find what you're looking for by grouping by name of file and not "hiding" files in folders. Maybe that's just me.  ;0) It probably also depends on the size of your project (if you have a lot of files in your project, perhaps folder organization is better). However, that is *NOT* the topic of this blog post! It's about partial classes!  =0)

As you may have guessed by now, the partial class will be called MessageProcessor and the MessageProcessor.cs file will contain all the variables and methods that are common between the various MessageTypes. Any method that is common to at least two MessageTypes belongs here. The main functionality of the code in the MessageProcessor.cs file is to receive the message, and then call the appropriate method to process it (which will be methods in the other .cs files).

OK, now for the code:

First, the "controlling" MessageProcessor.cs partial class file.

public partial class MessageProcessor
{
// Note that the constructor is only in this "main" class file
public MessageProcessor()
{
// initialize stuff here, probably having to do with setting up
// for receiving messages, and whatever else all three message types
// have in common for
}
// Next, we'll have all methods that are common to all types of Messages
// The first being processing the incoming message and calling the appropriate methods
public bool ProcessMessage(MyMessage message)
{
bool IsOK = true;
switch (message.MessageType)
{
// Incidents
case "NewIncident":
IsOK = this.ProcessNewIncident(message);
break;
case "UpdateIncident":
IsOK = this.ProcessIncidentUpdate(message);
break;
case "CloseIncident":
IsOK = this.ProcessIncidentClose(message);
break;
// Resources
case "NewResource":
IsOK = this.ProcessNewResource(message);
break;
case "AssignResource":
IsOK = this.ProcessResourceAssignment(message);
break;
case "UpdateResource":
IsOK = this.ProcessResourceUpdate(message);
break;
// Requests
case "RequestForTransfer":
IsOK = this.ProcessTransferRequest(message);
break;
default:
// some default behavior or
// log error or ignore or whatever
break;
}
return IsOK;
}
// Add additional methods that are common across all MessageTypes
}

Name the next file MessageIncidentProcessor.cs

public partial class MessageProcessor
{
protected bool ProcessNewIncident(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
protected bool ProcessIncidentUpdate(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
protected bool ProcessIncidentClose(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
// Add additional methods that are only used for Incidents
}

The next file is called MessageResourceProcessor.cs

public partial class MessageProcessor
{
protected bool ProcessNewResource(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
protected bool ProcessResourceAssignment(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
protected bool ProcessResourceUpdate(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
// Add additional methods that are only used for Resources
}

And, lastly, call this file MessageRequestProcessor.cs

public partial class MessageProcessor
{
protected bool ProcessTransferRequest(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
return IsOK;
}
// Add additional methods that are only used for Requests
}

Well, now let's say that the Requests need an additional constructor. Let's change the MessageRequestProcessor.cs file to look like this:

public partial class MessageProcessor
{
private bool SendResponse = false;

// An additional constructor needed for Requests
public MessageProcessor(bool sendResponse) : base()
{
this.SendResponse = sendResponse;
}
protected bool ProcessTransferRequest(MyMessage message)
{
bool IsOK = true;
// Necessary code to process, setting IsOK as needed
if (this.SendResponse)
{
// code to send the response
}
return IsOK;
}
// Add additional methods that are only used for Requests
}

Now, how does this structure help co-worker cooperation? If I'm working on Incidents and another developer on my Team is working on Resources, we don't have to worry about stepping on each other's toes and making sure that we are careful with TFS check-ins and check-outs. TFS normally makes this fairly painless, but not always. Sometimes the process of merging code that I'm checking in with code that another developer has checked in, does not go easily. I'd rather avoid it.  This way, my co-worker checks out the MessageResourceProcessor.cs file and I check out the MessageIncidentProcessor.cs file ... and we're both happy!!  =0)

I hope this post has given you some good ideas ... Happy Coding!  =0)

Friday, January 31, 2020

Merge Tables And Sum Columns

A question on the MSDN Forums prompted me to write the topic of today's blog post. The question, in a nutshell, involved how to merge data into one DataTable from two DataTables originating from two different sources and provide a sum of a common column in that new DataTable.

I will illustrate the concept with an example from my fictitious ride-sharing system that I used in two previous blog posts:

https://geek-goddess-bonnie.blogspot.com/2019/09/parsing-data-with-metadata.html
https://geek-goddess-bonnie.blogspot.com/2019/12/create-dictionary-using-linq.html

The RideShare application maintains a database containing all the cars for each of the Companies that subscribe to their Service. One of the Services that this RideShare application provides is to keep track of the mileage that each car has driven every day in its database.  Each Car Company will send information to RideShare from every car that the company operates. This is a very simplistic version of this hypothetical system, because in reality we'd keep a lot more information (such as a TripID, probably a begin/end time for each Trip, etc.).  But since I'm only using this for an example of how to "Merge and Sum", I'm not going to bother with all that. I'll only be interested in the TripMileage and the TripDate (since we'll be summing all Mileages for each Date for each Car).

Let's say that RideShare is a startup company and currently only has two Customers, GeeksRide and UberGeeks. The XML for the DataSet that RideShare has retrieved from its database for today's trips, looks like this:

<RideShareOwnerDataSet>
<CompanyInfo>
<CarCompanyID>1</CarCompanyID>
<CarCompany>GeeksRide</CarCompany>
</CompanyInfo>
<CompanyInfo>
<CarCompanyID>2</CarCompanyID>
<CarCompany>UberGeek</CarCompany>
</CompanyInfo>
<CarInfo>
<CarNumberID>1</CarNumberID>
<CarCompanyID>1</CarCompanyID>
<CarNumber>C001</CarNumber >
<TripMileage>111.1</TripMileage >
<TripDate>2019-01-31</TripDate >
</CarInfo>
<CarInfo>
<CarNumberID>1</CarNumberID>
<CarCompanyID>2</CarCompanyID>
<CarNumber>C201</CarNumber >
<TripMileage>121.1</TripMileage >
<TripDate>2019-01-31</TripDate >
</CarInfo>
<CarInfo>
<CarNumberID>2</CarNumberID>
<CarCompanyID>2</CarCompanyID>
<CarNumber>C202</CarNumber >
<TripMileage>222.2</TripMileage >
<TripDate>2019-01-31</TripDate >
</CarInfo>
<CarInfo>
<CarNumberID>3</CarNumberID>
<CarCompanyID>2</CarCompanyID>
<CarNumber>C203</CarNumber >
<TripMileage>333.3</TripMileage >
<TripDate>2019-01-31</TripDate >
</CarInfo>
</RideShareOwnerDataSet>

During the course of the day, RideShare receives data from it's two customers (perhaps after every trip, or perhaps at intervals during the day ... that's totally up to the Customers as to when they send data to RideShare). So, let's say that RideShare receives data from UberGeek. It looks like this:

<RideShareOwnerDataSet>
<CompanyInfo>
<CarCompanyID>2</CarCompanyID>
<CarCompany>UberGeek</CarCompany>
</CompanyInfo >
<CarInfo>
<CarNumberID>2</CarNumberID>
<CarCompanyID>2</CarCompanyID>
<CarNumber>C202</CarNumber >
<TripMileage>22.2</TripMileage >
<TripDate>2019-01-31</TripDate >
</CarInfo>
<CarInfo>
<CarNumberID>4</CarNumberID>
<CarCompanyID>2</CarCompanyID>
<CarNumber>C204</CarNumber >
<TripMileage>42.2</TripMileage >
<TripDate>2019-01-31</TripDate >
</CarInfo>
</RideShareOwnerDataSet>

UberGeek is reporting the mileage from two different cars, one (ID 2) is already in the database and the other (ID 4) is a brand new car. We need to merge this new information with the data from the database, and add up all mileage, so that each car has only one row that contains the total mileage for this day so far.

I decided to try my hand at doing this with LINQ.

I chose to simply create a new DataSet/DataTable to hold the revised data, rather than updating the existing DataSet (since a new DataTable was the original problem posted to the Forum).

I am also using a Typed DataSet as it is much better in terms of usability for just about anything you want to do with DataSets. Since I'm providing the XML above, it can easily be used to create a Typed DataSet. See the second link to my blog above (Create Dictionary Using LINQ), and scroll down the section "XML Data and a Typed DataSet) for info about how to do this if you don't already know.

So, let's start looking at code. I took the above XML and saved it to two different XML files and then read them into two DataSets.

RideShareOwnerDataSet dsRideShare = new RideShareOwnerDataSet();
RideShareOwnerDataSet dsCustomerRide = new RideShareOwnerDataSet();

dsRideShare.ReadXml("RideShareFromDatabase.xml");
dsCustomerRide.ReadXml("RideShareFromCustomer.xml");

var dsResult = this.MergeAndSumData(dsRideShare, dsCustomerRide);

You can then use the dsResult DataSet to update your database, however that is beyond the scope of this post.

Here is the MergeAndSumData() method:

private RideShareOwnerDataSet MergeAndSumData(RideShareOwnerDataSet dsDatabase, RideShareOwnerDataSet dsCustomer)
{
RideShareOwnerDataSet dsResult = new RideShareOwnerDataSet();
RideShareOwnerDataSet.CarInfoDataTable dtResult = dsResult.CarInfo;

// First, we Merge the CarInfo DataTables from both DataSets into one DataTable.
var dtMerged = dsDatabase.CarInfo.Copy() as RideShareOwnerDataSet.CarInfoDataTable;
dtMerged.Merge(dsCustomer.CarInfo);

var test =
dtMerged.Where(row => !row.IsTripDateNull())
.GroupBy(row => new { row.CarCompanyID, row.CarNumberID, row.CarNumber, row.TripDate }, (key, group) => new
{
key.CarNumberID,
key.CarCompanyID,
key.CarNumber,
key.TripDate,
// And Sum the Mileages for each car here:
TripMileage = group.Sum(s => s.TripMileage)
})
.Select(row => dsResult.CarInfo.LoadDataRow(new object[]
{
row.CarNumberID,
row.CarCompanyID,
row.CarNumber,
row.TripDate,
row.TripMileage
}, LoadOption.Upsert))
.ToList();

// Since we are returning the entire DataSet, we might as well put the original CompanyInfo data in its DataTable
// in case the calling method might need to have both DataTables contain data.
dsResult.CompanyInfo.Merge(dsDatabase.CompanyInfo);

return dsResult;
}

A couple of things to note about the above method:

  • We're using a .ToList() at the end of the LINQ query, but we really don't care about the List (which ends up in the variable called test) ... it's used simply because the act of running .ToList() then executes the query. Until then, LINQ hasn't done anything and dsResult.CarInfo will still be empty.
  • The order that the column variables are used in the LoadDataRow() is important. It corresponds to the order that columns are defined in the DataTable.

That's all there is to it. And while the LINQ looks fairly straightforward now that I've written it, I had not used LoadDataRow much in the past, and never had I used it in a LINQ query! So, this post took a lot of experimenting and Googling. Which is why it took me so long to finish writing it!! 

And now you know the rest of the story. 

Happy Coding!!  =0)

Monday, December 30, 2019

Create a Dictionary using LINQ

Dictionaries are useful to use in your code, and occasionally I need to look up a word I don't know!!  Oh wait ... that's a different kind of dictionary!  ;0)

Say that you have some data that you'd like to put into a Dictionary. One reason for using a Dictionary might be to make it easy to process one group of items at a time. Another reason might be for easily finding data based on the dictionary key. The data could be in a file or maybe it's been retrieved from TCP or from a database. It doesn't matter. Data that might look something like this:

MyFirstGroup:
item1

MySecondGroup:
item2
item3
item4

Or maybe delimited data, like this:

|CARCOMPANY=GeeksRide|CARNUMBER=C121|CARNUMBER=C122|CARCOMPANY=UberGeek|CARNUMBER=C133|

The above string may look a bit familiar to regular readers of my blog. It's similar to data coming from a fictional ride-sharing system that I "invented" in order to write this blog post about parsing data: https://geek-goddess-bonnie.blogspot.com/2019/09/parsing-data-with-metadata.html

Oh, and how about XML? Here's some XML (in this case, representing a DataSet):

<RideShareOwnerDataSet>
  <CompanyInfo>
    <CarCompanyID>1</CarCompanyID>
    <CarCompany>GeeksRide</CarCompany>
  </CompanyInfo>
  <CompanyInfo>
    <CarCompanyID>2</CarCompanyID>
    <CarCompany>UberGeek</CarCompany>
  </CompanyInfo>
  <CarInfo>
    <CarNumberID>1</CarNumberID>
    <CarCompanyID>1</CarCompanyID>
    <CarNumber>C121</CarNumber>
  </CarInfo>
  <CarInfo>
    <CarNumberID>2</CarNumberID>
    <CarCompanyID>1</CarCompanyID>
    <CarNumber>C122</CarNumber>
  </CarInfo>
  <CarInfo>
    <CarNumberID>3</CarNumberID>
    <CarCompanyID>2</CarCompanyID>
    <CarNumber>C133</CarNumber>
  </CarInfo>
</RideShareOwnerDataSet>

Of course, depending on the data, the processing in your application could be quite different. So, let's take a look at how we would handle these three very different types of data.

In all of these examples, I am making use of LINQ syntax to create the Dictionary. In two of these examples, I make use of LINQ's .Aggregate() function. Here is a good link that explains how .Aggregate works:

https://stackoverflow.com/questions/7105505/linq-aggregate-algorithm-explained

Lines of Data

For this first example, I put the data into a text file, that I called GroupAndItem.txt. We can use the File.ReadLines() to read each line of data and utilize the .Aggregate() function of LINQ to make it pretty easy to dump this data into a Dictionary:

string key = "";
var dcSectionItems = File.ReadLines("GroupAndItem.txt")
    .Aggregate(new Dictionary<string, List<string>>(), (a, s) => {
        var i = s.IndexOf(':');
        if (i < 0) a[key].Add(s);
        else a[(key = s.Substring(0, i))] = new List<string>();
    
        return a;
    });


And then we can show the data by iterating the Dictionary utilizing Console.WriteLine() for display.

foreach (var kvp in dcSectionItems)
{
    Console.WriteLine("{0}:", kvp.Key); // section name
    foreach (var item in kvp.Value)
    {
        Console.WriteLine("\t{0}", item); // item name
    }
}


Delimited Data

Rather than reading from a file this time, I simply used a string to hold the pipe-delimited data. In practice, this data could be coming in from TCP, a web service or a database. Notice that GeeksRide has 2 cars, while UberGeek has only one. Again, I make use of the LINQ .Aggregate() function to create the Dictionary:

string pipeCompanyCars = "|CARCOMPANY=GeeksRide|CARNUMBER=C121|CARNUMBER=C122|CARCOMPANY=UberGeek|CARNUMBER=C133|";

string[] entries = pipeCompanyCars.Split(new char[] { '|' }, StringSplitOptions.RemoveEmptyEntries);
key = "";
var dcCompanyCars = entries
    .Aggregate(new Dictionary<string, List<string>>(), (a, s) => {
        var i = s.IndexOf("CARCOMPANY=");
        if (i >= 0) a[(key = s.Substring(11))] = new List<string>();
        else
        {
            i = s.IndexOf("CARNUMBER=");
            if (i >= 0) a[key].Add(s.Substring(10));
        }
        return a;
    });


And, again, show the data in the Dictionary the same way:

foreach (var kvp in dcCompanyCars)
{
    Console.WriteLine("{0}:", kvp.Key); // car company name
    foreach (var item in kvp.Value)
    {
        Console.WriteLine("\t{0}", item); // car numbers
    }
}



XML Data


I almost exclusively utilize XML with DataSets. Most of the time, the DataSets in my applications are Typed DataSets. But, every once in a while, I'll use a plain old DataSet for one-time uses. My first example with XML data here will simply use a plain old DataSet.

The XML I'm using could have just as easily been retrieved from an .xml file to fill the DataSet using ds.ReadXml("FileName.xml"), but here I am filling the DataSet from an XML string. Here's the string and the code to read it into a DataSet :

string xmlCompanyCars =
    "<RideShareOwnerDataSet>" +
      "<CompanyInfo>" +
        "<CarCompanyID>1</CarCompanyID>" +
        "<CarCompany>GeeksRide</CarCompany>" +
      "</CompanyInfo >" +
      "<CompanyInfo>" +
        "<CarCompanyID>2</CarCompanyID>" +
        "<CarCompany>UberGeek</CarCompany>" +
      "</CompanyInfo >" +
      "<CarInfo>" +
        "<CarNumberID>1</CarNumberID>" +
        "<CarCompanyID>1</CarCompanyID>" +
        "<CarNumber>C121 </CarNumber >" +
      "</CarInfo>" +
      "<CarInfo>" +
        "<CarNumberID>2</CarNumberID>" +
        "<CarCompanyID>1</CarCompanyID>" +
        "<CarNumber>C122 </CarNumber >" +
      "</CarInfo>" +
      "<CarInfo>" +
        "<CarNumberID>3</CarNumberID>" +
        "<CarCompanyID>2</CarCompanyID>" +
        "<CarNumber>C133 </CarNumber >" +
      "</CarInfo>" +
    "</RideShareOwnerDataSet >";

DataSet ds = new DataSet();
StringReader sr = new StringReader(xmlCompanyCars);
ds.ReadXml(sr, XmlReadMode.InferSchema);

If you use XML strings to fill DataSet a lot, instead of the above code, I recommend creating a handy Extension method, like the following:

// Extension methods for DataSet:
public static void FillWithXml(this DataSet ds, string XML)
{
    StringReader sr = new StringReader(XML);
    ds.ReadXml(sr, XmlReadMode.InferSchema);
    ds.AcceptChanges();
}

Here, you'd use it like this:

ds.FillWithXml(xmlCompanyCars);

Here's the rest of the code for this XML example:

// Now let's get that DataSet into a Dictionary!
// We do need a "dummy" DataTable for this to work
DataTable dt = new DataTable();
dt.Columns.Add("CarCompany");
dt.Columns.Add("CarNumber");

var dcCompanyCarsFromDataSet = ds.Tables[0].AsEnumerable()
    .Join(ds.Tables[1].AsEnumerable(),
        rowCompany => rowCompany.Field<string>("CarCompanyID"), rowCar => rowCar.Field<string>("CarCompanyID"),
        (rowCompany, rowCar) =>
        {
            DataRow newRow = dt.NewRow();
            newRow["CarCompany"] = rowCompany.Field<string>("CarCompany");
            newRow["CarNumber"] = rowCar.Field<string>("CarNumber");
            return newRow;
        })
    .GroupBy(rowKey => rowKey.Field<string>("CarCompany"), rowValue => rowValue.Field<string>("CarNumber"))
    .ToDictionary(rowKey => rowKey.Key, rowValue => rowValue.ToList());

Notice the syntax for accessing the columns of data in LINQ with a regular DataSet. You *must* specify the type of data with .Field<string>("ColumnName"). There is no way around this (unless you use a Typed DataSet ... more on that in my final example). Also notice that I did *NOT* use the .Aggregate() function here. I could not figure out how to do it that way (and I doubt that it can be done, but if any of my faithful readers want to give it a try ... go for it! Leave me a comment if you figure it out). But the .GroupBy() and the .ToDictionary() work just fine!

One more thing that you may notice, and that is the CarCompanyID and the CarNumberID. In the XML, you can clearly see that they are numbers, most likely an int. But, because I read the XML in to a regular DataSet (and even though I used XmlReadMode.InferSchema), it did not infer that those numbers were anything but a string. Which is OK in this case, but just something to be aware of. We could have added Tables and Columns to that DataSet before we read in the XML data, but why bother? If I wanted that, I'd use a Typed DataSet.

And use a similar foreach, as before, to iterate the Dictionary:

foreach (var kvp in dcCompanyCarsFromDataSet)
{
    Console.WriteLine("{0}:", kvp.Key); // car company name
    foreach (var item in kvp.Value)
    {
        Console.WriteLine("\t{0}", item); // car numbers
    }
}

XML Data and a Typed DataSet

To experiment with Typed DataSets if you've never used them, since we have available XML, you can use ds.WriteSchema("RideShareOwnerDataSet.xsd") method to create an .xsd file. Then add the .xsd file to your solution and run the MSDataSetGenerator to create a Typed DataSet. Here's one of my blog posts about how to do that:

https://geek-goddess-bonnie.blogspot.com/2010/04/create-xsd.html

The relevant (to this post) part of that blog post is this:

Simply add that .xsd to your DataSet project, right-click on the .xsd and choose "Run Custom Tool". This is what will generate the Typed DataSet for you. If that option doesn't show up in your context menu, choose Properties instead and type "MSDataSetGenerator" in the Custom Tool property. After that, any time you make a change to the .xsd in the XML Editor and save the change, the Typed DataSet gets regenerated.

I edited the .xsd to change the datatypes of the IDs to an int, instead of a string. Go ahead and do that if you'd like to.

// We still need a "dummy" table
DataTable dt = new DataTable();
dt.Columns.Add("CarCompany");
dt.Columns.Add("CarNumber");

var dcCompanyCarsFromTypedDataSet = dsRideShareOwner.CompanyInfo
    .Join(dsRideShareOwner.CarInfo,
        // Notice that we can use the Typed syntax for the columns now
        rowCompany => rowCompany.CarCompanyID, rowCar => rowCar.CarCompanyID,
        (rowCompany, rowCar) =>
        {
            // Use the Typed syntax here too
            DataRow newRow = dt.NewRow();
            newRow["CarCompany"] = rowCompany.CarCompany;
            newRow["CarNumber"] = rowCar.CarNumber;
            return newRow;
        })
    // The GroupBy still needs to use the untyped syntax, because these Rows are from the untyped "dummy" DataTable
    .GroupBy(rowKey => rowKey.Field<string>("CarCompany"), rowValue => rowValue.Field<string>("CarNumber"))
    .ToDictionary(rowKey => rowKey.Key, rowValue => rowValue.ToList());

You'll notice that the LINQ syntax is quite different for Typed DataSet versus non-typed. We do *not* have to specify Field<string> for the columns we are using, since we can use the Typed property to access the column data: rowCompany.CarCompanyID, instead of rowCompany.Field<string>("CarCompanyID").  Much cleaner.

And, as before, iterate the Dictionary:

foreach (var kvp in dcCompanyCarsFromTypedDataSet)
{
    Console.WriteLine("{0}:", kvp.Key); // car company name
    foreach (var item in kvp.Value)
    {
        Console.WriteLine("\t{0}", item); // car numbers
    }
}

So, that's it. It's been a fun experiment for me as I wrote this blog post. I hope that you, Dear Reader, can put this to good use.

Happy Coding!!  =0)