PageRenderTime 313ms CodeModel.GetById 2ms app.highlight 296ms RepoModel.GetById 1ms app.codeStats 0ms

/articles/hdinsight-dotnet-avro-serialization.md

https://github.com/deaquino/azure-content
Markdown | 1314 lines | 1074 code | 240 blank | 0 comment | 0 complexity | ac89c072fea1008e0d0aba443c326bc4 MD5 | raw file
   1<properties linkid="hdinsight-dotnet-avro-serialization" urlDisplayName="HDInsight Microsoft .NET Library for Serialization with Avro" pageTitle="Serialize data with the Microsoft .NET Library for Avro | Azure" metaKeywords="" description="Learn how Azure HDInsight uses Avro to serialize big data." metaCanonical="" services="hdinsight" documentationCenter="" title="Serialize data with the Microsoft .NET Library for Avro " authors="bradsev" solutions="" manager="paulettm" editor="cgronlun" />
   2
   3<tags ms.service="hdinsight" ms.workload="big-data" ms.tgt_pltfrm="na" ms.devlang="na" ms.topic="article" ms.date="01/01/1900" ms.author="bradsev" />
   4
   5
   6# Serialize data with the Microsoft .NET Library for Avro
   7
   8##Overview
   9This topic shows how to use the Microsoft .NET Library for Avro to serialize objects and other data structures into streams in order to persist them to memory, a database or a file, and also how to deserialize them to recover the original objects. 
  10
  11###Apache Avro
  12The Microsoft .NET Library for Avro implements the Apache Avro data serialization system for the Microsoft.NET environment. Apache Avro provides a compact binary data interchange format for serialization. It uses [JSON](http://www.json.org) to define language agnostic schema that underwrites language interoperability. Data serialized in one language can be read in another. Currently C, C++, C#, Java, PHP, Python, and Ruby are supported. Detailed information on the format can be found in the [Apache Avro Specification](http://avro.apache.org/docs/current/spec.html). Note that the current version of the Microsoft .NET Library for Avro does not support the Remote Procedure Calls (RPC) part of this specification.
  13
  14The serialized representation of an object in Avro system consists of two parts: schema and actual value. The Avro schema describes the language independent data model of the serialized data with JSON. It is present side-by-side with a binary representation of data.  Having the schema separate from the binary representation permits each object to be written with no per-value overheads, making serialization fast and the representation small. 
  15
  16###The Hadoop scenario 
  17Apache Avro serialization format is widely used in Azure HDInsight and other Apache Hadoop environments. Avro provides a convenient way to represent complex data structures within a Hadoop MapReduce job. The format of Avro files has been designed to support the distributed MapReduce programming model. The key feature that enables the distribution is that the files are “splittable” in the sense that one can seek any point in a file and start reading from a particular block. 
  18 
  19###Serialization in the Microsoft .NET Library for Avro
  20The .NET Library for Avro supports two ways of serializing objects:
  21
  22- **reflection**: The JSON schema for the types is automatically built from the data contract attributes of the .NET types to be serialized. 
  23- **generic record**:The A JSON schema is explicitly specified in a record represented by the [**AvroRecord**](http://msdn.microsoft.com/en-us/library/microsoft.hadoop.avro.avrorecord.aspx) class when no .NET types are present to describe the schema for the data to be serialized. 
  24
  25When the data schema is known to both the writer and reader of the stream, the data can be sent without its schema. But when this is not the case, the schema must be shared using an Avro container file. Other parameters such as the codec used for data compression can be specified. These scenarios are outlined in more detail and illustrated in the code examples below.
  26
  27
  28###Microsoft .NET Library for Avro prerequisites
  29- [Microsoft .NET Framework v4.0](http://www.microsoft.com/en-us/download/details.aspx?id=17851)
  30- [Newtonsoft Json.NET](http://james.newtonking.com/json) (v5.0.5 or later) 
  31
  32Note that the Newtonsoft.Json.dll dependency is downloaded automatically with with the installation of the Microsoft .NET Library for Avro, the procedure for which is provided in the following section.
  33
  34###Microsoft .NET Library for Avro installation
  35The Microsoft .NET Library for Avro is distributed as a NuGet Package that can be installed from Visual Studio using the following procedure: 
  36
  37- Select the **Project** tab -> **Manage NuGet Packages...**
  38- Search for "Microsoft.Hadoop.Avro" in the **Online Search** box.
  39- Click the **Install** button next to **Microsoft .NET Library for Avro**. 
  40
  41Note that the Newtonsoft.Json.dll (>= .5.0.5) dependency is also downloaded automatically with with the Microsoft .NET Library for Avro.
  42 
  43
  44##Guide to the samples
  45Five examples provided in this topic each illustrate different scenarios supported by the Microsoft .NET Library for Avro. 
  46
  47The first two show how to serialize and deserialize data into memory stream buffers using reflection and generic records. The schema in these two cases is assumed to be shared between the readers and writers out-of-band so that the schema does not need to be serialized with the data in an Avro container file. 
  48
  49The third and fourth examples show how to serialize and deserialize data into memory stream buffers using reflection and generic record with Avro object container files. When data is stored in an Avro container file, its schema is always stored with it because the schema must be shared for deserialization.
  50
  51The sample containing the first four examples can be downloaded from [Azure code samples](http://code.msdn.microsoft.com/windowsazure/Serialize-data-with-the-86055923) site.
  52
  53The fifth and final example shows how to how to use a custom compression codec for object container files. A sample containing the code for this example can be downloaded from the  [Azure  code samples](http://code.msdn.microsoft.com/windowsazure/Serialize-data-with-the-67159111) site.
  54
  55The Microsoft .NET Library for Avro is designed to work with any stream. In these examples, data is manipulated using memory streams rather than file streams or databases for simplicity and consistency. The approach taken in a production environment will depend on the exact scenario requirements, data source and volume, performance constraints, and other factors.
  56
  57 * <a href="#Scenario1">**Serialization with reflection**</a>: The JSON schema for types to be serialized is automatically built from the data contract attributes.
  58 * <a href="#Scenario2">**Serialization with generic record**</a>: The JSON schema is explicitly specified in a record when no .NET type is available for reflection.
  59 * <a href="#Scenario3">**Serialization using object container files with reflection**</a>: The JSON schema is implicitly serialized with the data and shared using an Avro container file.
  60 * <a href="#Scenario4">**Serialization using object container files with generic record**</a>: The JSON schema is explicitly serialized with the data and shared using an Avro container file.
  61 * <a href="#Scenario5">**Serialization using object container files with a custom compression codec**</a>: The JSON schema is serialized with data and shared using an Avro container file with a customized .NET implementation of the deflate data compression codec.
  62
  63
  64<h2> <a name="Scenario1"></a>Serialization with reflection</h2>
  65 
  66The JSON schema for the types can be automatically built by Microsoft .NET Library for Avro using reflection from the data contract attributes of the C# objects to be serialized. Microsoft .NET Library for Avro creates an [**IAvroSeralizer<T>**](http://msdn.microsoft.com/en-us/library/dn627341.aspx) to identify the fields to be serialized.
  67
  68In this example objects (a **SensorData** class with a member **Location** struct) are serialized to a memory stream and this stream is in turn deserialized. The result is then compared to the initial instance to confirm that the **SensorData** object recovered is identical to original.
  69
  70The schema in this example is assumed to be shared between the readers and writers, so the Avro object container format is not required. For an example of how to serialize and deserialize data into memory buffers using reflection with the object container format when the schema must be serialized with the data, see <a href="#Scenario3">Serialization using object container files with reflection.</a>
  71
  72    namespace Microsoft.Hadoop.Avro.Sample
  73    {
  74        using System;
  75        using System.Collections.Generic;
  76        using System.IO;
  77        using System.Linq;
  78        using System.Runtime.Serialization;
  79        using Microsoft.Hadoop.Avro.Container;
  80
  81        //Sample Class used in serialization samples
  82        [DataContract(Name = "SensorDataValue", Namespace = "Sensors")]
  83        internal class SensorData
  84        {
  85            [DataMember(Name = "Location")]
  86            public Location Position { get; set; }
  87
  88            [DataMember(Name = "Value")]
  89            public byte[] Value { get; set; }
  90        }
  91
  92        //Sample struct used in serialization samples
  93        [DataContract]
  94        internal struct Location
  95        {
  96            [DataMember]
  97            public int Floor { get; set; }
  98
  99            [DataMember]
 100            public int Room { get; set; }
 101        }
 102
 103        //This class contains all methods demonstrating
 104        //the usage of Microsoft .NET Library for Avro
 105        public class AvroSample
 106        {
 107
 108            //Serialize and deserialize sample data set represented as an object using Reflection
 109            //No explicit schema definition is required - schema of serialized objects is automatically built
 110            public void SerializeDeserializeObjectUsingReflection()
 111            {
 112
 113                Console.WriteLine("SERIALIZATION USING REFLECTION\n");
 114                Console.WriteLine("Serializing Sample Data Set...");
 115
 116                //Create a new AvroSerializer instance and specify a custom serialization strategy AvroDataContractResolver
 117                //for serializing only properties attributed with DataContract/DateMember
 118                var avroSerializer = AvroSerializer.Create<SensorData>();
 119
 120                //Create a Memory Stream buffer
 121                using (var buffer = new MemoryStream())
 122                {
 123                    //Create a data set using sample Class and struct 
 124                    var expected = new SensorData { Value = new byte[] { 1, 2, 3, 4, 5 }, Position = new Location { Room = 243, Floor = 1 } };
 125
 126                    //Serialize the data to the specified stream
 127                    avroSerializer.Serialize(buffer, expected);
 128
 129
 130                    Console.WriteLine("Deserializing Sample Data Set...");
 131
 132                    //Prepare the stream for deserializing the data
 133                    buffer.Seek(0, SeekOrigin.Begin);
 134
 135                    //Deserialize data from the stream and cast it to the same type used for serialization
 136                    var actual = avroSerializer.Deserialize(buffer);
 137
 138                    Console.WriteLine("Comparing Initial and Deserialized Data Sets...");
 139
 140                    //Finally, verify that deserialized data matches the original one
 141                    bool isEqual = this.Equal(expected, actual);
 142
 143                    Console.WriteLine("Result of Data Set Identity Comparison is {0}", isEqual);
 144
 145                }
 146            }
 147
 148            //
 149            //Helper methods
 150            //
 151
 152            //Comparing two SensorData objects
 153            private bool Equal(SensorData left, SensorData right)
 154            {
 155                return left.Position.Equals(right.Position) && left.Value.SequenceEqual(right.Value);
 156            }
 157
 158
 159
 160            static void Main()
 161            {
 162
 163                string sectionDivider = "---------------------------------------- ";
 164
 165                //Create an instance of AvroSample Class and invoke methods
 166                //illustrating different serializing approaches
 167                AvroSample Sample = new AvroSample();
 168
 169                //Serialization to memory using Reflection
 170                Sample.SerializeDeserializeObjectUsingReflection();
 171
 172                Console.WriteLine(sectionDivider);
 173                Console.WriteLine("Press any key to exit.");
 174                Console.Read();
 175            }
 176        }
 177    }
 178    // The example is expected to display the following output: 
 179    // SERIALIZATION USING REFLECTION
 180    //
 181    // Serializing Sample Data Set...
 182    // Deserializing Sample Data Set...
 183    // Comparing Initial and Deserialized Data Sets...
 184    // Result of Data Set Identity Comparison is True
 185    // ----------------------------------------
 186    // Press any key to exit.
 187
 188
 189<h2> <a name="Scenario2"></a>Serialization with a generic record</h2>
 190
 191A JSON schema can be explicitly specified in a generic record when reflection cannot be used because the data cannot be represented using .NET classes with a data contract. This method is generally slower than using reflection and serializers for specific C# class. In such cases, the schema for the  data may also be dynamic because it is not be known until compile-time. Data represented as Comma Separated Values (CSV) files whose schema is unknown until it is transformed to the Avro format at run-time is an example of this sort of dynamic scenario.
 192
 193This example shows how to create and use an [**AvroRecord**](http://msdn.microsoft.com/en-us/library/microsoft.hadoop.avro.avrorecord.aspx) to explicitly specify a JSON schema, how to populate it with the data, then serialize and deserialize it. The result is then compared to the initial instance to confirm that the record recovered is identical to original.
 194
 195The schema in this example is assumed to be shared between the readers and writers, so the Avro object container format is not required. For an example of how to serialize and deserialize data into memory buffers using a generic record with the object container format when the schema must be included with the serialized data, see <a href="#Scenario4">Serialization using object container files with generic record</a> example.
 196
 197
 198	namespace Microsoft.Hadoop.Avro.Sample
 199	{
 200    using System;
 201    using System.Collections.Generic;
 202    using System.IO;
 203    using System.Linq;
 204    using System.Runtime.Serialization;
 205    using Microsoft.Hadoop.Avro.Container;
 206    using Microsoft.Hadoop.Avro.Schema;
 207
 208    //This class contains all methods demonstrating
 209    //the usage of Microsoft .NET Library for Avro
 210    public class AvroSample
 211    {
 212
 213        //Serialize and deserialize sample data set using Generic Record.
 214        //Generic Record is a special class with the schema explicitly defined in JSON.
 215        //All serialized data should be mapped to the fields of Generic Record,
 216        //which in turn will be then serialized.
 217        public void SerializeDeserializeObjectUsingGenericRecords()
 218        {
 219            Console.WriteLine("SERIALIZATION USING GENERIC RECORD\n");
 220            Console.WriteLine("Defining the Schema and creating Sample Data Set...");
 221
 222            //Define the schema in JSON
 223            const string Schema = @"{
 224                                ""type"":""record"",
 225                                ""name"":""Microsoft.Hadoop.Avro.Specifications.SensorData"",
 226                                ""fields"":
 227                                    [
 228                                        { 
 229                                            ""name"":""Location"", 
 230                                            ""type"":
 231                                                {
 232                                                    ""type"":""record"",
 233                                                    ""name"":""Microsoft.Hadoop.Avro.Specifications.Location"",
 234                                                    ""fields"":
 235                                                        [
 236                                                            { ""name"":""Floor"", ""type"":""int"" },
 237                                                            { ""name"":""Room"", ""type"":""int"" }
 238                                                        ]
 239                                                }
 240                                        },
 241                                        { ""name"":""Value"", ""type"":""bytes"" }
 242                                    ]
 243                            }";
 244
 245            //Create a generic serializer based on the schema
 246            var serializer = AvroSerializer.CreateGeneric(Schema);
 247            var rootSchema = serializer.WriterSchema as RecordSchema;
 248
 249            //Create a Memory Stream buffer
 250            using (var stream = new MemoryStream())
 251            {
 252                //Create a generic record to represent the data
 253                dynamic location = new AvroRecord(rootSchema.GetField("Location").TypeSchema);
 254                location.Floor = 1;
 255                location.Room = 243;
 256
 257                dynamic expected = new AvroRecord(serializer.WriterSchema);
 258                expected.Location = location;
 259                expected.Value = new byte[] { 1, 2, 3, 4, 5 };
 260
 261                Console.WriteLine("Serializing Sample Data Set...");
 262
 263                //Serialize the data
 264                serializer.Serialize(stream, expected);
 265
 266                stream.Seek(0, SeekOrigin.Begin);
 267
 268                Console.WriteLine("Deserializing Sample Data Set...");
 269
 270                //Deserialize the data into a generic record
 271                dynamic actual = serializer.Deserialize(stream);
 272
 273                Console.WriteLine("Comparing Initial and Deserialized Data Sets...");
 274
 275                //Finally, verify the results
 276                bool isEqual = expected.Location.Floor.Equals(actual.Location.Floor);
 277                isEqual = isEqual && expected.Location.Room.Equals(actual.Location.Room);
 278                isEqual = isEqual && ((byte[])expected.Value).SequenceEqual((byte[])actual.Value);
 279                Console.WriteLine("Result of Data Set Identity Comparison is {0}", isEqual);
 280            }
 281        }
 282
 283        static void Main()
 284        {
 285
 286            string sectionDivider = "---------------------------------------- ";
 287
 288            //Create an instance of AvroSample Class and invoke methods
 289            //illustrating different serializing approaches
 290            AvroSample Sample = new AvroSample();
 291
 292            //Serialization to memory using Generic Record
 293            Sample.SerializeDeserializeObjectUsingGenericRecords();
 294
 295            Console.WriteLine(sectionDivider);
 296            Console.WriteLine("Press any key to exit.");
 297            Console.Read();
 298        }
 299    }
 300	}
 301    // The example is expected to display the following output: 
 302    // SERIALIZATION USING GENERIC RECORD
 303    //
 304    // Defining the Schema and creating Sample Data Set...
 305    // Serializing Sample Data Set...
 306    // Deserializing Sample Data Set...
 307    // Comparing Initial and Deserialized Data Sets...
 308    // Result of Data Set Identity Comparison is True
 309    // ----------------------------------------
 310    // Press any key to exit.
 311
 312
 313<h2> <a name="Scenario3"></a>Serialization using object container files and serialization with reflection</h2>
 314
 315This example is similar to scenario in the <a href="#Scenario1"> first example</a> where the schema is implicitly specified with reflection, except that here the schema is not assumed to be known to the reader that deserializes it. The **SensorData** objects to be serialized and its implicitly specified schema are stored in an object container file represented by the [**AvroContainer**](http://msdn.microsoft.com/en-us/library/microsoft.hadoop.avro.container.avrocontainer.aspx) class. 
 316
 317The data is serialized in this example with [**SequentialWriter<SensorData>**](http://msdn.microsoft.com/en-us/library/dn627340.aspx) and deserialized with [**SequentialReader<SensorData>**](http://msdn.microsoft.com/en-us/library/dn627340.aspx). The result then is compared to the initial instances to insure identity.
 318
 319The data in object container file is compressed using the default [**Deflate**][deflate-100] compression codec from .NET Framework 4.0. See the <a href="#Scenario5"> last example</a> in this topic to learn how to use a more recent and superior version of the [**Deflate**][deflate-110] compression codec available in .NET Framework 4.5.
 320
 321    namespace Microsoft.Hadoop.Avro.Sample
 322    {
 323        using System;
 324        using System.Collections.Generic;
 325        using System.IO;
 326        using System.Linq;
 327        using System.Runtime.Serialization;
 328        using Microsoft.Hadoop.Avro.Container;
 329
 330        //Sample Class used in serialization samples
 331        [DataContract(Name = "SensorDataValue", Namespace = "Sensors")]
 332        internal class SensorData
 333        {
 334            [DataMember(Name = "Location")]
 335            public Location Position { get; set; }
 336
 337            [DataMember(Name = "Value")]
 338            public byte[] Value { get; set; }
 339        }
 340
 341        //Sample struct used in serialization samples
 342        [DataContract]
 343        internal struct Location
 344        {
 345            [DataMember]
 346            public int Floor { get; set; }
 347
 348            [DataMember]
 349            public int Room { get; set; }
 350        }
 351
 352        //This class contains all methods demonstrating
 353        //the usage of Microsoft .NET Library for Avro
 354        public class AvroSample
 355        {
 356
 357            //Serializes and deserializes sample data set using Reflection and Avro Object Container Files
 358            //Serialized data is compressed with Deflate codec
 359            public void SerializeDeserializeUsingObjectContainersReflection()
 360            {
 361
 362                Console.WriteLine("SERIALIZATION USING REFLECTION AND AVRO OBJECT CONTAINER FILES\n");
 363
 364                //Path for Avro Object Container File
 365                string path = "AvroSampleReflectionDeflate.avro";
 366
 367                //Create a data set using sample Class and struct
 368                var testData = new List<SensorData>
 369                        {
 370                            new SensorData { Value = new byte[] { 1, 2, 3, 4, 5 }, Position = new Location { Room = 243, Floor = 1 } },
 371                            new SensorData { Value = new byte[] { 6, 7, 8, 9 }, Position = new Location { Room = 244, Floor = 1 } }
 372                        };
 373
 374                //Serializing and saving data to file
 375                //Creating a Memory Stream buffer
 376                using (var buffer = new MemoryStream())
 377                {
 378                    Console.WriteLine("Serializing Sample Data Set...");
 379
 380                    //Create a SequentialWriter instance for type SensorData which can serialize a sequence of SensorData objects to stream
 381                    //Data will be compressed using Deflate codec
 382                    using (var w = AvroContainer.CreateWriter<SensorData>(buffer, Codec.Deflate))
 383                    {
 384                        using (var writer = new SequentialWriter<SensorData>(w, 24))
 385                        {
 386                            // Serialize the data to stream using the sequential writer
 387                            testData.ForEach(writer.Write);
 388                        }
 389                    }
 390
 391                    //Save stream to file
 392                    Console.WriteLine("Saving serialized data to file...");
 393                    if (!WriteFile(buffer, path))
 394                    {
 395                        Console.WriteLine("Error during file operation. Quitting method");
 396                        return;
 397                    }
 398                }
 399
 400                //Reading and deserializing data
 401                //Creating a Memory Stream buffer
 402                using (var buffer = new MemoryStream())
 403                {
 404                    Console.WriteLine("Reading data from file...");
 405
 406                    //Reading data from Object Container File
 407                    if (!ReadFile(buffer, path))
 408                    {
 409                        Console.WriteLine("Error during file operation. Quitting method");
 410                        return;
 411                    }
 412
 413                    Console.WriteLine("Deserializing Sample Data Set...");
 414
 415                    //Prepare the stream for deserializing the data
 416                    buffer.Seek(0, SeekOrigin.Begin);
 417
 418                    //Create a SequentialReader for type SensorData which will derserialize all serialized objects from the given stream
 419                    //It allows iterating over the deserialized objects because it implements IEnumerable<T> interface
 420                    using (var reader = new SequentialReader<SensorData>(
 421                        AvroContainer.CreateReader<SensorData>(buffer, true)))
 422                    {
 423                        var results = reader.Objects;
 424
 425                        //Finally, verify that deserialized data matches the original one
 426                        Console.WriteLine("Comparing Initial and Deserialized Data Sets...");
 427                        int count = 1;
 428                        var pairs = testData.Zip(results, (serialized, deserialized) => new { expected = serialized, actual = deserialized });
 429                        foreach (var pair in pairs)
 430                        {
 431                            bool isEqual = this.Equal(pair.expected, pair.actual);
 432                            Console.WriteLine("For Pair {0} result of Data Set Identity Comparison is {1}", count, isEqual);
 433                            count++;
 434                        }
 435                    }
 436                }
 437
 438                //Delete the file
 439                RemoveFile(path);
 440            }
 441
 442            //
 443            //Helper methods
 444            //
 445
 446            //Comparing two SensorData objects
 447            private bool Equal(SensorData left, SensorData right)
 448            {
 449                return left.Position.Equals(right.Position) && left.Value.SequenceEqual(right.Value);
 450            }
 451
 452            //Saving memory stream to a new file with the given path
 453            private bool WriteFile(MemoryStream InputStream, string path)
 454            {
 455                if (!File.Exists(path))
 456                {
 457                    try
 458                    {
 459                        using (FileStream fs = File.Create(path))
 460                        {
 461                            InputStream.Seek(0, SeekOrigin.Begin);
 462                            InputStream.CopyTo(fs);
 463                        }
 464                        return true;
 465                    }
 466                    catch (Exception e)
 467                    {
 468                        Console.WriteLine("The following exception was thrown during creation and writing to the file \"{0}\"", path);
 469                        Console.WriteLine(e.Message);
 470                        return false;
 471                    }
 472                }
 473                else
 474                {
 475                    Console.WriteLine("Can not create file \"{0}\". File already exists", path);
 476                    return false;
 477
 478                }
 479            }
 480
 481            //Reading a file content using given path to a memory stream
 482            private bool ReadFile(MemoryStream OutputStream, string path)
 483            {
 484                try
 485                {
 486                    using (FileStream fs = File.Open(path, FileMode.Open))
 487                    {
 488                        fs.CopyTo(OutputStream);
 489                    }
 490                    return true;
 491                }
 492                catch (Exception e)
 493                {
 494                    Console.WriteLine("The following exception was thrown during reading from the file \"{0}\"", path);
 495                    Console.WriteLine(e.Message);
 496                    return false;
 497                }
 498            }
 499
 500            //Deleting file using given path
 501            private void RemoveFile(string path)
 502            {
 503                if (File.Exists(path))
 504                {
 505                    try
 506                    {
 507                        File.Delete(path);
 508                    }
 509                    catch (Exception e)
 510                    {
 511                        Console.WriteLine("The following exception was thrown during deleting the file \"{0}\"", path);
 512                        Console.WriteLine(e.Message);
 513                    }
 514                }
 515                else
 516                {
 517                    Console.WriteLine("Can not delete file \"{0}\". File does not exist", path);
 518                }
 519            }
 520
 521            static void Main()
 522            {
 523
 524                string sectionDivider = "---------------------------------------- ";
 525
 526                //Create an instance of AvroSample Class and invoke methods
 527                //illustrating different serializing approaches
 528                AvroSample Sample = new AvroSample();
 529
 530                //Serialization using Reflection to Avro Object Container File
 531                Sample.SerializeDeserializeUsingObjectContainersReflection();
 532
 533                Console.WriteLine(sectionDivider);
 534                Console.WriteLine("Press any key to exit.");
 535                Console.Read();
 536            }
 537        }
 538    }
 539    // The example is expected to display the following output:
 540    // SERIALIZATION USING REFLECTION AND AVRO OBJECT CONTAINER FILES
 541    //
 542    // Serializing Sample Data Set...
 543    // Saving serialized data to file...
 544    // Reading data from file...
 545    // Deserializing Sample Data Set...
 546    // Comparing Initial and Deserialized Data Sets...
 547    // For Pair 1 result of Data Set Identity Comparison is True
 548    // For Pair 2 result of Data Set Identity Comparison is True
 549    // ----------------------------------------
 550    // Press any key to exit.
 551  
 552
 553<h2> <a name="Scenario4"></a>Serialization using object container files and serialization with generic record</h2>
 554
 555This example is similar to scenario in the <a href="#Scenario2"> second example</a> where the schema is explicitly specified with JSON, except that here the schema is not assumed to be known to the reader that deserializes it. 
 556
 557The test data set is collected into a list of [**AvroRecord**](http://msdn.microsoft.com/en-us/library/microsoft.hadoop.avro.avrorecord.aspx) objects using an explicitly defined JSON schema and then stored in an object container file represented by the [**AvroContainer**](http://msdn.microsoft.com/en-us/library/microsoft.hadoop.avro.container.avrocontainer.aspx) class. This container file creates a writer that is used to serialize the data, uncompressed, to a memory stream that is then saved to a file. It is the [**Codex.Null**](http://msdn.microsoft.com/en-us/library/microsoft.hadoop.avro.container.codec.null.aspx) parameter used when creating the reader that specifies this data will not be compressed. 
 558
 559The data is then read from the file and deserialized into a collection of objects. This collection is compared to the initial list of Avro records to confirm that they are identical.
 560
 561
 562    namespace Microsoft.Hadoop.Avro.Sample
 563    {
 564        using System;
 565        using System.Collections.Generic;
 566        using System.IO;
 567        using System.Linq;
 568        using System.Runtime.Serialization;
 569        using Microsoft.Hadoop.Avro.Container;
 570        using Microsoft.Hadoop.Avro.Schema;
 571
 572        //This class contains all methods demonstrating
 573        //the usage of Microsoft .NET Library for Avro
 574        public class AvroSample
 575        {
 576
 577            //Serializes and deserializes sample data set using Generic Record and Avro Object Container Files
 578            //Serialized data is not compressed
 579            public void SerializeDeserializeUsingObjectContainersGenericRecord()
 580            {
 581                Console.WriteLine("SERIALIZATION USING GENERIC RECORD AND AVRO OBJECT CONTAINER FILES\n");
 582
 583                //Path for Avro Object Container File
 584                string path = "AvroSampleGenericRecordNullCodec.avro";
 585
 586                Console.WriteLine("Defining the Schema and creating Sample Data Set...");
 587
 588                //Define the schema in JSON
 589                const string Schema = @"{
 590                                ""type"":""record"",
 591                                ""name"":""Microsoft.Hadoop.Avro.Specifications.SensorData"",
 592                                ""fields"":
 593                                    [
 594                                        { 
 595                                            ""name"":""Location"", 
 596                                            ""type"":
 597                                                {
 598                                                    ""type"":""record"",
 599                                                    ""name"":""Microsoft.Hadoop.Avro.Specifications.Location"",
 600                                                    ""fields"":
 601                                                        [
 602                                                            { ""name"":""Floor"", ""type"":""int"" },
 603                                                            { ""name"":""Room"", ""type"":""int"" }
 604                                                        ]
 605                                                }
 606                                        },
 607                                        { ""name"":""Value"", ""type"":""bytes"" }
 608                                    ]
 609                            }";
 610
 611                //Create a generic serializer based on the schema
 612                var serializer = AvroSerializer.CreateGeneric(Schema);
 613                var rootSchema = serializer.WriterSchema as RecordSchema;
 614
 615                //Create a generic record to represent the data
 616                var testData = new List<AvroRecord>();
 617
 618                dynamic expected1 = new AvroRecord(rootSchema);
 619                dynamic location1 = new AvroRecord(rootSchema.GetField("Location").TypeSchema);
 620                location1.Floor = 1;
 621                location1.Room = 243;
 622                expected1.Location = location1;
 623                expected1.Value = new byte[] { 1, 2, 3, 4, 5 };
 624                testData.Add(expected1);
 625
 626                dynamic expected2 = new AvroRecord(rootSchema);
 627                dynamic location2 = new AvroRecord(rootSchema.GetField("Location").TypeSchema);
 628                location2.Floor = 1;
 629                location2.Room = 244;
 630                expected2.Location = location2;
 631                expected2.Value = new byte[] { 6, 7, 8, 9 };
 632                testData.Add(expected2);
 633
 634                //Serializing and saving data to file
 635                //Create a MemoryStream buffer
 636                using (var buffer = new MemoryStream())
 637                {
 638                    Console.WriteLine("Serializing Sample Data Set...");
 639
 640                    //Create a SequentialWriter instance for type SensorData which can serialize a sequence of SensorData objects to stream
 641                    //Data will not be compressed (Null compression codec)
 642                    using (var writer = AvroContainer.CreateGenericWriter(Schema, buffer, Codec.Null))
 643                    {
 644                        using (var streamWriter = new SequentialWriter<object>(writer, 24))
 645                        {
 646                            // Serialize the data to stream using the sequential writer
 647                            testData.ForEach(streamWriter.Write);
 648                        }
 649                    }
 650
 651                    Console.WriteLine("Saving serialized data to file...");
 652
 653                    //Save stream to file
 654                    if (!WriteFile(buffer, path))
 655                    {
 656                        Console.WriteLine("Error during file operation. Quitting method");
 657                        return;
 658                    }
 659                }
 660
 661                //Reading and deserializng the data
 662                //Create a Memory Stream buffer
 663                using (var buffer = new MemoryStream())
 664                {
 665                    Console.WriteLine("Reading data from file...");
 666
 667                    //Reading data from Object Container File
 668                    if (!ReadFile(buffer, path))
 669                    {
 670                        Console.WriteLine("Error during file operation. Quitting method");
 671                        return;
 672                    }
 673
 674                    Console.WriteLine("Deserializing Sample Data Set...");
 675
 676                    //Prepare the stream for deserializing the data
 677                    buffer.Seek(0, SeekOrigin.Begin);
 678
 679                    //Create a SequentialReader for type SensorData which will derserialize all serialized objects from the given stream
 680                    //It allows iterating over the deserialized objects because it implements IEnumerable<T> interface
 681                    using (var reader = AvroContainer.CreateGenericReader(buffer))
 682                    {
 683                        using (var streamReader = new SequentialReader<object>(reader))
 684                        {
 685                            var results = streamReader.Objects;
 686
 687                            Console.WriteLine("Comparing Initial and Deserialized Data Sets...");
 688
 689                            //Finally, verify the results
 690                            var pairs = testData.Zip(results, (serialized, deserialized) => new { expected = (dynamic)serialized, actual = (dynamic)deserialized });
 691                            int count = 1;
 692                            foreach (var pair in pairs)
 693                            {
 694                                bool isEqual = pair.expected.Location.Floor.Equals(pair.actual.Location.Floor);
 695                                isEqual = isEqual && pair.expected.Location.Room.Equals(pair.actual.Location.Room);
 696                                isEqual = isEqual && ((byte[])pair.expected.Value).SequenceEqual((byte[])pair.actual.Value);
 697                                Console.WriteLine("For Pair {0} result of Data Set Identity Comparison is {1}", count, isEqual.ToString());
 698                                count++;
 699                            }
 700                        }
 701                    }
 702                }
 703
 704                //Delete the file
 705                RemoveFile(path);
 706            }
 707
 708            //
 709            //Helper methods
 710            //
 711
 712            //Saving memory stream to a new file with the given path
 713            private bool WriteFile(MemoryStream InputStream, string path)
 714            {
 715                if (!File.Exists(path))
 716                {
 717                    try
 718                    {
 719                        using (FileStream fs = File.Create(path))
 720                        {
 721                            InputStream.Seek(0, SeekOrigin.Begin);
 722                            InputStream.CopyTo(fs);
 723                        }
 724                        return true;
 725                    }
 726                    catch (Exception e)
 727                    {
 728                        Console.WriteLine("The following exception was thrown during creation and writing to the file \"{0}\"", path);
 729                        Console.WriteLine(e.Message);
 730                        return false;
 731                    }
 732                }
 733                else
 734                {
 735                    Console.WriteLine("Can not create file \"{0}\". File already exists", path);
 736                    return false;
 737
 738                }
 739            }
 740
 741            //Reading a file content using given path to a memory stream
 742            private bool ReadFile(MemoryStream OutputStream, string path)
 743            {
 744                try
 745                {
 746                    using (FileStream fs = File.Open(path, FileMode.Open))
 747                    {
 748                        fs.CopyTo(OutputStream);
 749                    }
 750                    return true;
 751                }
 752                catch (Exception e)
 753                {
 754                    Console.WriteLine("The following exception was thrown during reading from the file \"{0}\"", path);
 755                    Console.WriteLine(e.Message);
 756                    return false;
 757                }
 758            }
 759
 760            //Deleting file using given path
 761            private void RemoveFile(string path)
 762            {
 763                if (File.Exists(path))
 764                {
 765                    try
 766                    {
 767                        File.Delete(path);
 768                    }
 769                    catch (Exception e)
 770                    {
 771                        Console.WriteLine("The following exception was thrown during deleting the file \"{0}\"", path);
 772                        Console.WriteLine(e.Message);
 773                    }
 774                }
 775                else
 776                {
 777                    Console.WriteLine("Can not delete file \"{0}\". File does not exist", path);
 778                }
 779            }
 780
 781            static void Main()
 782            {
 783
 784                string sectionDivider = "---------------------------------------- ";
 785
 786                //Create an instance of AvroSample Class and invoke methods
 787                //illustrating different serializing approaches
 788                AvroSample Sample = new AvroSample();
 789
 790                //Serialization using Generic Record to Avro Object Container File
 791                Sample.SerializeDeserializeUsingObjectContainersGenericRecord();
 792
 793                Console.WriteLine(sectionDivider);
 794                Console.WriteLine("Press any key to exit.");
 795                Console.Read();
 796            }
 797        }
 798    }
 799    // The example is expected to display the following output:
 800    // SERIALIZATION USING GENERIC RECORD AND AVRO OBJECT CONTAINER FILES
 801    //
 802    // Defining the Schema and creating Sample Data Set...
 803    // Serializing Sample Data Set...
 804    // Saving serialized data to file...
 805    // Reading data from file...
 806    // Deserializing Sample Data Set...
 807    // Comparing Initial and Deserialized Data Sets...
 808    // For Pair 1 result of Data Set Identity Comparison is True
 809    // For Pair 2 result of Data Set Identity Comparison is True
 810    // ----------------------------------------
 811    // Press any key to exit.
 812
 813
 814<h2> <a name="Scenario5"></a>Serialization using object container files with a custom compression codec</h2>
 815
 816The example below shows how to use a custom compression codec for Avro object container files. The [Avro Specification](http://avro.apache.org/docs/current/spec.html#Required+Codecs) allows usage of an optional compression codec (in addition to **Null** and **Deflate** defaults). This example is not implementing completely new codec such Snappy (mentioned as a supported optional codec in [Avro Specification](http://avro.apache.org/docs/current/spec.html#snappy)). It shows how to use the .NET Framework 4.5  implementation of the [**Deflate**][deflate-110] codec which provides a better compression algorithm based on the [zlib](http://zlib.net/) compression library than the default .NET Framework 4.0 version.
 817
 818
 819    // 
 820    // This code needs to be compiled with the parameter Target Framework set as ".NET Framework 4.5"
 821    // to ensure the desired implementation of Deflate compression algorithm is used
 822    // Ensure your C# Project is set up accordingly
 823    //
 824
 825    namespace Microsoft.Hadoop.Avro.Sample
 826    {
 827        using System;
 828        using System.Collections.Generic;
 829        using System.Diagnostics;
 830        using System.IO;
 831        using System.IO.Compression;
 832        using System.Linq;
 833        using System.Runtime.Serialization;
 834        using Microsoft.Hadoop.Avro.Container;
 835
 836        #region Defining objects for serialization
 837        //Sample Class used in serialization samples
 838        [DataContract(Name = "SensorDataValue", Namespace = "Sensors")]
 839        internal class SensorData
 840        {
 841            [DataMember(Name = "Location")]
 842            public Location Position { get; set; }
 843
 844            [DataMember(Name = "Value")]
 845            public byte[] Value { get; set; }
 846        }
 847
 848        //Sample struct used in serialization samples
 849        [DataContract]
 850        internal struct Location
 851        {
 852            [DataMember]
 853            public int Floor { get; set; }
 854
 855            [DataMember]
 856            public int Room { get; set; }
 857        }
 858        #endregion
 859
 860        #region Defining custom codec based on .NET Framework V.4.5 Deflate
 861        //Avro.NET Codec class contains two methods 
 862        //GetCompressedStreamOver(Stream uncompressed) and GetDecompressedStreamOver(Stream compressed)
 863        //which are the key ones for data compression.
 864        //To enable a custom codec one needs to implement these methods for the required codec
 865
 866        #region Defining Compression and Decompression Streams
 867        //DeflateStream (class from System.IO.Compression namespace that implements Deflate algorithm)
 868        //can not be directly used for Avro because it does not support vital operations like Seek.
 869        //Thus one needs to implement two classes inherited from Stream
 870        //(one for compressed and one for decompressed stream)
 871        //that use Deflate compression and implement all required features 
 872        internal sealed class CompressionStreamDeflate45 : Stream
 873        {
 874            private readonly Stream buffer;
 875            private DeflateStream compressionStream;
 876
 877            public CompressionStreamDeflate45(Stream buffer)
 878            {
 879                Debug.Assert(buffer != null, "Buffer is not allowed to be null.");
 880
 881                this.compressionStream = new DeflateStream(buffer, CompressionLevel.Fastest, true);
 882                this.buffer = buffer;
 883            }
 884
 885            public override bool CanRead
 886            {
 887                get { return this.buffer.CanRead; }
 888            }
 889
 890            public override bool CanSeek
 891            {
 892                get { return true; }
 893            }
 894
 895            public override bool CanWrite
 896            {
 897                get { return this.buffer.CanWrite; }
 898            }
 899
 900            public override void Flush()
 901            {
 902                this.compressionStream.Close();
 903            }
 904
 905            public override long Length
 906            {
 907                get { return this.buffer.Length; }
 908            }
 909
 910            public override long Position
 911            {
 912                get
 913                {
 914                    return this.buffer.Position;
 915                }
 916
 917                set
 918                {
 919                    this.buffer.Position = value;
 920                }
 921            }
 922
 923            public override int Read(byte[] buffer, int offset, int count)
 924            {
 925                return this.buffer.Read(buffer, offset, count);
 926            }
 927
 928            public override long Seek(long offset, SeekOrigin origin)
 929            {
 930                return this.buffer.Seek(offset, origin);
 931            }
 932
 933            public override void SetLength(long value)
 934            {
 935                throw new NotSupportedException();
 936            }
 937
 938            public override void Write(byte[] buffer, int offset, int count)
 939            {
 940                this.compressionStream.Write(buffer, offset, count);
 941            }
 942
 943            protected override void Dispose(bool disposed)
 944            {
 945                base.Dispose(disposed);
 946
 947                if (disposed)
 948                {
 949                    this.compressionStream.Dispose();
 950                    this.compressionStream = null;
 951                }
 952            }
 953        }
 954
 955        internal sealed class DecompressionStreamDeflate45 : Stream
 956        {
 957            private readonly DeflateStream decompressed;
 958
 959            public DecompressionStreamDeflate45(Stream compressed)
 960            {
 961                this.decompressed = new DeflateStream(compressed, CompressionMode.Decompress, true);
 962            }
 963
 964            public override bool CanRead
 965            {
 966                get { return true; }
 967            }
 968
 969            public override bool CanSeek
 970            {
 971                get { return true; }
 972            }
 973
 974            public override bool CanWrite
 975            {
 976                get { return false; }
 977            }
 978
 979            public override void Flush()
 980            {
 981                this.decompressed.Close();
 982            }
 983
 984            public override long Length
 985            {
 986                get { return this.decompressed.Length; }
 987            }
 988
 989            public override long Position
 990            {
 991                get
 992                {
 993                    return this.decompressed.Position;
 994                }
 995
 996                set
 997                {
 998                    throw new NotSupportedException();
 999                }
1000            }
1001
1002            public override int Read(byte[] buffer, int offset, int count)
1003            {
1004                return this.decompressed.Read(buffer, offset, count);
1005            }
1006
1007            public override long Seek(long offset, SeekOrigin origin)
1008            {
1009                throw new NotSupportedException();
1010            }
1011
1012            public override void SetLength(long value)
1013            {
1014                throw new NotSupportedException();
1015            }
1016
1017            public override void Write(byte[] buffer, int offset, int count)
1018            {
1019                throw new NotSupportedException();
1020            }
1021
1022            protected override void Dispose(bool disposing)
1023            {
1024                base.Dispose(disposing);
1025
1026                if (disposing)
1027                {
1028                    this.decompressed.Dispose();
1029                }
1030            }
1031        }
1032        #endregion
1033
1034        #region Define Codec
1035        //Define the actual codec class containing the required methods for manipulating streams:
1036        //GetCompressedStreamOver(Stream uncompressed) and GetDecompressedStreamOver(Stream compressed)
1037        //Codec class uses classes for comressed and decompressed streams defined above
1038        internal sealed class DeflateCodec45 : Codec
1039        {
1040
1041            //We merely use different IMPLEMENTATION of Deflate, so the CodecName remains "deflate"
1042            public static readonly string CodecName = "deflate";
1043
1044            public DeflateCodec45()
1045                : base(CodecName)
1046            {
1047            }
1048
1049            public override Stream GetCompressedStreamOver(Stream decompressed)
1050            {
1051                if (decompressed == null)
1052                {
1053                    throw new ArgumentNullException("decompressed");
1054                }
1055
1056                return new CompressionStreamDeflate45(decompressed);
1057            }
1058
1059            public override Stream GetDecompressedStreamOver(Stream compressed)
1060            {
1061                if (compressed == null)
1062                {
1063                    throw new ArgumentNullException("compressed");
1064                }
1065
1066                return new DecompressionStreamDeflate45(compressed);
1067            }
1068        }
1069        #endregion
1070
1071        #region Define modified Codec Factory
1072        //Define modified Codec Factory to be used in Reader
1073        //It will catch the attempt to use "deflate" and provide Custom Codec 
1074        //For all other cases it will rely on the base class (CodecFactory)
1075        internal sealed class CodecFactoryDeflate45 : CodecFactory
1076        {
1077
1078            public override Codec Create(string codecName)
1079            {
1080                if (codecName == DeflateCodec45.CodecName)
1081                    return new DeflateCodec45();
1082                else
1083                    return base.Create(codecName);
1084            }
1085        }
1086        #endregion
1087
1088        #endregion
1089
1090        #region Sample Class with demonstration methods
1091        //This class contains methods demonstrating
1092        //the usage of Microsoft .NET Library for Avro
1093        public class AvroSample
1094        {
1095
1096            //Serializes and deserializes sample data set using Reflection and Avro Object Container Files
1097            //Serialized data is compressed with the Custom compression codec (Deflate of .NET Framework 4.5)
1098            //
1099            //This sample uses Memory Stream for all operations related to serialization, deserialization and
1100            //Object Container manipulation, though File Stream could be easily used.
1101            public void SerializeDeserializeUsingObjectContainersReflectionCustomCodec()
1102            {
1103
1104                Console.WriteLine("SERIALIZATION USING REFLECTION, AVRO OBJECT CONTAINER FILES AND CUSTOM CODEC\n");
1105
1106                //Path for Avro Object Container File
1107                string path = "AvroSampleReflectionDeflate45.avro";
1108
1109                //Create a data set using sample Class and struct
1110                var testData = new List<SensorData>
1111                        {
1112                            new SensorData { Value = new byte[] { 1, 2, 3, 4, 5 }, Position = new Location { Room = 243, Floor = 1 } },
1113                            new SensorData { Value = new byte[] { 6, 7, 8, 9 }, Position = new Location { Room = 244, Floor = 1 } }
1114                        };
1115
1116                //Serializing and saving data to file
1117                //Creating a Memory Stream buffer
1118                using (var buffer = new MemoryStream())
1119                {
1120                    Console.WriteLine("Serializing Sample Data Set...");
1121
1122                    //Create a SequentialWriter instance for type SensorData which can serialize a sequence of SensorData objects to stream
1123                    //Here the custom Codec is introduced. For convenience the next commented code line shows how to use built-in Deflate.
1124                    //Note, that because the sample deals with different IMPLEMENTATIONS of Deflate, built-in and custom codecs are interchangeable
1125                    //in read-write operations
1126                    //using (var w = AvroContainer.CreateWriter<SensorData>(buffer, Codec.Deflate))
1127                    using (var w = AvroContainer.CreateWriter<SensorData>(buffer, new DeflateCodec45()))
1128                    {
1129                        using (var writer = new SequentialWriter<SensorData>(w, 24))
1130                        {
1131                            // Serialize the data to stream using the sequential writer
1132                            testData.ForEach(writer.Write);
1133                        }
1134                    }
1135
1136                    //Save stream to file
1137                    Console.WriteLine("Saving serialized data to file...");
1138                    if (!WriteFile(buffer, path))
1139                    {
1140                        Console.WriteLine("Error during file operation. Quitting method");
1141                        return;
1142                    }
1143                }
1144
1145                //Reading and deserializing data
1146                //Creating a Memory Stream buffer
1147                using (var buffer = new MemoryStream())
1148                {
1149                    Console.WriteLine("Reading data from file...");
1150
1151                    //Reading data from Object Container File
1152                    if (!ReadFile(buffer, path))
1153                    {
1154                        Console.WriteLine("Error during file operation. Quitting method");
1155                        return;
1156                    }
1157
1158                    Console.WriteLine("Deserializing Sample Data Set...");
1159
1160                    //Prepare the stream for deserializing the data
1161                    buffer.Seek(0, SeekOrigin.Begin);
1162
1163                    //Because of SequentialReader<T> constructor signature an AvroSerializerSettings instance is required
1164                    //when Codec Factory is explicitly specified
1165                    //You may comment the line below if you want to use built-in Deflate (see next comment)
1166                    AvroSerializerSettings settings = new AvroSerializerSettings();
1167
1168                    //Create a SequentialReader for type SensorData which will derserialize all serialized objects from the given stream
1169                    //It allows iterating over the deserialized objects because it implements IEnumerable<T> interface
1170                    //Here the custom Codec Factory is introduced.
1171                    //For convenience the next commented code line shows how to use built-in Deflate
1172                    //(no explicit Codec Factory parameter is required in this case).
1173                    //Note, that because the sample deals with different IMPLEMENTATIONS of Deflate, built-in and custom codecs are interchangeable
1174                    //in read-write operations
1175                    //using (var reader = new SequentialReader<SensorData>(AvroContainer.CreateReader<SensorData>(buffer, true)))
1176                    using (var reader = new SequentialReader<SensorData>(
1177                        AvroContainer.CreateReader<SensorData>(buffer, true, settings, new CodecFactoryDeflate45())))
1178                    {
1179                        var results = reader.Objects;
1180
1181                        //Finally, verify that deserialized data matches the original one
1182                        Console.WriteLine("Comparing Initial and Deserialized Data Sets...");
1183                        bool isEqual;
1184                        int count = 1;
1185                        var pairs = testData.Zip(results, (serialized, deserialized) => new { expected = serialized, actual = deserialized });
1186                        foreach (var pair in pairs)
1187                        {
1188                            isEqual = this.Equal(pair.expected, pair.actual);
1189                            Console.WriteLine("For Pair {0} result of Data Set Identity Comparison is {1}", count, isEqual.ToString());
1190                            count++;
1191                        }
1192                    }
1193                }
1194
1195                //Delete the file
1196                RemoveFile(path);
1197            }
1198        #endregion
1199
1200            #region Helper Methods
1201
1202            //Comparing two SensorData objects
1203            private bool Equal(SensorData left, SensorData right)
1204            {
1205                return left.Position.Equals(right.Position) && left.Value.SequenceEqual(right.Value);
1206            }
1207
1208            //Saving memory stream to a new file with the given path
1209            private bool WriteFile(MemoryStream InputStream, string path)
1210            {
1211                if (!File.Exists(path))
1212                {
1213                    try
1214                    {
1215                        using (FileStream fs = File.Create(path))
1216                        {
1217                            InputStream.Seek(0, SeekOrigin.Begin);
1218                            InputStream.CopyTo(fs);
1219                        }
1220                        return true;
1221                    }
1222                    catch (Exception e)
1223                    {
1224                        Console.WriteLine("The following exception was thrown during creation and writing to the file \"{0}\"", path);
1225                        Console.WriteLine(e.Message);
1226                        return false;
1227                    }
1228                }
1229                else
1230                {
1231                    Console.WriteLine("Can not create file \"{0}\". File already exists", path);
1232                    return false;
1233
1234                }
1235            }
1236
1237            //Reading a file content using given path to a memory stream
1238            private bool ReadFile(MemoryStream OutputStream, string path)
1239            {
1240                try
1241                {
1242                    using (FileStream fs = File.Open(path, FileMode.Open))
1243                    {
1244                        fs.CopyTo(OutputStream);
1245                    }
1246                    return true;
1247                }
1248                catch (Exception e)
1249                {
1250                    Console.WriteLine("The following exception was thrown during reading from the file \"{0}\"", path);
1251                    Console.WriteLine(e.Message);
1252                    return false;
1253                }
1254            }
1255
1256            //Deleting file using given path
1257            private void RemoveFile(string path)
1258            {
1259                if (File.Exists(path))
1260                {
1261                    try
1262                    {
1263                        File.Delete(path);
1264                    }
1265                    catch (Exception e)
1266                    {
1267                        Console.WriteLine("The following exception was thrown during deleting the file \"{0}\"", path);
1268                        Console.WriteLine(e.Message);
1269                    }
1270                }
1271                else
1272                {
1273                    Console.WriteLine("Can not delete file \"{0}\". File does not exist", path);
1274                }
1275            }
1276            #endregion
1277
1278            static void Main()
1279            {
1280
1281                string sectionDivider = "---------------------------------------- ";
1282
1283                //Create an instance of AvroSample Class and invoke methods
1284                //illustrating different serializing approaches
1285                AvroSample Sample = new AvroSample();
1286
1287                //Serialization using Reflection to Avro Object Container File using Custom Codec
1288                Sample.SerializeDeserializeUsingObjectContainersReflectionCustomCodec();
1289
1290                Console.WriteLine(sectionDivider);
1291                Console.WriteLine("Press any key to exit.");
1292                Console.Read();
1293            }
1294        }
1295    }
1296    // The example is expected to display the following output:
1297    // SERIALIZATION USING REFLECTION, AVRO OBJECT CONTAINER FILES AND CUSTOM CODEC
1298    //
1299    // Serializing Sample Data Set...
1300    // Saving serialized data to file...
1301    // Reading data from file...
1302    // Deserializing Sample Data Set...
1303    // Comparing Initial and Deserialized Data Sets...
1304    // For Pair 1 result of Data Set Identity Comparison is True
1305    //For Pair 2 result of Data Set Identity Comparison is True
1306    // ----------------------------------------
1307    // Press any key to exit.
1308
1309
1310
1311[deflate-100]: http://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.100).aspx
1312[deflate-110]: http://msdn.microsoft.com/en-us/library/system.io.compression.deflatestream(v=vs.110).aspx
1313
1314