Video Generation Over the Cloud
 
 
Our Services:
Cloud Development
Amazon Web Services Google App Engine
Development Services
Flash/Flex C++/Win32/ActiveX DirectShow Java PHP
 
 
Contact Us
Objective

Madeyoum Ltd. is looking to change the way people and businesses generate video content and marketing materials.

Madeyoum required a cloud based architecture and implementation for their innovative, patent-pending video generation technology, in the quest to create an online video generation platform.

Such an architecture, needs to support robust and scalable web-services, manage complex video processing flows, stream video out of a CDN and handle a mixed solution of on-cloud machines and dedicated servers.

Challenges

Madeyoum’s online technology is a complex architecture of web-servers, video processing and conversion machines, video generation algorithms and video streaming capabilities.
Its unique technology combination requires mixed usage of Linux and Windows processing machines and support for both on-cloud machines and dedicated servers.

The main challenge faced by the company was to create a scalable and robust implementation for its video processing flow technology, capable of handling variable loads.
The solution needed to allow Madeyoum to generate different kinds of videos and serve users out of different web domains while utilizing the same service infrastructure.

Solution

Madeyoum LTD partnered with Wiseman Software to lead the architecture, design and implementation of its cloud infrastructure.

Following a detailed analysis of Madeyoum’s infrastructure requirements and research of several cloud solutions, Wiseman Software recommended using AWS (Amazon Web Services) cloud infrastructure for the implementation of Madeyoum’s web-service and video processing system.

AWS were chosen because of their cloud infrastructure maturity; scaling capabilities and the ability to implement a mixed cloud and dedicated servers system and the extensive support for video hosting and streaming.

The technology chosen for the infrastructure was Java™ based Spring Framework 3.0 and Hibernate® over Tomcat deployment.

The complete implementation solution was crafted based upon Wiseman Software’s Cloud-First development methodology.

Based on the key elements of the methodology, AWS cloud infrastructure services were chosen wherever appropriate and fault tolerance plan was designed.

 

Click to Enlarge
Figure 1 – Schematic architecture and design of Madeyoum’s Video Generation web-service over AWS cloud services infrastructure
Webservers and Database

The main concern regarding the web front-end servers and database was to provide a scalable and robust solution, capable of serving hundreds to thousands of concurrent video creation activities.
In order to utilize computational resources to the maximum, a single video processing flow system was implemented, capable of generating different types of videos.

AWS Elastic Beanstalk Tomcat deployment platform was chosen as the service front-end for its ease of operation and built-in scalability, monitoring, notifications and health check capabilities.
Each of the web domains owned by Madeyoum, such as EstateVideoOnline.com targeting the real-estate virtual tours market, was deployed on a seperate beanstalk application stack.
The web application was built using Java™ Spring Framework 3.0 MVC platform and Hibernate 3.0 ORM infrastructure, harnessing the power of AWS ElastiCache memcached based solution, as its session caching system and Hibernate second level cache.

The main system database resides on AWS RDS MySQL deployment.
The database is configured for robustness and fault tolerance, as a multi-zone deployment, and designed for performance boosting using Read Replica servers.

Video Generation Processing

Madeyoum’s video generation algorithm flow is composed of several processing steps.
The different steps include raw video and image conversion processing, video generation processing and output video conversion processing.

One of the challenges in this project was to develop the processing flow infrastructure.
The first version of the infrastructure was implemented based on AWS SQS queueing system and AWS SimpleDB.
Multiple processing machines, cloud based and dedicated, were listening on the queue, waiting for new jobs to arrive and updating SimpleDB with the processing jobs status.
Once a processor finished processing a job out of the queue, it updated the job state in the SimpleDB repository and sent AWS SNS notification for further job handling.

In order to prevent excessive polling actions over the SQS jobs queues, a solution combining SQS messages and SNS notifications was implemented.

 

Click to Enlarge
Figure 2 – Coordinating the processing activities by combining SQS and SNS. Processing information is stored in SimpleDB tables

The second version of the video processing flow system was implemented using Amazon Simple Workflow Service (SWF) for greater simplicity.
The new infrastructure includes EC2 cloud machines and dedicated servers deployed with Java™ Flow framework activities and workflow implementations.
Additional monitoring and maintenance jobs and workflow implementations were added using the SWF infrastructure, allowing a more robust and fault tolerant service.

 

Click to Enlarge
Figure 3 – Coordinating processing activities and system maintenance using AWS SWF
Broadcasting and Streaming Videos

Madeyoum offers users the option to play and stream their generated videos to a vast selection of devices and available media players.
This capability is feasible due to the combined usage of AWS S3 for video storage with AWS CloudFront for progressive download and RTMP streaming.

 

Click to Enlarge
Figure 4 – Using the full strength of CloudFront CDN for RTMP streaming to Flash players & Progressive Download
Logging System

Distributed system logging is hard to maintain and manage. Collecting logs from different machines and displaying a unified and complete log picture is hard and tedious.

The system combines AWS SimpleDB as its logging container and log4j Java™ library as its logging facility, using a customized log4j appender.

Madeyoum’s Admin system offers visual display of current processing flow system’s status and allows deep and extensive log querying.
These vital and important admin and monitoring features were easily implemented using AWS SDK and AWS SimpleDB query language, based on the logs being continually collected on SimpleDB repositories.

Results

Madeyoum launched its first release for real-estate virtual tours video creation successfully and on time.
As evidence to its sophisticated and smart cloud implementation,
Madeyoum’s innovative platform and unique AWS cloud infrastructure was nominated among the leading 100 startups of the 2011 AWS Challenge.

About Wiseman Software

Wiseman Software is a boutique software solution provider, specializing in software architecture, complex web application development and cloud computing implementations.
Wiseman Software provides cloud development and consulting services to software companies looking to implement their technology over the cloud.
Whether it is a complex, high availability cloud infrastructure solution or a smaller cloud migration or development project,
Wiseman Software delivers software projects on time, supporting clients in their time-to-market considerations.

About Madeyoum

Madeyoum Ltd. revolutionizes the way people and businesses create video content, unleashing the power of video commercial creation to everyone.
Madeyoum’s proprietary patent pending technology, Dynamic Storyboard, offers users the ability to self-create professional video content with a unique story and message,
using images and videos taken from any available device (such as digital cameras, camcorders, mobile phones cameras, etc.) within minutes,
Madeyoum operates a chain of market oriented websites over AWS infrastructure – such as EstateVideoOnline.com for real-estate video virtual tours creation.

* Oracle, Java, J2EE, J2ME and MySQL are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

* Amazon Web Services, the "Powered by Amazon Web Services" logo, AWS, Amazon Elastic Beanstalk, Amazon Route53, Amazon CloudFront, Amazon S3, Amazon RDS, Amazon SWF, Amazon SimpleDB, Amazon SQS, Amazon SNS, Amazon Elasticache, Amazon EC2, are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries.

* Youtube, Google App Engine and Google Drive are trademarks or distinctive brand features of Google Inc.

* Apache ActiveMQ, ActiveMQ, Apache, Tomcat, Apache Tomcat are trademarks of The Apache Software Foundation.

* IBM and WebSphere are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide.

* SpringSource, Cloud Foundry, RabbitMQ are registered trademarks and/or trademarks of VMware, Inc. in the United States and/or other jurisdictions.

* Hibernate is a registered trademark of Red Hat, Inc.

* Adobe, the Adobe logo, Adobe Flash, Adobe Flex, Adobe Flash Builder, Adobe Flash, Adobe Flash Professional, Adobe AIR, Adobe Media Server are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States and/or other countries.

* Zend, Zend Technologies, Zend Framework, Zend.com, Zend Server, and Zend Studio, and associated logos and icons, are trademarks of Zend Technologies, Inc. and may be registered in certain jurisdictions.

* Microsoft, Windows, Visual C#, and Visual C++, DirectShow, Microsoft .NET, MFC, ActiveX, COM, ATL, Direct3D, DirectX, Win32, Internet Explorer, IIS, MSSQL, ASP, ASP.Net, GDI, GDI+, WMV and ActiveDirectory are either registered trademarks or trademarks of Microsoft Corporation in the U.S.A. and/or other countries/regions.

* All trademarks and registered trademarks appearing on wisemansoftware.com are the property of their respective owners.