AI-Powered Software Developer: A Comprehensive Guide to Boosting Productivity with Generative AI

 

Disclaimer: This content, including the title and image, was co-authored with Generative AI to enhance productivity, accelerate content creation and has been reviewed and refined by humans.

 

 

After completing my course on Generative AI Specialization, I was eager to put my newly acquired skills to the test. Utilizing my class notes as data, I applied prompt engineering techniques and leveraged tools/ platforms like Watsonx.ai, Hugging Face, and Google AI Studio to create this blog post. My focus is on how software developers can leverage Generative AI tools to boost productivity; I've experienced a remarkable 60-70% increase in efficiency while writing this blog.

Generative AI (Gen AI) enables software developers to enhance their daily workflows, improve efficiency, reduce errors, and free up time for more complex and creative tasks. By automating repetitive tasks, developers can significantly decrease manual efforts and accelerate development cycles.

Here's how Generative AI can augment a software developer's daily tasks:

• Reviewing Emails and Slack Channel Discussions
  Summarization: Generative AI models, Large Language Models (LLMs), excel at summarizing large amounts of content. Developers can utilize Gen AI tools to quickly summarize lengthy email threads or extensive Slack discussions, generating concise summaries that highlight key decisions and action items without needing to read every message. 
  Information Extraction: Gen AI can analyze communication to identify patterns and trends, facilitating the rapid extraction of valuable insights from large volumes of correspondence. Aiding in recognizing urgent issues and recurring themes in team discussions or customer feedback.
• Understanding New Feature Requirements
  Idea Generation and Brainstorming: When faced with new feature requirements, generative AI can serve as a content generator, suggesting algorithms, data structures, and approaches to solve specific problems, thereby accelerating the initial conceptualization phase of a project. 
  Natural Language Processing (NLP): Powered by deep learning, Gen AI models can comprehend the context of natural language input and generate appropriate responses, simplifying the parsing of complex or ambiguous feature requests.
• Interpreting Complex Technical Architecture Documents
  Summarization and Question Answering: Gen AI tools like Box AI can summarize documents and answer questions based on those sources, enabling developers to grasp key points or clarify architectural decisions quickly. This understanding facilitates the identification of dependencies and potential bottlenecks while also grasping the overarching design philosophy.
• Creating High-Level and Low-Level Design Documents
  Content and Outline Generation: Generative AI, particularly text-to-text models, can produce various text formats, including outlines and content for design documents. Developers can utilize tools like Watsonx.ai, ChatGPT, and Huggingface to generate suggestions for titles, content, and visuals, streamlining the drafting process. Tools like DALL-E and Midjourney can provide visual representations, although accuracy will depend on the precision of the prompt and the model's training data—an intriguing area for future development.
• Developing Code and Testing
  Code Generation: Tools like GitHub Copilot and Tab9 can create boilerplate code, complete code snippets, or generate new code from natural language prompts. IBM Watson Code Assistant offers real-time recommendations and autocomplete features, substantially reducing manual coding time.  
  Code Optimization and Refactoring: Code generators can not only produce optimized code but also suggest refactoring patterns that adhere to best practices, helping to modernize applications by identifying outdated components.  
  Debugging: Generative AI can assist in identifying and fixing simple bugs, as well as recognizing potential issues during code reviews. For instance, ChatGPT can locate and correct code errors and explain the necessary adjustments. Text-to-code tools can significantly reduce the time dedicated to debugging.  
  Testing: Generative models can synthesize test cases and data, simulating user behavior and enhancing software reliability and robustness. Tools like Diffblue Cover can generate extensive and effective test cases.  
  Documentation: AI tools can produce summaries and comments for code, boosting documentation and readability. Sphinx is capable of generating comprehensive documentation based on natural language descriptions.  
  Issue Reporting/Anomaly Detection: IBM Watson AIOps and Moogsoft AIOps analyze system logs and metrics for early issue detection and troubleshooting, facilitating proactive maintenance.
• Creating Technical Presentations and Transitioning Knowledge to the Team
  Content Creation for Presentations: Generative AI tools like ChatGPT can provide suggestions for presentation titles, content, and visuals, allowing developers to quickly draft technical presentations. 
  Learning and Explanatory Aids: Gen AI-based text generators can serve as educational tools, offering clear, step-by-step explanations of programming concepts. In knowledge transition sessions, AI can help generate detailed explanations on complex topics, aiding team members in grasping new systems more effectively.

Prompt Engineering for Developers To capitalize on these Gen AI capabilities, developers must cultivate their "prompt engineering" skills. This involves crafting clear and contextually appropriate prompts for AI models. Techniques like Chain-of-Thought (breaking down complex tasks) and Tree-of-Thought (structuring prompts hierarchically) can guide the model's reasoning process and yield more nuanced outputs. Tools like IBM Watsonx.ai Prompt Lab and others can assist in refining these prompts for optimal results.

Important Considerations for Developers: While the potential benefits of Gen AI are substantial, developers should proceed with caution:

• Use organization-approved tools to prevent exposure of sensitive corporate information that could inadvertently be included in model training data.
• Recognize that generated code may not always be flawless; models might not fully comprehend semantics, leaving room for errors, especially with complex code.
• Be aware of "hallucinations," where AI produces incorrect or nonsensical output. Acknowledge the ethical implications, including potential security vulnerabilities and biases in training data, which necessitate human oversight, thorough testing, and validation of AI-generated content.

Embrace Generative AI to unlock your developer superpowers!

Is software as we know changing dramatically? Welcome to the Agentic AI era.

Agentic AI is transforming traditional software development by replacing hardcoded control logic in applications. In conventional systems, business logic is typically coded by software developers and structured into a multi-tier architecture. This architecture consists of a front end that interacts with users, a middle tier that exposes business logic via APIs, and a backend where CRUD operations manage data storage. Other architectures, like microservices, break down an application into smaller, independent services, each focused on a specific business function. Let's limit this discussion to multi-tiered architecture to simplify our understanding of Agentic AI transforming the software business. (however, the same could be applied to other software architectures as well)

With the advent of Agentic AI, this traditional model is shifting. For instance, consider a scenario where a business needs to identify the top 10 suppliers in the country and display this information visually. In the past, a Python developer would spend an hour or two writing logic to query the database and generate a graph using libraries like Matplotlib.

In an Agentic AI world, the control logic is essentially eliminated. The developer only needs to provide a simple prompt, such as "Fetch me the top 10 suppliers in the country for XXX and show them in a bar graph." That's the entirety of the task.

If the requirement shifts to pulling supplier details for the top performer in each state segment, there's no need for code revision. Instead, the prompt is simply modified, and the underlying LLM or RAG handles the data processing. This approach eliminates the need for control logic changes, testing, and new releases, significantly saving time, resources, and effort.

This revolution in software development will redefine how developers approach their work. Companies will increasingly seek developers skilled in training purpose-built models, leading to the creation of lightweight applications. Furthermore, the reliance on specific vendors and APIs for backend and data management will diminish, fostering a more democratized development environment.

For anyone aspiring to enter software development, it's essential to adapt and refocus their skills by:

- Embracing the role of assistive developers who leverage AI tools for faster coding. For example - Watson Code Assistant

- Shifting their architectural mindset to build and design solutions using Agentic AI methodologies.

Also, as we see this shift happening very fast, current software developers must adapt to this new Agentic AI era that is taking the industry by storm.

Intelligent Document Processing with IBM ADP

 

Documents come in a variety of shapes and sizes, encompassing structured, semi-structured, and unstructured content. They may feature images as backgrounds or foregrounds, formatting elements like checkboxes, and various combinations aiming to enhance legibility. From printed and handwritten texts to scanned images and complex tables, the possibilities are endless. The challenge of processing such documents and converting their content into a digital format can be associated to gold mining in the digital realm. You might wonder how this process works.

Once a document is converted into a digital format, it becomes possible to establish data relationships, contextualize information, and operate more efficiently with minimal human intervention. This is particularly beneficial in business contexts, where processing bills and payments can be significantly expedited. For instance, imagine automatically extracting the "Total Amount Due" from an invoice and processing the payment if it falls below a certain threshold; if it exceeds that amount, it can be flagged for human review. It's clear that the value of Document Processing is substantial.

Let's explore the "Intelligence" component of document processing. The first step is to identify the type of document, followed by extracting content based on its classification. This process requires embedding intelligence within the system. Just as humans gradually develop an understanding of a document—classifying it and honing their processing skills—computer systems are engineered to enhance their comprehension and content extraction capabilities over time. This field is referred to as Intelligent Document Processing, and IBM Automation Document Processing (ADP) is a standout product that excels in this area.

To comprehend how such systems are constructed, let's break it down from the top. Documents are classified based on their basic structure: they can be categorized as unstructured text, semi-structured (such as invoices or phone bills containing tables), or structured (like paragraphs in an article). The classification process involves making intelligent decisions based on the quantity of information elements present in the document. Once classified, Optical Character Recognition (OCR) activates to convert all content into digital attributes. Next, the system identifies the specific document type or class and extracts the corresponding key-value pairs (such as customer names, addresses, account numbers, etc.), Tables. This extraction utilizes a combination of AI and traditional techniques, resulting in a JSON output. In a comprehensive setup, this JSON can be parsed and linked to a workflow for further processing. For instance, the JSON might include the "Total Amount Due" along with its value, which can be utilized to trigger automated payment processing or further review of the invoice.

IBM ADP is an incredibly flexible product that allows supervisors to utilize a Designer environment for configuring the system. This includes setting up document classes, defining keys, and customizing extraction methods. Additionally, users can train models, test extraction accuracy, and once they are satisfied with the results, deploy the setup into the Runtime environment for live, automated intelligent document processing.

ADP 24.0.1 released Dec 2024 is the latest stable version,  please refer to the links in the reference section below for more information. To make it more interesting here are a few technologies it leverages

• OCR for document scanning
• AI/ ML for document & page classification, KVP extraction, fixing scanning issues
• Natural Language Extraction using IBM SystemT a declarative information extraction for defining custom extraction handlers 

Reference links

• ADP Product page https://www.ibm.com/products/document-processing
• ADP Product Documentation https://www.ibm.com/docs/en/cloud-paks/cp-biz-automation/24.0.1?topic=automation-document-processing
• Multi-part series of blogs on Table Extraction https://medium.com/@praveenmidde/table-extraction-in-content-capture-domain-why-what-and-how-8a3c90c79874 

No-Code Internet QoS (Quality of Service) Monitoring

Monitoring the Internet Bandwidth Quality of Service is essential to gain confidence in the service provider. 

There are several solutions available for enterprise-scale bandwidth monitoring.  This article is focused on a solution for home users. It is quick and straightforward. No coding is required. 

In 2018, I worked on a pet project that involved setting up a monitoring system for my internet bandwidth. I used a Raspberry Pi and installed all the necessary software. However, one of my biggest challenges was accessing the dashboard outside my home network. It was complex, and I had to rely on local/on-prem environments. After thinking, I realized that leveraging cloud services could help me overcome this bottleneck. I'm excited to explore this idea further and see how it can improve my project.

• Needs the ability to access the monitoring dashboard from anywhere, so it should be hosted on a cloud service. 
• Needs the ability to run an agent on the network to monitor 24x7, so a lightweight computer like Rasberry Pi will work. 

Software components 

• Docker standalone/local/on-prem runtime: To run the Telegraf agent container that collects the stats like download, upload, etc., and pushes them to a time-series database. 
• InfluxDB Cloud instance: Cloud-based time series database to collect and store the internet speed stats 
• Grafana Cloud instance: Cloud-based visualization instance to plot beautiful graphs of the internet speed. 

Hardware components 

• Raspberry Pi is connected to the home network and set up to run forever. Don't worry; its power consumption is very minimal. 

Setup

• Login to the Influx DB Cloud portal with your Google account https://cloud2.influxdata.com/signup 
• Setup "Company" and "Organization" You can give any name 
• Create a "Bucket" - Imagin this to be a store 
• We would use the Telegraf agent, so move to the Telegraf tab and create a configuration.
• Select the bucket you created earlier, and from the filter type "Internet Speed" select the tile. 
• The above steps are going to create a Telegraf config file and provide instructions that are important to note down. 
• INFLUX_TOKEN is required for the Telegraf agent to successfully write data into the cloud Influx DB 
• The start command of Telegraf provides the reference path for the config file. 
• Download Telegraf agent https://portal.influxdata.com/downloads/. There are several types, but we would be using the docker image. 
• On the Raspberry Pi, install Docker and open a terminal. 
• run the commands provided in the above link 

# Debian-based image

docker pull telegraf

•  Start the container 

• docker run -e INFLUX_TOKEN=<Token> -p 8086:8086 -d -it --name telegraf telegraf --config https://<URL>

• The above command is going to start the container with the config file location provided as https URL and start pushing the internet speed stats to the influx bucket 

• Verify data in the Influx bucket 
• Navigate to the influx Buckets tab and click on the bucket you created 
• Select the measurement drop-down and select the internet_speed that you already created 
• Select the fields and write a simple query to inspect the data that is collected
• This data is pushed from the Raspberry Pi on your home network 
• Configure Grafana visualization 
• Log into the Grafana Cloud portal https://grafana.com/auth/sign-in and launch the Grafana Cloud instance. 
• We need to pull the data from Influx Cloud Bucket and plot the visualization to create the Influx data source. 
• In the URL, use the influx cloud server URL, select Basic Authentication, and provide your login credentials. 
• Provide the influx organization you created earlier and select the bucket you created earlier. 
• This setup needs an Influx token to access the data from Grafana. So open the Influx Cloud instance and configure an API token with full access. 
• Copy the token and use that in the Grafana Data Source config 
• Now test the connection, and you should see that all is good
• Configre Grafana Dashboard 
• Click Grafana >> Dashboard and New 
• Select a pane and open the configuration 
• Make sure the Time series is selected on the top-right 
• Select the Data Source that you just created 
• Now the challenge here is to write a Flux Language Query. It took some learning for me so I am providing a basic query that you can use and modify (refer: https://docs.influxdata.com/influxdb/v2.7/query-data/get-started/query-influxdb/) 
• from(bucket: "Internet_mon")

  |> range(start: -30d)

  |> drop(columns: ["tag", "host", "server_id", "source"])

  |> filter(fn: (r) => r._measurement == "internet_speed" )

  |> filter(fn: (r) => r._field == "download")

  |> yield(name: "DownloadSpeed")
• The visualization starts showing the data 

That's it. You can now start playing around to create more widgets like the following. Observe your Internet Bandwidth Quality of Service in real time to understand current and past behaviors. 

Share the link and monitor your home network from anywhere. Note that the cloud instances used are on free plans. 

Enjoy!!

More interesting ideas 

• Set alerts using the "Threshold" concept in Grafana - Whenever the internet download speed drops or disconnects, you would like to get an email/tweet. 
• If the downtime continues for more than a threshold amount of time, say, 10 minutes or more, you should send an email to your service provided with detailed logs. 

Understanding application Observability and the difference with Application Process Monitoring is very important for a software/systems developer. Check this out https://youtu.be/dXohgVATzDc 

Please comment with your inputs/concerns/more ideas and how you could solve some complex problems with observability. 

Secure your personal secrets

In this digital world and internet era, personal sensitive information is a synonym for personal secrets. Everything that you are not supposed to share with others, that includes your browsing trends, habits, passwords, credit/debit cards details, PINs, OTPs, emails, personal or official files on your computer and so on. The list is long.

How to Secure yourself on the internet is a pressing question many ask or think these days especially after seeing the current trend of cyber attacks. Here are most definitive precautions one must take to keep their personal sensitive information secure. 

To generalize it is essential to protect both Physical secrets and Digital secrets. As anyone would guess, Physical secrets are secured with a lock and key. The challenge is securing digital secrets due to its shareable and easy reachable nature in the cyber world.

Digital secrets are to be secured during storage, transmission, revoking access after use and finally secure deletion when they expire or no longer required. With the advancements of internet-based utility applications in our society, the number of layman handling secrets increased exponentially in the last few years. This trend opened doors for hackers and social companies to exploit these vulnerable people to loot money, sell their habits, trends, thoughts, and opinions without their knowledge.

Understanding the digital information from the time a person starts touching a digital device is very important. Sometimes even the legal full name of a person could be classified as a secret and should not be disclosed. One might think that by completely isolating from the cyber world they could secure themselves, but that is not true. Governments, friends, colleague and other entities in social & personal life are injecting your information into the cyber world without your knowledge. Therefore it is mandatory to understand how to secure digital secrets than struggling to isolate them. 

Let us look at various entities of cybersecurity from a layman’s perspective

• Basic ethics - Before we jump into more detailed aspects of security lets look into some essential items to remember. Always update your digital device software to the latest and greatest version that offers full protection from virus and malware. Buy phones/systems/OS that promises security updates, never install software from an unknown source. All electronic devices should have all endpoints scanned automatically for any virus or malware or adware, detected and removed. All endpoints mean, network (both wired and wireless), CD drives, USB ports, cloud drives and so on. Experts believe that following these high ethics would eliminate the requirement of installing  Anti-virus software.
• Password - One of the most common entity that even a layman know these days. However, this is the most miss understood entity as well. Passwords should be very long, easy to remember and tough to guess, should not have any words from the dictionary, should contain special characters, should not use the same password for multiple websites/services, should be changed every 60 days and so on. All electronic devices/services need a password, not shared with ANYONE. If multiple people share a device, configure multiple login credentials for exclusive access. Sharing a password could expose the general password creation pattern, opening the options to guess more sensitive passwords like bank accounts.
• Password Manager - Secure passwords are hard to maintain as today average number of services a person uses in his lifetime are more than 100. So it is essential to get a password manager to store, update passwords of all the services. There are different kinds of password managers available in the market. A recommended password manager is KeyPass (http://keepass.info)  which is a free non-cloud based password manager. It can store, username, passwords, individual notes like secret questions and answers used for a website, bank account numbers, mobile app PINs, debit/credit card numbers, CVVs and so on. Setup one powerful password for the password manager and should never forget because losing this could jeopardize all passwords stored in there. KeyPass creates an encrypted file with extension "kdbx" that could be decrypted only with the master password. So it is safe to keep this file on any electronic device or cloud storage without having a fear of someone reading your secrets. A user has to download this file to his local device (computer/tab/phone) open with the application (keypass/macpass/keypassDroid), use the master password and read your secrets before using them. Yes, there is a bit of inconvenience of using multiple apps before accessing but this is a worthy price to pay for security.  
• Encryption -
• • Encrypting data stored on electronic devices - Digital information stored on computers, tabs and phones should be encrypted. Turn on Android and Apple phones disk encryption option; this would make the data on the phone unreadable when accessed directly without unlocking the phone using the correct password. On the similar lines, there is FileVault software for Mac user, BitLocker for Windows and man open source tools for linux. It is must to encrypt all devices a person uses.
  • Encrypting data stored in cloud services - This would protect your information stored in cloud services like Dropbox, Google Drive, and so on. Cloud data would be secure, even if the provider's security breached. One such service is boxcryptor.com it would keep your data on cloud safe from everyone - google, boxcryptor and even any third party cannot read your content.
  • Encrypting emails - Emails stored in providers server have to be kept safe as they contain many secrets - reset passwords, PINs, banking transactions, bank account statements. Even if hackers break the email provider's security, your data that is encrypted would be safe as it could be decrypted only by you who is the owner, who encrypted it can read it. Many tools/ services proved this facility, but the one recommended is protonmail.com . It has zero-access encryption that means the emails sent, received and stored in their servers are encrypted during transmission when originated from protonMail and also encrypted and stored in the mailbox so that even the company themselves cannot decrypt/read them.
• Multi-Factor Authentication - Having a strong password is single-factor authentication. Someone steals it, can gain access to your account. However, securing your account with an additional factor could enhance the security, this is called 2FA (2 Factor Authentication) and its provided by the service provider like Google, Amazon, Twitter, ProtonMail and so on. Mostly, you need to key-in your password and a PIN/ Code that is sent to your mobile or accessed via a mobile app like Google Authenticator, to gain access to your account.  
• Protecting your devices against snooping - First, let’s understand few facts. It is very much possible for someone to snoop into your electronic device or network traffic while you browse the internet. Imagine someone trying to look into your network packets that are transferred from your device to your bank while you are trying to perform a transaction. Many argue that all banking sites and many others use https which is secure. Snooping happens at various intensity, some hackers snoop to steal bank card details to perform financial crimes, some snoop to know browsing/buying habits. Big online shopping players like Amazon uses HTTPS right from its first page till you pay. However, still, remember Amazon collects your browsing habits for analysis and push ads. So HTTPS, in this case, is protecting you from external interference only. Many small players leverage HTTPS only while paying online so all other traffic like selecting your products can be snooped in clear text by anyone. HTTPS is controlled by the website that offers a service (like a bank/shopping site) there are cases where a lot of information other than HTTPS leaked which contributes to the overall corpus that a hacker builds over time and uses it for attacks. Special care should be taking while connecting to a public WIFI; it is very much possible that someone is always watching the traffic. So avoid transacting secrets on public WIFI. Now, the question is, how do we access the internet securely without exposing our secrets - passwords, browsing habits? Fortunately, there is a solution to secure everything between a device and a service endpoint (like a bank/shopping site). That is called VPN - Virtual Private Network, it encrypts all traffic, detects & stops intrusion dynamically. Is VPN the magic bullet? May not be, but they make it harder to associate online data to a particular user/computer. VPN creates a secure pipe from the start to the end in which encrypted data transfer. The moment a hacker tries to breach this pipe, it dismantles and recreates the pipe using a different route. Therefore, it is safe to use public WIFI via VPN.  Even ISP (Internet Service Providers like BSNL, Airtel, etc.,) cannot collect your browsing habits, and it also allows a user to connect to the internet from a different geo-location (other than his physical location). One trusted VPN service that is free (also can be bought) is protonvpn.com it can be installed on computers/tabs/smartphones, the free VPN has only 1 VPN server out of 3 to choose for one device at a time. If you pay, you can connect to the unlimited number of VPN servers across the world with fast internet speeds. One might argue that the VPN servers themselves can access the content. That is not true as Proton VPN uses a zero-knowledge protocol which means the data is encrypted with the user's password (proton mail password) as the key so that neither the company nor the third-party could decrypt. There are few countries where using VPN is illegal please do your research before subscribing/installing in your device. 

To conclude, there are various steps to follow to secure digital secrets

1. Update your device software to the latest.

2. Exclusive password protect your device, user and services

3. Install Password Manager (like KeyPass) and follow password ethics

4. Enable 2FA on all accounts that support it.

5. Encrypt all your devices local data (like FileVault)

6. Encrypt your cloud storage (like boxcryptor)

7. Use encrypted inbox (like ProtonMail) for all bank/debit/credit card/wallet emails

8. Always use VPN (like protonVPN) while accessing financial transactions

9. Never give your phone number/email ID associated with your bank accounts to anyone who should not know it.

10. Never share secrets - Passwords, PINs, card numbers, dates and so on

Remember, enhancing security would only make it difficult for hackers to gain control of your secrets. It is just like, locking your home secures your valuables but cannot guarantee absolute safety.

Considerations to implement Cloud Integration

Every company has a Cloud offering and most of the businesses have concluded or considering to move to the cloud due to its benefits.  Based on an IBM Survey, customers are 83% more likely to use a Hybrid cloud.

Hybrid Cloud is an integration of applications/resources/services that are spread across different cloud vendors or cloud applications integrated with on-prem assets.

So it is essential to understand Cloud Integration.

First, let's list out different types of cloud integration possibilities

• Integrating two applications within the same cloud
• Integrating a cloud application with on-prem resources
• Integrating two SaaS applications hosted on different cloud vendors

 On-prem and Cloud have their own benefits so the industry is migrating towards best of both worlds called Dynamic Hybrid Cloud which is a combination of both (on-prem & public cloud) to balance risk and speed of deployment. Also, the end users migration to mobile and social usage patterns is forcing traditional businesses to leverage Public clouds to deploy System of Engagement (SOE) i.e customer UI and integrate them with System of Records (SOR) resources hosted on-prem for better security.

Public Cloud Low energy Pay as you use Elastic

Private Cloud Customization flexibility Highly secure Management flexibility

Attributes to consider for Cloud integration (*refer OpenGroup)

• Scalable: Should be elastic to absorb the changing workloads
• Robust: Should be fault tolerant, adapt to failures and recovery fast
• Flexible: Capability to handle changes between cloud vendors and on-prem environments.
• Secured: Communication channels and data at rest should be encrypted.
• Configurable: Define configurable interfaces via web browser.

Generally, Cloud Integration is targeted at the Application level or at the Data level.

• When trying to integrate at the Data level a design pattern called Pipe & Filter could be considered. The piping and the filter code transforms and moves the data from one cloud to another depending on the source (producer/from Cloud) and sink (consumer/to Cloud) data format requirements. Similar to ETL (Extract Transform Load) process.
• Application integration is the key to choosing the best cloud integration option for a given situation so it is required to cover all three could integration possibilities listed above.
  
  • Application to Application integration in the same cloud
    
    • API driven - REST based and Messaging (MQ)
    • REST is a simple lightweight web services communication (often considered as an alternative to CORBA). It's platform and language independent. Does not have security, session, message formats making it less restrictive. It's a client-server model based on resources that are accessed via simple POST/GET/PUT calls. It is a stateless service which means the calls are atomic and do not depend on previous calls. The Server is not concerned about the Client and the Client will have everything required to fulfill the call. REST is used for synchronous communication. 
    • Messaging is required for different business applications to communicate effectively using a common method. when there are multiple applications integrated, there is a need for them to be active/up and running to receive the communication and respond. In reality, applications will go down but it is important to have the communication queued and safely handled without duplicating/missing the communication. This is where Messaging Queues comes into the context that guarantees "assured message delivery and Time independent processing". IBM MQ Light is a Messaging software that is available as a developer sandbox and in IBM Bluemix. A developer can seamlessly deploy the application developed & tested in the sandbox, into IBM Bluemix (or even IBM Websphere MQ). Mostly considered for asynchronous communication.
  • Application in a Cloud to on-prem resource
    
    • A common scenario for this integration is, SOE (System of Engagement) hosted in a cloud integrated with SOR (System of Records) that is hosted on on-prem. Some examples of SOE are UI for a bank that is used by the customers to check their bank accounts. Example of a SOR could be the relational database that stores financial transactions/customer information.
    • IBM Secure Gateway in IBM Bluemix is the preferred way to connect a public cloud application to the on-prem resource. This establishes a secure tunnel between IBM Bluemix organization and remote on-prem resource.
  • Applications hosted on different Cloud vendors
    
    • Integration between two SaaS applications. Considering the advancements happening the PaaS and SaaS areas, most of the cloud vendors are self-sufficient or very quickly enhancing with new/missing services, therefore this integration is not popular in the market. However, IBM Websphere Cast Iron Live in IBM Bluemix could be leveraged for integrating two SaaS applications hosted in different Cloud vendors.
    • One example scenario could be to synchronize events in CRM calendar system with customers Google Calendar to enable customers to see when a meeting is scheduled on the CRM ticket etc.,

To conclude, here are the considerations that could be adopted for Cloud Integration situations

• SaaS to SaaS integration - consider IBM Websphere Cast Iron Live
• SaaS to on-prem - consider IBM Websphere Cast Iron Live
• Public Bluemix to on-prem - consider IBM Secure Gateway
• Public Bluemix to IaaS - consider IBM Secure Gateway

 

IBM Watson Analytics for data insights - explained with a home infrastructure monitoring system

Gone are the days when we would spend countless hours developing intuitive dashboard software that displays/monitor key performance indicator (KPIs) to help the executives steer the company. IBM Watson Analytics moved several steps ahead by digesting any type of data for any domain, deriving relationships between the data, often making it a child's play to explore the data and finally providing a way to predict the future based on existing data patterns.

If you know what your data is about, leave the rest to Watson. All of this is on the cloud (no installation required) and free to try. Trust me it will not take more than 10 to 15 minutes to experience the power. Here is where you need to click https://www.ibm.com/watson-analytics and try it for FREE!!

To understand more let’s scale down from a corporate dataset to a home-based dataset. Like electric power and cooking gas, the internet became the new normal and essential supply for most homes. Though busy with our day jobs and personal work at home, it becomes important for us to monitor all of these essential services that will make our living comfortable and help us deliver better. 

We often have questions like …

- When was my electric power supply hit peek voltage, how many times it happened in the last year and when can it happen again this year so that I would keep my high-value electric equipment safe.

- How is my broadband speed changing over a time period, when was it last disconnected, how many times it got disconnected so that I could measure the quality of service?

- How are the temperature and humidity changing in my house and does that have any relationship to the seasonal flu

- How are my security cameras performing, its essential to get a notification when any of it is disconnected

At a high level, there are two components that could be explored to answer the questions like the above

1. A Raspberry Pi doing all the 24X7 work for you. Attached with SenseHAT for temperature and humidity sensors for continuous monitoring, logging, and reporting in real-time

2. IBM Watson Analytics to load the historical data, explore and predict future outcomes.

Raspberry Pi uses

- Node-Red service with Dashboard and SenseHAT modules added

- speedtest_cli to measure the bandwidth speed from command line

- OpenWeatherMap REST APIs for reporting the location weather

- email / SMS integration for notifications.

Node-Red orchestration displays a live real-time dashboard and logs  the data into CSV files

In the Node-Red drag-drop the required nodes and build the flow that …

- Checks the internet status at regular time intervals and log the ‘Internet UP/DOWN” status

- Collects the bandwidth upload and download speeds on regular intervals and logs them

- Collects the Raspberry Pi temperature on regular intervals and logs them

- Collects the ambient temperature and humidity on regular intervals and logs them

- Collects the location temperature and humidity details on regular intervals by calling the OpenWeatherMap REST API

- Constantly keeps track of local disk space and automatically moves CSV files to an external disk

This list can go on and on like monitoring the security cameras and automatically clean up the cloud storage space that stores the recording/motion detected photos etc.,

The Node-Red Dashboard module is used to build a simple auto-refreshing dashboard that is a  single page view of the key indicators.Here is a github project of the same that could be used for free: https://github.com/srinivasgnv/cmu

Here is an example of the real-time dashboard

When all of this data get’s collected for, let’s say for a few months/year(s) users can perform analytics on this data to derive insights and answer cognitive questions like above. To do this register to IBM Watson Analytics and follow these simple steps.

- Upload the CSV files (make sure there is a heading for each file describing the columns )

- Use the Explore option where Watson digests the dataset and automatically comes up with interesting relationships in the data

- Provides an option to ask questions and get answers

- Predict future outcomes.

All of this is without writing a single line of code. Yes, you read it correctly. You have to try this to experience the power of Watson Analytics.